-
Notifications
You must be signed in to change notification settings - Fork 0
WIPACrepo/DumpStream
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
James Bellinger This is the tree for the to-NERSC lta chain. The chain consists of 4 main parts: archivecontrol rest server: (which uses apachetest.conf and rest.py) jade-lta server: (BundleScanner.py) nersc server: (NERSC_Client.py) dumpercontrol (DumpControl.py) jadetools contains BundleScanner.py and various utility programs we run in Madison nersctools contains NERSC_Client.py and runs at NERSC dumper contains python and bash scripts for reading Pole disks In addition we have in this directory, though not in git (for obvious reasons) a rotating backup of the sqlite3 database on archivecontrol. archivecontrol runs its rest server from /opt/testing/rest archivecontrol uses apache, and in /etc/httpd/conf.d has my apachetest.conf to point queries to the right code archivecontrol uses the Flask utility to handle the interface between sqlite3 and python. The tool rest.py manages the processing, and rest.wsgi tells apache where to look for the rest.py file config.py tells Flask what the initial settings are. HISTORICAL NOTE: I was pointed to the mod_wsgi framework for using the REST server. However, it wasn't obvious what to use for communication, so I expanded the initial curl test scripts to handle the interactions. These are ugly as home-made sin, and at some point I'd like to get rid of them in favor of something more bulletproof--and maybe with some authentication. Patrick has some things that would probably work. Because the object was to get something running quickly, and get stuff to NERSC before we got too far behind the eight-ball, this does not (yet) use any of the file catalog or Patrick's (new) REST framework. In the medium term, the only things I expect to use are the dumper/DumpControl.py and the jadetools/GlueLTA.py (and the Utils.py) In the long term, DumpControl and GlueLTA will probably have to be more tightly integrated into the LTA suite. The Pole disk dumping uses dumper/DumpControl.py and some helper bash scripts. It runs on jade03 I did something quite crazy and non-pythonic to keep the footprint small: I use a preprocessor instead of a python library. This achieves the same result--keeping utility code in one place and consistent for all tools (except the rest.py wsgi handler)--but means I only have one script, as a rule, to schlepp around. In retrospect, that was crazy, but I was in an experimental mood, and it doesn't hurt readability/maintainability. I'll change it to the right way as needed: (only GlueLTA and DumpControl).
About
Scripts to manage and monitor the dumping of Pole tapes and transfer of the data to NERSC
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published