Skip to content

Scripts to manage and monitor the dumping of Pole tapes and transfer of the data to NERSC

Notifications You must be signed in to change notification settings

WIPACrepo/DumpStream

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

James Bellinger
This is the tree for the to-NERSC lta chain.  The
chain consists of 4 main parts:
archivecontrol rest server:	(which uses apachetest.conf and rest.py)
jade-lta server:		(BundleScanner.py)
nersc server:			(NERSC_Client.py)
dumpercontrol			(DumpControl.py)

jadetools	contains BundleScanner.py and various utility
   programs we run in Madison
nersctools	contains NERSC_Client.py and runs at NERSC
dumper		contains python and bash scripts for reading
   Pole disks

In addition we have in this directory, though not in git (for
obvious reasons) a rotating backup of the sqlite3 database
on archivecontrol.

archivecontrol runs its rest server from /opt/testing/rest

archivecontrol uses apache, and in /etc/httpd/conf.d has my
apachetest.conf to point queries to the right code

archivecontrol uses the Flask utility to handle the interface
between sqlite3 and python.  The tool rest.py manages the
processing, and rest.wsgi tells apache where to look for the
rest.py file  config.py tells Flask what the initial settings
are.

HISTORICAL NOTE:  I was pointed to the mod_wsgi framework
for using the REST server.  However, it wasn't obvious
what to use for communication, so I expanded the initial
curl test scripts to handle the interactions.  These are
ugly as home-made sin, and at some point I'd like to get
rid of them in favor of something more bulletproof--and
maybe with some authentication.  Patrick has some things
that would probably work.
Because the object was to get something running quickly,
and get stuff to NERSC before we got too far behind the
eight-ball, this does not (yet) use any of the file catalog
or Patrick's (new) REST framework.
In the medium term, the only things I expect to use are
the dumper/DumpControl.py and the jadetools/GlueLTA.py
(and the Utils.py)
In the long term, DumpControl and GlueLTA will probably
have to be more tightly integrated into the LTA suite.

The Pole disk dumping uses dumper/DumpControl.py and some
helper bash scripts.  It runs on jade03

I did something quite crazy and non-pythonic to keep the footprint
small:  I use a preprocessor instead of a python library.  This
achieves the same result--keeping utility code in one place and
consistent for all tools (except the rest.py wsgi handler)--but
means I only have one script, as a rule, to schlepp around.
In retrospect, that was crazy, but I was in an experimental mood,
and it doesn't hurt readability/maintainability.
I'll change it to the right way as needed: (only GlueLTA and
DumpControl).

About

Scripts to manage and monitor the dumping of Pole tapes and transfer of the data to NERSC

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published