• View
• Edit
• History
• Print

Grid

These notes for working with the MIPP software on the grid were started my Mark Messier (16 Jan 2008). They are based on telephone and e-mail conversations with Holger Meyer.

Making a grid release

To work on the grid, one first needs a working grid release. Holger is currently the expert. It requires tagging a release (R08.01.14, for example) and then making a tarball of the release and its associated binaries and libraries. There is a script for this in the repository originally due to Andre Lebedev. I'm not sure where it is or what its called. Perhaps an expert can update these two last sentences.

Submitting grid

Playing nice

There is not much documentation on grid use and some "do"s and "don't"s (mostly don'ts) have been discovered mostly related to DCACHE. Jobs which require moving many small files need to be staggered so that the number of files in play at any one time is kept to a reasonably small level. Also, while documentation on DCACHE says otherwise, moving roughly 10 Tb of data through public DCACHE in a week also seems to cause problems. The solutions in both cases seems to be limiting the throughput by hand. The scheduler doesn't seem to be able to detect and limit this usage automatically.

Running jobs

1. log onto fnpcsrv1.fnal.gov using your kerberos ticket
2. cd testRel and then use the "setrel" command to setup the environment
3. cd to BatchProc/grid
4. The command of interest is "gridSubmitPass" for the pass1, pass2, ... rescor passes. It requires the release to be specified (for example R08.01.14_Linux2.6). A list of releases can be obtained by doing an ls on some directory (help!). The command gridSubmitPass provides a good help message if you do a gridSubmitPass --help.
5. You will need to make a log file directory in /farm/e907_stage01/log/PASSNAME_DATE where PASSNAME is the name of the pass you are running and DATE is the submission date.
6. Output gets written to mass storage in the reco3/FNAL/PASSNAME directory