  Reconstruction of one run of CLEO III raw data can take up to 9 days to
complete using a single processor. This is an administrative nightmare, and
even minor failures result in reprocessing the entire run, which wastes time,
money and CPU power. We leveraged the ability of the CLEO III software
infrastructure to read and write multiple file formats to perform
reconstruction of a single run using several CPUs in parallel. Using the Sun
Grid Engine and some Perl scripts, we assign roughly equal-sized chunks of
events to different CPUs. The Raw data are read from an Objectivity/DB
database, but the reconstruction output is written to a temporary file, not the
database. This takes about 6 hours. Once all the chunks have been analyzed,
they are gathered together in event-number order and injected into
Objectivity/DB. This process takes an additional 6 to 18 hours, depending on
run size. A web-based monitoring tool displays the status of reconstruction.
Many benefits accrue from this process, including a dramatic increase in
efficiency, a 20% increase in luminosity processed per week, more predictable
and manageable processor farm load, reduction in the time to stop the processor
farm from up to 9 days to less than 24 hours, superior fault tolerance, quicker
feedback and repair times for bugs in the reconstruction code, and faster
turn-around of early runs for data quality and software correctness checks.
