Job Chaining in Three Steps
Job Chaining is a convenient tool used for automating the preparation of input files for compute jobs (pre-processing), launching the compute job, then archiving the new output files (post-processing).
The following 3 steps describe how to set up a basic job chain to run on the pacman HPC system at ARSC.
Step 1 - Create a Pre-processing Job Submission Script
To begin the job chain, a pre-processing job submission script needs to be created to batch_stage all archived input files, then copy them to your $CENTER. Following the copy, this pre-processing script then automatically submits your regular job submission script to the queue.
#!/bin/bash #PBS -q transfer #PBS -l walltime=24:00:00 #PBS -l nodes=1:ppn=1 #PBS -j oe cd $PBS_O_WORKDIR batch_stage $ARCHIVE/results.tar /usr/bin/rcp "bigdip-s:$ARCHIVE/results.tar" $CENTER || exit 1 # Submit your regular job to the compute nodes qsub my_job.pbs
Step 2 - Modify Your Existing Job Submission Script
Your regular job submission script should remain the same, plus a few new lines at the end of the file. These new lines test whether your compute process completed successfully. If the process ended successfully, a post-processing job will be automatically submitted to the transfer queue to copy the output files from $CENTER to $ARCHIVE.
#PBS -q standard_4 #PBS -l nodes=1:ppn=4 #PBS -l walltime=8:00:00 #PBS -j oe cd $PBS_O_WORKDIR # Run the executable: mpirun ./a.out # Check for error status: # If the application exited successfully, submit the post processing # script. # if [[ $? -eq 0 ]]; then qsub postProcess.pbs else echo "Error: The application exited with an error!" fi
Step 3 - Create a Post-processing Job Submission Script
If your compute job completes successfully, the script from step 2 will submit the postProcess.pbs script to the queue automatically. This postProcess.pbs script will copy the output files back to $ARCHIVE.
#!/bin/bash #PBS -q transfer #PBS -l walltime=24:00:00 #PBS -l nodes=1:ppn=1 #PBS -j oe cd $PBS_O_WORKDIR tar -cvf mydata.tar $CENTER/myOutputFiles /usr/bin/rcp $CENTER/mydata.tar "bigdip-s:$ARCHIVE/" || exit 1
A more complex example using job chaining is described in the ARSC HPC Users' Newsletter Issue 322.