ARSC HPC Users' Newsletter Issue 433 2014-06-18

ARSC HPC Users' Newsletter Issue 433 2014-06-18

A publication of the Arctic Region Supercomputing Center.

1 Segmenting Work With Job Arrays

Ben Bettisworth, ARSC Research Projects Assistant

1.1 What Are Job Arrays?

The purpose of a job array is to break up the work in a compute job into distinct parts for the job scheduler, letting the scheduler know that the components can be executed separately. This is especially useful if a job is exceptionally large but can be broken apart, so the execution of the program components can take up all idle cores dynamically.

1.2 Using Job Arrays

To specify a job array and its size, use the -t flag, which specifies the range of task ids for the job array. Below is an example of submitting such a job from the command line:

qsub -q standard_4 -t 1-10 -l walltime=8:00:00,nodes=1,ppn=4 ./myprog

Here is the same job submitted in a script:

#PBS -q standard_4
#PBS -l nodes=1:ppn=4
#PBS -l walltime=8:00:00
#PBS -t 1-10


To specify the maximum number of jobs to be run concurrently, append % followed by the number desired. For example, to limit the number of running tasks in the job above to 2, the new command would be:

qsub -q standard_4 -t 1-10%2 -l walltime=8:00:00,nodes=1,ppn=4 ./myprog

For all of this to be useful, you will likely want to apply the $PBS_ARRAY_INDEX variable, available for scripts. This variable informs the current process as to which index it owns in the array, so it can know which part of the work it must handle. For example, if you have a set of files workfile.[1-10].csv then the following script could be used to run myprog on each of those files:

#PBS -q standard_4
#PBS -l nodes=1:ppn=4
#PBS -l walltime=8:00:00
#PBS -t 1-10

./myprog workfile.${PBS_ARRAY_INDEX}.csv

1.3 Caveats

  • We have recently observed a problem with short-lived array jobs confusing scheduler operation. Consequently, you may experience difficulties if you submit array jobs that run for less than fifteen minutes. One option is to find a way to increase the run time of the array jobs, such as running multiple instances of your program sequentially inside the pbs script. (This would require doing some math with the PBS_ARRAY_INDEX variable to ensure each command received an appropriate index value.) Or consider taking some other approach which does not involve job arrays, such as running multiple serial executables simultaneously on a single node.
  • This approach should not be used on MPI jobs, as the number of running jobs will change over time. If you had constructed the array with -t 1-100%5, then the maximum is set to five job components, but there may be only four running at a time, or even just one.
  • Job arrays are not guaranteed to run in any specific order.
  • The wall time applies to each job component individually and is not cumulative.
  • To delete a job array with the qdel command, you will again need to pass in the array range with the -t flag.

2 Online Resources For Principal Investigators

PI's should be aware of the wealth of information available to them about their projects through our Extranet web portal. To view this information, log in using your UA username and password. From the "Memberships" pane, select the "Details" link for any project for which you are a PI.

The details page for a project contains summary information about the project in the "Project Details" pane. Under this you will find the "Project Allocation" pane, which provides colorful graphs illustrating project allocation, CPU load, and job quantity data for the current fiscal year. Separate graphs distinguish between system-total and per-user data.

The right side of the page contains a "Project Members" pane, allowing you to view project membership, as well as request the removal or addition of members. Underneath this are tables that represent (1) total data storage usage (for example, how much disk space your project is occupying in the $CENTER directory), (2) data storage usage by user, and (3) top user statistics, which reveal those users burning the most CPU time and quantity of jobs for each system.

3 Summer Tours Presentations Have Begun

ARSC will continue to host the seasonal ARSC Summer Tours every Wednesday, until August 13. So feel free to invite friends and family who might be interested in learning about ARSC, the technology we manage, and the research we support.

Tours takes place in the Butrovich machine room viewing area (in the hallway on the lowest level) with a digital slide presentation, talk, and Q&A, which totals about thirty minutes. Afterwards, interested visitors are guided though a brief walk-through tour of the machine room, to show them the ARSC storage equipment and supercomputers (for groups with less than 7 people, 8 years of age and older). Registration is not required.

4 ARSC Training

This Friday, ARSC will host a three-hour training session entitled "Introduction to the R Programming Language" with UAF Ph.D. candidate Katrina Bennet.

No experience necessary! Learn to hone the power of this elegant statistical and graphical programming language to enhance your data presentation skills.

We intended to advertise this training in the newsletter to encourage participation; however, interest in R Training is so great that the class is already completely full! Nevertheless, it is still possible to register for the waiting list. This training will take place on Friday, June 20 from 9am to 12pm in WRRB 009.

We are considering hosting a second class in August for those who cannot attend this session. Please contact ARSC if you are interested in attending this training at a future date.

5 More Information

5.1 Editor

Christopher Howard

5.2 Credits

Oralee Nudson, ARSC Lead User Consultant. Reviewer and insider source for ARSC news and tips.

Azara Mohammadi, ARSC Publications Staff. Reviewer.

5.3 Publication Schedule

The newsletter is usually released on the third Wednesday of each month.

5.5 Archived Newsletters

5.6 Questions, Comments, And Submissions

Need advice? Submit a question about HPC or ARSC software, and we will feature it in a Q&A section in the newsletter.

Back to Top