ARSC T3E Users' Newsletter 133, December 19, 1997

qalter Dynamically Adjusts the size of NQS Requests

With the "qalter" command, you may dynamically reduce the size of an NQS request, thus making PEs available to other users.

This would be considered an act of good citizenship, and could lead to better system throughput. As an example, the following request runs two jobs requiring 40 and 22 PEs respectively. The qalter call releases 18 PEs to other users:


######################################################################
#QSUB -q mpp
#QSUB -l mpp_p=40                           # prog_1 needs 40 PEs
#QSUB -l mpp_t=6:00:00                      # 6 hours for both runs
#QSUB -s /bin/ksh
cd /tmp/userid/run

mpprun -n 40 ./prog_1                       # run in 40 PEs

qalter -l mpp_p=22                          # release 18 PEs 

mpprun -n 22 ./prog_2                       # run in 22 PEs

######################################################################

Limitations of qalter:

  • It can not move a request to a different queue. For instance, were the above call replaced by qalter -l mpp_p=8 the request (on yukon) would continue to reside in the large queue (21-40 PEs) and not be moved to the small queue (2-9 PEs). It would thus continue to contribute to the global maximum of 2 large jobs.
  • It does not signal NQS to re-scan the queues, which only happens on events such as the arrival or deletion of a request. Thus, though 18 PEs might be freed, a waiting 18 PE job will not be started until some other event causes a re-scan. (Nor will the the termination of interactive jobs cause a re-scan.) On yukon, an hourly cron job sends a null request to NQS to force a re-scan. As the use of qalter increases, we will probably increase the frequency of these events.
  • Users may only decrease (not increase) the size of a request.

Despite these limitations, we heartily encourage you to use qalter in every qsub script if it could release PEs (even a few!) for other user jobs. For additional information see "man qalter."

All New Queues on Yukon

We have revised yukon's queue structure, as described in news queues on the ARSC T3E, yukon .

We have removed all explicit references to PE numbers and run-time limits from the queue names. This change will allow us to adjust these limits without needing to create new queues. No changes to user scripts are necessary to use the new queues.

The new queues are listed below:


  Name of        Number    Maximum jobs   Time limit    When queue is
  queue          of PEs     per queue      on queue      executable
 -------------  --------  ------------   ----------      ------------
  grand         65 - 90        1          4  hours        always

  xlarge        41 - 64        1          4  hours        always

  large         21 - 40        2          8  hours        always

  medium        11 - 20        2          8  hours        always

  small          2 - 10        2          8  hours        always

  Qgrand        65 - 90        1         30 minutes       always

  Qxlarge       41 - 64        1         30 minutes       always

  Qlarge        21 - 40        2         30 minutes       always

  Qmedium       11 - 20        2         30 minutes       always

  Qsmall         2 - 10        2         30 minutes       always

  single              1        5          4 hours         always

Note that with one exception (unless you count "1") we have eliminated the T3D legacy of power of two processor numbers. We actively encourage users to use algorithms which do not require a particular number of processors.

All users should run in the minimum number of processors which provide the necessary performance level, in terms of either memory requirements or runtime.

Users will notice that the short run-time limit has been increased from 10 minutes to 30 minutes. This should allow more significant development or test runs.

These quick queues, identified by the "Q" at the start of the name, are also prioritized so that once the resources become available they will run in advance of longer work. (ARSC will monitor usage of the quick queues to ensure this feature is used as intended. Please do not string multiple short jobs together.)

The larger queues which require a considerable fraction of the machine may require the halting of other queues so that sufficient PEs become free for them to run. ARSC monitors the submission of these jobs and periodically holds other jobs so this queue can proceed. Typically these hold and release events occur in the time before test times or prior to weekends. Users should read "news downtime" on the system to determine if such a session will be taken.

These details are likely to change to suit the nature of the usage patterns of the system. We actively encourage users to provide feedback to ARSC on this queue structure.

Random-Number Generators

[ Once again, a quick-tip question has inspired a full-blown article. Many thanks to Sung-Eun Choi of the University of Washington for this contribution. ]

It's pretty easy to get sequences of pseudo-random numbers that do not overlap -- more difficult is getting sequences that are not correlated. My response below is for "non-overlapping and non-correlated sequences." There is also the question of whether correlation actually matters for a particular simulation. Finally, when I say "random," I of course mean pseudo-random.

>From my investigations, there are (at least) two ways in which random numbers are used. The first way (which is not addressed by the "quick-tip" question) involves using a sequence that needs to be "random" over the entire problem. For example, you have a data structure that must be initialized to some random state. I won't elaborate on how to do this, but you can use preprocessing or other tricks to avoid complete serialization (on multiple PEs) if, for example, you need to re-initialize your data structure during the program.

The second usage requires a large number of independent, random sequences. For example, each element in your data structure represents a "body" with some random behavior which is independent of the other bodies.

Notice that this problem doesn't depend on the number of PEs, rather the number of bodies. Assuming we have at our disposal at least one generator that produces good, extremely long random number sequences (see Marsaglia and others for some really good generators, some with periods of > 2^1000), one way to guarantee that we don't get overlap or correlation in the sequence is to slice up the one extremely long random sequence into to a bunch of shorter (but still long), non-overlapping sequences.

There are theoretical results on other ways to do this, but here is a somewhat realistic scenario that I could imagine implementing. We have a (binary) file with every kth element of the random sequence -- k is the maximum required length of each subsequence to be used by each "body" in the simulation. Assume that all PEs have b bodies (k*b <= period). Each PE maintains b "seeds", the ith of which is initialized to the (PE*b+i-1)th element in the file. File access is fast and easy when the file is binary. In general, the seeds themselves may be a set of values that represent state. On any PE, each time a body b_i requires a random number, the generator is called with the corresponding seed and the returned value(s) becomes the new seed. All this can be done completely in parallel with no synchronization between PEs other than agreeing at initialization which PE "owns" what.

Notice that this method is not truly scalable. The file we use needs to be generated off-line, which means either the sequence needs to be enumerated or the generator must have a way to move ahead more than one element in the sequence. Moreover, as longer simulations are performed, longer sequences are required, which adds to the problem. Of course people continue to work on random number generators and computers are getting faster, so perhaps such a method will scale.

Editor's Note: Additional Concerns about Random Numbers

Repeatable parallel runs A salient feature of pseudo-random number generators running on serial platforms is that, given the same seed, they repeat the same sequence on every run. This characteristic is vital for debugging.

It is desirable to be able to repeat the sequence in parallel codes as well, but this is more complicated. Take an example of a Monte Carlo code, farming out work items, where the programmer wants the same results independent of the number of processors. Since work items are "farmed out" at run-time, based on run-time characteristics, it is not known which work item will be performed on which processor.

To solve this problem, a point in the sequence can be assigned to each work item which ensures that the same results will be produced independent of the number of processors used. Naturally the starting point much be chosen so that there are no repeated sequences and this mechanism will depend on the actual computation's treatment of random numbers.

Agreement between serial and parallel versions Another possible constraint is that the serial and parallel codes agree. Sometimes the method of dealing with random numbers in the serial code is hard to replicate in parallel. The best approach is to make both the serial and parallel codes use the same technique, which means that you must replicate the method used in parallel in the serial code. The same constraint is often applied to other compatibility issues.

Rudimentary tests for randomness Whatever technique is used, it is always a good idea to check the quality of the random numbers used. A simple method is to keep the mean and standard deviation of the numbers generated and verify that they are correct and acceptable.

Human Understanding It is also interesting to consider the human understanding of what is random. Most people have a poor understanding of this in everyday life which probably accounts for the belief in physic phenomenon and good fortune. A recent study of chosen lottery numbers shows people try to make them evenly spaced within the available set, while the actual draw produces sequences. The following is an interesting book on this subject: "Innumeracy : Mathematical Illiteracy and Its Consequences" by John Allen Paulos.

Further reading on random numbers A useful reference may be found at:

http://nhse.cs.rice.edu/NHSEreview/RNG

This a review of various serial and parallel random number generation techniques are presented along with a discussion on what makes a good random number generator.

Last-Minute Gift Ideas from ARSC Staff !

Again we polled ARSC staff. Thanks for these seasonal suggestions:

  • Chocolate always works (even if they don't like it, they can freeze it and give it to someone else)
  • Gloves
  • Silly socks!!
  • My last minute, all purpose, generic gift tends to be the Swiss Army Knife (small) w/ key chain or similar. (Boring, I know...)
  • Either the small pen lights key chains, or items I see in the free catalogue on airplanes, which has an interesting set of items for the distressed traveler who needs gifts.
  • Just one idea: www.godiva.com =)
  • USR Palm Pilot PDA
  • Legos!
  • Gift certificates for just about anything. Movie rentals and dinners at nice restaurants are good.
  • One pound of coffee (in a festive container) from the local coffee roaster.
  • A miniature, hand-carved wooden totem pole with raven. I found it at a gift shop which primarily serves tourists, but which has some quality goods along with the junk.
  • "Duke Nukem' 3-D" or "Total Annihilation"
  • Make a personal coupon book for your spouse or other loved-one. Here's the idea, where you state, "this coupon good for..."
    • "A 1-hour foot-massage",
    • "One unqualified 'Yes'",
    • "Silence. I will immediately shut up--and my feelings won't be hurt."
  • Woolen socks and other warm cozy things.
  • A bottle of wine or fine spirits.
  • Go to a used-book store and buy every title by the recipient's favorite mystery writer.
  • Goods from a museum store.

Next Newsletter: January 9, 1998!

The next issue, according to our nominal biweekly schedule, should come out on Jan. 2 at which time Guy will be in England, Tom in Colorado, and many of our readers happily eschewing email. So, I think we'll have to survive without T3E news for a week.

Solstice is a nice turning point at Fairbanks' latitude, so may we wish you Happy Hanukkah, Joyeux Noel, Happy New Year, Festive Solstice, and Happy Holidays.

Quick-Tip Q & A


A: {{ You're writing a monte-carlo simulation, which will use very long
      sequences of pseudo-random numbers, to run on multiple PEs.  How
      can you ensure that the generated sequences will not overlap? }}

  # Here are three relatively quick "tips."  Note that the second,
  # "sequence splitting" was described in detail in an earlier article.
  # 
  # These are taken from the previously mentioned reference
  # ("http://nhse.cs.rice.edu/NHSEreview/RNG"), "Random Number Generators
  # for Parallel Computers," by Paul D. Coddington:


  ## Starting with a good serial random-number generator:
  #   
  #  o Leapfrog -
  # 
  #      The sequence is partitioned among the processors
  #      in a cyclic fashion, like a deck of cards dealt to
  #      card players.
  # 
  # 
  #  o Sequence Splitting -
  # 
  #      The sequence is partitioned among processors in a
  #      block fashion, by splitting it into
  #      non-overlapping contiguous sections.
  # 
  # 
  #  o Independent Sequences -
  # 
  #      For some generators, the initial seeds can be
  #      chosen in such a way as to produce long period
  #      independent subsequences on each processor."



Q: On the T3E, how can you determine your program's memory usage?

[ Answers, questions, and tips graciously accepted. ]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top