ARSC T3E Users' Newsletter 140, April 3, 1998

Disk Quotas Now Enforced on Yukon

[ This is from yukon, "news quota". Important news for Yukon users! ]

During system downtime on March 31, 1998, the quota system was put into enforce mode. (Previously, it had been in warning mode.) Most users will not notice this change, as their use of disk is considerably lower than the quota limit. We encourage all users to remain conservative in their use of disk resources.

In enforce mode, an operation that attempts to create a file which would put you over quota will be aborted. The file will be truncated to prevent your total disk usage from exceeding your quota.

NQS jobs, as well as interactive jobs, will be aborted if they attempt to exceed quota.

All users have the same quota, and quotas are enforced by user, not by project. Quotas are applied to each file system independently, and the current limits are:


        /tmp :                    10GBytes
        /u1  :                    10GBytes
        /u2  :                    10GBytes

The quota limit , given above, is a hard limit, which may not be exceeded. In addition, the system supports a warning level which is 90% of the hard limit.

Users of Cray-supported shells (csh, ksh, sh) will receive notice printed to the console when they exceed either the warning level or reach their hard limit. Users of the tcsh (and other non-Cray supported shells) should add a call to the Cray utility, quotamon , to their .cshrc, or other appropriate startup script, in order to receive such notices. However, ARSC testing shows that, even with quotamon running, quota notices to tcsh users will be intermittent .

When you receive a quota warning or enforcement message, you need to remove obsolete files, ftp inactive files to denali for backup to CRL, and otherwise reduce your use of disk. Results from the operation that attempted to exceed quota must be considered suspect.

Some useful commands:


        quota -v      to see your quota setting and current usage.

        du            reports on disk usage.

Please contact consultants at consult@arsc.edu or 907-450-8602 for any assistance with quota or storage questions.

GRMVIEW Formatter

Call me a Luddite, but I miss the T3D's mppview utility. It gave a simple visualization of the current job layout on the torus, using blazingly fast, and good enough, ASCII Graphics. I like to see what's going on in the torus, and how my job fits in.

This article gives a simple script for presenting grmview output in a friendlier format. First, an example of the normal grmview -q output:


  Exec Queue: 5 entries total. 4 running, 1 queued
    uid   gid  acid    Label Size  BasePE   ApId Command           Note
   3648  3628  3628     -      62       0   1312 cz                  -
   3939  3945  3945     -      20    0x40   1317 pkv.20              -
   3325  3363  3363     -       4    0x54   1321 les                 -
   1235   206   206     -       3    0x58   4842 circle              -

    uid   gid  acid    Label Size   ApId Command         Reason
   1235   206   206     -       7   5349 circle          Ap. limit

Below is the corresponding formatted output. It begins with a legend into the graph of the torus. I can quickly scan for my UID (1235) and see that my 3-PE job is running, and my 7 PE job is blocked waiting for PEs (in the grmview output, this is denoted "Ap. limit", for "application PE limit reached").

Yukon is configured with 94 PEs available for parallel applications, which explains all the apparently unused PEs at the bottom of the graph (those are the command and OS PEs). Knowing this, the graph tells me that there are 5 unused PEs (2 stranded between jobs A and B, plus numbers 92-93). If possible, I should kill the 7 PE job, resubmit it as a 5 PE job, and let the system migrate jobs B, C, and D to make room for it to run.


       UID  SIZE BasePE      COMMAND
       ==== ==== =========== ==============
   A - 3648 62   0   [    0] cz
   B - 3939 20   64  [ 0x40] pkv.20     
   C - 3325 4    84  [ 0x54] les
   D - 1235 3    88  [ 0x58] circle
   E - 1235 7    BLOCKED     5349
  --------------------------------------------------
   0  
    A....A....A....A....A....A....A....A..
   8  
  ..A....A....A....A....A....A....A....A..
   16 
  ..A....A....A....A....A....A....A....A..
   24 
  ..A....A....A....A....A....A....A....A..
   32 
  ..A....A....A....A....A....A....A....A..
   40 
  ..A....A....A....A....A....A....A....A..
   48 
  ..A....A....A....A....A....A....A....A..
   56 
  ..A....A....A....A....A....A    -....-
   64 
    B....B....B....B....B....B....B....B..
   72 
  ..B....B....B....B....B....B....B....B..
   80 
  ..B....B....B....B    C....C....C....C
   88 
    D....D....D    -....-....-....-....-..
   96 
  ..-....-....-....-....-....-....-....-
  --------------------------------------------------

Here's the script itself. Feel free to grab and modify, but it comes with no warranty.


#!/usr/local/bin/perl
# 
# Prints a grid showing what job is assigned to each PE.  
# 
# Tom Baring, ARSC, Dec. 1997
############################################################

$MAXPES = 104;                 # Number of PEs on the system

&init;
&getConfig;
&showConfig;

############################################################
# Initialize array of job codes
############################################################
sub init {
  for ($i=0; $i<$MAXPES; $i++) {
    $pe[$i] = "-";
  }
}

############################################################
# Get current job assignment configuration from grmview.
# Each PE will be labeled with a job code letter or, if 
# unused, will retain its unused flag.  Ap. limited jobs
# will be marked "blocked".
############################################################
sub getConfig {

  # Jobs will be coded in the display using the alphabet
  @possCodes = split (' ', "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z");

  # Look at each line of the grmview output, sorted by base 
  # pe number.  Is field 5 in current version.

  foreach (`grmview -qt
sort +5`) {
    $c = shift (@possCodes);
    if (m
Ap\. limit
) {

      # This job is blocked waiting for pes

      ($uid{$c},$gid{$c},$acid{$c},$label{$c},$size{$c},$command{$c}) = split;
      $basePEstr{$c} = sprintf ("%-11s", "BLOCKED"); 
    }
    else {

      # Job is running.

      ($uid{$c},$gid{$c},$acid{$c},$label{$c},$size{$c},
                         $basePE{$c},$apid{$c},$command{$c},$note{$c}) = split;

      $basePEstr{$c} = sprintf ("%-4s[%5s]", hex($basePE{$c}), $basePE{$c}); 

      for ($i=hex($basePE{$c}); $i<hex($basePE{$c}) + $size{$c}; $i++) {
        $pe[$i] = $c;
      }
    }

    # Keep track of all user ids.  
    $uidlst .= " " . $uid{$c};
  }
}


############################################################
# Print PE grid and legend
############################################################
sub showConfig {

  # Print legend header

  printf ("\n");
  printf (" %1s   %-4s %-4s %-11s %-14s\n", 
            " ", "UID", "SIZE", "BasePE", "COMMAND");
  printf (" %1s   %-4s %-4s %-11s %-14s\n", 
            " ", "====", "====", "===========", "==============");


  # Print legend data

  foreach (keys(%uid)) {
    printf (" %1s - %-4s %-4s %-11s %-30s\n", 
            $_ ,$uid{$_}, $size{$_}, $basePEstr{$_},$command{$_});
  }

  
  # Print the grid of PEs

  printf ("--------------------------------------------------\n");
  for ($i=0; $i<$MAXPES; $i++) {
    printf (" %-3d
  ", $i) unless ($i % 8);
    printf ("%2s%1s%2s",  $pe[$i] eq $pe[int($i-1)] ? ".." : "  " , $pe[$i], 
                         $pe[$i] eq $pe[int($i+1)] ? ".." : "  " );
    printf ("\n") unless (($i + 1) % 8);
  }
  printf ("--------------------------------------------------\n");
}
#####################################################################

Please let me know if you come up with any good modifications to this script.

NPACI HPC Forum

[ Jay Boisseau of SDSC sent us this announcement. The HPC Forum sounds like a valuable contribution to our community, with a mission similar to that of this newsletter. By sharing our experience, we may avoid repeating mistakes, learn a lot, and move the community forward. ]

Thousands of HPC users around the country and the world are attempting to use the latest hardware and software to solve the biggest problems in science. Such leading-edge users sometimes face problems that their prior experience never anticipated, their colleagues never addressed, and possibly even the experts at the national centers are unfamiliar with.

To address these problems, NPACI has established the HPC Forum to enable the HPC user community to ask questions and discuss issues regarding any and all HPC topics. The purpose of HPC Forum is to enable and accelerate the efforts of HPC users by providing a venue for users to ask questions and discuss problems. Experts from the NPACI staff will read HPC Forum (and ask questions themselves as well as answer them!), but we hope everyone using HPC techniques, including researchers, staff from other centers, applications developers, and hardware vendors, will look to HPC Forum as a valuable resource for finding answers and developing solutions. We will actively recruit the participation of the entire HPC community in this endeavor.

HPC Forum is open to everyone and is provided by NPACI as a public service. We hope that it provides you with a valuable source of information and that you will in turn contribute your own expertise and experiences to help the community.

If you have any questions, comments, or suggestions, please use the 'Submit Questions/Comments' link on the HPC Forum main page, which is at the URL:

http://www.npaci.edu/Forum

Quick-Tip Q & A


A: {{ How can I determine if my T3E job was checkpointed, held, and
      released in the middle of its run? }}


  Use qsub's "-j <filename>" option.  

  "-j <filename>" instructs NQS to store the "job log" to the file
  <filename>. This file will contain a lot of nitty-gritty data on
  NQS's handling of the job. Among the information will be the time(s)
  at which it was held and released, and (if it was running when held)
  the time(s) at which it was checkpointed.

  Example qsub script:

      #QSUB -q mpp
      #QSUB -l mpp_p=3
      #QSUB -l mpp_t=30:00
      #QSUB -j my.job.log

      cd /u1/uaf/user/Progs/
      date
      mpprun -n3 ./prog
      date


  Excerpt from the resultant file, "my.job.log"

      [...]
      04/02 14:13:58 Running in queue <Qsmall>.
      04/02 14:14:53 Request hold initiated.
      04/02 14:15:55 Checkpointed to </ckpt/nqs/chkpnt/++2zd++1>, id=<6093>.
      04/02 14:17:14 Request released.
      04/02 14:17:15 Resumed.
      04/02 14:17:15 Restarted, ID=<6093>.
      [...]

A: BONUS ANSWER:  A reader sent yet another way to delete the ^M's
   from an FTPed file:

        tr -d \\015 < in > out

Q: When I first learned to program, I used "bubble-sort."  Now I use
   "quick-sort."  Isn't there any faster-yet sorting algorithm?
[ Answers, questions, and tips graciously accepted. ]
Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top