ARSC T3E Users' Newsletter 197, June 9, 2000

CUG Report

The following excerpts are from the trip reports of two ARSC staff members, Liam Forbes and Kurt Carlson, who attended the recent Holland CUG. They're intended to give the flavor of the CUG and share some information relevant to readers of this newsletter. This is extracted from hand-written notes taken during talks... if you want definitive info, visit and see numerous white papers and other documents.

General Impressions

Kurt Carlson:

The format was different from previous CUGs. Monday and Tuesday was reserved for Cray exclusively. Thursday and Friday was reserved for SGI exclusively. Wednesday was a mixed day. Two days with Cray was not enough. In my opinion Wednesday was a bust. Cray was not very talkative with SGI present. Considering the conference agenda was set before the divorce of Cray from SGI the format was better than one could reasonably expect, but some restructuring of the conference is warranted. I would personally prefer to see CUG return to "Cray Users Group". While there is certainly considerable value in vendor time with SGI, that should be an SGI user's group. CUG needs to retain some visualization, hopefully that can be from member presentations.

The "CUG Night Out" was a Canal Cruise on the 'Prins van Oranje' (built 1908). From Leiderdorp, we toured Dutch countryside... drawbridges, windmills, houses, flowers. Food was provided, of course, and a good opportunity to chat with others. Interesting feature to note: the water level in the canal was frequently above the surrounding land. An observation on Dutch housing (from cruise and walking about): many had large picture windows without blinds, one could see through the house from the streets... considering the art objects on many sills it seemed intentionally "open".

Liam Forbes:

The Grand Hotel Huis ter Duin was very nice. To see a view of the beach outside the hotel, you can go to a camera operated by Oregon State University, at: "". The tulips were not in bloom though, but there were cows & cyclists everywhere (draw your own connections). My biggest impression from this year's CUG is that the SGI/Cray split is causing FUD [fear, unease, dissention] which will be difficult to overcome. The Board has quite a challenge in front of it to guide CUG in the new "multiple vendor" world. I don't think that CUG will be dropping either vendor. Whether that is a good thing or bad thing is yet to be seen.


Cray Inc. Corporate Direction

  • mission: Provide the world's most powerful supercomputers to help solve our customer's most challenging problems.
  • 600 systems at 200 sites
  • Cray SV1, Cray T3E, Cray MTA, Cray SV2 (future)
  • Game Plan
    • Partner closely w/ customers
    • Deliver Cray T3E, Cray SV1 enhancements (2H00, 1H01 respectively)
    • Commercialize Cray MTA (no intention to merge w/ T3E,T90 architectures)
    • Aggressively meet Cray SV2 availability schedule
    • Become highly visible and vocal again
    • Become profitable & sustain it
    • Work to capture/recapture every high-end customer
  • Product Plans
      T90 -> SV1 -> SV2
             T3E -> SV2
                   -> SV2 -> SV3
      MTA -> MTA-2 -> MTA-3 -> .....
  • Cray & CUG Would like to have non-disclosure type meetings within the context of the SUMMITs to be able to candidly discuss product development & design.
  • Cray & SGI
    • $36million note due by end of this year
    • as service agreements come up for renewal SGI & Cray will convert contracts to Cray Inc. In the meantime, SGI is handling contract invoicing for Cray.
    • technology agreement covering intellectual property rights, patents, etc...
      • some intellectual property is transferred to Cray;
      • some is licensed to Cray w/out restrictions;
      • some is licensed to Cray w/ restrictions (ex: what Cray can do to develop the T3E).
    • transitional operations agreement for facilities, invoicing, contracts, etc...
  • MTA ==uniform shared memory multiprocessor w/ latency tolerance using fine-grain multithreading; no data caches or other hierarchy; no memory bandwidth bottleneck, absent hot-spots
  • every 64-bit memory word also has a full/empty bit; load & store can act as receive & send respectively; also implement locks & atomic updates
  • 16 protection domains (address spaces) one is used by the operating system; rest can be used to multiprogram the processor; limit the number of "big" jobs per processor
  • multithreaded w/ 3 operations per 64-bit instruction
  • 31 general purpose 64-bit registers per stream
  • paged program address space
  • segmented data address space
  • data addressability to byte level
  • explicit dependence lookahead
  • multiple orthogonally generated condition codes
  • explicit branch target registers
  • speculative loads
  • unprivileged traps
  • memory
    • hashed to logical addresses;
    • then interleaved among physical memory unit numbers and offsets;
    • number of memory units can be a power of 2 times any factor of 315;
    • 1, 2, 4GB of memory per processor;
    • supports over 1 memory reference per cycle per CPU.
Cray T3E
  • Manufacturing Status:
    • T3E-1200E
      • 600 MHz;
      • R6 chip: 25% faster interconnect;
      • rev 4 PCB
      • 64 mbit DRAMS at 256MB & 512MB per PE;
  • Engineering Status:
    • improvements in newly manufactured PEs
      • 675 MHz - T3E 2500F;
      • LC136 beta complete - ran OS on full margins;
      • new PCB;
    • GigaRing Enhancements:
      • higher density FC disk "DD330"
        • 36GBytes;
        • same form factor;
      • higher densities as devices become available;
      • there is a project on the SV1 team looking at GigE for GigaRing;
      • probably via bridges, not nodes;
      • support for IBM 3840escon, STK9840 drives
    • Software Status:
  • resiliency improvements
    • checkpoint/restart (2.0.2)
  • utilization improvements
    • psched enhancements (2.0.4)
      • gang scheduling & prime job
      • load balancer
    • migrate on swap (2.0.4)
    • scheduler performance improvements (2.0.5)
      • mini-launch
    • automatically compress dumps on completion & read compressed dumps
      • w/ crashmk
    • MPI 2 being implemented piece by piece based on customer requests; no plans to be fully compliant just to be compliant
    • Parallel Programming Language R&D
      • Co-arrays syntax - explicitly expresses replication of data images; will see on SV2.

    SV2 architecture combines T3E scalability w/ sustained, high bandwidth vector performance

CUG Members meeting to discuss some proposed modifications

moving vendor name from bylaws into handbook -

since '95/'96 vendor has changed named frequently which continually made the bylaws out of date weren't able to make bylaws changes fast enough to keep up w/ industry changes put "vendor = SGI & Cray Inc." in handbook and put "vendor as defined in handbook" in bylaws

Cray Hardware Overview
  • 3 efforts: SV1, SV2, MTA
  • Reviewed management structure
  • Targets: absolute bandwidth 10x microprocessor based architecture with 100x stride bandwidth
  • T3E expertise available to SV2
  • SV3 will be SV2 compatible
  • MTA-2 (working name PEAHI) in 2H2001
  • Not merging MTA/SV2, are leveraging (e.g., IOS)
  • SGI had end-of-lifed T3E, Cray has revived:

    600mhz to 675mhz and enabling adaptive routing

  • Reviewed timing of SV1E, SV1EX, and SV2
  • Reviewed SV2 speeds
  • SV2 (Unicos/sv) is being simulated now (on Origin)
  • SV2 will be 20x T90 price/performance
Cray Programming Environment
  • Goals:
    1. Performance enhancements on SV1 & T3E
    2. language compatibility across all platforms including legacy
    3. key features of standards (note: NOT standards conformance)
    4. interfaces on SV1 & T3E to move toward SV2
  • MPT (Message Passing Toolkit) 1.4 6/2000
  • PE 3.4: F90 potential for 20% performance improvement w/recompile
  • PE 3.5 4Q2000
  • SV2 cross compilation on IOS, modules environment
  • PVM dead, HPF Fortran likely dead (not CRAY sponsored)
  • Debugger already client/server model
  • Perf tools to follow c/s model There will be an SV1 workshop 23-25 October 2000 in Minnesota.

MPI_SM_POOL Variable and "Remote Protocol Queue"

There are a number of environment variables which you can set to control resources available to your MPI jobs. They're documented in "man mpi" on the T3E.

An ARSC user recently received this error message:

-MPI- FATAL: Remote protocol queue full

This is the Cray discussion in a problem report on the subject:

The "Remote protocol queue full" error means the shared memory queue area used for buffering messages has filled up. This typically occurs when a PE(s) is sending messages to another PE faster than that PE can receive them. The solution is to use environment variable MPI_SM_POOL to increase the size of the shared memory pool. The default size for MPI_SM_POOL is 8192 bytres. See section 2.7 of "Message Passing Toolkit: MPI Programmer's Manual" for additional information. MPI will be changed in a future release so that send operations will block when a "queue full" condition occurs.

Quick-Tip Q & A

A:{{ What's the best defense against bears?

We got five answers from four readers in Minnesota, Mississippi, and Alaska:

  • Always take someone along who runs slower than you do....
  • A well balanced stock portfolio
  • The Green Bay Packers front four!
  • A really big gun.
  • A loyal pet wolverine.
Assuming 4-legged bears with claws are your concern, and your wolverine's loyalty is doubtful, here's additional advice. This is from a flier by Alaskan bear expert, Larry Aumiller, entitled, "Bear Facts":

Close Encounters: What To Do If you see a bear, avoid it if you can. Give the bear every opportunity to avoid you. If you do encounter a bear at close distance, remain calm. Attacks are rare. Chances are, you are not in danger. Most bears are interested only in protecting food, cubs or their "personal space." Once the threat is removed, they will move on. Remember the following:

Identify Yourself: Let the bear know you are human. Talk to the bear in a normal voice. Wave your arms. [...] You may try to back away slowly diagonally, but if the bear follows, stop and hold your ground.

Don't Run: You can't outrun bear. They have been clocked at speeds up to 35 mph, and like dogs, they will chase fleeing animals. Bears often make bluff charges, sometimes to within 10 feet of their adversary, without making contact. Continue waving your arms and talking to the bear. If the bear gets too close, raise your voice and be more aggressive. Bang pots and pans. Use noisemakers. Never imitate bear sounds or make a high-pitched squeal.

If Attacked: If a bear actually makes contact, surrender! Fall to the ground and play dead. Lie flat on your stomach, or curl up in a ball with your hands behind your neck. Typically, a bear will break off its attack once it feels the threat has been eliminated. [...] In rare instances, particularly with black bears, an attacking bear may perceive a person as food. If the bear continues biting you long after you assume a defensive posture, it likely is a predatory attack. Fight back vigorously.

Protection Firearms should never be used as an alternative to common-sense approaches to bear encounters. If you are inexperienced with a firearm in emergency situations, you are more likely to be injured by a gun than a bear. It is illegal to carry firearms in some of Alaska's national parks, so check before you go.

A .300-Magnum rifle or a 12-gauge shotgun with rifled slugs are appropriate weapons if you have to shoot a bear. Heavy handguns such as a .44-Magnum may be inadequate in emergency situations, especially in untrained hands.

Defensive aerosol sprays which contain capsicum (red pepper extract) have been used with some success for protection against bears. These sprays may be effective at a range of 6-8 yards. If discharged upwind or in a vehicle, they can disable the user. Take appropriate precautions. If you carry a spray can, keep it handy and know how to use it.

Q: I re-use the same qsub script in different directories, copying it around as I change data sets.

The first command in the script is a "cd" right back to the directory from which I submit the script (the "cd" is required because NQS always starts the job in my home directory).

Thus, I must update the "cd" command in my script whenever I copy it to a new directory. It's BAD NEWS when I forget! Any advice?

[ Answers, questions, and tips graciously accepted. ]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top