• Storage Locations and Environment Variables
• Long Term Storage Introduction
• Access to bigdipper
• SAM-QFS file management utilities
  • archive
  • release
  • sfind
  • stage
  • batch_stage
  • sls
  • sdu
• Saving One Tape Archive Copy
• Long Term Storage Best Practices
• Policies

Several environment variables are defined on each ARSC system to provide convenient access to the available storage directories. We recommend you use these environment variables in batch scripts and on the command line to improve portability of your work within the ARSC environment. Because the variables have identical names and refer to the same types of storage on every system, it is possible to move scripts among ARSC systems with minimal changes by using the storage environment variables rather than explicit paths.

Home directories are intended to contain files used for customizing your basic working environment only. Your home directory path is stored in the $HOME environment variable on every system. Home directories are unique to each platform and are routinely backed up. Small data files specific to a particular system belong here such as account configuration .cshrc/.login or .profile files. In many cases, if an executable compiled for a system is small enough, $HOME would be a logical place to store the executable.

Home directories are limited in size, should you require a larger $HOME quota, please contact User Support with your request.

Work directories are typically mounted locally or over a fast network connection. Following the "local cache" concept, doing your work in $WORKDIR instead of $ARCHIVE (where disk space is NFS mounted from a remote machine) will allow for faster I/O and save network bandwidth. $WORKDIR is available on most systems and always refers to the same style of storage space: a fast file system addressable by all nodes on that system.

Files in $WORKDIR or $SCRATCH that have not been accessed in over 30 days are subject to purging (i.e. removal). In rare circumstances purging may occur in less than 30 days should a filesystem become full, etc. Purging is performed in order to maintain adequate free space in the high-performance filesystem for all users to effectively share the space.

As work directories are not backed up , please copy data that needs to be backed up to archive storage as soon as possible.

Larger $HOME and $WORKDIR quota requests may be sent to User Support .

For more details on local storage configurations please see 'news storage' on the system you are using:

• Pacman

Archive directories are intended for data to be stored long term. Files in this directory are automatically backed up to tape after several hours. This filesystem provides the slowest access overall, however no quotas are set on this filesystem. See the Long Term Storage Introduction and Best Practices for a more information.

Bigdipper is a Sun SPARC Enterprise T5440 Server providing long term, backed up data storage for ARSC resources. Bigdipper is the /archive filesystem for pacman. The server utilizes SAM-QFS for managing user files stored in the corresponding archive file system. SAM-QFS consists of "online" disk storage, "offline" tape storage, and a set of daemons managing file status.

When a file is saved to the archive filesystem, the file is initially "online". Shortly thereafter, two copies of the file are automatically made on separate tapes. The file will remain "online" if sufficient disk space is available, otherwise the file will be taken "offline" and will be removed from disk storage, leaving two copies on tape. If the file is "online", it is immediately accessible to the user. If the file is "offline", then the user can request a copy of the file to be brought "online" by the SAM-QFS system. Please read more on the stage and batch_stage commands to request files to be brought back "online". The staging process is highly encouraged, especially when potentially working with thousands of "offline" files.

Users of pacman have an account on bigdipper, where they can more efficiently manage their files. The SAM-QFS commands listed below can be used to manage data from bigdipper.

The following commands must be executed on bigdipper where your files are stored. For more information, please read the man page for the particular command (e.g. man sls .

archive

The "archive" command issues a request for a file, or group of files matching wildcards (when * or ? are specified), to be copied to tape. Please note that the return of this command does not mean the copy to tape has finished. Check for file transfer completion with the sfind or sls commands. Also, disk space is not released unless the release command is executed.

SAM-QFS will archive files automatically. This normally occurs within approximately an hour of creation or modification.

Usage:

archive filename

After a file has been archived, three copies exist -- two on tape and one on disk. The "release" command will remove the copy of the "online" file if the "offline" (tape) copies have already been made. A directory listing will remain for the now "offline" file. To use the file after issuing the "release" command, the file will need to be copied from tape back to disk (e.g. manually by stage. )

SAM-QFS may release files automatically over time depending on the level of activity on the file system.

Usage:

release filename

The "sfind" command finds files with the requested attributes. Searchable attributes include:

• -offline (file copied to tape and disk space released)
• -online (copy exists on disk)
• -archdone (all archive/stage steps completed)
• -copies 2 (both copies exist on tape)

In most cases, "sfind" uses the same keywords as sls -DK.

Usage:

sfind -offline

Examples :

Search for offline files with $ARCHIVE

bigdipper % sfind $ARCHIVE -offline

Search for files that have two copies on tape:

bigdipper % sfind $ARCHIVE -copies 2

The "stage" command initiates a request to copy an "offline" (tape copy only) file to be placed "online" (on disk). Only online files may be read, copied, etc. Giving advanced notice that the file will be needed is a way to ensure the file is "online" and ready to use when requested. Otherwise, an attempt to read the file will automatically stage the file, although it may take some time for the tape to mount and a new "online" copy of the file to be placed on disk.

Please note that the return of this command does not mean the copy to disk has finished. Check for "online" file completion with the sfind or sls commands.

When staging large numbers of files, the batch_stage command may perform more optimally. Therefore, consider using "batch_stage" over the "stage" command in this situation.

Usage:

stage filename

Examples :

The following sfind command will find all of your f90 files and stage them to disk, then will echo the filenames to the terminal:

bigdipper % sfind . -name "*.f90" -offline -exec stage {} \; -exec echo {} \;

The "batch_stage" command brings a set of files "online". Files are staged in the order they are written to tape. This minimizes tape seeks and typically reduces the amount of time it takes to bring multiple files back "online".

Usage:

batch_stage filenames

Examples :

To stage several files by name, simply list all the files that need to be staged following the "batch_stage" command:

bigdipper % batch_stage fileone filetwo filethree

Wildcards may also be used. The following command will stage all the files in the $ARCHIVE/data directory:

bigdipper % batch_stage $ARCHIVE/data/*

A list of files to be staged can also be supplied through stdin, making "batch_stage" ideal to use in conjunction with the "find" command:

bigdipper % find $ARCHIVE/somedirectory/ -name \*.nc | batch_stage -i

The "sls" command is an extended version of the "ls" command which lists files including their SAM-QFS attributes. The -D option will show most of the SAM-QFS attributes.

Usage:

sls filename

Examples:

Show detailed description of SAM information for all files in $ARCHIVE

bigdipper % sls -D $ARCHIVE

Show two lines of output with SAM information:

bigdipper % sls -2 $ARCHIVE/myfile

Usage:

sdu directory

Examples:

Show summary (-s) of usage for each subdirectory in $ARCHIVE in kilobytes (-k):

bigdipper % sdu -sk $ARCHIVE/*

Show summary (-s) of usage for $ARCHIVE directory in human readable form (-h):

bigdipper % sdu -sh $ARCHIVE

By default, files written to $ARCHIVE or /projects are written to two tapes to ensure that a single tape failure doesn't result in data loss.   In cases where data can be easily regenerated users can request that only one tape copy of files be generated by creating a directory called "one_copy".  

e.g.

% mkdir $ARCHIVE/one_copy

Any files copied or created in this directory will have a single tape copy generated rather than two copies. 

NOTE: files moved from one $ARCHIVE directory to a "one_copy" subdirectory using the "mv" will maintain two copies.

The following command can be used to determine which files in a directory have only one tape copy generated:

% sfind . -copy 2 -a ! -copy 3

Similarly the following command will list files with have two tape copies:

% sfind . -copy 2 -a -copy 3

The following is a list of practices which can improve the performance and usability of long term storage at ARSC.

• Storage resources are finite Be judicious in the data you store. Periodically review your data holdings and carefully consider what you need; remove what you do not.
  
• Avoid large numbers of files Having hundreds of thousands of files in your $ARCHIVE can drastically degrade the performance of the long term storage daemons. When possible save related files into a single tar file.
  
• Large transfers to $ARCHIVE When copying more than one terabyte of data into $ARCHIVE limit the number of streams (e.g. cp's) to one. This will allow the archiving daemon to keep up with the creation of tape copies while leaving tape drives available for other users.
  
  Faster transfer speeds can be achieved by using rcp instead of cp. For example:
  

  /usr/bin/rcp results.tar "bigdip-s:$ARCHIVE"

• File transfers from $ARCHIVE When transferring large numbers of files from $ARCHIVE to $WORKDIR or a remote location, staging the files using the batch_stage prior to the transfer may drastically reduce the time required to complete the transfer.
  
  Faster transfer speeds can be achieved by using rcp instead of cp. For example:
  

  /usr/bin/rcp "bigdip-s:$ARCHIVE/results.tar" $WORKDIR

• Using batch_stage When using batch_stage , run only one instance at a time. There are a limited number of tape drives shared by all users, therefore when the demand for tape drives is greater than the number of tape drives available, performance becomes drastically worse for everyone.
  
• Use tar files If you have many files which will be used together (such as source code), use a tar file to store all of the files together.
  
• Creating tar files When creating a tar file from files stored in $ARCHIVE it is most effective to do this directly on bigdipper. The batch_stage command should be used to bring all files online prior to issuing the tar command.
  
• Large fIles While terabyte and larger files can be accommodated, it is generally best to keep file sizes under 250 gigabytes in order to reduce chances of problems during transfer, archiving and staging.
  
• Save to $ARCHIVE as you work it is best to make copies of the files to $ARCHIVE as they are created, rather than saving the work up and copying them all at once. The archive system works better with regular smaller volumes of new data than it does with occasional floods of new data. This practice will also minimize the impact of a catastrophic failure of the temporary filesystem(s) on the HPC system.
  
• Verify transfers are successful When transferring files between systems, it is best to confirm that the copy matches before removing the source. Use of various utilities (' sum -r , ' md5 , etc.) can be used to check integrity of the copy and source. It is best to use these utilities on the system where the file resides in order to avoid NFS related issue (e.g. cache effects).