ARSC HPC Users' Newsletter 361, May 5, 2007



Introduction to strace, Part I of II

[ by: Craig Stephenson ]

When it comes to elusive bugs and unexpected runtime behavior, strace is an invaluable tool to keep in your toolbox. strace reports all interaction between a process and the kernel, including file reads and writes, memory allocation, network activity, and signals. It can provide many clues as to why a program has quit abruptly, hung, or produced bad results. As an added benefit, source code is not required to analyze a program with strace.

This article is geared toward the Linux version of strace, which is available on midnight, nelchina and ARSC Linux workstations. There is an executable called "strace" on iceberg and iceflyer, but it has a completely different purpose.

Suppose you are running a poorly documented executable, "dataProcessor," given to you by a colleague without accompanying source code. Assume you know this program processes an input file and uses it to produce a corresponding output file. However, the command syntax was never explained to you.

Hoping to find some usage examples, you type:

  % dataProcessor -h
  terminate called after throwing an instance of 'std::bad_alloc'
    what():  St9bad_alloc

Not very helpful. Clearly, this program didn't have the type of error checking that you would expect because it crashed. When the command is prefixed with strace, the following output is produced:

  % strace dataProcessor -h
  execve("dataProcessor", ["dataProcessor", "-h"], [/* 40 vars */]) = 0
  brk(0)                                  = 0x502000
  mmap(NULL, 4096, PROT_READ
MAP_ANONYMOUS, -1, 0) = 0x2aaaaaac5000
  open("-h", O_RDONLY)                    = -1 ENOENT (No such file or directory)
  --- SIGABRT (Aborted) @ 0 (0) ---
  +++ killed by SIGABRT +++
  Process 7242 detached

(Note: The output of strace can be quite large. Throughout this article, strace output has been abbreviated for the sake of illustration. In most cases, it's beneficial to filter the output of strace with grep or to write the output to a file by redirecting stderr. Also, modifiers can be supplied to the "-e" option to restrict strace's output to filesystem activities, signals, network functions, etc. A complete list of these modifiers can be found in the strace man page.)

In the example above, it appears the dataProcessor program is treating "-h" as the name of a file. Furthermore, the "O_RDONLY" parameter passed to the "open" function shows that the program is expecting the name of an input file as its first argument. There is no file named "-h", so the open function returns -1 to indicate an error has occurred, and sets the errno variable to "ENOENT", which means "No such file or directory". For more information on the errno variable, please reference the errno man page. In light of this discovery, you might presume that the second argument is the name of the output file to be created by the program. To test your theory, you issue the following command:

  % strace dataProcessor myDataFile.out
  execve("dataProcessor", ["dataProcessor", "", "myDataFile.out"], [/* 40 vars */]) = 0
  brk(0)                                  = 0x502000
  mmap(NULL, 4096, PROT_READ
MAP_ANONYMOUS, -1, 0) = 0x2aaaaaac5000
  open("", O_RDONLY) = 3
  lseek(3, 0, SEEK_END)                   = 14629
  lseek(3, 0, SEEK_CUR)                   = 14629
  lseek(3, 0, SEEK_SET)                   = 0

After the typical strace output scrolls past the screen, it displays the aforementioned "open" function call corresponding to the input file you specified, but then appears to hang at an incomplete "read" function call several lines later.

What does this mean? If the input file being processed is not particularly large, you might want to take a moment to see if you're trying to read it from $ARCHIVE_HOME. If so, you may need to bring it on-line before trying to read it, to avoid a long delay. And, indeed, in this particular example, "" did live on $ARCHIVE_HOME and happened to be offline. Therefore, the file needed to be brought online, i.e., staged from ARSC's tape library. (For I/O intensive data processing, it's best to copy the input files to $WORKDIR and then run the program from within that filesystem.)

Now that you've gained a basic understanding of how the hypothetical dataProcessor command works, have copied the input file to $WORKDIR, and are currently in the $WORKDIR file system, it should be safe to issue the same command used in the previous example:

  % dataProcessor myDataFile.out

The program does not take long to finish, but "myDataFile.out" is 0 bytes, and there were no errors. Now what? Run strace!

  % strace dataProcessor myDataFile.out
  execve("dataProcessor", ["dataProcessor", "", "myDataFile.out"], [/* 40 vars */]) = 0
  brk(0)                                  = 0x502000
  mmap(NULL, 4096, PROT_READ
MAP_ANONYMOUS, -1, 0) = 0x2aaaaaac5000
  open("myDataFile.out", O_WRONLY
O_TRUNC, 0666) = 4
  close(4)                                = -1 EDQUOT (Disk quota exceeded)
  close(3)                                = 0
  exit_group(0)                           = ?
  Process 7313 detached

You are over quota, but without strace, it could have taken quite a while to realize this. Ultimately, the dataProcessor program is to blame for its neglect towards error checking, but without the source code available, strace provides a means to sort through the wreckage.

Here are two useful options to try out on your own:

  strace -r <command>   
      -- prefix each line of strace output with a relative timestamp
  strace -p <PID>       
      -- attach strace to a process that's already running

(Part II of this series will cover more advanced features of strace as well as how to use strace with MPI.)


Shell "dot" files

[ by: Oralee Nudson ]

To enable customization of your working environment, Unix shells support personalization using the ".rc", ".login", and ".profile" files. The content of these "dot" files overrides system-wide shell setting defaults as specified in /etc/csh.login and /etc/profile. Creating aliases, umask values, shortcut commands (for example making a temporary recycling bin for deleted files), or writing shell scripts to automate repetitive tasks are a few of the common customizations created from within the "dot" files.

The main difference between these three "dot" files is the time at which each file is executed. In general, the run control file (for example .rc or .bshrc for Bourne shell) is read every time a new shell is invoked, whereas the ".login" or ".profile" files are read only during shell login. The following is a chart showing the order in which different shells execute the "dot" files for both invocation and login sessions.


Invoked: ~/.cshrc                        
Login:   /etc/login  


Invoked: /etc/csh.cshrc  
Login:   /etc/csh.cshrc  


Invoked: ~/.profile  
  "ENV" environment variable setting(*)
Login:   /etc/profile  


Invoked: ~/.bashrc       
Login:   /etc/profile  

(*) Korn shell executes the ~/.profile file, which may optionally set a value
    for the environment variable, "ENV."  If ENV is set, then it must
    be be the path to a script.  This script will then be executed
    whenever the shell is invoked, and is typically set to the path
    of the .kshrc file.

So, if you're interested in customizing your working environment and you'd like the commands to be read every time a new shell is opened, the data should be placed in the appropriate invoked session file. Common examples include alias commands such as "alias c 'clear' " in the .cshrc file, or "umask 077" for default permission bits.

If you'd like to have the "dot" file contents executed just once during the initial login, the commands should be placed in an appropriate ".login" or ".profile" file. Setting terminal characteristics with "stty" or initializing terminals with "tset" are examples of values you may want to set during your initial login.

The best way to change your login shell on an ARSC system is to send an email to or call the ARSC Consultants and tell us your preferred default shell. We can easily switch your user account login shell on any machine. This may help avoid future headaches when attempting to track down and debug errors related to inconsistencies with login shells, pbs or loadleveler script errors, modifications made to the "dot" files we may not see initially, and errors related to special setups such as modules. Otherwise, If you would like to experiment with a temporary shell, type the new shell's name (e.g. csh, tcsh, bash, or ksh) in your current window's command prompt.

Many shells exist, and not all are supported on every machine. Examples of various startup commands included in different shell "dot" files are widely available on the web, including in past Quick-Tips: > csh (under "C," for "csh," in the Quick-Tip Index) , > ksh , > zsh , > Favorite shell prompts , > Favorite shell aliases .


Congrats to "Make Clean" (ARSC Mud Volleyball Team)

Make Clean performed surprising well in the UAF all-campus mud volleyball tournament last week. The team was comprised of ARSC staff members Alec Bennett, Oralee Nudson, Jill Ladegard, and Newsletter co-editor, Don Bahls, plus two ARSC users, who, since we have the utmost respect for all our users, shall remain nameless.

It warmed from 45 degrees to a balmy 55 by the end of the tournament.

Make Clean managed 4th (or so) out of 14 teams (or so).


Quick-Tip Q & A

# Was it the full moon?  The muse definitely struck.  A big thanks
# for all responses to the previous TWO (2!) questions: 

A: [[ If I have an array of double-precision floating point values in C
   [[ and I want to write each value out to a file in an ASCII format 
   [[ so that when they are read back in I have no loss of precision, 
   [[ how would I do it?  I'd like the solution to avoid writing 
   [[ characters that don't affect the precision, like trailing 0's.  
   [[ (This question might be of interest to Fortran users as well).

  # Orion Lawlor

          fprintf(f,"%.17g ",&arr[i]); 
  is what you want.  This leaves 17 decimal digits, which is just enough
  to capture all the bits of a double, but leaves off trailing zeros.
  Every double I've tried this way writes and reads unchanged, and I've
  tried millions of random doubles.  Note that "%.16g" gets the last
  digit wrong fairly frequently, so you do need at least 17 digits!

  There's also a new format character in C99, "%a", that writes and
  reads double-precision floats in *hex*, not decimal.  So
  This isn't at all human-readable, but it's more compact than %.17g,
  and guaranteed bit-for-bit exact.

  # Lorin Hochstein

  As a quick-and-dirty hack, you can reintepret the floating-point
  values as integers and write them out that way (the code below writes
  them out in hexadecimal format):

  #include <stdio.h>
  #include <assert.h>
  #include <math.h>

  * Write a double-precision floating point number to a text file
  void write_double(FILE *output, double dvalue)
          /* This only works if a double is exactly twice the size of an int */

          int *p = (int*)&dvalue;
          fprintf(output,"%x %x\n",*p,*(p+1));

  * Read a double-precision floating point number from a text file
  double read_double(FILE *input)
          int ipair[2] = {0,0};
          /* This code only works if a double is exactly twice the size of an int */

          fscanf(input,"%x %x\n",ipair,ipair+1);
          return *(double*)ipair;

  int main ()
          const double PI = 4 * atan(1);
          double dval;
          FILE *fp = fopen("temp.txt","w");
          fp = fopen("temp.txt","r");
          dval = read_double(fp);
          return 0;

  However, this method is horribly non-portable, and the ASCII string
  representation will be meaningless to a human being looking at the
  text file.

  A better solution is to use David Gay's floating-point conversion
  code, which is in netlib:

  # Chris Petrich
  Try this:

    double a = exp(1e-10)-1.0;
    double b = 18.5;
    printf("%.30g %.30g\n",a,b);

  the number of digits (here "30") should be larger than the max number
  of significant decimal digits of the mantissa (16 for 64-bit floats)
  plus whatever it takes if %g choses to format a number with leading
  zeros (e.g. 0.003), i.e. somewhere around 6 or 8 characters I think.
  Basically anything larger than like 25 should work.

  # Sean Ziegeler
  This is actually harder to do that you would think because
  fractional values are represented in base-2 and you want to print
  out in base-10. For example, the value 7 is most likely 7.0000001
  (in single precision, even more zeros in double-precision).

  The C++ output stream operator tries to compromise and assumes that
  a large number of zeros followed a different digit (or a large
  number of 9's followed by a different digit) is most likely all
  zeros and truncates at the first zero.  If you are using pure C,
  then I know of no built-in routine to do this.  You may be able to
  find some math libraries out there with specialized output routines,
  or write your own stripped-down, customized printf that can decided
  where to draw the line based on your specific requirements.

A: [[ When I accidentally "cat" a binary file and my terminal gets messed
   [[ up, how can I make it usable again?  And why doesn't "stty sane"
   [[  seem to help?

  # Greg Newby and Martin Luthi gave the same answer:

  Use "reset".

  # Sean Ziegeler
  Most of the time it is your terminal program that gets confused by
  the binary.  In many cases, you can "reset" your terminal and it will
  be ok.  For example in KDE's terminal program (Konsole) you can go to
  the menu item: Edit->Reset and Clear Terminal.  gnome-terminal has a
  similar menu item.  Some versions of xterm provide menus where you must
  control-click, for example.  The man page for your terminal program
  on your platform should tell you how to access such menus.

  That said, I've seen it in such bad shape that nothing short of closing
  the terminal program and logging back in would fix it.

  # Brad Havel (self-described "uncouth user") offers this:
  (I figured this one out when I was in college and had a draconian
  system admin who allowed a single login session per user.  If you
  fubared the session you were essentially locked out until you could
  get him to bounce the session and of course the sessions didn't time
  out either, so...)

  "cat" another binary file, piped through "more," then hit <CNTL-C>
  when you see readable text at the pause.  For example:
  [=> cat /bin/bash 
  This will pause after each page and you can just abort the command.
  From what I've seen, this will allow you to keep a "sane" terminal
  session and bring the text back to being human readable.

Q: It's an easy Unix task to create the union of two simple lists:
      cat old.txt additions.txt 
   But how can I create the difference?  I.e., I want to expunge every
   line from "old.txt" which has a duplicate in "trash_em.txt".

[[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top