ARSC HPC Users' Newsletter 344, July 20, 2006
Converting Byte Order of Binary Data Files on the Cray XD1
[ Thanks to Ed Korkven of ARSC for this article, the first in a Series on Porting Codes From the Cray X1. ]
In anticipation of the September 30 decommissioning of Klondike, ARSC's Cray X1, ARSC is urging Klondike users to begin planning for that transition. One of the major concerns that users may have is the portability of data files produced on the X1. Text files (i.e. those created with the Fortran OPEN option FORM='FORMATTED') are not a problem since by default the X1 creates ASCII text files and any machine to which users would port those files will read ASCII text files transparently, perhaps with minor end-of-line differences for non-Unix machines. But what about binary data files (i.e. those created with the OPEN option FORM='UNFORMATTED')?
The X1 differs from many earlier Cray machines in that by default it uses 32-bit IEEE standard floating point with two's complement arithmetic. This is an advantage since this format is commonly used on most other platforms likely to be targets for porting from the X1. (We would appreciate hearing of any exceptions.) Even when porting to another IEEE platform however, there are some issues to consider. In this and following newsletters we will discuss strategies for doing these ports.
In this issue, we will begin to take a look at the "byte order" of the contents of a binary data file. The byte order of a machine refers to the order in which it stores multi-byte data items in memory, and then in data files. Klondike is a "big-endian" machine. If moving to another big-endian machine, byte order will not be a consideration. One notable CPU set with opposite byte order however, is the x86 family. This includes the AMD chips that are prevalent in many machines, notably, Cray's XT3 and XD1 platforms. Reading unformatted X1 files on such computers will require some kind of special handling either on the X1 or on the target.
ARSC's XD1 machine has a very handy feature built in to its Portland Group Fortran compiler that automatically converts big-endian to little-endian and vice versa when UNFORMATTED data is read or written. Here is an example. The following is an X1 program that writes a binary data file called "output.dat".
program writer implicit none integer*4 :: out_i4 = X'deadbeef' real*4 :: out_r4 = X'cafefeed' integer*8 :: out_i8 = X'ceded0dace0faded' real*8 :: out_r8 = X'deafbeadedefface' open(11,file='output.dat', form='unformatted', access='sequential') write(11) out_i8, out_i4, out_r4, out_r8 close(11) end program writer
Compile this code like this: ftn -o writer writer.f90 And then execute: ./writer
Looking at the output with the od utility reveals its contents: od -x output.dat
0000000 0000 0018 cede d0da ce0f aded dead beef 0000020 cafe feed deaf bead edef face 0000 0018
Running the same program on the XD1 gives a different result:
0000000 0018 0000 aded ce0f d0da cede beef dead 0000020 feed cafe face edef bead deaf 0018 0000
Let's now read that file on the XD1 with this program:
program reader implicit none integer*4 :: in_i4, out_i4 = X'deadbeef' real*4 :: in_r4, out_r4 = X'cafefeed' integer*8 :: in_i8, out_i8 = X'ceded0dace0faded' real*8 :: in_r8, out_r8 = X'deafbeadedefface' open(11,file='output.dat', form='unformatted', access='sequential') read(11) in_i8, in_i4, in_r4, in_r8 close(11) if (in_i4 .eq. out_i4) print*, 'Integer*4 matches.' if (in_r4 .eq. out_r4) print*, 'Real*4 matches.' if (in_i8 .eq. out_i8) print*, 'Integer*8 matches.' if (in_r8 .eq. out_r8) print*, 'Real*8 matches.' end program reader
If we compile the reader program with: pgf90 -o reader reader.f90 we get no "match" messages during execution because the X1 big-endian output is read as little-endian values. Recompiling with:
pgf90 -byteswapio -o reader reader.f90gives the result we hoped for:
Integer*4 matches. Real*4 matches. Integer*8 matches. Real*8 matches.
So the Portland Group compiler provides a simple solution to byte order differences between big-endian and little-endian machines. And this example will hopefully motivate X1 users to start thinking about the porting process -- for data as well as code. ARSC staff will be happy to assist Klondike users during the transition.
Handel-C Programming Language Class, for FPGA Applications
ARSC will offer training for the Handel-C programming language and DK toolkit from Celoxica. The class will use Celoxica's RC-10 reconfigurable computing boards.
Class schedule: Tuesday July 25 through Friday July 28 9:00 am - 5:00 pm each day West Ridge Research Building (WRRB) room 104
This is a class for electrical & computing engineers and others interested in using field-programmable gate arrays (FPGAs) for a variety of applications. Celoxica's Handel-C is one of several programming toolkits that ARSC is investigating this summer as part of an ongoing study of high-performance reconfigurable computing.
Space is limited. Please email Greg Newby (newby AT arsc.edu) to register. There is no fee to attend this training.
Details on the class are available at:
The class will take place in the EPRS teaching laboratory, on the UAF campus, West Ridge Research Building (WRRB) room 104. Lunch and snacks will be provided.
Parallel Performance Evaluation Tools Workshop
This course is next week, already! For more information and to register, see:
Quick-Tip Q & A
A:[[ I have an existing library written in C/C++ that I would like to [[ call from a python script. Is there a way to call this existing [[ code without reimplementing the code in python? # # Thanks to Brad Chamberlain: # While I don't have firsthand experience with it, I do know that the Babel package out of LLNL is designed to support C/C++/Python interoperability (as well as Fortran and Java). For more information, see: http://www.llnl.gov/CASC/components/babel.html # # Editor's response # SWIG (http://www.swig.org/) is a tool which connects C/C++ to high level languages such as Python, Ruby, Java, Tcl, PHP. The steps are: 1) Build an interface file for your code so SWIG knows what portions of the code should be available in the python modules. 2) Run swig on the interface file to create a wrapped C file and the corresponding python module. 3) Compile the wrapped C file and other C source files. 4) Build a shared library using the object files. There is a basic tutorial here: http://www.swig.org/tutorial.html Q: I want to search for certain keywords in a monstrous text file. But for each "hit," I want to see, not just the line containing the word, but a few lines **preceding** it as well. I think I could write a perl script to do this, but is there another way?
[[ Answers, Questions, and Tips Graciously Accepted ]]
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669 Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678 Arctic Region Supercomputing Center University of Alaska Fairbanks PO Box 756020 Fairbanks AK 99775-6020
Subscribe to (or unsubscribe from) the e-mail edition of the
ARSC HPC Users' Newsletter.
Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.