ARSC HPC Users' Newsletter 407, September 28, 2009

Gnuplot Magic, Part 2 -- Color Tricks

[ By Anton Kulchitsky ]

Let us talk a little about how color represents values of a function. If we have just two colors: black and white, we could mix these two colors in some proportion to represent different values in some range (creating a grayscale palette). Suppose our function ranges from 0 to 1. We can write a mapping of this function range [0:1] to a color range [black:white] so that every value from the function range has its own representation on the [black:white] range. If we use numbers from 0 to 1 to represent the intensity of white, a map of [0:1] to [0:1] will represent the mapping between function values to the grayscale. If the function range is different from [0:1], it can be rescaled to [0:1], and the previous mapping will apply. This is how gnuplot actually does the grayscale function values representation.

The color mapping is a little bit more tricky. Indeed, now we need to make a mapping between a 1D range of function value variation and its 3D (for example RGB, red-green-blue) color representation. Actually, gnuplot allows 2 different models of color representation: RGB and HSV (hue-saturation-value). We need to define some continuous change in colors when changing function values.

There are 2 different ways to define the mapping from the plot range into the color spectrum in gnuplot: (1) We can rely on gnuplot's sense of color and define some basic values and their corresponding colors and allow gnuplot to interpolate smoothly in between. This is arguably the easiest way to define the palette. The palette can be defined either in the script or in a separate file using a special format. The last option is useful when a specific color scheme needs to be emulated in gnuplot or different color schemes are to be evaluated. (2) We can define three functions that map [0:1] to [0:1] similar to the technique for grayscale demonstrated above but for 3 distinct color dimensions either in RGB or HSV model. Usually, it is easier to define smooth and natural functions in the HSV model rather than in RGB. Of course, HSV and RGB representations are absolutely interchangable.

Let us run gnuplot and try a few relatively simple examples. Let us use the second way to define the mapping. Type the following line into gnuplot:

  set palette model HSV functions gray,1,1

The set palette command is used to define a palette. Keyword model accepts one of two arguments: RGB or HSV. We used HSV in this example. Keyword functions defines the mapping from [0:1] to [0:1] using method 2 above. The dummy variable name in all of these three functions separated by comma, is gray, not x. Thus, the functions that we use define hue as a linear function varying from 0 to 1, while saturation and value are fixed at 1. We can run:

  test palette

to see the exact color mapping. You can see a nice rainbow in the strip below the graph. You also can see three plots of red, green, and blue for every color. The same rainbow with logarithmic scale can be displayed as follows:

  set palette model HSV functions log(gray+1)/log(2),1,1
  test palette

You can experiment with other functions and with the RGB model.

Let us now try the first method to set the palette. For example, we would like the middle of the range to be yellow, the minimum of the range to be blue, and the maximum to be red. In this case, we can try the following command:

  set palette model RGB defined ( 0 "blue", 0.5 "yellow", 1 "red" )

We just use the keyword defined instead of functions and specify the list of pairs. Use

  test palette

to see the color representation. It seems gnuplot does a pretty good job of interpolating the colors in between.

We can use predefined keywords for colors. To see the list of predefined color names, type

  show palette colornames

We can represent colors either by hexadecimal code or by three digits varying from 0 to 1 which are just intensities of the three values in the RGB or HSV models. Generally, the most convenient way besides names is hexadecimal representation. Usually, your favorite bitmap editing software like Photoshop or Gimp can return the hexadecimal code for any color. I usually prefer to run Gimp and pick the color to see its hexadecimal value. For example, red in the RGB model can be represented either by #FF0000 in hexadecimal or by a triple 1 0 0. Thus, the following commands are equivalent:

  set palette model RGB defined ( 0 "blue", 1/3. "green", 2/3. "yellow", 1 "red" )
  set palette model RGB defined ( 0 #0000FF, 1/3. #00FF00,  2/3. #FFFF00, 1 #FF0000 )
  set palette model RGB defined ( 0  0 0 1, 1/3.  0 1 0,  2/3.  1 1 0,  1  1 0 0 )

Try this with test palette. By the way, this is a pretty and very natural coloring for 3D functions. I suggest you take the example from part 1 of this series (see Example #2 in HPC News Issue 406) and use this palette for the plot. Just add one of the previous lines somewhere before the splot command there to create a beautiful picture.

I suggest you to do a simple exercise: Using the functions and defined keywords, define a grayscale scheme. You could accomplish this with the set palette gray command of course, but try to achieve the same result using colors.

We will continue the color discussion in the next part of this gnuplot series.

Debug Output for C Applications

[ By Don Bahls ]

When initially developing a new application it's not uncommon to want to print extra output so you can ensure that your code is doing what you expect. However for production codes, it makes sense to minimize the amount of output for the sake of performance. In the past I've used pre-processor tricks to allow the compiler to optimize out unnecessary "printf" statements.

e.g.

    #ifndef DEBUGGING
    #define DEBUGGING (1)
    #endif
    #define dprintf if ( DEBUGGING ) printf

When you want to disable the "dprintf" statements, simply compile with -DDEBUGGING=0.

e.g.

    gcc -DDEBUGGING=0 some.c -c

The pre-processor directive method works well if you are diligent about using an opening "{" and closing "}" (i.e. curly braces) when using "if" statements with dprintf. If you don't use curly braces consistently, you can run into an "if/else" ambiguity which can cause your code to run in unexpected ways.

An improvement to this idea would be a debug printing routine that:

  1. Lets the user change the amount of output produced without recompiling the application.
  2. Allows you to recompile and completely disable output if it's not needed to eliminate any performance issues caused by the print statements.

In this example we will use an environment variable to set the debugging level, but you could just as easily use a command line argument or get this information from a file.

To match the style of printf, we'll make the debug printf statement a variadic function (i.e. a function that can accept a variable number of arguments).

The new debugging printf definition looks like this:


    int dprintf(int min, const char * format, ...)

The "min" value is the minimum value at which a debugging message should be displayed. The "format" and "..." arguments match the standard printf definition. The elipsis operator in C specifies that a routine should accept a variable number of arguments.

To simplify the implementation of dprintf, the vprintf routine is used. The vprintf function handles a va_list (variable argument list).


    #include <stdio.h>
    #include <stdlib.h>
    #include <stdarg.h>
    
    #ifndef DPRINTF_OPTIMIZED
    #define DPRINTF_OPTIMIZED (0)  /* allows dprintf to be disabled */
    #endif
    
    int dprintf(int min, const char * format, ...)
    {
        if ( DPRINTF_OPTIMIZED != 0 ) 
        { return 0; }              /* allows dprintf to be optimized out */
        if ( min >= get_debug_level() )   /* should message print? */
        {
            int status;                   /* return status */
            va_list ap;                   /* variable argument list */
            va_start(ap, format);         /* macro for variable args */
            status = vprintf(format, ap); /* use va_list format of printf */
            va_end(ap);                   /* required to call va_end */
            return status;                /* return status from vprintf */ 
        }
        return 0; 
    }

To avoid the overhead of the getenv call in the "get_debug_level" routine a static int is used. The getenv function only needs to be called one time, otherwise the previously retrieved value is returned.


    #define DPRINTF_MIN_VAL (-100000) /* initial value for debug level */
    
    int get_debug_level()   /* returns the debug level of the application */
    {
        static int level=DPRINTF_MIN_VAL; /* initialize a static variable */
        if ( level == DPRINTF_MIN_VAL )   /* if it's the default read env var */
        {
            char * dbg=getenv("DEBUG_LEVEL");  /* read DEBUG_LEVEL env var */
            if ( dbg != NULL ) { level=atoi(dbg); } /* convert to an int */
            else { level=0; }                       /* otherwise return 0 */
        }
        return level;                     /* return the level */
    }

Here's an example showing the use of dprintf:


    midnight % cat test.c 
    #include <dprintf.h>
    #include <stdio.h>
    #include <stdlib.h>
    
    int main()
    {   int ii;
        for(ii=-3; ii<3; ++ii) 
        { dprintf(ii, "debug message level=%d\n", ii); }
    }

Set the "DEBUG_LEVEL" environment variable to show more or less output:


    midnight % export DEBUG_LEVEL=1 
    midnight % ./test 
    debug message level=1
    debug message level=2

To completely disable debugging output compile with -DDPRINTF_OPTIMIZED.


    midnight % gcc -I. -DDPRINTF_OPTIMIZED test.c -o test

At higher optimization levels, the compiler should completely remove the call to dprintf.


    # 
    # For an overview of how to do something similar with Fortran namelists,
    # see:
    # /arsc/support/news/hpcnews/hpcnews371/index.xml#article1
    #
    #
    
    /* complete code */
    
    #ifndef DEBUG_PRINTF__
    #define DEBUG_PRINTF__
    
    #include <stdio.h>
    #include <stdlib.h>
    #include <stdarg.h>
    
    #define DPRINTF_MIN_VAL (-100000)     /* initial value for debug level */
    
    #ifndef DPRINTF_OPTIMIZED
    #define DPRINTF_OPTIMIZED (0)         /* allows dprintf to be disabled */
    #endif
    
    int get_debug_level()   /* returns the debug level of the application */
    {
        static int level=DPRINTF_MIN_VAL; /* initial a static variable */
        if ( level == DPRINTF_MIN_VAL )   /* if it's the default read env var */
        {
            char * dbg=getenv("DEBUG_LEVEL");  /* read DEBUG_LEVEL env var */
            if ( dbg != NULL ) { level=atoi(dbg); } /* convert to an int */
            else { level=0; }                       /* otherwise return 0 */
        }
        return level;                     /* return the level */
    }
    
    int dprintf(int min, const char * format, ...)
    {
        if ( DPRINTF_OPTIMIZED != 0 ) 
        { return 0; }              /* allows dprintf to be optimized out */
        if ( min >= get_debug_level() )   /* should message print? */
        {
            int status;                   /* return status */
            va_list ap;                   /* variable argument list */
            va_start(ap, format);         /* macro for variable args */
            status = vprintf(format, ap); /* use va_list format of printf */
            va_end(ap);                   /* required to call va_end */
            return status;                /* return status from vprintf */ 
        }
        return 0; 
    }
    
    #endif /* DEBUG_PRINTF__ */

Lee Higbie Retirement

This week ARSC said "goodbye" to HPC Specialist Lee Higbie. Lee and his wife Betty packed up a motorhome and headed down the Alaska Highway to enjoy their retirement in Colorado after five years at ARSC. Lee was a frequent contributor to this newsletter and we will miss not only Lee but his well-written articles. Lee and Betty are both writers and we look forward to reading their published work, hopefully soon. In addition to writing and enjoying (another!) retirement, Lee plans to work with his son in a new business venture. We wish all the best to Lee and thank him for his friendship and service to ARSC.

CLE 2.2 Slated for Pingo

On October 1, 2009 Pingo will be taken down for an update from Cray Linux Environment (CLE) 2.1 to CLE 2.2. CLE is the Linux-based operating system of the Cray XT systems. Cray recommends that users recompile applications that were compiled on CLE 2.1. The downtime is scheduled for 0900 - 2200 AKDT.

Quick-Tip Q & A


A:[[ I have a cron job that invokes a script, update.pl, every hour.
  [[ Sometimes this script takes more than an hour to run, causing two
  [[ of these script processes to overlap, which makes them interfere with
  [[ each other.  Is there a way I can tell this update.pl cron job to run
  [[ ONLY if there is not already an update.pl process running, to prevent
  [[ the processes from overlapping?

#
# Why make things complicated?  Greg Newby's solution shows that
# straight-forward solutions are effective:
#

It's old-school, but an easy solution is to add three simple pieces
to update.pl:

a) check whether an arbitrary file (such as $HOME/.update.pl.running)
   exists.  If it does, exit.
b) otherwise, create the file
c) remove the file upon exit

The danger is that if update.pl bombs, the file won't get removed and
the next program won't run.  So, add some output to step (a) that will
get sent via email from cron, so you know when this happens.

#
# There are other means to achieve interprocess communication, as Rich
# Griswold points out:
#

You can modify update.pl to use a lock file or some other interprocess
communication (IPC) mechanism to check if an instance is already
running.  There are a lot of tips for doing this in Perl at
http://www.perlmonks.org/index.pl?node_id=280064.

#
# With no modifications to update.pl itself, Ryan Czerwiec shows how
# shell tricks can be used to accomplish this:
#

Cron typically runs sh unless you override it, so you can include sh
commands to modify its behavior.  You'd have a line that looks something
like this:

* * * * * $PATHNAME/update.pl

You can modify it to something like:

* * * * * a=$(ps -elf 
 grep -w update.pl 
 grep -cv grep) ; if [[ $a == 0 ]] ; then $PATHNAME/update.pl ; fi

Then, a new update.pl will run only if there isn't another in the
process list.  If you want to account for the possibility that you might
be running multiple cron jobs, and you only want to avoid a job
overlapping itself, but it's ok for update.pl's from different cron jobs
to run simultaneously:

* * * * * a=$(ps -elf 
 grep -w update.pl 
 grep -v grep 
  gawk '{print $5}' 
 grep -cw $$ ) ; if ...

will work.

#
# Finally, Jed Brown's response adds new tricks and reliable solutions:
#

The quickest solution is to put

 pgrep update.pl 

 update.pl --whatever --required-args --here

in your crontab.  The pgrep program is part of the handy procps package
which is available/installed on many distros, procps.sf.net.  For this
purpose, you can replace it with $(ps xo comm= 
 grep update.pl)
or similar.

The trouble with the quick solution is that you can get false positives
which might prevent your job from running at all.  A more reliable
solution is to use lock files.  For example, update.pl writes it's PID
into /tmp/$USER-update.pl.lock with failure if the file exists.  In the
case of failure, you confirm that the PID in the file corresponds to a
running instance and exit early.  The lock file should be removed at the
end of update.pl.

It gets messy if, e.g., it is valid for a user to have multiple
update.pl instances running as long as they are called with
non-conflicting arguments.  If this is necessary, you could use a daemon
to keep track of the running update.pl instances and only let new
instances execute if they don't conflict.



Q: At the start of each semester, I need to show a new wave of students
  the basics of using Linux.  I usually start with commands like cd,
  cat, cp, vi, etc., but I know they would benefit from learning dozens
  of other commands I use frequently but can never remember off the top
  of my head.  Do you have any tips or tools I could use to help
  brainstorm what commands to show them?
 

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top