Using IOFSL over NewMadeleine


  • boost with packages date_time,program_options,regex,thread,test:
 $ ./ --with-libraries=date_time,program_options,regex,thread,test --prefix=$PWD/install/
 $ ./bjam  --prefix=$PWD/install/ --libdir=$PWD/install/lib --includedir=$PWD/install/include install
  • autoconf 2.61 (or higher)
  • flex 2.4.35

Configuring NewMadeleine

You need to compile PM2 with a flavor featuring the following options:

  • nmad: build_both tag_huge
  • pioman: build_both pthread_lock dont_poll

For example, here's a typical nmad flavor:

  • –flavor=nmad
  • –builddir=$flavor
  • –modules=init –modules=nmad –modules=ntbx –modules=pioman –modules=puk –modules=tbx
  • –init=build_both –init=opt
  • –nmad=build_both –nmad=ibverbs –nmad=mx –nmad=opt –nmad=strat_custom –nmad=tag_huge
  • –ntbx=build_both –ntbx=opt
  • –pioman=build_both –pioman=dont_poll –pioman=pthread_lock
  • –puk=build_both –puk=opt
  • –tbx=build_both –tbx=opt
  • –appli=opt

Then, you can compile PM2 (export PM2_FLAVOR=nmad; cd $PM2_ROOT; make)

Configuring IOFSL

First, you have to get a fresh copy of IOFSL:

 git clone iofsl

The repository does not include the scripts needed to configure the sourcecode for building. Use the ./prepare script to generate these files. Note that you need a recent version of autoconf (we are using version 2.63).

Since IOFSL expects the “classical” BMI implementation (ie. the one from PVFS), we need to fix the using the following patch. This patch also contains a temporary “fix” in IOFSL state machine so that it can compile. I guess, this will be fixed soon in the git repository.

You can then configure IOFSL using the following command line:

 ./configure --prefix=$PWD/install --with-bmi=$PM2_ROOT/build/x86_64/$PM2_FLAVOR CFLAGS="`pm2-config --cflags`" LDFLAGS="`pm2-config --libs`" LIBS="`pm2-config --libs`"

You also may have to export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`pm2-config –libdir tbx`

Configuring MPICH2

You will need the followinf packages:

  • autoconf 2.63 (or higher)
  • automake 1.11 (or higher)
  • libtool 2.2.6 (or higher)

get a fresh copy from MPICH2 svn and run ./maint/updatefiles

Remark: on the Borderline cluster, subversion is too old and won't retrieve all the files (especially svn:externals files). You should svn co on another machine with a recent version of subversion and then copy the files to Borderline.

You can configure MPICH2 so that it uses the NMad library as a network driver:

 ./configure --with-device=ch3:nemesis:newmad --with-newmad=$PM2_ROOT

Remark: For now (ie. MPICH2 rev. 5975), you have to apply this patch to MPICH2 in order to make it work with NMad.

You can configure MPICH2 so that it uses IOFSL for MPI-IO:

 export CFLAGS="$CFLAGS -I$IOFSL_ROOT/include"
 ./configure --enable-romio --with-file-system=ufs+zoidfs

Getting all the needed packages on the Borderline cluster

I have compiled and installed all these packages. You can find them in the /home/bordeaux/ftrahay/soft/ directory

You can add the following lines to your .bashrc file to use then automatically:

 # add boost libs
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/bordeaux/ftrahay/soft/boost_1_39_0/lib
 # add CUnit libs
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/bordeaux/ftrahay/soft/CUnit-2.1-0/lib
 # add nmad libs
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/bordeaux/ftrahay/build/x86_64/nmad/lib/
 #add libtool
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/bordeaux/ftrahay/soft/libtool-2.2.6/install/lib/
 #add openpa
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/bordeaux/ftrahay/soft/openpa-1.0.2/lib/
 # add various up to date packages
 export PATH=/home/bordeaux/ftrahay/soft/autoconf-2.63/bin/:$PATH
 export PATH=/home/bordeaux/ftrahay/soft/automake-1.11/bin:$PATH
 export PATH=/home/bordeaux/ftrahay/soft/libtool-2.2.6/install/bin:$PATH


First, you need to start the IOFSL server. Create a config_file like this:

frontend { 
   type = "bmi";
   log {
     default = "DEBUG";

Then, start the server:

$ $IOFSL_ROOT/bin/iofwd --config config_file
[20100226T113351.266134/3] (default) Booting IOFWDMain...
[20100226T113351.266289/3] (iofwdfrontend) Server listening on nm://0a00000dca25
# nmad: strategy = strategy-default
Using system uts nodename "joe1" for local url, might need to be superseded by a network-valid name
# nmad: driver name = driver-tcp; url = joe1:37354
# laucher: local url = '0a00000eca25'
[20100226T113351.318000/3] (default) About to run IOFWDMain...
[20100226T113351.318035/3] (iofwdfrontend) IOFW thread running

Before starting your application, you need to specify the IOFSL server to use with the ZOIDFS_ION_NAME env variable. You need to specify the protocol (nm that stands for NewMadeleine) and the url provided by NewMadeleine:

export ZOIDFS_ION_NAME="nm://0a00000eca25"
using_iofsl_over_newmadeleine.txt · Last modified: 2010/02/26 11:40 by trahay
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki