Revisión de 15:57 23 ene 2023

Compilation at CIMA's HPC 'hydra'

Installation steps using modipsl

Go to the installation directory [INSTALLDIR] (for hydra [INSTALLDIR]=/share/ORCHIDEE)

$ cd [INSTALLDIR]

Instalation requirements: svn, ksh.

Getting the IPSL-models suite (trunk), following this link

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl

Going to the util folder:

$ cd modipsl/util

Getting the off-line version of ORCHIDEE (see components in file mod.def)

$ ./model ORCHIDEE_trunk >& run_get_model.log

We got some issues with the ports, thus, it needs to be manually done:

$ cd ../modeles

Getting IOIPSL:

$ mkdir -p IOIPSL/src 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src

Getting XIOS library

$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Getting ORCHIDEE:

By the end one should have:

$ ls
IOIPSL  ORCHIDEE  XIOS

Going back to util to proceed

$ cd ../util

These are the final versions of the codes (on 2022 January the 23rd)

• ORCHIDEE: 7867
• ORCHIDEE_OL: 7867
• IOIPSL: 6273
• XIOS: 2439

In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named arch or ARCH with files of the likes of arch-[compiler]_[machine] with 3 different extensions: .path, .fcm and .env. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify .path and .env).

• arch-[compiler]_[machine].path: Determines the paths of the compilers and libraries
• arch-[compiler]_[machine].fcm: Determines the flags of the compilers
• arch-[compiler]_[machine].env: Determines the environtment for the compilation (e.g. module instructions)

Preparing compilation environment for hydra. One need to add into AA_make.gdef specifications for hydra (at the end of file):

#-Q- ifort_hydra   #- Global definitions for hydra, ifort
#-Q- ifort_hydra LIB_MPI = MPI1
#-Q- ifort_hydra LIB_MPI_BIS = MPI1
#-Q- ifort_hydra FCM_ARCH = ifort_hydra
#-Q- ifort_hydra   M_K = make
#-Q- ifort_hydra   P_C = cpp
#-Q- ifort_hydra   P_O = -P -C -traditional $(P_P)
#-Q- ifort_hydra   F_C = /usr/local/bin/mpif90 -c -cpp
#-Q- ifort_hydra   #-D- MD    F_D = -g
#-Q- ifort_hydra   #-D- MN    F_D =
#-Q- ifort_hydra   #-P- I4R4  F_P = -i4
#-Q- ifort_hydra   #-P- I4R8  F_P = -i4 -r8
#-Q- ifort_hydra   #-P- I8R8  F_P = -i8 -r8
#-Q- ifort_hydra   F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
#-Q- ifort_hydra   F_L = /usr/local/bin/mpif90
#-Q- ifort_hydra   M_M = 0
#-Q- ifort_hydra   L_X = 0
#-Q- ifort_hydra   L_O =
#-Q- ifort_hydra   A_C = ar -rs
#-Q- ifort_hydra   A_G = ar -x
#-Q- ifort_hydra   C_C = cc -c
#-Q- ifort_hydra   C_O =
#-Q- ifort_hydra   C_L = cc
#-Q- ifort_hydra   #-
#-Q- ifort_hydra   NCDF_INC = /usr/local/include
#-Q- ifort_hydra   NCDF_LIB = -L/usr/local/lib -lnetcdf -lnetcdff
#-Q- ifort_hydra  #-

• Creation of the configuration arch files for the compilation of each component

XIOS

• • going there:

$ cd ../modeles/XIOS/arch

• • Using CICLAD's ifort compilation as reference

$ cp arch-ifort_CICLAD.fcm arch-ifort_hydra.fcm

• • Adapting values (following ifort bug issue for version 12.0.3, ifort bug):

%CCOMPILER           /usr/local/bin/mpicc 
%FCOMPILER           /usr/local/bin/mpif90 
%LINKER        /usr/local/bin/mpif90 -nofor-main

%BASE_CFLAGS    -std=c++0x 
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS 
%DEV_CFLAGS     -g -traceback 
%DEBUG_CFLAGS   -DBZ_DEBUG -g -traceback -fno-inline 

%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3 
%DEV_FFLAGS     -g -O2 -traceback 
%DEBUG_FFLAGS   -g -traceback 

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++ 

%CPP            /usr/local/bin/mpicc -EP 
%FPP            cpp -P 
%MAKE           gmake

• • Getting paths:

$ cp arch-ifort_CICLAD.path arch-ifort_hydra.path

• • Adapting values:

NETCDF_INCDIR="-I/usr/local/include" 
NETCDF_LIBDIR="-L/usr/local/lib" 
NETCDF_LIB="-lnetcdf -lnetcdff" 
MPI_INCDIR="" 
MPI_LIBDIR="" 
MPI_LIB="" 
HDF5_INCDIR="" 
HDF5_LIBDIR="" 
HDF5_LIB=""

ORCHIDEE

• • Going there

$ cd ../../ORCHIDEE/arch

• • Using CICLAD's ifort compilation as reference

$ cp arch-ifort_CICLAD.fcm arch-ifort_hydra.fcm

• • Adapting values:

%COMPILER            /usr/local/bin/mpif90 
%LINK                /usr/local/bin/mpif90
%AR                  ar 
%MAKE                gmake
%FPP_FLAGS           -P -traditional
%FPP_DEF
%BASE_FFLAGS         -i4 -r8 -fp-model precise 
%PROD_FFLAGS         -O3 
%DEV_FFLAGS          -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds 
%DEBUG_FFLAGS        -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -check bounds -check all
%MPI_FFLAGS 
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -i4 -r8
%MPI_LD 
%OMP_LD              -openmp

• • getting paths

$ cp arch-ifort_CICLAD.path arch-ifort_hydra.path

• • Adapting values:

NETCDF_LIBDIR="/usr/local/lib -lnetcdf -lnetcdff -lstdc++" 
NETCDF_INCDIR=/usr/local/include 
IOIPSL_INCDIR=../../../lib
IOIPSL_LIBDIR=../../../lib 
XIOS_INCDIR=${ORCHDIR}/../XIOS/inc 
XIOS_LIBDIR="${ORCHDIR}/../XIOS/lib -lxios"

Bulding & compiling

• • Going back to util to proceed

$ cd ../../../util

• • Installing the code:

$ ./ins_make -t ifort_hydra >& run_ins_make.log

• • Going there:

$ cd ../config/ORCHIDEE_OL

• • No need to loading the right environment since in hydra its by default
  • Compiling XIOS in sequential (in Makefile, adding no-parallel compilation):

./make_xios --netcdf_lib netcdf4_seq (...)

• • Compiling:

$ gmake >& run_gmake.log

• • Checking if everything went fine:

$ cat -n run_gmake.log | grep Error
$ cat -n run_gmake.log | grep undefined
$ ls ../../bin/ 
orchidee_ol  xios_server.exe

Compiling rebuild

Due to the fact that hydra does not have the netcdf libraries compiled in parallel, ORCHDEE output is written with multiple files being a file for each process. In order to overcome this output. IPSL does prepare a tool to join the files and provide the expected output. This tool is called rebuild (a Kornshell script which encapsulates de exacutable flio_rbld) and it comes with the IOIPSL libraries. The IOIPSL libraries, are a set of old libraries developed by the IPSL (M. A. Foujuls, J. Polcher among others) to deal with the netCDF data access for all the models developed at the IOIPSL. Compiling IOIPSL (following install rebuild. It has to be compiled independently. Here it is explained how it has to be done.

• Go to the instalation directory [INSTALLDIR] (at hydra share)

$ cd [INSTALLDIR]

• getting the last version of the code

$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_IOIPSLtools

• going to prepare compilation/installation

$ cd modipsl_IOIPSLtools/util

• preparing environment

./model IOIPSL_PLUS

• We got some issues with the ports, thus, it needs to be manually done:

$ cd ../modeles

• Getting IOIPSL (NOTE: notice the difference in library's version from the previous one):

$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_3 IOIPSL

• Comming back to the util folder

$ cd ../util

• Creation of the compilation environment for `hydra' using 'ifort' inside AA_make.gdef as in the full compilation

$ cp ../../modipsl/util/AA_make.gdef ./

• Preparing makefiles

$ ./ins_make -t "ifort_hydra" -p I4R8

• going to src folder to compile

$ cd ../modeles/IOIPSL/src

• cleaning (you never know...)

$ make clean

• compiling

$ make >& run_make.log

• Checking no errors:

$ cat -n run_make.log | grep Error

• compilation of tools (file concatenation)

$ cd ../tools 

• cleaning (you never know...)

$ make clean

• compiling

$ make >& run_make.log

• Checking no errors:

$ cat -n run_make.log | grep Error

• One should have:

$ ls ../../../bin/ 
flio_rbld  rebuild

debug Instalation

Getting a debugging copy for test purposes

Instalation steps using modipsl

• Go to the instalation directory [INSTALLDIR] (/share in hydra)

$ cd [INSTALLDIR]

• Getting the IPSL-models suite (r 3379)

$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg

• Once we got the codes, we need to copy the specificities of the installation for hydra

$ cp ../../modipsl/util/AA_make.gdef ./
$ cp ../../modipsl/modeles/XIOS/arch/*hydra* ../modeles/XIOS/arch/
$ cp ../../modipsl/modeles/ORCHIDEE/arch/*hydra* ../modeles/ORCHIDEE/arch/

• Everything is the same as for the installation, except that we need to indicate debugging compilation:

./ins_make -t ifort_hydra -d >& run_ins_make.log

• And on Makefile from config/ORCHIDEE_OL, change all the -prod by -debug

(...)
xios : 
        (cd  ../../modeles/XIOS ; ./make_xios --netcdf_lib netcdf4_seq \
--debug --arch ${FCM_ARCH} --job 8 ; cp bin/xios_server.exe ../../bin/. ; ) 
(...)
driver : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; )
(...)
driver_xios : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -xios2 -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; ) 
(...)

• Compiling:

$ gmake >& run_gmake.log

• Checking if everything went fine:

$ cat -n run_gmake.log | grep Error 
$ cat -n run_gmake.log | grep undefined
$ ls ../../bin/ 
orchidee_ol  xios_server.exe