Compilation at CIMA's HPC 'hydra'

Contenido

1 Installation steps using modipsl
2 Compiling rebuild
3 debug Instalation
- 3.1 Instalation steps using modipsl

Installation steps using modipsl

Go to the installation directory [INSTALLDIR] (for hydra [INSTALLDIR]=/share/ORCHIDEE)

$ cd [INSTALLDIR]

Instalation requirements: svn, ksh.

Getting the IPSL-models suite (trunk), following this link

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl

Going to the util folder:

$ cd modipsl/util

Getting the off-line version of ORCHIDEE (see components in file mod.def)

$ ./model ORCHIDEE_trunk >& run_get_model.log

We got some issues with the ports, thus, it needs to be manually done:

$ cd ../modeles

Getting IOIPSL:

$ mkdir -p IOIPSL/src 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src

Getting XIOS library

$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Getting ORCHIDEE:

By the end one should have:

$ ls
IOIPSL  ORCH  XIOS

Going back to util to proceed

$ cd ../util

These are the final versions of the codes (on 2022 January the 23rd)

ORCHIDEE: 7867
ORCHIDEE_OL: 7867
IOIPSL: 6273
XIOS: 2439

In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named arch or ARCH with files of the likes of arch-[compiler]_[machine] with 3 different extensions: .path, .fcm and .env. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify .path and .env).

arch-[compiler]_[machine].path: Determines the paths of the compilers and libraries
arch-[compiler]_[machine].fcm: Determines the flags of the compilers
arch-[compiler]_[machine].env: Determines the environtment for the compilation (e.g. module instructions)

There is a global file of configuration modipsl/util/AA_make.gdef, where one store the shared configuration for all the components.

NOTE: hydra's compilation will be based on ifort_MESOIPSL or the equivalent ifort_CICLAD

Preparing compilation environment for hydra. One need to add into AA_make.gdef specifications for hydra (at the end of file, for INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33):

To prepare an environment for compilation in hydra, one needs to execute (in this example we selected option 1:

$ source /opt/load-libs.sh

Then, in order to know the paths of the different components (to be changed in the intel_HYDRA configuration)...

$ which mpif90
/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90

$ nc-config --all
(...)
  --prefix        -> /opt/netcdf/netcdf-4/intel/2021.4.0
  --includedir    -> /opt/netcdf/netcdf-4/intel/2021.4.0/include
  --libdir        -> /opt/netcdf/netcdf-4/intel/2021.4.0/lib
  --version       -> netCDF 4.8.1

#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   #- Global definitions for hydra, ifort
#-Q- ifort_HYDRA LIB_MPI = MPI1
#-Q- ifort_HYDRA LIB_MPI_BIS = MPI1
#-Q- ifort_HYDRA FCM_ARCH = ifort_HYDRA
#-Q- ifort_HYDRA   M_K = make
#-Q- ifort_HYDRA   P_C = cpp
#-Q- ifort_HYDRA   P_O = -P -C -traditional $(P_P)
#-Q- ifort_HYDRA   F_C = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 -c -cpp
#-Q- ifort_HYDRA   #-D- MD    F_D = -g
#-Q- ifort_HYDRA   #-D- MN    F_D =
#-Q- ifort_HYDRA   #-P- I4R4  F_P = -i4
#-Q- ifort_HYDRA   #-P- I4R8  F_P = -i4 -r8
#-Q- ifort_HYDRA   #-P- I8R8  F_P = -i8 -r8
#-Q- ifort_HYDRA   F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
#-Q- ifort_HYDRA   F_L = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90
#-Q- ifort_HYDRA   M_M = 0
#-Q- ifort_HYDRA   L_X = 0
#-Q- ifort_HYDRA   L_O =
#-Q- ifort_HYDRA   A_C = ar -rs
#-Q- ifort_HYDRA   A_G = ar -x
#-Q- ifort_HYDRA   C_C = cc -c
#-Q- ifort_HYDRA   C_O =
#-Q- ifort_HYDRA   C_L = cc
#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   NCDF_INC = /opt/netcdf/netcdf-4/intel/2021.4.0/include
#-Q- ifort_HYDRA   NCDF_LIB = -L/opt/netcdf/netcdf-4/intel/2021.4.0/lib -lnetcdf -lnetcdff
#-Q- ifort_hydra  #-

Creation of the configuration arch files for the compilation of each component

XIOS

going there:

$ cd ../modeles/XIOS/arch

Using CICLAD's ifort compilation as reference

$ cp arch-ifort_CICLAD.env arch-ifort_HYDRA.env
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values (following ifort bug issue for version 12.0.3, ifort bug):

CCOMPILER           mpicc
%FCOMPILER           mpif90
%LINKER              mpif90 -nofor-main

%BASE_CFLAGS    -fPIC -std=c++11
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g
%DEBUG_CFLAGS   -DBZ_DEBUG -g -fno-inline

%BASE_FFLAGS    -fPIC -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           gmake

arch-ifort_HYDRA.path

Adapting values:

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdff -lnetcdf"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

arch-ifort_HYDRA.env

Adpating values

source /opt/load-libs.sh 1

ORCHIDEE

Going there

$ cd ../../ORCH/ORCHIDEE/arch

Using CICLAD's ifort compilation as reference

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values:

COMPILER            mpif90
%LINK                mpif90
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P -traditional
%FPP_DEF
%BASE_FFLAGS         -i4 -r8 -fp-model precise
%PROD_FFLAGS         -O3
%DEV_FFLAGS          -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds
%DEBUG_FFLAGS        -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -init=arrays -init=snan -check bounds,noarg_temp_created
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -i4 -r8
%MPI_LD
%OMP_LD              -openmp

arch-ifort_HYDRA.path

Adpating values

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdf -lnetcdff -lstdc++"

IOIPSL_INCDIR="-I${ORCHDIR}/../../lib -I${ORCHDIR}/../IOIPSL/inc"
IOIPSL_LIBDIR="-L${ORCHDIR}/../../lib -L${ORCHDIR}/../IOIPSL/lib"
IOIPSL_LIB="-lioipsl"

XIOS_INCDIR="-I${ORCHDIR}/../XIOS/inc"
XIOS_LIBDIR="-L${ORCHDIR}/../XIOS/lib"
XIOS_LIB="-lxios"

arch-ifort_HYDRA.env

Creation of file, since it does not exist

$ cp ../../../XIOS/arch/arch-ifort_HYDRA.env ./

IOIPSL

Using CICLAD's configuration

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values

%COMPILER            mpif90
%LINK                mpif90
%FPP                 mpif90 -E
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P
%FPP_DEF             KEY_NONE CPP_NETCDF4
%BASE_FFLAGS         -fPIC -i4 -r8 -auto -align all
%PROD_FFLAGS         -g -traceback -O3
%DEV_FFLAGS          -g -O1 -traceback
%DEBUG_FFLAGS        -g -traceback -check bounds -fp-model strict
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -fPIC -i4 -r8 -auto
%MPI_LD
%OMP_LD              -openmp

arch-ifort_HYDRA.fcm

Adapting values

ETCDF_INCDIR=$(nf-config --fflags)
NETCDF_LIBDIR=$(nf-config --flibs)
NETCDF_LIB=""

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

Bulding & compiling

Going back to util to proceed

Going there:

$ cd modispl/config/

Moving folder to the right position:

$ mv ../modeles/ORCH/ORCHIDEE_OL
$ mv ../modeles/ORCH/ORCHIDEE ../modeles/

Going there:

$ cd ORCHIDEE_OL

Loding environment:

source /opt/load-libs.sh 1

arch-ifort_HYDRA.env

Preparing environtment for compilation

$ cp ARCH/arch-ifort_CICLAD.env ARCH/arch-ifort_HYDRA.env

Compilation

Going there:

cd ../config/ORCHIDEE_OL/

Content:

# Compile and running environnemnet for HYDRA at CIMA
source /opt/load-libs.sh 1

Compiling XIOS in sequential (in Makefile, adding no-parallel compilation):

./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA >& run_compile.log

Checking if everything went fine:

$ cat -n run_compile.log | grep Error
$ cat -n run_compile.log | grep undefined
$ ls ../../bin/ 
orchideedriver_prod  orchidee_ol_prod  xios_server_prod.exe

Compiling rebuild

Due to the fact that hydra does not have the netcdf libraries compiled in parallel, ORCHDEE output is written with multiple files being a file for each process. In order to overcome this output. IPSL does prepare a tool to join the files and provide the expected output. This tool is called rebuild (a Kornshell script which encapsulates de exacutable flio_rbld) and it comes with the IOIPSL libraries. The IOIPSL libraries, are a set of old libraries developed by the IPSL (M. A. Foujuls, J. Polcher among others) to deal with the netCDF data access for all the models developed at the IOIPSL. Compiling IOIPSL (following install rebuild. It has to be compiled independently. Here it is explained how it has to be done.

Go to the instalation directory [INSTALLDIR] (at hydra share)

$ cd [INSTALLDIR]

getting the last version of the code

$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_IOIPSLtools

going to prepare compilation/installation

$ cd modipsl_IOIPSLtools/util

preparing environment

./model IOIPSL_PLUS

We got some issues with the ports, thus, it needs to be manually done:

$ cd ../modeles

Getting IOIPSL (NOTE: notice the difference in library's version from the previous one):

$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_3 IOIPSL

Comming back to the util folder

$ cd ../util

Creation of the compilation environment for `hydra' using 'ifort' inside AA_make.gdef as in the full compilation

$ cp ../../modipsl/util/AA_make.gdef ./

Preparing makefiles

$ ./ins_make -t "ifort_hydra" -p I4R8

going to src folder to compile

$ cd ../modeles/IOIPSL/src

cleaning (you never know...)

$ make clean

compiling

$ make >& run_make.log

Checking no errors:

$ cat -n run_make.log | grep Error

compilation of tools (file concatenation)

$ cd ../tools

cleaning (you never know...)

$ make clean

compiling

$ make >& run_make.log

Checking no errors:

$ cat -n run_make.log | grep Error

One should have:

$ ls ../../../bin/ 
flio_rbld  rebuild

debug Instalation

Getting a debugging copy for test purposes

Instalation steps using modipsl

Go to the instalation directory [INSTALLDIR] (/share/ORCHIDEE in hydra)

$ cd [INSTALLDIR]

Getting the IPSL-models suite (r 3379)

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg

Going to the util folder:

$ cd modipsl_dbg/util

Getting the off-line version of ORCHIDEE (see components in file mod.def)

$ ./model ORCHIDEE_trunk >& run_get_model.log

We should have:

$ ls ../modeles
IOIPSL  ORCHIDEE  XIOS

Getting all the environment files for each component (from the regular compilation)

$ cp AA_make.gdef AA_make.gdef.old
$ cp ../../modipsl/util/AA_make.gdef ./
$ cp ../../modipsl/modeles/IOIPSL/arch/arch-*HYDRA* ../modeles/IOIPSL/arch/
$ cp ../../modipsl/modeles/ORCHIDEE/arch/arch-*HYDRA* ../modeles/ORCHIDEE/arch/
$ cp ../../modipsl/modeles/XIOS/arch/arch-*HYDRA* ../modeles/XIOS/arch/

Everything is the same as for the installation, except that we need to indicate debugging compilation:

./ins_make -t ifort_hydra -d >& run_ins_make.log

And on Makefile from config/ORCHIDEE_OL, change all the -prod by -debug

(...)
xios : 
        (cd  ../../modeles/XIOS ; ./make_xios --netcdf_lib netcdf4_seq \
--debug --arch ${FCM_ARCH} --job 8 ; cp bin/xios_server.exe ../../bin/. ; ) 
(...)
driver : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; )
(...)
driver_xios : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -xios2 -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; ) 
(...)

Compiling:

$ gmake >& run_gmake.log

Checking if everything went fine:

$ cat -n run_gmake.log | grep Error 
$ cat -n run_gmake.log | grep undefined
$ ls ../../bin/ 
orchidee_ol  xios_server.exe

ORcompHYDRA hydra

Contenido

Installation steps using modipsl

XIOS

arch-ifort_HYDRA.fcm

arch-ifort_HYDRA.path

arch-ifort_HYDRA.env

ORCHIDEE

arch-ifort_HYDRA.fcm

arch-ifort_HYDRA.path

arch-ifort_HYDRA.env

IOIPSL

arch-ifort_HYDRA.fcm

arch-ifort_HYDRA.fcm

Bulding & compiling

arch-ifort_HYDRA.env

Compilation

Compiling rebuild

debug Instalation

Instalation steps using modipsl

Vistas

Herramientas personales

Navegación

Buscar

Herramientas