ORcompHYDRA hydra
Compilation at CIMA's HPC 'hydra'
CIMA's cluster named hydra, has the following compiler:
fortran: intel v 2021.4.0 mpi: 3.4.2
Contenido |
Installation steps using modipsl
Go to the installation directory [INSTALLDIR]
(for hydra [INSTALLDIR]=/share/ORCHIDEE
)
$ cd [INSTALLDIR]
Instalation requirements: svn, ksh.
Getting the IPSL-models suite (trunk), following this link
$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl
Going to the util
folder:
$ cd modipsl/util
Getting the off-line version of ORCHIDEE (see components in file mod.def
)
$ ./model ORCHIDEE_trunk >& run_get_model.log
We got some issues with the ports, thus, it needs to be manually done:
$ cd ../modeles
Getting IOIPSL:
$ mkdir -p IOIPSL/src $ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src
Getting XIOS library
$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS
Getting ORCHIDEE:
By the end one should have:
$ ls IOIPSL ORCH XIOS
Going back to util
to proceed
$ cd ../util
These are the final versions of the codes (on 2022 January the 23rd)
- ORCHIDEE: 7867
- ORCHIDEE_OL: 7867
- IOIPSL: 6273
- XIOS: 2439
In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named arch
or ARCH
with files of the likes of arch-[compiler]_[machine] with 3 different extensions: .path
, .fcm
and .env
. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify .path
and .env
).
-
arch-[compiler]_[machine].path
: Determines the paths of the compilers and libraries -
arch-[compiler]_[machine].fcm
: Determines the flags of the compilers -
arch-[compiler]_[machine].env
: Determines the environtment for the compilation (e.g.module
instructions)
There is a global file of configuration modipsl/util/AA_make.gdef
, where one store the shared configuration for all the components.
NOTE: hydra's compilation will be based on ifort_MESOIPSL
or the equivalent ifort_CICLAD
Preparing compilation environment for hydra
. One need to add into AA_make.gdef
specifications for hydra
(at the end of file, for INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
):
To prepare an environment for compilation in hydra
, one needs to execute (in this example we selected option 1
:
$ source /opt/load-libs.sh
Then, in order to know the paths of the different components (to be changed in the intel_HYDRA
configuration)...
$ which mpif90 /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 $ nc-config --all (...) --prefix -> /opt/netcdf/netcdf-4/intel/2021.4.0 --includedir -> /opt/netcdf/netcdf-4/intel/2021.4.0/include --libdir -> /opt/netcdf/netcdf-4/intel/2021.4.0/lib --version -> netCDF 4.8.1
#-Q- ifort_HYDRA #- #-Q- ifort_HYDRA #- Global definitions for hydra, ifort #-Q- ifort_HYDRA LIB_MPI = MPI1 #-Q- ifort_HYDRA LIB_MPI_BIS = MPI1 #-Q- ifort_HYDRA FCM_ARCH = ifort_HYDRA #-Q- ifort_HYDRA M_K = make #-Q- ifort_HYDRA P_C = cpp #-Q- ifort_HYDRA P_O = -P -C -traditional $(P_P) #-Q- ifort_HYDRA F_C = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 -c -cpp #-Q- ifort_HYDRA #-D- MD F_D = -g #-Q- ifort_HYDRA #-D- MN F_D = #-Q- ifort_HYDRA #-P- I4R4 F_P = -i4 #-Q- ifort_HYDRA #-P- I4R8 F_P = -i4 -r8 #-Q- ifort_HYDRA #-P- I8R8 F_P = -i8 -r8 #-Q- ifort_HYDRA F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise #-Q- ifort_HYDRA F_L = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 #-Q- ifort_HYDRA M_M = 0 #-Q- ifort_HYDRA L_X = 0 #-Q- ifort_HYDRA L_O = #-Q- ifort_HYDRA A_C = ar -rs #-Q- ifort_HYDRA A_G = ar -x #-Q- ifort_HYDRA C_C = cc -c #-Q- ifort_HYDRA C_O = #-Q- ifort_HYDRA C_L = cc #-Q- ifort_HYDRA #- #-Q- ifort_HYDRA NCDF_INC = /opt/netcdf/netcdf-4/intel/2021.4.0/include #-Q- ifort_HYDRA NCDF_LIB = -L/opt/netcdf/netcdf-4/intel/2021.4.0/lib -lnetcdf -lnetcdff #-Q- ifort_hydra #-
- Creation of the configuration
arch
files for the compilation of each component
XIOS
going there:
$ cd ../modeles/XIOS/arch
Using CICLAD's ifort compilation as reference
$ cp arch-ifort_CICLAD.env arch-ifort_HYDRA.env $ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm $ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
arch-ifort_HYDRA.fcm
Adapting values (following ifort bug issue for version 12.0.3, ifort bug):
CCOMPILER mpicc %FCOMPILER mpif90 %LINKER mpif90 -nofor-main %BASE_CFLAGS -fPIC -std=c++11 %PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS %DEV_CFLAGS -g %DEBUG_CFLAGS -DBZ_DEBUG -g -fno-inline %BASE_FFLAGS -fPIC -D__NONE__ %PROD_FFLAGS -O3 %DEV_FFLAGS -g -O2 -traceback %DEBUG_FFLAGS -g -traceback %BASE_INC -D__NONE__ %BASE_LD -lstdc++ %CPP mpicc -EP %FPP cpp -P %MAKE gmake
arch-ifort_HYDRA.path
Adapting values:
NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include" NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib" NETCDF_LIB="-lnetcdff -lnetcdf" MPI_INCDIR="" MPI_LIBDIR="" MPI_LIB="" HDF5_INCDIR="" HDF5_LIBDIR="" HDF5_LIB=""
arch-ifort_HYDRA.env
Adpating values
source /opt/load-libs.sh 1
ORCHIDEE
Going there
$ cd ../../ORCH/ORCHIDEE/arch
Using CICLAD's ifort compilation as reference
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm $ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
arch-ifort_HYDRA.fcm
Adapting values:
COMPILER mpif90 %LINK mpif90 %AR ar %MAKE gmake %FPP_FLAGS -P -traditional %FPP_DEF %BASE_FFLAGS -i4 -r8 -fp-model precise %PROD_FFLAGS -O3 %DEV_FFLAGS -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds %DEBUG_FFLAGS -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -init=arrays -init=snan -check bounds,noarg_temp_created %MPI_FFLAGS %OMP_FFLAGS -openmp -openmp-threadprivate compat %BASE_LD -i4 -r8 %MPI_LD %OMP_LD -openmp
arch-ifort_HYDRA.path
Adpating values
NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include" NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib" NETCDF_LIB="-lnetcdf -lnetcdff -lstdc++" IOIPSL_INCDIR="-I${ORCHDIR}/../../lib -I${ORCHDIR}/../IOIPSL/inc" IOIPSL_LIBDIR="-L${ORCHDIR}/../../lib -L${ORCHDIR}/../IOIPSL/lib" IOIPSL_LIB="-lioipsl" XIOS_INCDIR="-I${ORCHDIR}/../XIOS/inc" XIOS_LIBDIR="-L${ORCHDIR}/../XIOS/lib" XIOS_LIB="-lxios"
arch-ifort_HYDRA.env
Creation of file, since it does not exist
$ cp ../../../XIOS/arch/arch-ifort_HYDRA.env ./
IOIPSL
Using CICLAD's configuration
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm $ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
arch-ifort_HYDRA.fcm
Adapting values
%COMPILER mpif90 %LINK mpif90 %FPP mpif90 -E %AR ar %MAKE gmake %FPP_FLAGS -P %FPP_DEF KEY_NONE CPP_NETCDF4 %BASE_FFLAGS -fPIC -i4 -r8 -auto -align all %PROD_FFLAGS -g -traceback -O3 %DEV_FFLAGS -g -O1 -traceback %DEBUG_FFLAGS -g -traceback -check bounds -fp-model strict %MPI_FFLAGS %OMP_FFLAGS -openmp -openmp-threadprivate compat %BASE_LD -fPIC -i4 -r8 -auto %MPI_LD %OMP_LD -openmp
arch-ifort_HYDRA.fcm
Adapting values
ETCDF_INCDIR=$(nf-config --fflags) NETCDF_LIBDIR=$(nf-config --flibs) NETCDF_LIB="" MPI_INCDIR="" MPI_LIBDIR="" MPI_LIB="" HDF5_INCDIR="" HDF5_LIBDIR="" HDF5_LIB=""
Bulding & compiling
Going back to util
to proceed
Going there:
$ cd modispl/config/
Moving folder to the right position:
$ mv ../modeles/ORCH/ORCHIDEE_OL $ mv ../modeles/ORCH/ORCHIDEE ../modeles/
Going there:
$ cd ORCHIDEE_OL
Loding environment:
source /opt/load-libs.sh 1
arch-ifort_HYDRA.env
Preparing environtment for compilation
$ cp ARCH/arch-ifort_CICLAD.env ARCH/arch-ifort_HYDRA.env
Compilation
Going there:
cd ../config/ORCHIDEE_OL/
Content:
# Compile and running environnemnet for HYDRA at CIMA source /opt/load-libs.sh 1
Compiling XIOS in sequential (in Makefile
, adding no-parallel compilation):
./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA >& run_compile.log
Checking if everything went fine:
$ cat -n run_compile.log | grep Error $ cat -n run_compile.log | grep undefined $ ls ../../bin/ orchideedriver_prod orchidee_ol_prod xios_server_prod.exe
Compiling rebuild
Due to the fact that hydra
does not have the netcdf libraries compiled in parallel, ORCHDEE output is written with multiple files being a file for each process. In order to overcome this output. IPSL does prepare a tool to join the files and provide the expected output. This tool is called rebuild
(a Kornshell script which encapsulates de exacutable flio_rbld
) and it comes with the IOIPSL
libraries. The IOIPSL libraries, are a set of old libraries developed by the IPSL (M. A. Foujuls, J. Polcher among others) to deal with the netCDF data access for all the models developed at the IOIPSL. Compiling IOIPSL (following install rebuild. It has to be compiled independently. Here it is explained how it has to be done.
- Go to the instalation directory
[INSTALLDIR]
(at hydrashare
)
$ cd [INSTALLDIR]
- getting the last version of the code
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_IOIPSLtools
- going to prepare compilation/installation
$ cd modipsl_IOIPSLtools/util
- preparing environment
./model IOIPSL_PLUS
- We got some issues with the ports, thus, it needs to be manually done:
$ cd ../modeles
- Getting IOIPSL (NOTE: notice the difference in library's version from the previous one):
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_3 IOIPSL
- Comming back to the
util
folder
$ cd ../util
- Creation of the compilation environment for
`hydra'
using'ifort'
insideAA_make.gdef
as in the full compilation
$ cp ../../modipsl/util/AA_make.gdef ./
- Preparing makefiles
$ ./ins_make -t "ifort_hydra" -p I4R8
- going to src folder to compile
$ cd ../modeles/IOIPSL/src
- cleaning (you never know...)
$ make clean
- compiling
$ make >& run_make.log
- Checking no errors:
$ cat -n run_make.log | grep Error
- compilation of tools (file concatenation)
$ cd ../tools
- cleaning (you never know...)
$ make clean
- compiling
$ make >& run_make.log
- Checking no errors:
$ cat -n run_make.log | grep Error
- One should have:
$ ls ../../../bin/ flio_rbld rebuild
debug Instalation
Getting a debugging copy for test purposes
Instalation steps using modipsl
Go to the instalation directory [INSTALLDIR]
(/share/ORCHIDEE
in hydra
)
$ cd [INSTALLDIR]
Getting the IPSL-models suite (r 3379)
$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg
Going to the util
folder:
$ cd modipsl_dbg/util
Getting the off-line version of ORCHIDEE (see components in file mod.def
)
$ ./model ORCHIDEE_trunk >& run_get_model.log
We should have:
$ ls ../modeles IOIPSL ORCHIDEE XIOS
Getting all the environment files for each component (from the regular compilation)
$ cp AA_make.gdef AA_make.gdef.old $ cp ../../modipsl/util/AA_make.gdef ./ $ cp ../../modipsl/modeles/IOIPSL/arch/arch-*HYDRA* ../modeles/IOIPSL/arch/ $ cp ../../modipsl/modeles/ORCHIDEE/arch/arch-*HYDRA* ../modeles/ORCHIDEE/arch/ $ cp ../../modipsl/modeles/XIOS/arch/arch-*HYDRA* ../modeles/XIOS/arch/
Everything is the same as for the installation, except that we need to indicate debugging compilation:
./ins_make -t ifort_HYDRA -d >& run_ins_make.log
Compiling:
$ cd ../config/ORCHIDEE_OL $ ./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA -debug >& run_compile.log
Checking if everything went fine:
$ cat -n run_compile.log | grep Error $ cat -n run_compile.log | grep undefined $ ls ../../bin/ orchideedriver_debug orchidee_ol_debug xios_server_debug.exe