ORcompHYDRA hydra

De Wikicima
(Diferencias entre revisiones)
Saltar a: navegación, buscar
(→‎Installation steps using modipsl)
 
(No se muestran 34 ediciones intermedias realizadas por 2 usuarios)
Línea 1: Línea 1:
 
Compilation at CIMA's HPC 'hydra'
  
Compilation at CIMA's HPC 'hydra'
  +
  +
CIMA's cluster named hydra, has the following compiler:
  +
<pre>
  +
fortran: intel v 2021.4.0
  +
mpi: 3.4.2
  +
</pre>
   
 
= Installation steps using modipsl =
  
= Installation steps using modipsl =
−
* Go to the installation directory <code>[INSTALLDIR]</code> (for hydra <code>[INSTALLDIR]=/share/ORCHIDEE</code>)
 +
Go to the installation directory <code>[INSTALLDIR]</code> (for hydra <code>[INSTALLDIR]=/share/ORCHIDEE</code>)
 
<pre>$ cd [INSTALLDIR]</pre>
  
<pre>$ cd [INSTALLDIR]</pre>
   
−
* Getting the IPSL-models suite (trunk), following this [https://forge.ipsl.jussieu.fr/orchidee/wiki/DevelopmentActivities/ORCHIDEE-CNP/howtoUse link]
 +
Instalation requirements: <code>svn, ksh. </code>
  +
  +
Getting the IPSL-models suite (trunk), following this [https://forge.ipsl.jussieu.fr/orchidee/wiki/DevelopmentActivities/ORCHIDEE-CNP/howtoUse link]
 
<pre>$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl</pre>
  
<pre>$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl</pre>
 
<!-- icmc2022 -->
  
<!-- icmc2022 -->
   
−
* Going to the <code>util</code> folder:
 +
Going to the <code>util</code> folder:
 
<pre>$ cd modipsl/util</pre>
  
<pre>$ cd modipsl/util</pre>
   
−
* Getting the off-line version of ORCHIDEE (see components in file <code>mod.def</code>)
 +
Getting the off-line version of ORCHIDEE (see components in file <code>mod.def</code>)
 
<pre>$ ./model ORCHIDEE_trunk >& run_get_model.log</pre>
  
<pre>$ ./model ORCHIDEE_trunk >& run_get_model.log</pre>
   
−
* We got some issues with the ports, thus, it needs to be manually done:
 +
We got some issues with the ports, thus, it needs to be manually done:
 
<pre>$ cd ../modeles</pre>
  
<pre>$ cd ../modeles</pre>
   
−
* Getting IOIPSL:
 +
Getting IOIPSL:
 
<pre>$ mkdir -p IOIPSL/src
 
<pre>$ mkdir -p IOIPSL/src
 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src</pre>
  
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src</pre>
   
−
* Getting XIOS library
 +
Getting XIOS library
−
<pre>$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS</pre>
 +
<pre>
  +
$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS
  +
</pre>
   
−
* By the end one should have:
 +
Getting ORCHIDEE:
  +
<!--pre>
  +
$ svn co --username sechiba svn://forge.ipsl.jussieu.fr/orchidee/trunk ORCH
  +
</pre-->
  +
  +
  +
By the end one should have:
 
<pre>$ ls
  
<pre>$ ls
−
IOIPSL ORCHIDEE XIOS</pre>
 +
IOIPSL ORCH XIOS</pre>
   
−
* Going back to <code>util</code> to proceed
 +
Going back to <code>util</code> to proceed
 
<pre>$ cd ../util</pre>
  
<pre>$ cd ../util</pre>
 
<!--% username: sechiba
  
<!--% username: sechiba
 
passw: ipsl2000 -->
  
passw: ipsl2000 -->
−
* These are the final versions of the codes
 +
−
** ORCHIDEE: 4622
 +
These are the final versions of the codes (on 2022 January the 23rd)
−
** ORCHIDEE_OL: 4622
 +
* ORCHIDEE: 7867
−
** IOIPSL: 3379
 +
* ORCHIDEE_OL: 7867
−
** XIOS: 1043
 +
* IOIPSL: 6273
−
* Preparing compilation environment for <code>hydra</code>. One need to add into <code>AA_make.gdef</code> specifications for <code>hydra</code> (at the end of file):
 +
* XIOS: 2439
  +
  +
In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named <code>arch</code> or <code>ARCH</code> with files of the likes of arch-[compiler]_[machine] with 3 different extensions: <code>.path</code>, <code>.fcm</code> and <code>.env</code>. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify <code>.path</code> and <code>.env</code>).
  +
* <code>arch-[compiler]_[machine].path</code>: Determines the paths of the compilers and libraries
  +
* <code>arch-[compiler]_[machine].fcm</code>: Determines the flags of the compilers
  +
* <code>arch-[compiler]_[machine].env</code>: Determines the environtment for the compilation (e.g. <code>module</code> instructions)
  +
  +
There is a global file of configuration <code>modipsl/util/AA_make.gdef</code>, where one store the shared configuration for all the components.
  +
  +
'''NOTE:''' hydra's compilation will be based on <code>ifort_MESOIPSL</code> or the equivalent <code>ifort_CICLAD</code>
  +
  +
Preparing compilation environment for <code>hydra</code>. One need to add into <code>AA_make.gdef</code> specifications for <code>hydra</code> (at the end of file, for <code>INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33</code>):
  +
  +
To prepare an environment for compilation in <code>hydra</code>, one needs to execute (in this example we selected option <code>1</code>:
 
<pre>
  
<pre>
−
#-Q- ifort_hydra #- Global definitions for hydra, ifort
 +
$ source /opt/load-libs.sh
−
#-Q- ifort_hydra LIB_MPI = MPI1
 +
</pre>
−
#-Q- ifort_hydra LIB_MPI_BIS = MPI1
 +
−
#-Q- ifort_hydra FCM_ARCH = ifort_hydra
 +
Then, in order to know the paths of the different components (to be changed in the <code>intel_HYDRA</code> configuration)...
−
#-Q- ifort_hydra M_K = make
 +
−
#-Q- ifort_hydra P_C = cpp
 +
<pre>
−
#-Q- ifort_hydra P_O = -P -C -traditional $(P_P)
 +
$ which mpif90
−
#-Q- ifort_hydra F_C = /usr/local/bin/mpif90 -c -cpp
 +
/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90
−
#-Q- ifort_hydra #-D- MD F_D = -g
 +
−
#-Q- ifort_hydra #-D- MN F_D =
 +
$ nc-config --all
−
#-Q- ifort_hydra #-P- I4R4 F_P = -i4
 +
(...)
−
#-Q- ifort_hydra #-P- I4R8 F_P = -i4 -r8
 +
--prefix -> /opt/netcdf/netcdf-4/intel/2021.4.0
−
#-Q- ifort_hydra #-P- I8R8 F_P = -i8 -r8
 +
--includedir -> /opt/netcdf/netcdf-4/intel/2021.4.0/include
−
#-Q- ifort_hydra F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
 +
--libdir -> /opt/netcdf/netcdf-4/intel/2021.4.0/lib
−
#-Q- ifort_hydra F_L = /usr/local/bin/mpif90
 +
--version -> netCDF 4.8.1
−
#-Q- ifort_hydra M_M = 0
 +
</pre>
−
#-Q- ifort_hydra L_X = 0
 +
−
#-Q- ifort_hydra L_O =
 +
<pre>
−
#-Q- ifort_hydra A_C = ar -rs
 +
#-Q- ifort_HYDRA #-
−
#-Q- ifort_hydra A_G = ar -x
 +
#-Q- ifort_HYDRA #- Global definitions for hydra, ifort
−
#-Q- ifort_hydra C_C = cc -c
 +
#-Q- ifort_HYDRA LIB_MPI = MPI1
−
#-Q- ifort_hydra C_O =
 +
#-Q- ifort_HYDRA LIB_MPI_BIS = MPI1
−
#-Q- ifort_hydra C_L = cc
 +
#-Q- ifort_HYDRA FCM_ARCH = ifort_HYDRA
−
#-Q- ifort_hydra #-
 +
#-Q- ifort_HYDRA M_K = make
−
#-Q- ifort_hydra NCDF_INC = /usr/local/include
 +
#-Q- ifort_HYDRA P_C = cpp
−
#-Q- ifort_hydra NCDF_LIB = -L/usr/local/lib -lnetcdf -lnetcdff
 +
#-Q- ifort_HYDRA P_O = -P -C -traditional $(P_P)
  +
#-Q- ifort_HYDRA F_C = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 -c -cpp
  +
#-Q- ifort_HYDRA #-D- MD F_D = -g
  +
#-Q- ifort_HYDRA #-D- MN F_D =
  +
#-Q- ifort_HYDRA #-P- I4R4 F_P = -i4
  +
#-Q- ifort_HYDRA #-P- I4R8 F_P = -i4 -r8
  +
#-Q- ifort_HYDRA #-P- I8R8 F_P = -i8 -r8
  +
#-Q- ifort_HYDRA F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
  +
#-Q- ifort_HYDRA F_L = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90
  +
#-Q- ifort_HYDRA M_M = 0
  +
#-Q- ifort_HYDRA L_X = 0
  +
#-Q- ifort_HYDRA L_O =
  +
#-Q- ifort_HYDRA A_C = ar -rs
  +
#-Q- ifort_HYDRA A_G = ar -x
  +
#-Q- ifort_HYDRA C_C = cc -c
  +
#-Q- ifort_HYDRA C_O =
  +
#-Q- ifort_HYDRA C_L = cc
  +
#-Q- ifort_HYDRA #-
  +
#-Q- ifort_HYDRA NCDF_INC = /opt/netcdf/netcdf-4/intel/2021.4.0/include
  +
#-Q- ifort_HYDRA NCDF_LIB = -L/opt/netcdf/netcdf-4/intel/2021.4.0/lib -lnetcdf -lnetcdff
 
#-Q- ifort_hydra #-
  
#-Q- ifort_hydra #-
  +
 
</pre>
  
</pre>
 
* Creation of the configuration <code>arch</code> files for the compilation of each component
  
* Creation of the configuration <code>arch</code> files for the compilation of each component
   
 
== XIOS ==
  
== XIOS ==
−
** going there:
 +
going there:
 
<pre>$ cd ../modeles/XIOS/arch</pre>
  
<pre>$ cd ../modeles/XIOS/arch</pre>
−
** Using CICLAD's ifort compilation as reference
  
−
<pre>$ cp arch-ifort_CICLAD.fcm arch-ifort_hydra.fcm</pre>
  
−
** Adapting values (following ifort bug issue for version 12.0.3, [https://software.intel.com/en-us/forums/intel-c-compiler/topic/289446 ifort bug]):
  
−
<pre>%CCOMPILER /usr/local/bin/mpicc
 
−
%FCOMPILER /usr/local/bin/mpif90
 
−
%LINKER /usr/local/bin/mpif90 -nofor-main
  
   
−
%BASE_CFLAGS -std=c++0x
+
Using CICLAD's ifort compilation as reference
−
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
+
<pre>
−
%DEV_CFLAGS -g -traceback
+
$ cp arch-ifort_CICLAD.env arch-ifort_HYDRA.env
−
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
+
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
  +
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
  +
</pre>
   
−
%BASE_FFLAGS -D__NONE__
 +
=== arch-ifort_HYDRA.fcm ===
−
%PROD_FFLAGS -O3
+
Adapting values (following ifort bug issue for version 12.0.3, [https://software.intel.com/en-us/forums/intel-c-compiler/topic/289446 ifort bug]):
−
%DEV_FFLAGS -g -O2 -traceback
+
<pre>
−
%DEBUG_FFLAGS -g -traceback
+
CCOMPILER mpicc
  +
%FCOMPILER mpif90
  +
%LINKER mpif90 -nofor-main
  +
  +
%BASE_CFLAGS -fPIC -std=c++11
  +
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
  +
%DEV_CFLAGS -g
  +
%DEBUG_CFLAGS -DBZ_DEBUG -g -fno-inline
  +
  +
%BASE_FFLAGS -fPIC -D__NONE__
  +
%PROD_FFLAGS -O3
  +
%DEV_FFLAGS -g -O2 -traceback
  +
%DEBUG_FFLAGS -g -traceback
   
 
%BASE_INC -D__NONE__
  
%BASE_INC -D__NONE__
−
%BASE_LD -lstdc++
+
%BASE_LD -lstdc++
   
−
%CPP /usr/local/bin/mpicc -EP
+
%CPP mpicc -EP
−
%FPP cpp -P
+
%FPP cpp -P
−
%MAKE gmake</pre>
 +
%MAKE gmake
−
** Getting paths:
 +
</pre>
−
<pre>$ cp arch-ifort_CICLAD.path arch-ifort_hydra.path</pre>
 +
−
** Adapting values:
 +
=== arch-ifort_HYDRA.path ===
−
<pre>NETCDF_INCDIR="-I/usr/local/include"
+
Adapting values:
−
NETCDF_LIBDIR="-L/usr/local/lib"
+
<pre>
−
NETCDF_LIB="-lnetcdf -lnetcdff"
+
NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
−
MPI_INCDIR=""
+
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
−
MPI_LIBDIR=""
+
NETCDF_LIB="-lnetcdff -lnetcdf"
−
MPI_LIB=""
+
MPI_INCDIR=""
−
HDF5_INCDIR=""
+
MPI_LIBDIR=""
−
HDF5_LIBDIR=""
+
MPI_LIB=""
−
HDF5_LIB=""</pre>
 +
HDF5_INCDIR=""
  +
HDF5_LIBDIR=""
  +
HDF5_LIB=""
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.env ===
  +
Adpating values
  +
<pre>
  +
source /opt/load-libs.sh 1
  +
</pre>
   
 
== ORCHIDEE ==
  
== ORCHIDEE ==
−
** Going there
 +
−
<pre>$ cd ../../ORCHIDEE/arch</pre>
 +
Going there
−
** Using CICLAD's ifort compilation as reference
 +
<pre>$ cd ../../ORCH/ORCHIDEE/arch</pre>
−
<pre>$ cp arch-ifort_CICLAD.fcm arch-ifort_hydra.fcm</pre>
 +
−
** Adapting values:
 +
Using CICLAD's ifort compilation as reference
−
<pre>%COMPILER /usr/local/bin/mpif90
+
<pre>
−
%LINK /usr/local/bin/mpif90
 +
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
−
%AR ar
+
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.fcm ===
  +
  +
Adapting values:
  +
<pre>
  +
COMPILER mpif90
  +
%LINK mpif90
  +
%AR ar
 
%MAKE gmake
  
%MAKE gmake
 
%FPP_FLAGS -P -traditional
  
%FPP_FLAGS -P -traditional
 
%FPP_DEF
  
%FPP_DEF
−
%BASE_FFLAGS -i4 -r8 -fp-model precise
+
%BASE_FFLAGS -i4 -r8 -fp-model precise
−
%PROD_FFLAGS -O3
+
%PROD_FFLAGS -O3
−
%DEV_FFLAGS -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds
+
%DEV_FFLAGS -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds
−
%DEBUG_FFLAGS -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -check bounds -check all
 +
%DEBUG_FFLAGS -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -init=arrays -init=snan -check bounds,noarg_temp_created
−
%MPI_FFLAGS
+
%MPI_FFLAGS
 
%OMP_FFLAGS -openmp -openmp-threadprivate compat
  
%OMP_FFLAGS -openmp -openmp-threadprivate compat
 
%BASE_LD -i4 -r8
  
%BASE_LD -i4 -r8
−
%MPI_LD
+
%MPI_LD
−
%OMP_LD -openmp</pre>
 +
%OMP_LD -openmp
−
** getting paths
 +
</pre>
−
<pre>$ cp arch-ifort_CICLAD.path arch-ifort_hydra.path</pre>
 +
−
** Adapting values:
 +
=== arch-ifort_HYDRA.path ===
−
<pre>NETCDF_LIBDIR="/usr/local/lib -lnetcdf -lnetcdff -lstdc++"
+
Adpating values
−
NETCDF_INCDIR=/usr/local/include
+
<pre>
−
IOIPSL_INCDIR=../../../lib
 +
NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
−
IOIPSL_LIBDIR=../../../lib
+
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
−
XIOS_INCDIR=${ORCHDIR}/../XIOS/inc
+
NETCDF_LIB="-lnetcdf -lnetcdff -lstdc++"
−
XIOS_LIBDIR="${ORCHDIR}/../XIOS/lib -lxios"</pre>
 +
  +
IOIPSL_INCDIR="-I${ORCHDIR}/../../lib -I${ORCHDIR}/../IOIPSL/inc"
  +
IOIPSL_LIBDIR="-L${ORCHDIR}/../../lib -L${ORCHDIR}/../IOIPSL/lib"
  +
IOIPSL_LIB="-lioipsl"
  +
  +
XIOS_INCDIR="-I${ORCHDIR}/../XIOS/inc"
  +
XIOS_LIBDIR="-L${ORCHDIR}/../XIOS/lib"
  +
XIOS_LIB="-lxios"
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.env ===
  +
Creation of file, since it does not exist
  +
  +
<pre>
  +
$ cp ../../../XIOS/arch/arch-ifort_HYDRA.env ./
  +
</pre>
  +
  +
== IOIPSL ==
  +
Using CICLAD's configuration
  +
<pre>
  +
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
  +
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.fcm ===
  +
Adapting values
  +
  +
<pre>
  +
%COMPILER mpif90
  +
%LINK mpif90
  +
%FPP mpif90 -E
  +
%AR ar
  +
%MAKE gmake
  +
%FPP_FLAGS -P
  +
%FPP_DEF KEY_NONE CPP_NETCDF4
  +
%BASE_FFLAGS -fPIC -i4 -r8 -auto -align all
  +
%PROD_FFLAGS -g -traceback -O3
  +
%DEV_FFLAGS -g -O1 -traceback
  +
%DEBUG_FFLAGS -g -traceback -check bounds -fp-model strict
  +
%MPI_FFLAGS
  +
%OMP_FFLAGS -openmp -openmp-threadprivate compat
  +
%BASE_LD -fPIC -i4 -r8 -auto
  +
%MPI_LD
  +
%OMP_LD -openmp
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.fcm ===
  +
Adapting values
  +
  +
<pre>
  +
ETCDF_INCDIR=$(nf-config --fflags)
  +
NETCDF_LIBDIR=$(nf-config --flibs)
  +
NETCDF_LIB=""
  +
  +
MPI_INCDIR=""
  +
MPI_LIBDIR=""
  +
MPI_LIB=""
  +
  +
HDF5_INCDIR=""
  +
HDF5_LIBDIR=""
  +
HDF5_LIB=""
  +
</pre>
   
 
== Bulding & compiling ==
  
== Bulding & compiling ==
−
** Going back to <code>util</code> to proceed
 +
Going back to <code>util</code> to proceed
−
<pre>$ cd ../../../util</pre>
 +
<!--<pre>$ cd ../../../util</pre>
−
** Installing the code:
 +
−
<pre>$ ./ins_make -t ifort_hydra >& run_ins_make.log</pre>
 +
Installing the code:
−
** Going there:
 +
<pre>$ ./ins_make -t ifort_HYDRA >& run_ins_make.log</pre-->
−
<pre>$ cd ../config/ORCHIDEE_OL</pre>
 +
−
** No need to loading the right environment since in <code>hydra</code> its by default
 +
Going there:
−
** Compiling XIOS in sequential (in <code>Makefile</Code>, adding no-parallel compilation):
 +
<pre>$ cd modispl/config/
−
<pre>./make_xios --netcdf_lib netcdf4_seq (...)</pre>
 +
</pre>
−
** Compiling:
 +
−
<pre>$ gmake >& run_gmake.log</pre>
 +
Moving folder to the right position:
−
** Checking if everything went fine:
 +
<pre>
−
<pre>$ cat -n run_gmake.log | grep Error
 +
$ mv ../modeles/ORCH/ORCHIDEE_OL
−
$ cat -n run_gmake.log | grep undefined
 +
$ mv ../modeles/ORCH/ORCHIDEE ../modeles/
  +
</pre>
  +
  +
Going there:
  +
<pre>$ cd ORCHIDEE_OL</pre>
  +
  +
Loding environment:
  +
<pre>
  +
source /opt/load-libs.sh 1
  +
</pre>
  +
  +
=== arch-ifort_HYDRA.env ===
  +
Preparing environtment for compilation
  +
<pre>
  +
$ cp ARCH/arch-ifort_CICLAD.env ARCH/arch-ifort_HYDRA.env
  +
</pre>
  +
  +
== Compilation ==
  +
  +
Going there:
  +
<pre>
  +
cd ../config/ORCHIDEE_OL/
  +
</pre>
  +
  +
Content:
  +
<pre>
  +
# Compile and running environnemnet for HYDRA at CIMA
  +
source /opt/load-libs.sh 1
  +
</pre>
  +
  +
Compiling XIOS in sequential (in <code>Makefile</Code>, adding no-parallel compilation):
  +
<pre>./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA >& run_compile.log</pre>
  +
  +
Checking if everything went fine:
  +
<pre>$ cat -n run_compile.log | grep Error
  +
$ cat -n run_compile.log | grep undefined
 
$ ls ../../bin/
 
$ ls ../../bin/
−
orchidee_ol xios_server.exe</pre>
 +
orchideedriver_prod orchidee_ol_prod xios_server_prod.exe
  +
</pre>
   
 
= Compiling rebuild =
  
= Compiling rebuild =
Línea 204: Línea 205:
   
 
== Instalation steps using modipsl ==
  
== Instalation steps using modipsl ==
−
* Go to the instalation directory <code>[INSTALLDIR]</code> (<code>/share</code> in <code>hydra</code>)
 +
Go to the instalation directory <code>[INSTALLDIR]</code> (<code>/share/ORCHIDEE</code> in <code>hydra</code>)
 
<pre>$ cd [INSTALLDIR]</pre>
  
<pre>$ cd [INSTALLDIR]</pre>
−
* Getting the IPSL-models suite (r 3379)
 +
−
<pre>$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg</pre>
 +
Getting the IPSL-models suite (r 3379)
−
* Once we got the codes, we need to copy the specificities of the installation for <code>hydra</code>
 +
<pre>
−
<pre>$ cp ../../modipsl/util/AA_make.gdef ./
 +
$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg
−
$ cp ../../modipsl/modeles/XIOS/arch/*hydra* ../modeles/XIOS/arch/
 +
</pre>
−
$ cp ../../modipsl/modeles/ORCHIDEE/arch/*hydra* ../modeles/ORCHIDEE/arch/</pre>
 +
−
* Everything is the same as for the installation, except that we need to indicate debugging compilation:
 +
Going to the <code>util</code> folder:
−
<pre>./ins_make -t ifort_hydra -d >& run_ins_make.log</pre>
 +
<pre>$ cd modipsl_dbg/util</pre>
−
* And on <code>Makefile</code> from <code>config/ORCHIDEE_OL</code>, change all the <code>-prod</code> by <code>-debug</code>
 +
  +
Getting the off-line version of ORCHIDEE (see components in file <code>mod.def</code>)
  +
<pre>$ ./model ORCHIDEE_trunk >& run_get_model.log</pre>
  +
  +
We should have:
  +
<pre>
  +
$ ls ../modeles
  +
IOIPSL ORCHIDEE XIOS
  +
</pre>
  +
  +
Getting all the environment files for each component (from the regular compilation)
  +
  +
<pre style="shell">$ cp AA_make.gdef AA_make.gdef.old
  +
$ cp ../../modipsl/util/AA_make.gdef ./
  +
$ cp ../../modipsl/modeles/IOIPSL/arch/arch-*HYDRA* ../modeles/IOIPSL/arch/
  +
$ cp ../../modipsl/modeles/ORCHIDEE/arch/arch-*HYDRA* ../modeles/ORCHIDEE/arch/
  +
$ cp ../../modipsl/modeles/XIOS/arch/arch-*HYDRA* ../modeles/XIOS/arch/
  +
</pre>
  +
  +
Everything is the same as for the installation, except that we need to indicate debugging compilation:
  +
<pre>./ins_make -t ifort_HYDRA -d >& run_ins_make.log</pre>
  +
  +
<!--And on <code>Makefile</code> from <code>config/ORCHIDEE_OL</code>, change all the <code>-prod</code> by <code>-debug</code>
 
<pre>(...)
  
<pre>(...)
 
xios :
 
xios :
Línea 225: Línea 226:
 
driver_xios :
 
driver_xios :
 
(cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -xios2 -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; )
 
(cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -xios2 -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; )
−
(...)</pre>
 +
(...)</pre-->
−
* Compiling:
 +
−
<pre>$ gmake >& run_gmake.log</pre>
 +
Compiling:
−
* Checking if everything went fine:
 +
<pre>
−
<pre>$ cat -n run_gmake.log | grep Error
+
$ cd ../config/ORCHIDEE_OL
−
$ cat -n run_gmake.log | grep undefined
 +
$ ./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA -debug >& run_compile.log
  +
</pre>
  +
  +
Checking if everything went fine:
  +
<pre>$ cat -n run_compile.log | grep Error
  +
$ cat -n run_compile.log | grep undefined
 
$ ls ../../bin/
 
$ ls ../../bin/
−
orchidee_ol xios_server.exe</pre>
 +
orchideedriver_debug orchidee_ol_debug xios_server_debug.exe
  +
</pre>

Última revisión de 10:56 6 mar 2023

Compilation at CIMA's HPC 'hydra'

CIMA's cluster named hydra, has the following compiler: 

fortran: intel v 2021.4.0
mpi: 3.4.2

Contenido

[editar] Installation steps using modipsl

Go to the installation directory [INSTALLDIR] (for hydra [INSTALLDIR]=/share/ORCHIDEE) 

$ cd [INSTALLDIR]

Instalation requirements: svn, ksh.

Getting the IPSL-models suite (trunk), following this link 

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl

Going to the util folder: 

$ cd modipsl/util

Getting the off-line version of ORCHIDEE (see components in file mod.def) 

$ ./model ORCHIDEE_trunk >& run_get_model.log

We got some issues with the ports, thus, it needs to be manually done: 

$ cd ../modeles

Getting IOIPSL: 

$ mkdir -p IOIPSL/src 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src

Getting XIOS library 

$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Getting ORCHIDEE:


By the end one should have: 

$ ls
IOIPSL  ORCH  XIOS

Going back to util to proceed 

$ cd ../util

These are the final versions of the codes (on 2022 January the 23rd)

  • ORCHIDEE: 7867
  • ORCHIDEE_OL: 7867
  • IOIPSL: 6273
  • XIOS: 2439

In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named arch or ARCH with files of the likes of arch-[compiler]_[machine] with 3 different extensions: .path, .fcm and .env. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify .path and .env).

  • arch-[compiler]_[machine].path: Determines the paths of the compilers and libraries
  • arch-[compiler]_[machine].fcm: Determines the flags of the compilers
  • arch-[compiler]_[machine].env: Determines the environtment for the compilation (e.g. module instructions)

There is a global file of configuration modipsl/util/AA_make.gdef, where one store the shared configuration for all the components.

NOTE: hydra's compilation will be based on ifort_MESOIPSL or the equivalent ifort_CICLAD

Preparing compilation environment for hydra. One need to add into AA_make.gdef specifications for hydra (at the end of file, for INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33):

To prepare an environment for compilation in hydra, one needs to execute (in this example we selected option 1: 

$ source /opt/load-libs.sh

Then, in order to know the paths of the different components (to be changed in the intel_HYDRA configuration)... 

$ which mpif90
/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90

$ nc-config --all
(...)
  --prefix        -> /opt/netcdf/netcdf-4/intel/2021.4.0
  --includedir    -> /opt/netcdf/netcdf-4/intel/2021.4.0/include
  --libdir        -> /opt/netcdf/netcdf-4/intel/2021.4.0/lib
  --version       -> netCDF 4.8.1

#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   #- Global definitions for hydra, ifort
#-Q- ifort_HYDRA LIB_MPI = MPI1
#-Q- ifort_HYDRA LIB_MPI_BIS = MPI1
#-Q- ifort_HYDRA FCM_ARCH = ifort_HYDRA
#-Q- ifort_HYDRA   M_K = make
#-Q- ifort_HYDRA   P_C = cpp
#-Q- ifort_HYDRA   P_O = -P -C -traditional $(P_P)
#-Q- ifort_HYDRA   F_C = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 -c -cpp
#-Q- ifort_HYDRA   #-D- MD    F_D = -g
#-Q- ifort_HYDRA   #-D- MN    F_D =
#-Q- ifort_HYDRA   #-P- I4R4  F_P = -i4
#-Q- ifort_HYDRA   #-P- I4R8  F_P = -i4 -r8
#-Q- ifort_HYDRA   #-P- I8R8  F_P = -i8 -r8
#-Q- ifort_HYDRA   F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
#-Q- ifort_HYDRA   F_L = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90
#-Q- ifort_HYDRA   M_M = 0
#-Q- ifort_HYDRA   L_X = 0
#-Q- ifort_HYDRA   L_O =
#-Q- ifort_HYDRA   A_C = ar -rs
#-Q- ifort_HYDRA   A_G = ar -x
#-Q- ifort_HYDRA   C_C = cc -c
#-Q- ifort_HYDRA   C_O =
#-Q- ifort_HYDRA   C_L = cc
#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   NCDF_INC = /opt/netcdf/netcdf-4/intel/2021.4.0/include
#-Q- ifort_HYDRA   NCDF_LIB = -L/opt/netcdf/netcdf-4/intel/2021.4.0/lib -lnetcdf -lnetcdff
#-Q- ifort_hydra  #-
  • Creation of the configuration arch files for the compilation of each component

[editar] XIOS

going there: 

$ cd ../modeles/XIOS/arch

Using CICLAD's ifort compilation as reference 

$ cp arch-ifort_CICLAD.env arch-ifort_HYDRA.env
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

[editar] arch-ifort_HYDRA.fcm

Adapting values (following ifort bug issue for version 12.0.3, ifort bug): 

CCOMPILER           mpicc
%FCOMPILER           mpif90
%LINKER              mpif90 -nofor-main

%BASE_CFLAGS    -fPIC -std=c++11
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g
%DEBUG_CFLAGS   -DBZ_DEBUG -g -fno-inline

%BASE_FFLAGS    -fPIC -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           gmake

[editar] arch-ifort_HYDRA.path

Adapting values: 

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdff -lnetcdf"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

[editar] arch-ifort_HYDRA.env

Adpating values 

source /opt/load-libs.sh 1

[editar] ORCHIDEE

Going there 

$ cd ../../ORCH/ORCHIDEE/arch

Using CICLAD's ifort compilation as reference 

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

[editar] arch-ifort_HYDRA.fcm

Adapting values: 

COMPILER            mpif90
%LINK                mpif90
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P -traditional
%FPP_DEF
%BASE_FFLAGS         -i4 -r8 -fp-model precise
%PROD_FFLAGS         -O3
%DEV_FFLAGS          -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds
%DEBUG_FFLAGS        -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -init=arrays -init=snan -check bounds,noarg_temp_created
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -i4 -r8
%MPI_LD
%OMP_LD              -openmp

[editar] arch-ifort_HYDRA.path

Adpating values 

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdf -lnetcdff -lstdc++"

IOIPSL_INCDIR="-I${ORCHDIR}/../../lib -I${ORCHDIR}/../IOIPSL/inc"
IOIPSL_LIBDIR="-L${ORCHDIR}/../../lib -L${ORCHDIR}/../IOIPSL/lib"
IOIPSL_LIB="-lioipsl"

XIOS_INCDIR="-I${ORCHDIR}/../XIOS/inc"
XIOS_LIBDIR="-L${ORCHDIR}/../XIOS/lib"
XIOS_LIB="-lxios"

[editar] arch-ifort_HYDRA.env

Creation of file, since it does not exist 

$ cp ../../../XIOS/arch/arch-ifort_HYDRA.env ./

[editar] IOIPSL

Using CICLAD's configuration 

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

[editar] arch-ifort_HYDRA.fcm

Adapting values 

%COMPILER            mpif90
%LINK                mpif90
%FPP                 mpif90 -E
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P
%FPP_DEF             KEY_NONE CPP_NETCDF4
%BASE_FFLAGS         -fPIC -i4 -r8 -auto -align all
%PROD_FFLAGS         -g -traceback -O3
%DEV_FFLAGS          -g -O1 -traceback
%DEBUG_FFLAGS        -g -traceback -check bounds -fp-model strict
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -fPIC -i4 -r8 -auto
%MPI_LD
%OMP_LD              -openmp

[editar] arch-ifort_HYDRA.fcm

Adapting values 

ETCDF_INCDIR=$(nf-config --fflags)
NETCDF_LIBDIR=$(nf-config --flibs)
NETCDF_LIB=""

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

[editar] Bulding & compiling

Going back to util to proceed

Going there: 

$ cd modispl/config/

Moving folder to the right position: 

$ mv ../modeles/ORCH/ORCHIDEE_OL
$ mv ../modeles/ORCH/ORCHIDEE ../modeles/

Going there: 

$ cd ORCHIDEE_OL

Loding environment: 

source /opt/load-libs.sh 1

[editar] arch-ifort_HYDRA.env

Preparing environtment for compilation 

$ cp ARCH/arch-ifort_CICLAD.env ARCH/arch-ifort_HYDRA.env

[editar] Compilation

Going there: 

cd ../config/ORCHIDEE_OL/

Content: 

# Compile and running environnemnet for HYDRA at CIMA
source /opt/load-libs.sh 1

Compiling XIOS in sequential (in Makefile, adding no-parallel compilation): 

./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA >& run_compile.log

Checking if everything went fine: 

$ cat -n run_compile.log | grep Error
$ cat -n run_compile.log | grep undefined
$ ls ../../bin/ 
orchideedriver_prod  orchidee_ol_prod  xios_server_prod.exe

[editar] Compiling rebuild

Due to the fact that hydra does not have the netcdf libraries compiled in parallel, ORCHDEE output is written with multiple files being a file for each process. In order to overcome this output. IPSL does prepare a tool to join the files and provide the expected output. This tool is called rebuild (a Kornshell script which encapsulates de exacutable flio_rbld) and it comes with the IOIPSL libraries. The IOIPSL libraries, are a set of old libraries developed by the IPSL (M. A. Foujuls, J. Polcher among others) to deal with the netCDF data access for all the models developed at the IOIPSL. Compiling IOIPSL (following install rebuild. It has to be compiled independently. Here it is explained how it has to be done.

  • Go to the instalation directory [INSTALLDIR] (at hydra share) 
$ cd [INSTALLDIR]
  • getting the last version of the code 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_IOIPSLtools
  • going to prepare compilation/installation 
$ cd modipsl_IOIPSLtools/util
  • preparing environment 
./model IOIPSL_PLUS
  • We got some issues with the ports, thus, it needs to be manually done: 
$ cd ../modeles
  • Getting IOIPSL (NOTE: notice the difference in library's version from the previous one): 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_3 IOIPSL
  • Comming back to the util folder 
$ cd ../util
  • Creation of the compilation environment for `hydra' using 'ifort' inside AA_make.gdef as in the full compilation 
$ cp ../../modipsl/util/AA_make.gdef ./
  • Preparing makefiles 
$ ./ins_make -t "ifort_hydra" -p I4R8
  • going to src folder to compile 
$ cd ../modeles/IOIPSL/src
  • cleaning (you never know...) 
$ make clean
  • compiling 
$ make >& run_make.log
  • Checking no errors: 
$ cat -n run_make.log | grep Error
  • compilation of tools (file concatenation) 
$ cd ../tools
  • cleaning (you never know...) 
$ make clean
  • compiling 
$ make >& run_make.log
  • Checking no errors: 
$ cat -n run_make.log | grep Error
  • One should have: 
$ ls ../../../bin/ 
flio_rbld  rebuild

[editar] debug Instalation

Getting a debugging copy for test purposes

[editar] Instalation steps using modipsl

Go to the instalation directory [INSTALLDIR] (/share/ORCHIDEE in hydra) 

$ cd [INSTALLDIR]

Getting the IPSL-models suite (r 3379) 

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg

Going to the util folder: 

$ cd modipsl_dbg/util

Getting the off-line version of ORCHIDEE (see components in file mod.def) 

$ ./model ORCHIDEE_trunk >& run_get_model.log

We should have: 

$ ls ../modeles
IOIPSL  ORCHIDEE  XIOS

Getting all the environment files for each component (from the regular compilation) 

$ cp AA_make.gdef AA_make.gdef.old
$ cp ../../modipsl/util/AA_make.gdef ./
$ cp ../../modipsl/modeles/IOIPSL/arch/arch-*HYDRA* ../modeles/IOIPSL/arch/
$ cp ../../modipsl/modeles/ORCHIDEE/arch/arch-*HYDRA* ../modeles/ORCHIDEE/arch/
$ cp ../../modipsl/modeles/XIOS/arch/arch-*HYDRA* ../modeles/XIOS/arch/

Everything is the same as for the installation, except that we need to indicate debugging compilation: 

./ins_make -t ifort_HYDRA -d >& run_ins_make.log


Compiling: 

$ cd ../config/ORCHIDEE_OL
$ ./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA -debug >& run_compile.log

Checking if everything went fine: 

$ cat -n run_compile.log | grep Error 
$ cat -n run_compile.log | grep undefined
$ ls ../../bin/ 
orchideedriver_debug  orchidee_ol_debug  xios_server_debug.exe
Obtenido de «https://wiki.cima.fcen.uba.ar/mediawiki/index.php?title=ORcompHYDRA_hydra&oldid=2579»
Herramientas personales