ORcompHYDRA hydra

De Wikicima
Saltar a: navegación, buscar

Compilation at CIMA's HPC 'hydra'

Contenido

Installation steps using modipsl

Go to the installation directory [INSTALLDIR] (for hydra [INSTALLDIR]=/share/ORCHIDEE) 

$ cd [INSTALLDIR]

Instalation requirements: svn, ksh.

Getting the IPSL-models suite (trunk), following this link 

$ svn co --username icmc_users https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl

Going to the util folder: 

$ cd modipsl/util

Getting the off-line version of ORCHIDEE (see components in file mod.def) 

$ ./model ORCHIDEE_trunk >& run_get_model.log

We got some issues with the ports, thus, it needs to be manually done: 

$ cd ../modeles

Getting IOIPSL: 

$ mkdir -p IOIPSL/src 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_2/src IOIPSL/src

Getting XIOS library 

$ svn co -r 1043 https://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/trunk XIOS

Getting ORCHIDEE:


By the end one should have: 

$ ls
IOIPSL  ORCH  XIOS

Going back to util to proceed 

$ cd ../util

These are the final versions of the codes (on 2022 January the 23rd)

  • ORCHIDEE: 7867
  • ORCHIDEE_OL: 7867
  • IOIPSL: 6273
  • XIOS: 2439

In order to compile in a new machine, one need to prepare the files for the compilation of each component of the IPSL model. For each component there is a folder named arch or ARCH with files of the likes of arch-[compiler]_[machine] with 3 different extensions: .path, .fcm and .env. One need to adapt these files to the specifications and characteristics of the HPC. The recommendation is to copy the files for the desired compiler from a different machine (to get the right flags) and adapt them to the new HPC (usually one would only need to modify .path and .env).

  • arch-[compiler]_[machine].path: Determines the paths of the compilers and libraries
  • arch-[compiler]_[machine].fcm: Determines the flags of the compilers
  • arch-[compiler]_[machine].env: Determines the environtment for the compilation (e.g. module instructions)

There is a global file of configuration modipsl/util/AA_make.gdef, where one store the shared configuration for all the components.

NOTE: hydra's compilation will be based on ifort_MESOIPSL or the equivalent ifort_CICLAD

Preparing compilation environment for hydra. One need to add into AA_make.gdef specifications for hydra (at the end of file, for INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33):

To prepare an environment for compilation in hydra, one needs to execute (in this example we selected option 1: 

$ source /opt/load-libs.sh

Then, in order to know the paths of the different components (to be changed in the intel_HYDRA configuration)... 

$ which mpif90
/opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90

$ nc-config --all
(...)
  --prefix        -> /opt/netcdf/netcdf-4/intel/2021.4.0
  --includedir    -> /opt/netcdf/netcdf-4/intel/2021.4.0/include
  --libdir        -> /opt/netcdf/netcdf-4/intel/2021.4.0/lib
  --version       -> netCDF 4.8.1

#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   #- Global definitions for hydra, ifort
#-Q- ifort_HYDRA LIB_MPI = MPI1
#-Q- ifort_HYDRA LIB_MPI_BIS = MPI1
#-Q- ifort_HYDRA FCM_ARCH = ifort_HYDRA
#-Q- ifort_HYDRA   M_K = make
#-Q- ifort_HYDRA   P_C = cpp
#-Q- ifort_HYDRA   P_O = -P -C -traditional $(P_P)
#-Q- ifort_HYDRA   F_C = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90 -c -cpp
#-Q- ifort_HYDRA   #-D- MD    F_D = -g
#-Q- ifort_HYDRA   #-D- MN    F_D =
#-Q- ifort_HYDRA   #-P- I4R4  F_P = -i4
#-Q- ifort_HYDRA   #-P- I4R8  F_P = -i4 -r8
#-Q- ifort_HYDRA   #-P- I8R8  F_P = -i8 -r8
#-Q- ifort_HYDRA   F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
#-Q- ifort_HYDRA   F_L = /opt/mpich/mpich-3.4.2/intel/2021.4.0/bin/mpif90
#-Q- ifort_HYDRA   M_M = 0
#-Q- ifort_HYDRA   L_X = 0
#-Q- ifort_HYDRA   L_O =
#-Q- ifort_HYDRA   A_C = ar -rs
#-Q- ifort_HYDRA   A_G = ar -x
#-Q- ifort_HYDRA   C_C = cc -c
#-Q- ifort_HYDRA   C_O =
#-Q- ifort_HYDRA   C_L = cc
#-Q- ifort_HYDRA   #-
#-Q- ifort_HYDRA   NCDF_INC = /opt/netcdf/netcdf-4/intel/2021.4.0/include
#-Q- ifort_HYDRA   NCDF_LIB = -L/opt/netcdf/netcdf-4/intel/2021.4.0/lib -lnetcdf -lnetcdff
#-Q- ifort_hydra  #-
  • Creation of the configuration arch files for the compilation of each component

XIOS

going there: 

$ cd ../modeles/XIOS/arch

Using CICLAD's ifort compilation as reference 

$ cp arch-ifort_CICLAD.env arch-ifort_HYDRA.env
$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values (following ifort bug issue for version 12.0.3, ifort bug): 

CCOMPILER           mpicc
%FCOMPILER           mpif90
%LINKER              mpif90 -nofor-main

%BASE_CFLAGS    -fPIC -std=c++11
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g
%DEBUG_CFLAGS   -DBZ_DEBUG -g -fno-inline

%BASE_FFLAGS    -fPIC -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           gmake

arch-ifort_HYDRA.path

Adapting values: 

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdff -lnetcdf"
MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""
HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

arch-ifort_HYDRA.env

Adpating values 

source /opt/load-libs.sh 1

ORCHIDEE

Going there 

$ cd ../../ORCH/ORCHIDEE/arch

Using CICLAD's ifort compilation as reference 

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values: 

COMPILER            mpif90
%LINK                mpif90
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P -traditional
%FPP_DEF
%BASE_FFLAGS         -i4 -r8 -fp-model precise
%PROD_FFLAGS         -O3
%DEV_FFLAGS          -g -O2 -traceback -fp-stack-check -ftrapuv -check bounds
%DEBUG_FFLAGS        -fpe0 -O0 -g -traceback -fp-stack-check -ftrapuv -init=arrays -init=snan -check bounds,noarg_temp_created
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -i4 -r8
%MPI_LD
%OMP_LD              -openmp

arch-ifort_HYDRA.path

Adpating values 

NETCDF_INCDIR="-I /opt/netcdf/netcdf-4/intel/2021.4.0/include"
NETCDF_LIBDIR="-L/opt/netcdf/netcdf-4/intel/2021.4.0/lib"
NETCDF_LIB="-lnetcdf -lnetcdff -lstdc++"

IOIPSL_INCDIR="-I${ORCHDIR}/../../lib -I${ORCHDIR}/../IOIPSL/inc"
IOIPSL_LIBDIR="-L${ORCHDIR}/../../lib -L${ORCHDIR}/../IOIPSL/lib"
IOIPSL_LIB="-lioipsl"

XIOS_INCDIR="-I${ORCHDIR}/../XIOS/inc"
XIOS_LIBDIR="-L${ORCHDIR}/../XIOS/lib"
XIOS_LIB="-lxios"

arch-ifort_HYDRA.env

Creation of file, since it does not exist 

$ cp ../../../XIOS/arch/arch-ifort_HYDRA.env ./

IOIPSL

Using CICLAD's configuration 

$ cp arch-ifort_CICLAD.fcm arch-ifort_HYDRA.fcm
$ cp arch-ifort_CICLAD.path arch-ifort_HYDRA.path

arch-ifort_HYDRA.fcm

Adapting values 

%COMPILER            mpif90
%LINK                mpif90
%FPP                 mpif90 -E
%AR                  ar
%MAKE                gmake
%FPP_FLAGS           -P
%FPP_DEF             KEY_NONE CPP_NETCDF4
%BASE_FFLAGS         -fPIC -i4 -r8 -auto -align all
%PROD_FFLAGS         -g -traceback -O3
%DEV_FFLAGS          -g -O1 -traceback
%DEBUG_FFLAGS        -g -traceback -check bounds -fp-model strict
%MPI_FFLAGS
%OMP_FFLAGS          -openmp -openmp-threadprivate compat
%BASE_LD             -fPIC -i4 -r8 -auto
%MPI_LD
%OMP_LD              -openmp

arch-ifort_HYDRA.fcm

Adapting values 

ETCDF_INCDIR=$(nf-config --fflags)
NETCDF_LIBDIR=$(nf-config --flibs)
NETCDF_LIB=""

MPI_INCDIR=""
MPI_LIBDIR=""
MPI_LIB=""

HDF5_INCDIR=""
HDF5_LIBDIR=""
HDF5_LIB=""

Bulding & compiling

Going back to util to proceed

Going there: 

$ cd modispl/config/

Moving folder to the right position: 

$ mv ../modeles/ORCH/ORCHIDEE_OL
$ mv ../modeles/ORCH/ORCHIDEE ../modeles/

Going there: 

$ cd ORCHIDEE_OL

Loding environment: 

source /opt/load-libs.sh 1

Preparing environtment for compilation 

$ cp ARCH/arch-ifort_CICLAD.env ARCH/arch-ifort_HYDRA.env

Content of arch-ifort_HYDRA.env 

# Compile and running environnemnet for HYDRA at CIMA
source /opt/load-libs.sh 1

Compiling XIOS in sequential (in Makefile, adding no-parallel compilation): 

./compile_orchidee_ol.sh -netcdf_lib_seq -arch ifort_HYDRA >& run_compile.log

Compiling: 

$ gmake >& run_gmake.log

Checking if everything went fine: 

$ cat -n run_gmake.log | grep Error
$ cat -n run_gmake.log | grep undefined
$ ls ../../bin/ 
orchidee_ol  xios_server.exe

Compiling rebuild

Due to the fact that hydra does not have the netcdf libraries compiled in parallel, ORCHDEE output is written with multiple files being a file for each process. In order to overcome this output. IPSL does prepare a tool to join the files and provide the expected output. This tool is called rebuild (a Kornshell script which encapsulates de exacutable flio_rbld) and it comes with the IOIPSL libraries. The IOIPSL libraries, are a set of old libraries developed by the IPSL (M. A. Foujuls, J. Polcher among others) to deal with the netCDF data access for all the models developed at the IOIPSL. Compiling IOIPSL (following install rebuild. It has to be compiled independently. Here it is explained how it has to be done.

  • Go to the instalation directory [INSTALLDIR] (at hydra share) 
$ cd [INSTALLDIR]
  • getting the last version of the code 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_IOIPSLtools
  • going to prepare compilation/installation 
$ cd modipsl_IOIPSLtools/util
  • preparing environment 
./model IOIPSL_PLUS
  • We got some issues with the ports, thus, it needs to be manually done: 
$ cd ../modeles
  • Getting IOIPSL (NOTE: notice the difference in library's version from the previous one): 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/IOIPSL/tags/v2_2_3 IOIPSL
  • Comming back to the util folder 
$ cd ../util
  • Creation of the compilation environment for `hydra' using 'ifort' inside AA_make.gdef as in the full compilation 
$ cp ../../modipsl/util/AA_make.gdef ./
  • Preparing makefiles 
$ ./ins_make -t "ifort_hydra" -p I4R8
  • going to src folder to compile 
$ cd ../modeles/IOIPSL/src
  • cleaning (you never know...) 
$ make clean
  • compiling 
$ make >& run_make.log
  • Checking no errors: 
$ cat -n run_make.log | grep Error
  • compilation of tools (file concatenation) 
$ cd ../tools
  • cleaning (you never know...) 
$ make clean
  • compiling 
$ make >& run_make.log
  • Checking no errors: 
$ cat -n run_make.log | grep Error
  • One should have: 
$ ls ../../../bin/ 
flio_rbld  rebuild

debug Instalation

Getting a debugging copy for test purposes

Instalation steps using modipsl

  • Go to the instalation directory [INSTALLDIR] (/share in hydra) 
$ cd [INSTALLDIR]
  • Getting the IPSL-models suite (r 3379) 
$ svn co https://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl_dbg
  • Once we got the codes, we need to copy the specificities of the installation for hydra 
$ cp ../../modipsl/util/AA_make.gdef ./
$ cp ../../modipsl/modeles/XIOS/arch/*hydra* ../modeles/XIOS/arch/
$ cp ../../modipsl/modeles/ORCHIDEE/arch/*hydra* ../modeles/ORCHIDEE/arch/
  • Everything is the same as for the installation, except that we need to indicate debugging compilation: 
./ins_make -t ifort_hydra -d >& run_ins_make.log
  • And on Makefile from config/ORCHIDEE_OL, change all the -prod by -debug 
(...)
xios : 
        (cd  ../../modeles/XIOS ; ./make_xios --netcdf_lib netcdf4_seq \
--debug --arch ${FCM_ARCH} --job 8 ; cp bin/xios_server.exe ../../bin/. ; ) 
(...)
driver : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; )
(...)
driver_xios : 
        (cd ../../modeles/ORCHIDEE ; ./makeorchidee_fcm -xios2 -j 8 -parallel mpi -debug -arch ${FCM_ARCH} -driver ; ) 
(...)
  • Compiling: 
$ gmake >& run_gmake.log
  • Checking if everything went fine: 
$ cat -n run_gmake.log | grep Error 
$ cat -n run_gmake.log | grep undefined
$ ls ../../bin/ 
orchidee_ol  xios_server.exe
Obtenido de «https://wiki.cima.fcen.uba.ar/mediawiki/index.php?title=ORcompHYDRA_hydra&oldid=2346»