papa-deimos/instalacion

Web page of the instalation of the ocean model CROCO in CIMA's HPC 'hydra'

Installation

CIMA's HPC has a common folder where all the people has access. This is going to be the place where CROCO model will be installed.

The installation is going to be in the following INSTALLDIR=/share/CROCO folder.

Downloading the code (there is a GItlab repository, but by now, we are going to install closed releases)

$ wget https://gitlab.inria.fr/croco-ocean/croco/-/archive/v2.1.0/croco-v2.1.0.tar.gz

Creating the folder of the compilation (using intel compiler and shared memory dmpar) and where to deploy the code

$ mkdir $INSTALLDIR/v210
$ cd $INSTALLDIR/v210
$ mkdir -p intel/dmpar
$ git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco.git croco-v2.1.0
Cloning into 'croco'...
remote: Enumerating objects: 52646, done.
remote: Counting objects: 100% (448/448), done.
remote: Compressing objects: 100% (263/263), done.
remote: Total 52646 (delta 268), reused 317 (delta 185), pack-reused 52198 (from 1)
Receiving objects: 100% (52646/52646), 300.67 MiB | 4.59 MiB/s, done.
Resolving deltas: 100% (40426/40426), done.
Note: switching to '291e8cd11b2329ee171c7a02c52c9c481dda981e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false
$ ls croco
GRIF         create_config.bash  OBSTRUCTION  README.md         TEST_CASES
BENCH         MPI_NOLAND          OCEAN        requirements.txt  XIOS
CHANGELOG.md  MUSTANG             PISCES       SCRIPTS

Go to compile, first we set-up the environment for intel compilation

$ source /opt/load-libs.sh 1
The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
$ cd croco

We create an ASCII file in /share/CROCO/v210/intel with the specificities of the compilation with intel in hydra, name CROCO_hydra_intel.env (following the instructios from here tutorial 00)

#set environment variables for croco in HYDRA to be compiled with intel
source /opt/load-libs.sh 1
export CC=icc
export FC=ifort 
export F90=ifort 
export F77=ifort 
export NETCDF=/opt/netcdf/netcdf-4/intel/2021.4.0 
export PATH=$NETCDF/bin::${PATH} 
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}::${NETCDF}/lib

Loading the compilation environment

$ source CROCO_hydra_intel.env

BASIN

The simplest installation would be the 'basin' configuration. We follow these instructions basin. Assuming that the source code resides in CROCOsrc=/share/CROCO/v210/intel/serie/croco

$ CROCOsrc=/share/CROCO/v210/intel/serie/croco
$ cd ${CROCOsrc}
$ mkdir -p CONFIGS/BASIN
cp OCEAN/cppdefs.h CONFIGS/BASIN
cp OCEAN/param.h CONFIGS/BASIN
cp OCEAN/jobcomp CONFIGS/BASIN
cd CONFIGS/BASIN

We need to edit the configuration files.

Editing the file with the configuration

$ cp cppdefs.h cppdefs.h_orig
$ diff cppdefs.h cppdefs.h_orig 
18c18
< #define  BASIN           /* Basin Example */
---
> #undef  BASIN           /* Basin Example */
56c56
< #undef REGIONAL        /* REGIONAL Applications */
---
> #define REGIONAL        /* REGIONAL Applications */

Editing the file with the size of the domain and configuration of the parallel environment

$ cp param.h param.h_orig
$ diff param.h param.h_orig

Editing the file with the compilation

$ cp jobcomp jobcomp_orig
$ diff jobcomp jobcomp_orig 
34,35c34
< CROCOsrc=/share/CROCO/v210/intel/serie/croco
< SOURCE1=${CROCOsrc}/OCEAN
---
> SOURCE1=../croco/OCEAN
45c45
< FC=ifort
---
> FC=gfortran

And compile

$ ./jobcomp > jobcomp.log

If everything when fine, one should have:

$ tail -n 15 jobcomp.log 
mv a.out croco
                                   
                                   
    .,:looddddddd:            ,;'  
  .cdddo;;oo'....            .ldd; 
 ,ddddd,                    ;odoc. 
.oddddo:               .'';odoc    
'dddddd;':,  .;;  .;;. :ccdo;      
.oddddddddooooddoooddooodl.        
 .oddddddddddddddddddddddl;;       
  .,ldddddddddddddddddddoc:lolo'.. 
    ;ddc;::::::::::codo,     .';:, 
   .ccc             ,cc,           
                                   
CROCO is OK
$ ls -l croco
-rwxrwxr-x 1 lluis.fita cima 1012160 Jul 15 15:11 croco

IMPORTANT NOTES
1. do not run the model in the same place where you compiled the source
2. Always use a queue-system job to run simulations, never run without it
3. Always run in your salidas/ folder never in $HOME

UPWELLING

Compilation of the upwelling test case in distributed memory (dmpar) with intel.

$ $ source /opt/load-libs.sh 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33
$ mkdir /share/CROCO/v210/intel/dmpar/
$ git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco.git croco
$ cd croco
$ CROCOsrc=/share/CROCO/v210/intel/dmpar/croco

Copying configuration files

$ mkdir -p CONFIGS/Upwelling
cp OCEAN/cppdefs.h CONFIGS/Upwelling
cp OCEAN/param.h CONFIGS/Upwelling
cp OCEAN/jobcomp CONFIGS/Upwelling
cd CONFIGS/Upwelling

Configuring the case

$ cp cppdefs.h cppdefs.h_orig
$ diff cppdefs.h cppdefs.h_orig 
29c29
< #define  UPWELLING       /* Upwelling Example */
---
> #undef  UPWELLING       /* Upwelling Example */
56c56
< #undef REGIONAL        /* REGIONAL Applications */
---
> #define REGIONAL        /* REGIONAL Applications */
1037c1037
< # define MPI
---
> # undef  MPI

$ cp param.h param.h_orig
$ diff param.h param.h_orig 

$ cp jobcomp jobcomp_orig
$ diff jobcomp jobcomp_orig 
34,35c34
< CROCOsrc=/share/CROCO/v210/intel/dmpar/croco
< SOURCE1=${CROCOsrc}/OCEAN
---
> SOURCE1=../croco/OCEAN
46c45
< FC=ifort
---
> FC=gfortran
52,53c51,52
< MPILIB="-L/opt/mpich/mpich-3.4.2/intel/2021.4.0/lib"
< MPIINC="-I/opt/mpich/mpich-3.4.2/intel/2021.4.0/include"
---
> MPILIB=""
> MPIINC=""

Compiling in parallel

$ ./jobcomp >& jobcomp.log
$ tail -n 15 jobcomp.log
 mv a.out croco
                                   
                                   
    .,:looddddddd:            ,;'  
  .cdddo;;oo'....            .ldd; 
 ,ddddd,                    ;odoc. 
.oddddo:               .'';odoc    
'dddddd;':,  .;;  .;;. :ccdo;      
.oddddddddooooddoooddooodl.        
 .oddddddddddddddddddddddl;;       
  .,ldddddddddddddddddddoc:lolo'.. 
    ;ddc;::::::::::codo,     .';:, 
   .ccc             ,cc,           
                                   
CROCO is OK

$ ls -l croco
-rwxrwxr-x 1 lluis.fita cima 1184600 Jul 15 16:34 croco

IMPORTANT NOTES
1. do not run the model in the same place where you compiled the source
2. Always use a queue-system job to run simulations, never run without it
3. Always run in your salidas/ folder never in $HOME

Example of use

serie: BASIN

How to run a basic example

$ cd $HOME/salidas
$ mkdir CROCO
$ cd CROCO
$ mkdir tests
$ cd tests/
$ cp /share/tools/workflows/direct/CROCO/hydra/launch_ideal_test_serie.pbs ./

Link the necessary files and editng them if needed:

$ ln -s $CROCOsrc/CONFIGS/BASIN/croco ./
$ cp $CROCOsrc/TEST_CASES/croco.in.Basin ./

And execute

$ qsub launch_ideal_test_serie.pbs
70619.hydra
$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
(...)
70619.hydra                CROCO4L_ideal    lluis.fita             0 R larga 

Looking for the current step:

$ tail croco.out 
    2698   299.77778 8.942712777E-05-3.3148734E+00-3.3147840E+00 5.0400000E+16  0
    2699   299.88889 8.942810068E-05-3.3148728E+00-3.3147833E+00 5.0400000E+16  0
    2700   300.00000 8.942902900E-05-3.3148721E+00-3.3147827E+00 5.0400000E+16  0
      WRT_HIS -- wrote history fields into time record =   31 /   31
    2701   300.11111 8.942991352E-05-3.3148715E+00-3.3147820E+00 5.0400000E+16  0
    2702   300.22222 8.943075502E-05-3.3148708E+00-3.3147814E+00 5.0400000E+16  0
    2703   300.33333 8.943155431E-05-3.3148701E+00-3.3147807E+00 5.0400000E+16  0
    2704   300.44444 8.943231222E-05-3.3148695E+00-3.3147800E+00 5.0400000E+16  0
    2705   300.55556 8.943302959E-05-3.3148688E+00-3.3147794E+00 5.0400000E+16  0
    2706   300.66667 8.943370728E-05-3.3148682E+00-3.3147787E+00 5.0400000E+16  0

Once finished:

$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
(...)
70619.hydra                CROCO4L_ideal    lluis.fita      00:01:13 C larga          

$ tail croco.out 
    3239   359.88889 9.041893005E-05-3.3144938E+00-3.3144033E+00 5.0400000E+16  0
    3240   360.00000 9.041879739E-05-3.3144930E+00-3.3144026E+00 5.0400000E+16  0
      WRT_HIS -- wrote history fields into time record =   37 /   37

 MAIN - number of records written into history  file(s):   37
        number of records written into restart  file(s):    0


 MAIN: DONE

$ ls -lrt
total 15576
-rw-rw-r-- 1 lluis.fita cima     1412 Jul 15 15:40 launch_ideal_test_serie.pbs
lrwxrwxrwx 1 lluis.fita cima       55 Jul 15 15:44 croco -> /share/CROCO/v210/intel/serie/croco/CONFIGS/BASIN/croco
-rw-rw-r-- 1 lluis.fita cima     3361 Jul 15 15:46 croco.in.Basin
-rw------- 1 lluis.fita cima     3361 Jul 15 15:47 croco.in
-rw------- 1 lluis.fita cima      824 Jul 15 15:48 CROCO4L_ideal.o70619
-rw------- 1 lluis.fita cima 15653792 Jul 15 15:48 basin_his.nc
-rw------- 1 lluis.fita cima   274775 Jul 15 15:48 croco.out

dmpar: Upwelling

How to run a basic example

$ cd $HOME/salidas
$ mkdir CROCO
$ cd CROCO
$ mkdir mpi
$ cd mpi/
$ cp /share/tools/workflows/direct/CROCO/hydra/launch_ideal_test_mpi.pbs ./

Link the necessary files and editng them if needed:

$ ln -s $CROCOsrc/CONFIGS/Upwelling/croco ./
$ cp $CROCOsrc/TEST_CASES/croco.in.Upwelling ./

And execute

$ qsub launch_ideal_test_mpi.pbs
70649.hydra
$ qstat
Job ID                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
(...)
70645.hydra               CROCO4L_ideal_   lluis.fita             0 R larga 

Once finished

$ tail -3 croco.out 

 MAIN: DONE

$ ls -lrt
total 6956
lrwxrwxrwx 1 lluis.fita cima      59 Jul 15 16:42 croco -> /share/CROCO/v210/intel/dmpar/croco/CONFIGS/Upwelling/croco
-rw-rw-r-- 1 lluis.fita cima    1500 Jul 15 16:42 croco.in.Upwelling
-rw------- 1 lluis.fita cima    1500 Jul 15 16:43 croco.in
-rwxrwxr-x 1 lluis.fita cima    9328 Jul 15 16:43 a.out
-rw-rw-r-- 1 lluis.fita cima    1448 Jul 15 16:50 launch_ideal_test_mpi.pbs
-rw------- 1 lluis.fita cima  238630 Jul 15 16:50 CROCO4L_ideal_mpi.o70634
-rw------- 1 lluis.fita cima  717220 Jul 15 16:50 upwelling_rst.nc
-rw------- 1 lluis.fita cima 3316688 Jul 15 16:50 upwelling_his.nc
-rw------- 1 lluis.fita cima 2771248 Jul 15 16:50 upwelling_avg.nc
-rw------- 1 lluis.fita cima   42450 Jul 15 16:50 croco.out

RUNNING A REAL SIMULATION

v2.1: simulación climatológica

Ejemplo de uso en un entorno real de CROCO para simular un periodo corto.

Para eso necesitamos:

• Forzantes atmosféricos:
• Batimetría:

v2.1: simulación multianual

Ejemplo de uso en un entorno real de CROCO para simular mútliples años reciclando el forzante atmosférico.

Para eso necesitamos:

• Forzantes atmosféricos:
• Batimetría:

v1.1 - Nicolás Aubone modified

Here we run a real simulation with croco version 1.1

It's a simulation of one climatologic year in the San Matías Gulf, Argentina.

Be carefull, this model was modified from its original code (we add a bi-dimensional logarithmic bottom friction).

Creating the folder where to deploy the code (using intel compiler and shared memory dmpar):

$ INSTALLDIR=/share/CROCO
$ mkdir $INSTALLDIR/v11
$ cd $INSTALLDIR/v11
$ mkdir -p intel/dmpar
$ cd intel/dmpar
$ tar xvfz [CROCOv1.1_NicolasAubone].tar.gz 
$ ls croco
AGRIF            OCEAN                      run_simulation_openmp.bash
create_run.bash  PISCES                     SCRIPTS
CVTK             readme_version_croco.txt   TEST_CASES
DOC_SPHINX       run_multi_simulation.bash  XIOS
MPP_PREP         run_simulation.bash

Create the "CONFIGS" folder to store our different simulations there.

Create the "GSM_CLIMATOLOGY" folder to store our simulation config files and the model compilation.

Copy to the simulation forlder "GSM_CLIMATOLOGY" our configuration files to compile and run the model:

$ CROCOsrc=$INSTALLDIR/v110/intel/dmpar/croco
$ cd ${CROCOsrc}
$ mkdir -p CONFIGS/GSM_CLIMATOLOGY
$ cd CONFIGS/GSM_CLIMATOLOGY
cp [file_location]/cppdefs.h CONFIGS/BASIN
cp [file_location]/param.h CONFIGS/BASIN
cp [file_location]/jobcomp CONFIGS/BASIN

---

Edit cppdefs.h file to define a mpi run:

define MPI

---

Edit param.h file to set the mpi parallelization settings, in this case we use 60 nodes:

#ifdef MPI

integer NP_XI, NP_ETA, NNODES

parameter (NP_XI=10, NP_ETA=6, NNODES=NP_XI*NP_ETA)

---

Edit the compilation file "jobcomp" with hydra parameters:

ROOT_DIR=/share/CROCO/v110/intel/dmpar/croco

SOURCE=${ROOT_DIR}'/OCEAN'

FC=ifort

MPIF90="mpif90"

MPILIB="-L/opt/mpich/mpich-3.4.2/intel/2021.4.0/lib"

MPIINC="-I/opt/mpich/mpich-3.4.2/intel/2021.4.0/include"

---

Create an ASCII file in /share/CROCO/v110/intel with the specificities of the compilation with intel in hydra, name CROCO_hydra_intel.env (following the instructios from here tutorial 00)

#set environment variables for croco in HYDRA to be compiled with intel
source /opt/load-libs.sh 1
export CC=icc
export FC=ifort 
export F90=ifort 
export F77=ifort 
export NETCDF=/opt/netcdf/netcdf-4/intel/2021.4.0 
export PATH=$NETCDF/bin::${PATH} 
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}::${NETCDF}/lib

Loading the compilation environment

$ source /share/CROCO/v110/intel/CROCO_hydra_intel.env

Run the compilation (jobcomp file)

$ ./jobcomp >& jobcomp.log

If compilation is successful, you should have a croco executable in your directory.

You will also find a Compile directory containing the model source files.

Run the model

Copy de configuration file croco.in inside your simulation directory

$scp -p [croco.in] $CROCOsrc/CONFIGS/GSM_CLIMATOLOGY/

create a CROCO_FILES folder inside your simulation directory and copy the grid, forcing and ini files to it (croco_grd.nc,croco_clm.nc,croco_frc.nc and croco_ini.nc)

$ mkdir -p CROCO_FILES
$scp -p [grid,ini and forcing files] $CROCOsrc/CONFIGS/GSM_CLIMATOLOGY/CROCO_FILES

Copy a job script in my "salidas" folder and edit it:

cd $HOME/salidas/croco/mpi
cp /share/tools/workflows/direct/CROCO/hydra/launch_ideal_test_mpi.pbs ./launch_gsm_clim_mpi.pbs

# Amount of processors
Ntotprocs=60

# Name of the file with the input of the ideal case
idealn=croco.in 
CROCOsrc=/share/CROCO/v110/intel/dmpar/croco

copy the executable "croco" file and the config file "croco.in" to "salidas/croco/mpi":

$ cp /share/CROCO/v110/intel/dmpar/croco/CONFIGS/GSM_CLIMATOLOGY/croco ./
$ cp /share/CROCO/v110/intel/dmpar/croco/CONFIGS/GSM_CLIMATOLOGY/croco.in ./

And execute

$ qsub launch_gsm_clim_mpi.pbs