CROCO
(→v1.1 - Nicolás Aubone modified) |
(→v1.1 - Nicolás Aubone modified) |
||
| Línea 452: | Línea 452: | ||
------ |
------ |
||
Edit param.h file to set the mpi parallelization settings, in this case we use 60 nodes: |
Edit param.h file to set the mpi parallelization settings, in this case we use 60 nodes: |
||
| − | + | <PRE style="shell"> |
|
#ifdef MPI |
#ifdef MPI |
||
| + | |||
integer NP_XI, NP_ETA, NNODES |
integer NP_XI, NP_ETA, NNODES |
||
| − | parameter (NP_XI=10, NP_ETA=6, NNODES=NP_XI*NP_ETA) |
||
| − | parameter (NPP=60) |
||
| − | parameter (NSUB_X=1, NSUB_E=1) |
||
| + | parameter (NP_XI=10, NP_ETA=6, NNODES=NP_XI*NP_ETA) |
||
| + | </PRE> |
||
------ |
------ |
||
Revisión de 12:18 16 jul 2025
Web page of the instalation of the ocean model CROCO in CIMA's HPC 'hydra'
Contenido |
Installation
CIMA's HPC has a common folder where all the people has access. This is going to be the place where CROCO model will be installed.
The installation is going to be in the following INSTALLDIR=/share/CROCO folder.
Downloading the code (there is a GItlab repository, but by now, we are going to install closed releases)
$ wget https://gitlab.inria.fr/croco-ocean/croco/-/archive/v2.1.0/croco-v2.1.0.tar.gz
Creating the folder of the compilation (using intel compiler and shared memory dmpar) and where to deploy the code
$ mkdir $INSTALLDIR/v210 $ cd $INSTALLDIR/v210 $ mkdir -p intel/dmpar $ git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco.git croco-v2.1.0 Cloning into 'croco'... remote: Enumerating objects: 52646, done. remote: Counting objects: 100% (448/448), done. remote: Compressing objects: 100% (263/263), done. remote: Total 52646 (delta 268), reused 317 (delta 185), pack-reused 52198 (from 1) Receiving objects: 100% (52646/52646), 300.67 MiB | 4.59 MiB/s, done. Resolving deltas: 100% (40426/40426), done. Note: switching to '291e8cd11b2329ee171c7a02c52c9c481dda981e'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false $ ls croco GRIF create_config.bash OBSTRUCTION README.md TEST_CASES BENCH MPI_NOLAND OCEAN requirements.txt XIOS CHANGELOG.md MUSTANG PISCES SCRIPTS
Go to compile, first we set-up the environment for intel compilation
$ source /opt/load-libs.sh 1 The following libraries, compiled with Intel 2021.4.0 compilers, were loaded: * MPICH 3.4.2 * NetCDF 4 * HDF5 1.10.5 * JASPER 2.0.33 To change it please logout and login again. To load this libraries from within a script add this line to script: source /opt/load-libs.sh 1 $ cd croco
We create an ASCII file in /share/CROCO/v210/intel with the specificities of the compilation with intel in hydra, name CROCO_hydra_intel.env (following the instructios from here tutorial 00)
#set environment variables for croco in HYDRA to be compiled with intel
source /opt/load-libs.sh 1
export CC=icc
export FC=ifort
export F90=ifort
export F77=ifort
export NETCDF=/opt/netcdf/netcdf-4/intel/2021.4.0
export PATH=$NETCDF/bin::${PATH}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}::${NETCDF}/lib
Loading the compilation environment
$ source CROCO_hydra_intel.env
RUNNING CROCO TEST CASES
BASIN
The simplest installation would be the 'basin' configuration. We follow these instructions basin. Assuming that the source code resides in CROCOsrc=/share/CROCO/v210/intel/serie/croco
$ CROCOsrc=/share/CROCO/v210/intel/serie/croco
$ cd ${CROCOsrc}
$ mkdir -p CONFIGS/BASIN
cp OCEAN/cppdefs.h CONFIGS/BASIN
cp OCEAN/param.h CONFIGS/BASIN
cp OCEAN/jobcomp CONFIGS/BASIN
cd CONFIGS/BASIN
We need to edit the configuration files.
Editing the file with the configuration
$ cp cppdefs.h cppdefs.h_orig $ diff cppdefs.h cppdefs.h_orig 18c18 < #define BASIN /* Basin Example */ --- > #undef BASIN /* Basin Example */ 56c56 < #undef REGIONAL /* REGIONAL Applications */ --- > #define REGIONAL /* REGIONAL Applications */
Editing the file with the size of the domain and configuration of the parallel environment
$ cp param.h param.h_orig $ diff param.h param.h_orig
Editing the file with the compilation
$ cp jobcomp jobcomp_orig
$ diff jobcomp jobcomp_orig
34,35c34
< CROCOsrc=/share/CROCO/v210/intel/serie/croco
< SOURCE1=${CROCOsrc}/OCEAN
---
> SOURCE1=../croco/OCEAN
45c45
< FC=ifort
---
> FC=gfortran
And compile
$ ./jobcomp > jobcomp.log
If everything when fine, one should have:
$ tail -n 15 jobcomp.log
mv a.out croco
.,:looddddddd: ,;'
.cdddo;;oo'.... .ldd;
,ddddd, ;odoc.
.oddddo: .'';odoc
'dddddd;':, .;; .;;. :ccdo;
.oddddddddooooddoooddooodl.
.oddddddddddddddddddddddl;;
.,ldddddddddddddddddddoc:lolo'..
;ddc;::::::::::codo, .';:,
.ccc ,cc,
CROCO is OK
$ ls -l croco
-rwxrwxr-x 1 lluis.fita cima 1012160 Jul 15 15:11 croco
| IMPORTANT NOTES |
|---|
| 1. do not run the model in the same place where you compiled the source |
| 2. Always use a queue-system job to run simulations, never run without it |
3. Always run in your salidas/ folder never in $HOME
|
UPWELLING
Compilation of the upwelling test case in distributed memory (dmpar) with intel.
$ $ source /opt/load-libs.sh 1 The following libraries, compiled with Intel 2021.4.0 compilers, were loaded: * MPICH 3.4.2 * NetCDF 4 * HDF5 1.10.5 * JASPER 2.0.33 $ mkdir /share/CROCO/v210/intel/dmpar/ $ git clone --branch v2.1.0 https://gitlab.inria.fr/croco-ocean/croco.git croco $ cd croco $ CROCOsrc=/share/CROCO/v210/intel/dmpar/croco
Copying configuration files
$ mkdir -p CONFIGS/Upwelling cp OCEAN/cppdefs.h CONFIGS/Upwelling cp OCEAN/param.h CONFIGS/Upwelling cp OCEAN/jobcomp CONFIGS/Upwelling cd CONFIGS/Upwelling
Configuring the case
$ cp cppdefs.h cppdefs.h_orig
$ diff cppdefs.h cppdefs.h_orig
29c29
< #define UPWELLING /* Upwelling Example */
---
> #undef UPWELLING /* Upwelling Example */
56c56
< #undef REGIONAL /* REGIONAL Applications */
---
> #define REGIONAL /* REGIONAL Applications */
1037c1037
< # define MPI
---
> # undef MPI
$ cp param.h param.h_orig
$ diff param.h param.h_orig
$ cp jobcomp jobcomp_orig
$ diff jobcomp jobcomp_orig
34,35c34
< CROCOsrc=/share/CROCO/v210/intel/dmpar/croco
< SOURCE1=${CROCOsrc}/OCEAN
---
> SOURCE1=../croco/OCEAN
46c45
< FC=ifort
---
> FC=gfortran
52,53c51,52
< MPILIB="-L/opt/mpich/mpich-3.4.2/intel/2021.4.0/lib"
< MPIINC="-I/opt/mpich/mpich-3.4.2/intel/2021.4.0/include"
---
> MPILIB=""
> MPIINC=""
Compiling in parallel
$ ./jobcomp >& jobcomp.log
$ tail -n 15 jobcomp.log
mv a.out croco
.,:looddddddd: ,;'
.cdddo;;oo'.... .ldd;
,ddddd, ;odoc.
.oddddo: .'';odoc
'dddddd;':, .;; .;;. :ccdo;
.oddddddddooooddoooddooodl.
.oddddddddddddddddddddddl;;
.,ldddddddddddddddddddoc:lolo'..
;ddc;::::::::::codo, .';:,
.ccc ,cc,
CROCO is OK
$ ls -l croco
-rwxrwxr-x 1 lluis.fita cima 1184600 Jul 15 16:34 croco
| IMPORTANT NOTES |
|---|
| 1. do not run the model in the same place where you compiled the source |
| 2. Always use a queue-system job to run simulations, never run without it |
3. Always run in your salidas/ folder never in $HOME
|
Example of use
serie: BASIN
How to run a basic example
$ cd $HOME/salidas $ mkdir CROCO $ cd CROCO $ mkdir tests $ cd tests/ $ cp /share/tools/workflows/direct/CROCO/hydra/launch_ideal_test_serie.pbs ./
Link the necessary files and editng them if needed:
$ ln -s $CROCOsrc/CONFIGS/BASIN/croco ./ $ cp $CROCOsrc/TEST_CASES/croco.in.Basin ./
And execute
$ qsub launch_ideal_test_serie.pbs 70619.hydra $ qstat Job ID Name User Time Use S Queue ------------------------- ---------------- --------------- -------- - ----- (...) 70619.hydra CROCO4L_ideal lluis.fita 0 R larga
Looking for the current step:
$ tail croco.out
2698 299.77778 8.942712777E-05-3.3148734E+00-3.3147840E+00 5.0400000E+16 0
2699 299.88889 8.942810068E-05-3.3148728E+00-3.3147833E+00 5.0400000E+16 0
2700 300.00000 8.942902900E-05-3.3148721E+00-3.3147827E+00 5.0400000E+16 0
WRT_HIS -- wrote history fields into time record = 31 / 31
2701 300.11111 8.942991352E-05-3.3148715E+00-3.3147820E+00 5.0400000E+16 0
2702 300.22222 8.943075502E-05-3.3148708E+00-3.3147814E+00 5.0400000E+16 0
2703 300.33333 8.943155431E-05-3.3148701E+00-3.3147807E+00 5.0400000E+16 0
2704 300.44444 8.943231222E-05-3.3148695E+00-3.3147800E+00 5.0400000E+16 0
2705 300.55556 8.943302959E-05-3.3148688E+00-3.3147794E+00 5.0400000E+16 0
2706 300.66667 8.943370728E-05-3.3148682E+00-3.3147787E+00 5.0400000E+16 0
Once finished:
$ qstat
Job ID Name User Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
(...)
70619.hydra CROCO4L_ideal lluis.fita 00:01:13 C larga
$ tail croco.out
3239 359.88889 9.041893005E-05-3.3144938E+00-3.3144033E+00 5.0400000E+16 0
3240 360.00000 9.041879739E-05-3.3144930E+00-3.3144026E+00 5.0400000E+16 0
WRT_HIS -- wrote history fields into time record = 37 / 37
MAIN - number of records written into history file(s): 37
number of records written into restart file(s): 0
MAIN: DONE
$ ls -lrt
total 15576
-rw-rw-r-- 1 lluis.fita cima 1412 Jul 15 15:40 launch_ideal_test_serie.pbs
lrwxrwxrwx 1 lluis.fita cima 55 Jul 15 15:44 croco -> /share/CROCO/v210/intel/serie/croco/CONFIGS/BASIN/croco
-rw-rw-r-- 1 lluis.fita cima 3361 Jul 15 15:46 croco.in.Basin
-rw------- 1 lluis.fita cima 3361 Jul 15 15:47 croco.in
-rw------- 1 lluis.fita cima 824 Jul 15 15:48 CROCO4L_ideal.o70619
-rw------- 1 lluis.fita cima 15653792 Jul 15 15:48 basin_his.nc
-rw------- 1 lluis.fita cima 274775 Jul 15 15:48 croco.out
dmpar: Upwelling
How to run a basic example
$ cd $HOME/salidas $ mkdir CROCO $ cd CROCO $ mkdir mpi $ cd mpi/ $ cp /share/tools/workflows/direct/CROCO/hydra/launch_ideal_test_mpi.pbs ./
Link the necessary files and editng them if needed:
$ ln -s $CROCOsrc/CONFIGS/Upwelling/croco ./ $ cp $CROCOsrc/TEST_CASES/croco.in.Upwelling ./
And execute
$ qsub launch_ideal_test_mpi.pbs 70649.hydra $ qstat Job ID Name User Time Use S Queue ------------------------- ---------------- --------------- -------- - ----- (...) 70645.hydra CROCO4L_ideal_ lluis.fita 0 R larga
Once finished
$ tail -3 croco.out MAIN: DONE $ ls -lrt total 6956 lrwxrwxrwx 1 lluis.fita cima 59 Jul 15 16:42 croco -> /share/CROCO/v210/intel/dmpar/croco/CONFIGS/Upwelling/croco -rw-rw-r-- 1 lluis.fita cima 1500 Jul 15 16:42 croco.in.Upwelling -rw------- 1 lluis.fita cima 1500 Jul 15 16:43 croco.in -rwxrwxr-x 1 lluis.fita cima 9328 Jul 15 16:43 a.out -rw-rw-r-- 1 lluis.fita cima 1448 Jul 15 16:50 launch_ideal_test_mpi.pbs -rw------- 1 lluis.fita cima 238630 Jul 15 16:50 CROCO4L_ideal_mpi.o70634 -rw------- 1 lluis.fita cima 717220 Jul 15 16:50 upwelling_rst.nc -rw------- 1 lluis.fita cima 3316688 Jul 15 16:50 upwelling_his.nc -rw------- 1 lluis.fita cima 2771248 Jul 15 16:50 upwelling_avg.nc -rw------- 1 lluis.fita cima 42450 Jul 15 16:50 croco.out
RUNNING A REAL SIMULATION
v1.1 - Nicolás Aubone modified
Here we run a real simulation with croco version 1.1
It's a simulation of one climatologic year in the San Matías Gulf, Argentina.
Be carefull, this model was modified from its original code (we add a bi-dimensional logarithmic bottom friction).
Creating the folder where to deploy the code (using intel compiler and shared memory dmpar):
$ INSTALLDIR=/share/CROCO $ mkdir $INSTALLDIR/v11 $ cd $INSTALLDIR/v11 $ mkdir -p intel/dmpar $ cd intel/dmpar $ tar xvfz [CROCOv1.1_NicolasAubone].tar.gz $ ls croco AGRIF OCEAN run_simulation_openmp.bash create_run.bash PISCES SCRIPTS CVTK readme_version_croco.txt TEST_CASES DOC_SPHINX run_multi_simulation.bash XIOS MPP_PREP run_simulation.bash
Create the "CONFIGS" folder to store our different simulations there.
Create the "GSM_CLIMATOLOGY" folder to store our simulation config files and the model compilation.
Copy to the simulation forlder "GSM_CLIMATOLOGY" our configuration files to compile and run the model:
$ CROCOsrc=$INSTALLDIR/v110/intel/dmpar/croco
$ cd ${CROCOsrc}
$ mkdir -p CONFIGS/GSM_CLIMATOLOGY
$ cd CONFIGS/GSM_CLIMATOLOGY
cp [file_location]/cppdefs.h CONFIGS/BASIN
cp [file_location]/param.h CONFIGS/BASIN
cp [file_location]/jobcomp CONFIGS/BASIN
Edit cppdefs.h file to define a mpi run:
define MPI
Edit param.h file to set the mpi parallelization settings, in this case we use 60 nodes:
#ifdef MPI integer NP_XI, NP_ETA, NNODES parameter (NP_XI=10, NP_ETA=6, NNODES=NP_XI*NP_ETA)
Edit the compilation file "jobcomp" with hydra parameters:
ROOT_DIR=/share/CROCO/v110/intel/dmpar/croco
SOURCE=${ROOT_DIR}'/OCEAN'
FC=ifort
MPIF90="mpif90"
MPILIB="-L/opt/mpich/mpich-3.4.2/intel/2021.4.0/lib"
MPIINC="-I/opt/mpich/mpich-3.4.2/intel/2021.4.0/include"
Create an ASCII file in /share/CROCO/v110/intel with the specificities of the compilation with intel in hydra, name CROCO_hydra_intel.env (following the instructios from here tutorial 00)
#set environment variables for croco in HYDRA to be compiled with intel
source /opt/load-libs.sh 1
export CC=icc
export FC=ifort
export F90=ifort
export F77=ifort
export NETCDF=/opt/netcdf/netcdf-4/intel/2021.4.0
export PATH=$NETCDF/bin::${PATH}
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}::${NETCDF}/lib
Loading the compilation environment
$ source /share/CROCO/v110/intel/CROCO_hydra_intel.env
Run the compilation (jobcomp file)
$ ./jobcomp >& jobcomp.log
