Revisión de 15:46 25 jun 2025

Here we describe the installation and use of the ECCO model in CIMA-IFAECI's hydra HPC

Installation

We are going to follow the instructions from the 2025 ECCO summer school - EMU Installation, specifically to install the tools by our selves following these instructions.

First we look into the right GIThub repository, being

https://github.com/ECCO-GROUP/ECCO-EIS/tree/main/emu

We are going to install the model for all hydra users. So, our $INSTALLDIR will be:

INSTALLDIR=/share/EMU

Obtaining the code and installing $INSTALLDIR (we are going to use [NASA https://ecco.jpl.nasa.gov/drive/ Earthdata] user: lluisfita (using its WebDAV password). **NOTE:** MIT certificate is not well set-up, we needed to modify the script to keep going

cd $INSTALLDIR
wget https://raw.githubusercontent.com/ECCO-GROUP/ECCO-EIS/main/emu/emu_setup.sh
chmod +x ./emu_setup.sh
./emu_setup.sh

------------------------------------------------------------------------------
 This script sets up EMU, a collection of computational tools for analyzing
 the ECCO model (flux-forced version of ECCO Version 4 Release 4). The Tools 
 include the following;

   1) Sampling (samp); Evaluates state time-series from model output.
   2) Forward Gradient (fgrd); Computes model's forward gradient.
   3) Adjoint (adj); Computes model's adjoint gradient.
   4) Convolution (conv); Evaluates adjoint gradient decomposition.
   5) Tracer (trc); Computes passive tracer evolution.
   6) Budget (budg); Evaluates budget time-series from model output.
   7) Modified Simulation (msim); Re-runs model with modified input.
   8) Attribution (atrb); Evaluates state time-series by control type.
   9) Auxiliary (aux): Generates user input files for other EMU tools.

************************
 This script will install EMU's Programs (~1GB), its User Interface (~2MB), 
 and download its Input Files (~1TB) to user-specified directories. 

 Users should not move or alter these directories or their files unless 
 noted otherwise (e.g., conforming batch scripts pbs_*.sh for the host 
 system, installed in the User Interface directory). Once installed, 
 any user of the host system should be able to utilize the installed files 
 and programs; Separate installations for different users are not necessary. 

 Installation requires obtaining a NASA Earthdata account for downloading 
 files from https://ecco.jpl.nasa.gov/drive/. Enter your Earthdata 
 username and WebDAV password (not your Earthdata password) at the prompts 
 below. The WebDAV password can be found at this URL after logging in with 
 your Earthdata username and Earthdata password, or click the 'Back to 
 WebDAV Credentials' button when browsing files at the URL.

 See the README file that will be installed in the User Interface directory 
 for details of EMU, including instructions on how to use it.
************************

Press ENTER key to continue ... 


----------------------
Enter your Earthdata username: lluisfita
Enter your WebDAV password (*NOT* Earthdata password): LX7SPsj9N8U8puZA5whS


----------------------
Enter directory name (emu_dir) to download and set up EMU's Programs (~1 GB) 
or press the ENTER key to use EMU's default (emu_dir under the present directory) ... ?


EMU's Programs will be installed in 
/share/EMU/emu_dir

----------------------
Enter directory name (emu_userinterface_dir) to install EMU's User Interface
(~2 MB) or press the ENTER key to use EMU's default (emu_userinterface_dir 
under the present directory) ... ?


EMU's User Interface will be installed in 
/share/EMU/emu_userinterface_dir

----------------------
Enter directory name (emu_input_dir) to download up to 1.1 TB of EMU's Input 
Files or press the ENTER key to use EMU's default (emu_input_dir under the 
present directory) .... ? 


EMU's Input Files will be downloaded to 
/share/EMU/emu_input_dir

************************
NOTE: See *.log files in /share/EMU/emu_dir/temp_setup should this script fail.
************************

----------------------
EMU's Programs can be installed in two different ways;
   1) Compiling source code on host (native) 
   2) Using Singularity image (singularity)

Option 1) requires a TAF license to derive the MITgcm adjoint used by EMU's 
Adjoint Tool. Option 2) has compiled versions of the code in 
containerized form that do not require a separate TAF license to use.

Enter choice for type of EMU implementation ... (1 or 2)?
1

Implementation type choice is 1

----------------------
EMU uses batch scripts to run some of its tools in PBS (Portable 
Batch System). The PBS commands in these shell scripts (pbs_*.sh),
installed in EMU's User Interface directory (emu_userinterface_dir)
/share/EMU/emu_userinterface_dir
may need to be revised for different batch systems and/or different hosts. 
Alternatively, these shell scripts can be run interactively if sufficient 
resources are available.

Enter the command for submitting batch jobs (e.g., qsub, sbatch, 
bsub <, condor_submit, msub) or press the ENTER key to have EMU 
run its batch scripts interactively ... ?
qsub

Command to submit EMU's batch job scripts will be: qsub

----------------------
EMU's Input Files total 1.1 TB, of which (directory)
   175 GB (emu_ref) is needed by Sampling, Forward Gradient, Adjoint, Tracer, Budget, and Attribution
   195 GB (forcing) is needed by Forward Gradient, Adjoint, Modified Simultion
   380 GB (state_weekly) is needed by Tracer
   290 GB (emu_msim) is needed by Attribution
   (Convolution Tool uses results of the Adjoint Tool and files downloaded by default.)

Choose among the following to download ... 
   0) All Input Files (1.1 TB) 
   1) Files (~175 GB) needed for Sampling and Budget Tools
   2) Files (~195 GB) needed for Modified Simultion Tools
   3) Files (~370 GB) needed for Adjoint and Forward Gradient Tool
   4) Files (~465 GB) needed for Attribution Tool
   5) Files (~555 GB) needed for Tracer Tool
or press the ENTER key to skip this step, which can take a while
(~13 hours if downloading all input files.) 

EMU's Input Files can be downloaded later with shell script
   /share/EMU/emu_userinterface_dir/emu_input_setup.sh 
See 
   /share/EMU/emu_userinterface_dir/README_input_setup 
for additional detail, including options to download the input
in batch mode.

Enter Input Files download choice ... ?
0


----------------------
Choose number of CPU cores (nproc) for running MITgcm.
Choose among the following nproc ... 

13
36
48
68
72
96
192
360

Enter choice for nproc ... ?
48


Number of CPU cores to be used for MITgcm: 48

**********************
 End of user input for EMU setup 
 Rest of this script is conducted without user input.

----------------------
Download and compiling EMU on host system in directory 
/share/EMU/emu_dir


----------------------
Download and compiling MITgcm and its adjoint in 
/share/EMU/emu_dir/emu/exe/nproc
This can take a while (~30 minutes). 
Progress can be monitored in file  /share/EMU/emu_dir/temp_setup/emu_compile_mdl.log
  tail /share/EMU/emu_dir/temp_setup/emu_compile_mdl.log 

Now we will work in solving the problems found in the log files

/share/EMU/emu_dir/temp_setup/emu_compile_mdl.log

emu_dir/emu/native/emu_compile_mdl.sh: line 106: /usr/local/lib/global.profile: No such file or directory

Inside emu_dir/emu/native/emu_compile_mdl.sh we found the following lines, where the compilation environment is set-up

# Get the directory containing the script (full path to emu/native)
nativedir=$(dirname "$script_path")
(...)
# 7) Load module for compilation. 
source /usr/local/lib/global.profile
source ${nativedir}/set_modules.sh

In CIMA's hydra there is not /usr/local/lib/global.profile. File ${nativedir}/set_modules.sh assumes the existence of modules. hydra does not have modules, it uses on-porpouse scripts to set-up compilation environment... Wea are going to adapt </CODE>emu_compile_mdl.sh</CODE> to hydra characterstics

$ cp emu_dir/emu/native/emu_compile_mdl.sh emu_dir/emu/native/emu_compile_mdl_orig.sh
$ vim emu_dir/emu/native/emu_compile_mdl.sh
$ diff emu_dir/emu/native/emu_compile_mdl.sh emu_dir/emu/native/emu_compile_mdl_orig.sh
106,108c106,107
< #source /usr/local/lib/global.profile
< #source ${nativedir}/set_modules.sh
< source /opt/load-libs.sh 1
---
> source /usr/local/lib/global.profile
> source ${nativedir}/set_modules.sh

/share/EMU/emu_dir/temp_setup/download_emu_source.log

After solving this problem, setup shell abruptly stops without any message. However we found in emu_dir/temp_setup/download_emu_source.log

$ cat ./emu_dir/temp_setup/download_emu_source.log
Cloning into 'ECCO-EIS'...
mv: cannot move 'ECCO-EIS/emu' to './emu': Directory not empty

In the previous attempt we already cloned the code, therefore we are going to make sure, that we remove the folder before the clone is made, again modifying scripts, this time emu_setup.sh:

$ cp emu_setup.sh emu_setup_orig.sh
$ vim emu_setup.sh 
$ diff emu_setup.sh emu_setup_orig.sh
539,542d538
<     if test -d ECCO-EIS; then
<       echo "  Code has already previously been cloned... removing the pre-existing one" 
<       rm -rf ECCO-EIS
<       rm -rf emu
<     fi

Use