Wikicima - Contribuciones del usuario [es]

PALM

2026-10-01T19:32:45Z

Guido.tiscornia: /* Hydra, for this purpose */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5. That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>

== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Tools and links ==

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/

PALM

2026-10-01T19:01:49Z

Guido.tiscornia: /* Tools and links */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5. That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>

== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Tools and links ==

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/

PALM

2026-10-01T19:01:26Z

Guido.tiscornia: /* Pending */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5. That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>

== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T19:00:14Z

Guido.tiscornia: /* Performance */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5. That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>

== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:59:05Z

Guido.tiscornia: /* Setting up a case */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5. That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>

== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:56:38Z

Guido.tiscornia: /* Build */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:56:00Z

Guido.tiscornia: /* Environment script */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:55:06Z

Guido.tiscornia: /* Installation */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (at the moment <code>git clone</code> asks for credentials inside of PALM website, but the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

A practical tip: do not create the file by pasting a long <code>cat > file << EOF</code> block into the SSH terminal. Lines can get mixed up and the file ends up broken without it being obvious. Use <code>nano</code>/<code>vim</code>, or upload it with <code>scp</code>, and check it with <code>cat -n</code>.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:54:13Z

Guido.tiscornia: /* Environment script */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (<code>git clone</code> asks for credentials, the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

A practical tip: do not create the file by pasting a long <code>cat > file << EOF</code> block into the SSH terminal. Lines can get mixed up and the file ends up broken without it being obvious. Use <code>nano</code>/<code>vim</code>, or upload it with <code>scp</code>, and check it with <code>cat -n</code>.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:53:01Z

Guido.tiscornia: /* Hydra, for this purpose */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

== Installation ==

Download (<code>git clone</code> asks for credentials, the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

A practical tip: do not create the file by pasting a long <code>cat > file << EOF</code> block into the SSH terminal. Lines can get mixed up and the file ends up broken without it being obvious. Use <code>nano</code>/<code>vim</code>, or upload it with <code>scp</code>, and check it with <code>cat -n</code>.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:52:25Z

Guido.tiscornia: /* Hydra, for this purpose */

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

<code>~/salidas</code> is an NFS mount and is the same location as <code>/nfsmounts/storage/scratch/<user></code>. PALM was installed there, in <code>~/salidas/palm</code>.

== Installation ==

Download (<code>git clone</code> asks for credentials, the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

A practical tip: do not create the file by pasting a long <code>cat > file << EOF</code> block into the SSH terminal. Lines can get mixed up and the file ends up broken without it being obvious. Use <code>nano</code>/<code>vim</code>, or upload it with <code>scp</code>, and check it with <code>cat -n</code>.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

PALM

2026-10-01T18:50:07Z

Guido.tiscornia:

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue <code>larga</code>), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <code><user></code>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

== Starting point ==

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with <code>example_cbl</code> (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with <code>ifort</code> and 2026.0 with <code>ifx</code>), and the MPICH <code>mpif90</code> calls an <code>ifort</code> that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with <code>undefined symbol: __netcdf_MOD_nf90_put_var...</code>. After two attempts we switched to a pure gfortran build.

== Hydra, for this purpose ==

There is a single queue, <code>larga</code>. The nodes (as of July 2026; check with <code>pbsnodes</code>):

* 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 are free.
* 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
* 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses, do not use.

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

<code>~/salidas</code> is an NFS mount and is the same location as <code>/nfsmounts/storage/scratch/<user></code>. PALM was installed there, in <code>~/salidas/palm</code>.

== Installation ==

Download (<code>git clone</code> asks for credentials, the tarball does not):

<pre>
mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system
</pre>
=== Environment script ===

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this as <code>~/cargar_entorno_palm.sh</code>:

<pre>
#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
| grep -v "netcdf/netcdf-4/intel" \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
| grep -v "mpich" \
| tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"
</pre>
It has to be sourced in every new terminal. After loading it, the second line must say <code>gfortran</code>; if it does not, do not continue.

A practical tip: do not create the file by pasting a long <code>cat > file << EOF</code> block into the SSH terminal. Lines can get mixed up and the file ends up broken without it being obvious. Use <code>nano</code>/<code>vim</code>, or upload it with <code>scp</code>, and check it with <code>cat -n</code>.

=== Build ===

With the environment loaded:

<pre>
cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x
</pre>
<code>-p</code> is the install prefix, <code>-s</code> and <code>-t</code> force the system NetCDF (otherwise autodetection picks Intel’s), and <code>-x</code> cleans the previous build. When it finishes, the executables are in <code>~/salidas/palm/bin</code> and the default configuration is <code>~/salidas/palm/.palm.config.default</code>. RRTMG is linked in. Parallel NetCDF is not available on Hydra, but this does not matter much: the computation is parallel and only the file writing is serial.

=== Interactive test ===

<pre>
cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z
</pre>
It should end with <code>--> palmrun finished</code>. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

== Batch mode ==

PALM uses one configuration file per suffix, selected with <code>-c</code>. We work on a copy so the default one stays untouched:

<pre>
cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue larga|' $CFG
sed -i 's|^#%memory.*|%memory 2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG
</pre>
The block of PBS directives in the template already works on Hydra, and the <code>BDT:</code> block (queue <code>dataq</code>) is only used to transfer files from remote hosts, so it is left alone. <code>%memory</code> is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in <code>LD_LIBRARY_PATH</code> the job fails with <code>libmpi.so.12 => not found</code>. That is why the script includes the runtime line and why it is loaded through <code>%login_init_cmd</code>. To check this without wasting a real run, a small job that runs <code>ldd</code> on the executable is enough:

<pre>
#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
echo "PROBLEM:"; ldd $EXE | grep "not found"
else
echo "OK: all libraries resolve"
fi
</pre>
Submit it with <code>qsub</code> and read the log. The executable in <code>MAKE_DEPOSITORY_hydra</code> only exists after the first run with <code>-c hydra</code>.

=== Running ===

<pre>
source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z
</pre>
<code>-r</code> is the case, <code>-X</code> the total number of processes, <code>-T</code> the processes per node (nodes = X/T), <code>-t</code> the walltime in seconds, <code>-b</code> submits to the queue, and <code>-F</code> generates the job script without submitting it, which is handy for inspecting it. <code>-a "d3#"</code> is mandatory: without it <code>palmrun</code> aborts with <code>no activation string list given</code>. For two 128-core nodes use <code>-X 256 -T 128</code>. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example <code>-X 128 -T 64</code>.

Each new configuration recompiles once into its own <code>MAKE_DEPOSITORY</code>. The number PALM gives to its own runs (for example <code>case.650</code>) is not the PBS job ID.

== Setting up a case ==

Each case lives in <code>~/salidas/palm/JOBS/<case>/INPUT/</code>, with the namelist <code><case>_p3d</code> and, for urban cases, the static driver <code><case>_static</code> (without <code>.nc</code>). <code>palmrun</code> is always called from <code>~/salidas/palm</code>. Rules that caused problems:

* Only <code>_p3d</code> and <code>_static</code> files go in <code>INPUT/</code>. A backup there (for example <code>case_p3d.bak</code>) makes the run abort with <code>wrong ending</code>.
* <code>nx+1</code> and <code>ny+1</code> must factorize into 2, 3 and 5 (Temperton FFT). That is why 384 was used and not 400.
* PALM counts the maximum index (<code>nx = 383</code> means 384 points), while GEO4PALM counts points. If they do not match, the error is <code>static driver: horizontal dimension ... does not match model dimension</code>.
* Always set <code>data_output_2d_on_each_pe = .FALSE.</code> in <code>&runtime_parameters</code>. With the default value, <code>combine_plot_fields</code> breaks on NFS and outputs are lost.
* <code>_xy</code> output needs <code>section_xy</code>, for example <code>section_xy = 0</code>.
* The static driver must declare all five surface type variables, including <code>water_type</code>, even if one is entirely fill values (otherwise error <code>DRV0011</code>).

Minimal namelist used for the benchmark (derived from <code>example_cbl</code>):

<pre>
&initialization_parameters
nx = 383, ny = 383, nz = 95,
dx = 10.0, dy = 10.0, dz = 10.0,
dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
initializing_actions = 'set_constant_profiles',
ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
pt_vertical_gradient = 0.0, 1.0,
pt_vertical_gradient_level = 0.0, 800.0,
surface_heatflux = 0.1, bc_pt_b = 'neumann',
fft_method = 'temperton-algorithm',
/
&runtime_parameters
end_time = 120.0,
create_disturbances = .TRUE., dt_disturb = 150.0,
disturbance_energy_limit = 0.01,
data_output_2d_on_each_pe = .FALSE.,
netcdf_data_format = 2, restart_data_format = 'mpi',
dt_run_control = 0.0,
/
</pre>
== Monitoring and results ==

<pre>
qstat -u <user> # Q queued, R running, C completed
pbsnodes -l free # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc
</pre>
The <code>env LD_LIBRARY_PATH=</code> prefix is needed on Hydra because the system <code>ncdump</code> clashes with the NetCDF library that PALM’s environment puts first (<code>undefined symbol: nc_reclaim_data</code>). Output files get a suffix <code>.000</code>, <code>.001</code>, … per successful run. The <code>_cpu.NNN</code> file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (<code>ls</code>, <code>ncdump -h</code>) before considering a run good.

== Performance ==

384×384×96 domain at 10 m, one simulated hour:

* Bare solver (no city), 48 processes: 0.0807 µs per grid point and time step.
* With urban physics (urban surface, land surface, 3D radiation), 48 processes: 0.0961 µs, which is 19% more. About 3,170 s of wall time.
* With urban physics, 256 processes on 2 nodes: 0.2727 µs, about 8,870 s. This is slower than with 48.

The cause is the pressure solver, which uses all-to-all communication and collapses when it crosses nodes. For domains of this size it is not worth adding cores without measuring first. The wall time of another domain can be estimated as unit cost × number of points × number of time steps, valid only for 1 node and 48 processes.

== Pending ==

* Test restart in batch (it failed in local interactive mode, because of the <code>palmrun</code> script and not PALM).
* Scaling sweep with 96 and 128 processes inside a single node, and the multigrid solver.
* Nested domains and a real dynamic driver (WRF4PALM) on Hydra.
* Tell the administrators about the double oneAPI installation; there is no record that it has been fixed.

== Tools and links ==

* GEO4PALM, to build the static driver from global datasets. Its code needs patches to work with current libraries: use Miniforge/conda-forge, numpy below 2, and <code>pip --extra-index-url</code> (not <code>-i</code>). https://github.com/dongqi-DQ/GEO4PALM (confirm the exact URL from the WRF4PALM README).
* WRF4PALM, to build the dynamic driver from WRF output: https://github.com/dongqi-DQ/WRF4PALM
* <code>palm_csd</code> (ships with PALM), <code>palmtest</code> and <code>palmplot</code>.
* <code>ncview</code> and Python (xarray, netCDF4, matplotlib with the <code>Agg</code> backend) for looking at outputs. Pre- and post-processing were done on local machines.

Documentation and downloads:

* https://docs.palm-model.org/
* https://gitlab.palm-model.org/releases/palm_model_system
* Maronga et al. 2020, PALM 6.0 overview: https://gmd.copernicus.org/articles/13/1335/2020/
* Lin et al. 2021, WRF4PALM: https://doi.org/10.5194/gmd-14-2503-2021
* Lin et al. 2024, GEO4PALM: https://doi.org/10.5194/gmd-17-815-2024

Página principal

2026-10-01T18:49:43Z

Guido.tiscornia: /* Modelos */

== SCAD del CIMA (Cluster) ==
=== Reglamento de uso del SCAD del CIMA ===
Versión descargable: [[ Media:SCAD-Reglamento_de_uso.pdf‎ | SCAD-Reglamento_de_uso‎ ]]

==== General ====

*El siguiente reglamento puede ser modificado sin previo aviso y es responsabilidad de los usuarios mantenerse informados leyendo el mismo regularmente.

*El SCAD del CIMA sólo se puede utilizar para realizar corridas de procesos en paralelo con fines de investigación científica

*El SCAD es un recurso compartido por todos, por lo que debe usarse de forma RESPONSABLE y no se debe abusar del mismo

==== De su uso ====

*Está absolutamente prohibido navegar en internet o descargar archivos a través de los clusters. Sólo están permitidas las conexiones hacia los clusters a través de protocolos seguros (ssh, scp, etc.).

*La administración de las corridas (jobs) en los clusters se realiza a través de un sistema de colas (pbs). Esta prohibido lanzar corridas fuera del sistema de colas

*La cantidad de trabajos que pueden estar corriendo, depende de la cantidad y el tipo de nodos que hayan sido solicitados para cada una de las corridas.

*Sólo se puede utilizar para corridas de código paralelizado. Bajo ningún concepto se debe usar el nodo principal para correr programas seriales (Ej: pre y post procesamiento de los datos).

==== Almacenamiento ====

*Todos los archivos de salida deben ser almacenados en el directorio en el directorio de salidas de cada usuario ($HOME/salidas/). Los archivos pueden permanecer almacenados en este directorio durante un plazo máximo de 30 días al cabo de los cuales serán automáticamente eliminados.

*Cada usuario contará con un “$HOME” de poco tamaño para almacenar archivos permanentes (Ej: Programas, códigos, etc)

==== Agradecimiento ====

*Toda publicación que haya utilizado recursos del SCAD del CIMA debe contener el siguiente agradecimiento:
“Simulations were made with the high-performance computing clusters available at CIMA/UBA-CONICET, Argentina”

=== Autoridades ===
;Comité Asesor Científico:
: Claudio Menendez
: Juan Ruiz
: Celeste Saulo
: Silvina Solman
: Pedro Flombaum (Director CIMA)

;Coordinador:
: Juan Ruiz

;Supervisor:
: Gabriel Vieytes

=== Monitoreo ===

Página de monitoreo y estadísticas del SCAD del CIMA
*''''' https://scad.cima.fcen.uba.ar/

=== Solicitud de cuenta para uso del SCAD del CIMA ===
Para poder hacer uso de los cluster del SCAD del CIMA, es necesario solicitar la apertura de una cuenta personal enviando un mail a [mailto:soporte@cima.fcen.uba.ar soporte@cima.fcen.uba.ar].

=== Guía de usuario del SCAD del CIMA ===
Acceso a la Guía de usuario haciendo click [[Guía de usuario del SCAD del CIMA | aquí]]

=== Composición del SCAD del CIMA ===
Detalle de software y hardware que conforman el SCAD del cima haciendo click [[Composición del SCAD del CIMA | aquí ]]

=== Links ===
Links de referencia haciendo click [[Links | aquí ]]

=== FAQ ===
Preguntas frecuentes haciendo click [[FAQ | aquí ]]

== Datos ==
Acceso a [[papa-deimos]], servidor de almacenamiento y procesamiento de datos del CIMA compartido con el DCAO.

De los datos disponibles en el clúster SCAD (hydra) para simular (forzantes para modelos cómo el WRF, ORCHIDEE, ...) aquí [[DatosSCADSimular]]

Del grupo ILLAPA: 'Mecanismos del clima regional y sus impactos' descritos aquí [[DatosRCMSueloAtmosfera]]

== Reanalysis ==
'''Copia de los archivos para uso interno del CIMA'''. Datos Disponibles via FTP y DODS(**) de:
NCEP Climate Forecast System Reanalysis (CFSR)
ds093.0: 6-hourly Products / Reanalysis 1980-2010(*) / https://rda.ucar.edu/datasets/ds093.0/index.html

Para descargar los archivos via '''FTP''': ftp://usuario:password@datos.cima.fcen.uba.ar

Para abrir los archivos en forma remota '''DODS''': http://gds.cima.fcen.uba.ar:9090/dods/

'''Descripción y ayudas:''' https://www.cima.fcen.uba.ar/CFSR.html

'''Resolución horizontal'''

Sudamerica: 0.5° x 0.5° - 15°N a 65°S y 30°W a 85°W ( 110 x 160 Longitud/Latitud)

Global: 2.5° x 2.5° - 90°N a 90°S y 0°E a 357.5°E ( 144 x 73 Longitud/Latitud)

'''Resolución vertical''' en las variables de Nivel: 32 niveles de presión (mbar)

1000 975 950 925 900 875 850 825 800 775 750 700 650 600 550 500 450 400 350 300 250 225 200 175 150 125 100 70 50 30 20 10

'''Período''' 31 Años : 1980/01/01 a 2010/12/31

'''Frecuencia del Pronóstico''': 0 6 12 18 h

'''25 VARIABLES DE SUPERFICIE'''

'''6 VARIABLES DE PRESION'''

== Modelos ==

*[[CHIMERE]]

*[[CESM]]

*[[CROCO]]

*[[ECCO]]

*[[EMU]]

*[[LMDZ]]

*[[MARS]]

*[[ORCHIDEE]]

*[[PALM]]

*[[RegIPSL]]

*[[RCA]]

*[[WRF]]



== Mapeo de datos de secuenciación ==

* [[Mapeo de lecturas contra bases de datos de referencia]]

== Documentación ==
*[[Redes avanzadas]]

*[[Libros]]

*[[Memoria Compartida (e.g.OpenMP)]]

*[[anaconda]]

*[[llaves ssh]]

*[[entornos python]]

*[[Configuración correo electrónico | Configuración correo electrónico (mail)]]

*[[Configuración filtros webmail para avisos de vacaciones]]

*[[procesos zombies]]

*[[parTUXza]]

*[[Seguimiento simulaciones]]

*[[Spyder]]

*[[Jupyter Notebook]]

*[[Rstudio]]

*[[Cambio de contraseña]]

*[[VPN]]

* [[Café programación]]

== Reparaciones ==
*Documentación para ingreso de personal de service [[ Media:SEGUROS_CONTRAT_2016.pdf‎ | Seguros‎ ]]

*[[Empresas para service de AA]]

== Becarios ==

== Programa pasajes OPTAR-FCEN ==

==== General ====

En la Sub seccion '''Viajes''' de la Sección Trámites de la web oficial de la FCEN-UBA → https://exactas.uba.ar/ van a encontrar la Normativa vigente, el Instructivo y la Planilla de Solicitud del '''Programa Reserva y Pago de Pasajes Aéreos - Convenio FCEN-OPTAR'''

En el Instructivo van a encontrar QUÉ se debe hacer para que el uso del programa sea ''exitoso'', lo que viene a continuación son detalles clave a tener en cuenta pero no es la ''guía definitiva''.

Repito, LEAN EL INSTRUCTIVO PARA NO OLVIDAR PASOS CLAVE. E.g.: Una vez realizado el viaje el área requirente deberá enviar los pases de embarque correspondientes, a fin de que se pueda dar por realizado el viaje.

El programa puede ser utilizado tanto para que viaje gente de la FCEN como para que venga gente del extranjero a la FCEN -este último punto resulta clave por la evidente relación CIMA-Francia: para pasantías, para viajes de profesores invitades, autoridades IRD, etc.

==== Puntos clave a tener en cuenta para solicitar presupuesto y compra de pasaje vía el Programa OPTAR-FCEN ====

* la Planilla de Solicitud debe ser firmada por la Autoridad del Departamento, Instituto, Secretaria de quien viaja. En el caso del CIMA, la firma debe ser del Director del Instituto, en este caso el Prof. Pedro Flombaum

* la solicitud debe enviarse a hacienda@de.fcen.uba.ar, rosana@de.fcen.uba.ar, y a sicyt@de.fcen.uba.ar

* en el mail de solicitud debe ponerse en copia a la autoridad que firmó la planilla (en este caso el Prof. Flombaum)

* junto a la planilla completa con la firma de la autoridad debe adjuntarse una carta de invitación/aceptación -u otro documento- que avale la actividad que se llevará a cabo en el marco de la solicitud del pasaje

* en el cuerpo del correo se debe aclarar quién reintegrará el dinero a la facultad (la persona que viajará, el instituto al cual pertenece, institución que brinda becas, etc)

* el mail ''debería'' ser enviado desde la secretaría del instituto o departamento. En caso de que se tenga poco tiempo para hacer la solicitud, el correo lo puede enviar quien solicita presupuesto y posterior compra del pasaje, para que sea la primera persona en enterarse de las novedades

== Problemas habituales ==
* [[Phishing]]

== Compras ==
* [[Especificaciones servidor]]

== Direcciones de correo electrónico con dominio CIMA ==
=== ¿Quién debe tener una cuenta @cima.fcen.uba.ar? ===
* Todos los que tengan lugar de trabajo en el CIMA (investigadores, becaries, CPA, agentes externos).
* No incluye: pasantes temporales (estudiantes de grado, master, pasantes, visitantes). Salvo pedidos excepcionales que serán evaluados por el CD.

=== ¿Cuándo corresponde dar de baja una dirección? ===
* Las bajas se darán por jubilación, cambio de lugar de trabajo, finalización de doctorado (si no siguen en el CIMA)
Listas de correo
* Las listas de estamento correspondientes y a listas generales solo incluyen direcciones @cima.fcen.uba.ar

=== Acceso a servidores ===
* Los colaboradores externos podrán tener acceso a los servidores para trabajar con un investigador o grupo del CIMA (se otorgará el permiso con la cuenta de mail que indique la persona interesada). El pedido de alta y baja tiene que ser hecho por el investigador responsable.

== Logos CIMA e IFAECI ==
* CIMA oficial: [[ Media:CIMA-CONICET-UBA.png‎ | Logo CIMA - versión principal para colocar sobre fondo blanco en png ]]
* CIMA oficial: [[ Media:CIMA-CONICET-UBA.jpg‎ | Logo CIMA - versión principal para colocar sobre fondo blanco en jpg ]]
* CIMA Versión vectorial: [[ Media:logos-CIMA-UBA-CONICET.pdf‎ | Logos CIMA - versiones permitidas (principal y en gris o con fondo azul) - formato vectorial ]]
* IFAECI: [[ Media:IFAECI-01.png‎ | Logo IFAECI - con fondo azul en png ]]
* IFAECI: [[ Media:IFAECI-02.png‎ | Logo IFAECI - con fondo blanco en png ]]
* IFAECI tutelas: [[ Media:4tutelasIFAECI-logos-sept2024.png‎ | Banner con logos CNRS, CONICET, IRD y UBA - con fondo blanco en png ]]

Página principal

2026-10-01T18:49:20Z

Guido.tiscornia: /* Modelos */

== SCAD del CIMA (Cluster) ==
=== Reglamento de uso del SCAD del CIMA ===
Versión descargable: [[ Media:SCAD-Reglamento_de_uso.pdf‎ | SCAD-Reglamento_de_uso‎ ]]

==== General ====

*El siguiente reglamento puede ser modificado sin previo aviso y es responsabilidad de los usuarios mantenerse informados leyendo el mismo regularmente.

*El SCAD del CIMA sólo se puede utilizar para realizar corridas de procesos en paralelo con fines de investigación científica

*El SCAD es un recurso compartido por todos, por lo que debe usarse de forma RESPONSABLE y no se debe abusar del mismo

==== De su uso ====

*Está absolutamente prohibido navegar en internet o descargar archivos a través de los clusters. Sólo están permitidas las conexiones hacia los clusters a través de protocolos seguros (ssh, scp, etc.).

*La administración de las corridas (jobs) en los clusters se realiza a través de un sistema de colas (pbs). Esta prohibido lanzar corridas fuera del sistema de colas

*La cantidad de trabajos que pueden estar corriendo, depende de la cantidad y el tipo de nodos que hayan sido solicitados para cada una de las corridas.

*Sólo se puede utilizar para corridas de código paralelizado. Bajo ningún concepto se debe usar el nodo principal para correr programas seriales (Ej: pre y post procesamiento de los datos).

==== Almacenamiento ====

*Todos los archivos de salida deben ser almacenados en el directorio en el directorio de salidas de cada usuario ($HOME/salidas/). Los archivos pueden permanecer almacenados en este directorio durante un plazo máximo de 30 días al cabo de los cuales serán automáticamente eliminados.

*Cada usuario contará con un “$HOME” de poco tamaño para almacenar archivos permanentes (Ej: Programas, códigos, etc)

==== Agradecimiento ====

*Toda publicación que haya utilizado recursos del SCAD del CIMA debe contener el siguiente agradecimiento:
“Simulations were made with the high-performance computing clusters available at CIMA/UBA-CONICET, Argentina”

=== Autoridades ===
;Comité Asesor Científico:
: Claudio Menendez
: Juan Ruiz
: Celeste Saulo
: Silvina Solman
: Pedro Flombaum (Director CIMA)

;Coordinador:
: Juan Ruiz

;Supervisor:
: Gabriel Vieytes

=== Monitoreo ===

Página de monitoreo y estadísticas del SCAD del CIMA
*''''' https://scad.cima.fcen.uba.ar/

=== Solicitud de cuenta para uso del SCAD del CIMA ===
Para poder hacer uso de los cluster del SCAD del CIMA, es necesario solicitar la apertura de una cuenta personal enviando un mail a [mailto:soporte@cima.fcen.uba.ar soporte@cima.fcen.uba.ar].

=== Guía de usuario del SCAD del CIMA ===
Acceso a la Guía de usuario haciendo click [[Guía de usuario del SCAD del CIMA | aquí]]

=== Composición del SCAD del CIMA ===
Detalle de software y hardware que conforman el SCAD del cima haciendo click [[Composición del SCAD del CIMA | aquí ]]

=== Links ===
Links de referencia haciendo click [[Links | aquí ]]

=== FAQ ===
Preguntas frecuentes haciendo click [[FAQ | aquí ]]

== Datos ==
Acceso a [[papa-deimos]], servidor de almacenamiento y procesamiento de datos del CIMA compartido con el DCAO.

De los datos disponibles en el clúster SCAD (hydra) para simular (forzantes para modelos cómo el WRF, ORCHIDEE, ...) aquí [[DatosSCADSimular]]

Del grupo ILLAPA: 'Mecanismos del clima regional y sus impactos' descritos aquí [[DatosRCMSueloAtmosfera]]

== Reanalysis ==
'''Copia de los archivos para uso interno del CIMA'''. Datos Disponibles via FTP y DODS(**) de:
NCEP Climate Forecast System Reanalysis (CFSR)
ds093.0: 6-hourly Products / Reanalysis 1980-2010(*) / https://rda.ucar.edu/datasets/ds093.0/index.html

Para descargar los archivos via '''FTP''': ftp://usuario:password@datos.cima.fcen.uba.ar

Para abrir los archivos en forma remota '''DODS''': http://gds.cima.fcen.uba.ar:9090/dods/

'''Descripción y ayudas:''' https://www.cima.fcen.uba.ar/CFSR.html

'''Resolución horizontal'''

Sudamerica: 0.5° x 0.5° - 15°N a 65°S y 30°W a 85°W ( 110 x 160 Longitud/Latitud)

Global: 2.5° x 2.5° - 90°N a 90°S y 0°E a 357.5°E ( 144 x 73 Longitud/Latitud)

'''Resolución vertical''' en las variables de Nivel: 32 niveles de presión (mbar)

1000 975 950 925 900 875 850 825 800 775 750 700 650 600 550 500 450 400 350 300 250 225 200 175 150 125 100 70 50 30 20 10

'''Período''' 31 Años : 1980/01/01 a 2010/12/31

'''Frecuencia del Pronóstico''': 0 6 12 18 h

'''25 VARIABLES DE SUPERFICIE'''

'''6 VARIABLES DE PRESION'''

== Modelos ==

*[[CHIMERE]]

*[[CESM]]

*[[CROCO]]

*[[ECCO]]

*[[EMU]]

*[[LMDZ]]

*[[MARS]]

*[[ORCHIDEE]]

*[[PALM & PALM4U]]

*[[RegIPSL]]

*[[RCA]]

*[[WRF]]



== Mapeo de datos de secuenciación ==

* [[Mapeo de lecturas contra bases de datos de referencia]]

== Documentación ==
*[[Redes avanzadas]]

*[[Libros]]

*[[Memoria Compartida (e.g.OpenMP)]]

*[[anaconda]]

*[[llaves ssh]]

*[[entornos python]]

*[[Configuración correo electrónico | Configuración correo electrónico (mail)]]

*[[Configuración filtros webmail para avisos de vacaciones]]

*[[procesos zombies]]

*[[parTUXza]]

*[[Seguimiento simulaciones]]

*[[Spyder]]

*[[Jupyter Notebook]]

*[[Rstudio]]

*[[Cambio de contraseña]]

*[[VPN]]

* [[Café programación]]

== Reparaciones ==
*Documentación para ingreso de personal de service [[ Media:SEGUROS_CONTRAT_2016.pdf‎ | Seguros‎ ]]

*[[Empresas para service de AA]]

== Becarios ==

== Programa pasajes OPTAR-FCEN ==

==== General ====

En la Sub seccion '''Viajes''' de la Sección Trámites de la web oficial de la FCEN-UBA → https://exactas.uba.ar/ van a encontrar la Normativa vigente, el Instructivo y la Planilla de Solicitud del '''Programa Reserva y Pago de Pasajes Aéreos - Convenio FCEN-OPTAR'''

En el Instructivo van a encontrar QUÉ se debe hacer para que el uso del programa sea ''exitoso'', lo que viene a continuación son detalles clave a tener en cuenta pero no es la ''guía definitiva''.

Repito, LEAN EL INSTRUCTIVO PARA NO OLVIDAR PASOS CLAVE. E.g.: Una vez realizado el viaje el área requirente deberá enviar los pases de embarque correspondientes, a fin de que se pueda dar por realizado el viaje.

El programa puede ser utilizado tanto para que viaje gente de la FCEN como para que venga gente del extranjero a la FCEN -este último punto resulta clave por la evidente relación CIMA-Francia: para pasantías, para viajes de profesores invitades, autoridades IRD, etc.

==== Puntos clave a tener en cuenta para solicitar presupuesto y compra de pasaje vía el Programa OPTAR-FCEN ====

* la Planilla de Solicitud debe ser firmada por la Autoridad del Departamento, Instituto, Secretaria de quien viaja. En el caso del CIMA, la firma debe ser del Director del Instituto, en este caso el Prof. Pedro Flombaum

* la solicitud debe enviarse a hacienda@de.fcen.uba.ar, rosana@de.fcen.uba.ar, y a sicyt@de.fcen.uba.ar

* en el mail de solicitud debe ponerse en copia a la autoridad que firmó la planilla (en este caso el Prof. Flombaum)

* junto a la planilla completa con la firma de la autoridad debe adjuntarse una carta de invitación/aceptación -u otro documento- que avale la actividad que se llevará a cabo en el marco de la solicitud del pasaje

* en el cuerpo del correo se debe aclarar quién reintegrará el dinero a la facultad (la persona que viajará, el instituto al cual pertenece, institución que brinda becas, etc)

* el mail ''debería'' ser enviado desde la secretaría del instituto o departamento. En caso de que se tenga poco tiempo para hacer la solicitud, el correo lo puede enviar quien solicita presupuesto y posterior compra del pasaje, para que sea la primera persona en enterarse de las novedades

== Problemas habituales ==
* [[Phishing]]

== Compras ==
* [[Especificaciones servidor]]

== Direcciones de correo electrónico con dominio CIMA ==
=== ¿Quién debe tener una cuenta @cima.fcen.uba.ar? ===
* Todos los que tengan lugar de trabajo en el CIMA (investigadores, becaries, CPA, agentes externos).
* No incluye: pasantes temporales (estudiantes de grado, master, pasantes, visitantes). Salvo pedidos excepcionales que serán evaluados por el CD.

=== ¿Cuándo corresponde dar de baja una dirección? ===
* Las bajas se darán por jubilación, cambio de lugar de trabajo, finalización de doctorado (si no siguen en el CIMA)
Listas de correo
* Las listas de estamento correspondientes y a listas generales solo incluyen direcciones @cima.fcen.uba.ar

=== Acceso a servidores ===
* Los colaboradores externos podrán tener acceso a los servidores para trabajar con un investigador o grupo del CIMA (se otorgará el permiso con la cuenta de mail que indique la persona interesada). El pedido de alta y baja tiene que ser hecho por el investigador responsable.

== Logos CIMA e IFAECI ==
* CIMA oficial: [[ Media:CIMA-CONICET-UBA.png‎ | Logo CIMA - versión principal para colocar sobre fondo blanco en png ]]
* CIMA oficial: [[ Media:CIMA-CONICET-UBA.jpg‎ | Logo CIMA - versión principal para colocar sobre fondo blanco en jpg ]]
* CIMA Versión vectorial: [[ Media:logos-CIMA-UBA-CONICET.pdf‎ | Logos CIMA - versiones permitidas (principal y en gris o con fondo azul) - formato vectorial ]]
* IFAECI: [[ Media:IFAECI-01.png‎ | Logo IFAECI - con fondo azul en png ]]
* IFAECI: [[ Media:IFAECI-02.png‎ | Logo IFAECI - con fondo blanco en png ]]
* IFAECI tutelas: [[ Media:4tutelasIFAECI-logos-sept2024.png‎ | Banner con logos CNRS, CONICET, IRD y UBA - con fondo blanco en png ]]

PALM

2026-10-01T18:46:52Z

Guido.tiscornia:

PALM

2026-10-01T18:44:28Z

Guido.tiscornia: Página creada con «Prueba PALM»

Prueba PALM

Página principal

2026-10-01T18:44:04Z

Guido.tiscornia: /* Modelos */

WRF

2025-06-24T15:21:15Z

Guido.tiscornia: /* Unavailable variables when running metgrid */

= Model description =
Weather Research and Forecast model ([http://www2.mmm.ucar.edu/wrf/users/ WRF], Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this [http://www2.mmm.ucar.edu/wrf/users/docs/technote/v4_technote.pdf pdf]

= WRF and CIMA =
CIMA has an HPC called <code>hydra</code> to perform high intensive modelling efforts with WRF.

== hydra ==
The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

=== Selection of compilers & libraries ===

It does not have the <code>module</code> software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script <code>source /opt/load-libs.sh</code>

Example selecting intel 2021.4.0, mpich and netcdf 4

<pre style="shell">
$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
</pre>

== Compilation ==
Multiple different versions are available and pre-compiled in <code>hydra</code>, with the folloing structure:

<pre style="shell">
/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}
</pre>



== Forcings ==

=== Atmospheric forcings ===
This will change once <code>papa-deimos</code> is fully operational

Provide the atmospheric conditions to the model at a given date.

* There is a shared space called <code>/share</code>
* At <code>hydra</code> all forcings are at:
<pre>/share/DATA/</pre>
* ''ERA-Interim''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim ERA-Interim] forcings are:
<pre>/share/DATA/re-analysis/ERA-Interim/</pre>
** Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
*** <code>ERAI_pl[YYYY][MM]_[var1]-[var2].grib</code>: pressure levels variables (all levels). <code>GRIB</code> codes as:
**** 129: geopotential
**** 157: relative humidty
**** 130: Temperature
**** 131: u-wind
**** 132: v-wind
*** <code>ERAI_sfc[YYYY][MM].grib</code>: all surface levels variables (step 0)
** To download data:
*** Generate files from ECMWF web-page [https://apps.ecmwf.int/auth/login/?back=http%3A%2F%2Fapps.ecmwf.int%2Fdatasets%2Fdata%2Finterim-full-daily%2Flevtype%253Dsfc%2F ERA-Interim]
*** Go to the folder in <code>hydra</code>
<pre>cd /share/DATA/re-analysis/ERA-Interim/</pre>
*** get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:
<pre>$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib</pre>
*** Rename file according to its content
<pre>$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib</pre>

* ''ERA5''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5 ERA5] forcings are:
<pre>/share/DATA/re-analysis/ERA5/</pre>

* 'NCEP-NNRP1'
** Thus, full [https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html NCEP-NNRP1] forcings are ('''NOTE: no land data!!!'''):
<pre>/share/DATA/re-analysis/NCEP_NNRP/</pre>

=== Morphological forcings ===
Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

* In WRF there is a huge amount of data ans sources at different resolutions. At <code>hydra</code> everything is already there at:
<pre>/share/GEOG/</pre>
* They are ready to be use

= GENERIC Model use =
WRF has two main parts:
* ''WPS'': Generation of the domain, initial and boundary condition: Runs 4 programs:
** <code>geogrid</code>: domain generation
** <code>ungrib</code>: unpack atmospheric forcing \verb+grib+ files
** <code>metgrid</code>: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
** <code>real</code>: generation of the initial and boundary conditions using \verb+metgrid+ output
* ''WRF'': model it self: <code>wrf.exe</code>

At <code>hydra</code> all the code is already compiled at <code>/opt/wrf</code> (with folders <code>WPS</code> and <code>WRFV3</code>)
As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory
<pre>/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF</pre>

{| class="wikitable" style="margin:auto"
|-
! IMPORTANT NOTES
|-
| * No simulation has to be run in the <CODE>$HOME</CODE> directory, use <CODE>salidas/</CODE> instead
|-
| * Always run all the components of the model using the queue
|}

'''Recommendations:'''
* Do not run the WPS/WRF components at the same folder where the code resides. Always run it on another place and link the exectuables and the necessary files
* Split the execution of the model in 2 different places:
** Where the model components run
** Where the outputs of the model components are to be stored and kept
** For example
{| class="wikitable" style="margin:auto"
|-
| '''WRFrunning:''' || <CODE>salidas/estudios/[ExperimentName]/[Simulation]</CODE>
|-
| '''WRFstorage:''' || <CODE>salidas/EXPS/[ExperimentName]/[Simulation]</CODE>
|-
| Inside <CODE>WRFrunning</CODE>, <CODE>WRFstorage</CODE> || <CODE>geogrid/, ungrib/, metgrid/, run/, ...</CODE>
|}

* Dimension the amount of nodes/cores/walltime accordingly to the needs of your simulations
* Make use of restarts, try to avoid to run simulations for more than a week

== WPS ==
Let's assume that we work in a folder called <code>$WORKDIR</code> (at users' <code>${HOME}</code> at <code>hydra</code>) and we will work with a given WRF version located in <code>$WRFversion</code>. As example, let's create a two nested domain of ''30 km'' for the entire South America and a second one of ''4.285 km'' for Córdoba mountain ranges (see the [[WRF/namelist.wps]])



=== geogrid ===

It is used to generate the domain

* Create a folder for the geogrid section
<pre style="shell">
$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid
</pre>

* Link and copy all the necessary files (do not forget to add <code>opt_geogrid_tbl_path = './'</code> into <code>&geogrid</code> section) in the namelist!!):
<pre style="shell">
$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./
</pre>

* Domain configuration is done via <code>namelist.wps</code> (more information at: [https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/user_guide_V3.9/users_guide_chap3.html WPS user guide])
<pre style="shell">
$ vim namelist.wps
</pre>

In hydra, the path to the <code>geographic data</code>a has to be defined in the <code>namelist.wps</code> as follows:
<pre style="shell">
geog_data_path = '/share/GEOG/'
</pre>

Once it is defined, run it (all the necessary <code>PBS script files</code> are available in <code>/share/WRF</code>, you only need to change amount of processes and users's email):
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs
</pre>

* This will create the domain files, one for each domain.
<pre style="shell">
geo_em.d[nn].nc
</pre>

* Some variables from the geogrid files:
** <code>LANDMASK</code>: sea-land mask
** <code>XLAT_M</code>: latitude on mass point
** <code>XLONG_M</code>: longitude on mass point
** <code>HGT_M</code>: orographical height

The definition of the domain it is not a simple task and one needs to take into account a large amount of details, [[/DomainDefinition|here]] you can find a page with some indications

=== ungrib ===
Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the [https://gitlab.in2p3.fr/ipsl/lmd/intro/regipsl/regipsl/-/wikis/Tools/nc2ps nc2wps] tool. Which is already compiled in: <code>/share/tools/RegIPSL/tools/nc2wps</code>.

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir ungrib
$ cd ungrib
</pre>

* Linking necessary files from compiled source
<pre stye="shell">
$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./
</pre>

* Edit the namelist <code>namelist.wps</code> to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)
<pre style="shell">
$ vim namelist.wps
</pre>

* Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA
<pre style="shell">
$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./
$ cd ..
</pre>

* Re-link files for WRF with its own script
<pre style="shell">
./link_grib.csh GribDir/*
</pre>

* Should appear:
<pre style="shell">
$ ls GRIBFILE.AA*
GRIBFILE.AAA GRIBFILE.AAB GRIBFILE.AAC GRIBFILE.AAD
</pre>

* We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder <code>Variable_Tables</code>. In this example we use ECMWF ERA-Interim at pressure levels, thus we link
<pre style="shell">
$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable
</pre>

* We need to take the domain file and get the right dates
<pre style="shell">
$ cp ../geogrid/nameslist.wps ./
</pre>

* Files can be unpacked using the PBS job file (user's email has to be changed and the output folder called <CODE>odir</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_ungrib_intel.pbs ./
$ vim launch_ungrib_intel.pbs
$ qsub launch_ungrib_intel.pbs
</pre>

* If everything went fine, should appear:
<pre>FILE:[YYYY]-[MM]-[DD]_[HH]</pre>
* And...
<pre>$ tail run_ungrib.log
(...)
**********
Done deleting temporary files.
**********

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== metgrid ===
Horizontal interpolation of atmospheric forcing data at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir metgrid
$ cd metgrid
</pre>

* Getting necessary files (do not forget <code> opt_metgrid_tbl_path = './'</code>, into <code>&metgrid</code> section in namelist !! to avoid: <code>ERROR: Could not open file METGRID.TBL</code>)

<pre style="shell">
$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL
</pre>

* Getting the <code>ungrib</code> output
<pre style="shell">
$ ln -s ../ungrib/FILE* ./
</pre>

** Link the domains of simulation
<pre style="shell">
$ ln -s ../geogrid/geo_em.d* ./
</pre>

** Link the namelist from <code>ungrib</code> (to make sure we are using the same!)
<pre style="shell">
$ ln -s ../ungrib/namelist.wps ./
</pre>

* Get the PBS (job queue script) to run the <code>metrid.exe</code> (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>ungrib</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_metgrid_intel.pbs ./
$ vim launch_metgrid_intel.pbs
$ qsub launch_metgrid_intel.pbs
</pre>

* If everything went fine one should have
<pre>
met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc
</pre>

* And...
<pre style="shell">
$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of metgrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== real ===
Vertical interpolation of atmospheric forcing at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)
<pre style="shell">
$ mkdir run
$ cd run
</pre>

* Link all the necessary files from WRF (it already links all the necessary to run the model)
<pre style="shell">
$ ln -s $WRFversion/WRF/run/* ./
</pre>

* Remove and copy the configuration file (<code>namelist.input</code>)
<pre style="shell">
$ rm namelist.input
$ cp $WRFversion/WRF/run/namelist.input ./
</pre>

* Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here [[WRF/namelist.input]]

<pre style="shell">
$ vim namelist.input
</pre>

NOTE: If you are using the LCZ classification scheme, the following changes must be done in the <code>namelist.input</code> to properly simulate the domains with the data.
<pre style="shell">
&physics
use_wudapt_lcz = 1
num_land_cat = 61 #If you chose to run geogrid directly from WPS v4.5
num_land_cat = 41 #If you chose to use w2w
</pre>
When using LCZs, it is recommended to use the urban parameterization options BEP or BEP+BEM (<code>sf_urban_physics = 2 or 3</code>, respectively). In case you use SLUCM model (<code>sf_urban_physics = 1</code>) the lowest model level has to be above the highest building height.

* Linking the <code>metgrid</code> generated files
<pre style="shell">
$ ln -s ../metgrid/met_em.d*.nc ./
</pre>

* Getting the PBS job script for <code>real.exe</code>

<pre>$ cp /share/WRF/run_real.pbs ./</pre>
* Getting a bash script to run executables in hydra (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>metgrid</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_real_intel.pbs ./
$ vim launch_real_intel.pbs
$ qsub /share/WRF/launch_pbs.bash
</pre>

* If everything went fine one should have (the basic ones):
<pre>
wrfbdy_d[nn] wrfinput_d[nn] ...
</pre>

** <code>wrfbdy_d01</code>: Boundary conditions file (only for the first domain)
** <code>wrfinput_d[nn]</code>: Initial conditions file for each domain
** <code>wrffdda_d[nn]</code>: Nudging file [optional]
** <code>wrflowinp_d[nn]</code>: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
** And...
<pre style="shell">$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT
</pre>

== WRF ==
* Simulation (look on [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm\#_Description_of_Namelist description of namelist] for namelist configuration/specifications).

* Getting the necessary PBS job (same folder for <code>real</code>)
<pre style="shell">$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs
</pre>

* If everything went fine one should have (the basic ones):

<pre>outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] ...
</pre>

** <code>wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: simulation output (at η=(p-ptop)/(psfc-ptop) levels)
** <code>wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: restart file (to continue simulation)
** <code>wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file with extremes from internal integration [optional]
** <code>wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file at vertical pressure levels [optional]
** <code>namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: All the parameters used for the simulation
** <code>stations</code>: folder with the time-series files (<code>tslist</code>)
* And...
<pre style="shell">
tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF
</pre>

* While ''running'' one can check the status on regarding on the <code>rsl.error.0000</code> file, e.g.:
<pre>$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain 1: 3.86105 elapsed seconds
</pre>

= WRF: known errors =

== CFL ==
* But if something went ''wrong'':
** [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition CFL]: At <code>wrfout/[InitialDATE]-[EndDATE]/</code> there are the files <code>rsl.[error/out].[nnnn]</code> (two per cpu). Use to appear <code>SIGSEV segmentation fault</code>, and to look different sources, usually <code>cfl</code>. e.g. (usually look for the largest <code>rsl</code> file (after the <code>0000</code>); [<code>$ ls -rS rsl.error.*</code>])
<pre>$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...)
d01 2012-12-01_01:30:00 33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours
d01 2012-12-01_01:30:00 MAX AT i,j,k: 100 94 21 vert_cfl,w,d(eta)= 5.897325 17.00738 3.2057911E-02
(...)</pre>

== NO working restart ==
Since a given version there is a need to include a new namelist parameter (in <code>&time_control</code> section) [[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm#restart WRF restart]] in order to make available the option to continue a simulation from a given restart
<PRE>
override_restart_timers = .true.
</PRE>

If we want to get values at the output files at the time of the restart, you need to add at the (<code>&time_control</code> section) the parameter
<PRE>
write_hist_at_0h_rst = .true.
</PRE>

== Setting e_we and e_sn for nested domains ==
<PRE>
ERROR: For nest 2, (e_we-s_we+1) must be one greater than an integer multiple of the parent_grid_ratio of "X".
</PRE>

This error indicates that condition of relation between e_we (it could be also e_sn) related to the parent_grid_ratio is not being accomplished.

For nested domains, e_we and e_sn must be one greater than an integer multiple of the nest's parent_grid_ratio. This is to ensure that a nest always starts and ends on a parent grid point. (i.e., e_we=n*parent_grid_ratio+1 for some integer n).

For example if parent_grid_ratio = 3, e_we could be 10 (or 13, 16, 19, etc).
This is because 10 - 1 = 9
And 9 % 3 = 0 (nine is multiple of 3)

== Unavailable variables when running metgrid ==
<PRE>
2025-06-24 12:03:22.797 --- INFORM: Couldn't find ST010040 at level 200100.000000 to fill level 40.000000 of ST.
2025-06-24 12:03:22.797 --- INFORM: Couldn't find [VARIABLE] at level 200100.000000 to fill level 10.000000 of [VARIABLE].
</PRE>

This error indicates that condition not all of the necessary variables where found in the files generated by ungrib. A possible cause for this is that when generating the links of the ungrib input, not all the files (and so, variables) where selected. Be aware that running ungrib will still figure as "successful", but only a few variables will have been processed. It is suggested taking a look in the ungrib.exe log where a table showing variables found are marked with an "X".

= Additional information =
For more additional information and further details visit [[WRFextras]]

= WRF4L: Lluís' WRF work-flow management =
For information about Lluí's WRF work-flow management visit here [[WRF4L]]

= WRF URBAN =
For information about urban simulation using WRF 4.5.0 or higher visit here [[WRF-URBAN]]

= CDXWRF: WRF for CORDEX =
A new module developed in CIMA to attain CORDEX variable demands visit here [[CDXWRF]]

= WRFles =
LES simulations with WRF model here [[WRFles]]

WRF

2025-06-24T15:19:35Z

Guido.tiscornia: /* WRF: known errors */

= Model description =
Weather Research and Forecast model ([http://www2.mmm.ucar.edu/wrf/users/ WRF], Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this [http://www2.mmm.ucar.edu/wrf/users/docs/technote/v4_technote.pdf pdf]

= WRF and CIMA =
CIMA has an HPC called <code>hydra</code> to perform high intensive modelling efforts with WRF.

== hydra ==
The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

=== Selection of compilers & libraries ===

It does not have the <code>module</code> software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script <code>source /opt/load-libs.sh</code>

Example selecting intel 2021.4.0, mpich and netcdf 4

<pre style="shell">
$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
</pre>

== Compilation ==
Multiple different versions are available and pre-compiled in <code>hydra</code>, with the folloing structure:

<pre style="shell">
/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}
</pre>



== Forcings ==

=== Atmospheric forcings ===
This will change once <code>papa-deimos</code> is fully operational

Provide the atmospheric conditions to the model at a given date.

* There is a shared space called <code>/share</code>
* At <code>hydra</code> all forcings are at:
<pre>/share/DATA/</pre>
* ''ERA-Interim''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim ERA-Interim] forcings are:
<pre>/share/DATA/re-analysis/ERA-Interim/</pre>
** Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
*** <code>ERAI_pl[YYYY][MM]_[var1]-[var2].grib</code>: pressure levels variables (all levels). <code>GRIB</code> codes as:
**** 129: geopotential
**** 157: relative humidty
**** 130: Temperature
**** 131: u-wind
**** 132: v-wind
*** <code>ERAI_sfc[YYYY][MM].grib</code>: all surface levels variables (step 0)
** To download data:
*** Generate files from ECMWF web-page [https://apps.ecmwf.int/auth/login/?back=http%3A%2F%2Fapps.ecmwf.int%2Fdatasets%2Fdata%2Finterim-full-daily%2Flevtype%253Dsfc%2F ERA-Interim]
*** Go to the folder in <code>hydra</code>
<pre>cd /share/DATA/re-analysis/ERA-Interim/</pre>
*** get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:
<pre>$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib</pre>
*** Rename file according to its content
<pre>$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib</pre>

* ''ERA5''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5 ERA5] forcings are:
<pre>/share/DATA/re-analysis/ERA5/</pre>

* 'NCEP-NNRP1'
** Thus, full [https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html NCEP-NNRP1] forcings are ('''NOTE: no land data!!!'''):
<pre>/share/DATA/re-analysis/NCEP_NNRP/</pre>

=== Morphological forcings ===
Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

* In WRF there is a huge amount of data ans sources at different resolutions. At <code>hydra</code> everything is already there at:
<pre>/share/GEOG/</pre>
* They are ready to be use

= GENERIC Model use =
WRF has two main parts:
* ''WPS'': Generation of the domain, initial and boundary condition: Runs 4 programs:
** <code>geogrid</code>: domain generation
** <code>ungrib</code>: unpack atmospheric forcing \verb+grib+ files
** <code>metgrid</code>: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
** <code>real</code>: generation of the initial and boundary conditions using \verb+metgrid+ output
* ''WRF'': model it self: <code>wrf.exe</code>

At <code>hydra</code> all the code is already compiled at <code>/opt/wrf</code> (with folders <code>WPS</code> and <code>WRFV3</code>)
As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory
<pre>/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF</pre>

{| class="wikitable" style="margin:auto"
|-
! IMPORTANT NOTES
|-
| * No simulation has to be run in the <CODE>$HOME</CODE> directory, use <CODE>salidas/</CODE> instead
|-
| * Always run all the components of the model using the queue
|}

'''Recommendations:'''
* Do not run the WPS/WRF components at the same folder where the code resides. Always run it on another place and link the exectuables and the necessary files
* Split the execution of the model in 2 different places:
** Where the model components run
** Where the outputs of the model components are to be stored and kept
** For example
{| class="wikitable" style="margin:auto"
|-
| '''WRFrunning:''' || <CODE>salidas/estudios/[ExperimentName]/[Simulation]</CODE>
|-
| '''WRFstorage:''' || <CODE>salidas/EXPS/[ExperimentName]/[Simulation]</CODE>
|-
| Inside <CODE>WRFrunning</CODE>, <CODE>WRFstorage</CODE> || <CODE>geogrid/, ungrib/, metgrid/, run/, ...</CODE>
|}

* Dimension the amount of nodes/cores/walltime accordingly to the needs of your simulations
* Make use of restarts, try to avoid to run simulations for more than a week

== WPS ==
Let's assume that we work in a folder called <code>$WORKDIR</code> (at users' <code>${HOME}</code> at <code>hydra</code>) and we will work with a given WRF version located in <code>$WRFversion</code>. As example, let's create a two nested domain of ''30 km'' for the entire South America and a second one of ''4.285 km'' for Córdoba mountain ranges (see the [[WRF/namelist.wps]])



=== geogrid ===

It is used to generate the domain

* Create a folder for the geogrid section
<pre style="shell">
$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid
</pre>

* Link and copy all the necessary files (do not forget to add <code>opt_geogrid_tbl_path = './'</code> into <code>&geogrid</code> section) in the namelist!!):
<pre style="shell">
$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./
</pre>

* Domain configuration is done via <code>namelist.wps</code> (more information at: [https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/user_guide_V3.9/users_guide_chap3.html WPS user guide])
<pre style="shell">
$ vim namelist.wps
</pre>

In hydra, the path to the <code>geographic data</code>a has to be defined in the <code>namelist.wps</code> as follows:
<pre style="shell">
geog_data_path = '/share/GEOG/'
</pre>

Once it is defined, run it (all the necessary <code>PBS script files</code> are available in <code>/share/WRF</code>, you only need to change amount of processes and users's email):
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs
</pre>

* This will create the domain files, one for each domain.
<pre style="shell">
geo_em.d[nn].nc
</pre>

* Some variables from the geogrid files:
** <code>LANDMASK</code>: sea-land mask
** <code>XLAT_M</code>: latitude on mass point
** <code>XLONG_M</code>: longitude on mass point
** <code>HGT_M</code>: orographical height

The definition of the domain it is not a simple task and one needs to take into account a large amount of details, [[/DomainDefinition|here]] you can find a page with some indications

=== ungrib ===
Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the [https://gitlab.in2p3.fr/ipsl/lmd/intro/regipsl/regipsl/-/wikis/Tools/nc2ps nc2wps] tool. Which is already compiled in: <code>/share/tools/RegIPSL/tools/nc2wps</code>.

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir ungrib
$ cd ungrib
</pre>

* Linking necessary files from compiled source
<pre stye="shell">
$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./
</pre>

* Edit the namelist <code>namelist.wps</code> to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)
<pre style="shell">
$ vim namelist.wps
</pre>

* Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA
<pre style="shell">
$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./
$ cd ..
</pre>

* Re-link files for WRF with its own script
<pre style="shell">
./link_grib.csh GribDir/*
</pre>

* Should appear:
<pre style="shell">
$ ls GRIBFILE.AA*
GRIBFILE.AAA GRIBFILE.AAB GRIBFILE.AAC GRIBFILE.AAD
</pre>

* We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder <code>Variable_Tables</code>. In this example we use ECMWF ERA-Interim at pressure levels, thus we link
<pre style="shell">
$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable
</pre>

* We need to take the domain file and get the right dates
<pre style="shell">
$ cp ../geogrid/nameslist.wps ./
</pre>

* Files can be unpacked using the PBS job file (user's email has to be changed and the output folder called <CODE>odir</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_ungrib_intel.pbs ./
$ vim launch_ungrib_intel.pbs
$ qsub launch_ungrib_intel.pbs
</pre>

* If everything went fine, should appear:
<pre>FILE:[YYYY]-[MM]-[DD]_[HH]</pre>
* And...
<pre>$ tail run_ungrib.log
(...)
**********
Done deleting temporary files.
**********

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== metgrid ===
Horizontal interpolation of atmospheric forcing data at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir metgrid
$ cd metgrid
</pre>

* Getting necessary files (do not forget <code> opt_metgrid_tbl_path = './'</code>, into <code>&metgrid</code> section in namelist !! to avoid: <code>ERROR: Could not open file METGRID.TBL</code>)

<pre style="shell">
$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL
</pre>

* Getting the <code>ungrib</code> output
<pre style="shell">
$ ln -s ../ungrib/FILE* ./
</pre>

** Link the domains of simulation
<pre style="shell">
$ ln -s ../geogrid/geo_em.d* ./
</pre>

** Link the namelist from <code>ungrib</code> (to make sure we are using the same!)
<pre style="shell">
$ ln -s ../ungrib/namelist.wps ./
</pre>

* Get the PBS (job queue script) to run the <code>metrid.exe</code> (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>ungrib</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_metgrid_intel.pbs ./
$ vim launch_metgrid_intel.pbs
$ qsub launch_metgrid_intel.pbs
</pre>

* If everything went fine one should have
<pre>
met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc
</pre>

* And...
<pre style="shell">
$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of metgrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== real ===
Vertical interpolation of atmospheric forcing at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)
<pre style="shell">
$ mkdir run
$ cd run
</pre>

* Link all the necessary files from WRF (it already links all the necessary to run the model)
<pre style="shell">
$ ln -s $WRFversion/WRF/run/* ./
</pre>

* Remove and copy the configuration file (<code>namelist.input</code>)
<pre style="shell">
$ rm namelist.input
$ cp $WRFversion/WRF/run/namelist.input ./
</pre>

* Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here [[WRF/namelist.input]]

<pre style="shell">
$ vim namelist.input
</pre>

NOTE: If you are using the LCZ classification scheme, the following changes must be done in the <code>namelist.input</code> to properly simulate the domains with the data.
<pre style="shell">
&physics
use_wudapt_lcz = 1
num_land_cat = 61 #If you chose to run geogrid directly from WPS v4.5
num_land_cat = 41 #If you chose to use w2w
</pre>
When using LCZs, it is recommended to use the urban parameterization options BEP or BEP+BEM (<code>sf_urban_physics = 2 or 3</code>, respectively). In case you use SLUCM model (<code>sf_urban_physics = 1</code>) the lowest model level has to be above the highest building height.

* Linking the <code>metgrid</code> generated files
<pre style="shell">
$ ln -s ../metgrid/met_em.d*.nc ./
</pre>

* Getting the PBS job script for <code>real.exe</code>

<pre>$ cp /share/WRF/run_real.pbs ./</pre>
* Getting a bash script to run executables in hydra (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>metgrid</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_real_intel.pbs ./
$ vim launch_real_intel.pbs
$ qsub /share/WRF/launch_pbs.bash
</pre>

* If everything went fine one should have (the basic ones):
<pre>
wrfbdy_d[nn] wrfinput_d[nn] ...
</pre>

** <code>wrfbdy_d01</code>: Boundary conditions file (only for the first domain)
** <code>wrfinput_d[nn]</code>: Initial conditions file for each domain
** <code>wrffdda_d[nn]</code>: Nudging file [optional]
** <code>wrflowinp_d[nn]</code>: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
** And...
<pre style="shell">$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT
</pre>

== WRF ==
* Simulation (look on [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm\#_Description_of_Namelist description of namelist] for namelist configuration/specifications).

* Getting the necessary PBS job (same folder for <code>real</code>)
<pre style="shell">$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs
</pre>

* If everything went fine one should have (the basic ones):

<pre>outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] ...
</pre>

** <code>wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: simulation output (at η=(p-ptop)/(psfc-ptop) levels)
** <code>wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: restart file (to continue simulation)
** <code>wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file with extremes from internal integration [optional]
** <code>wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file at vertical pressure levels [optional]
** <code>namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: All the parameters used for the simulation
** <code>stations</code>: folder with the time-series files (<code>tslist</code>)
* And...
<pre style="shell">
tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF
</pre>

* While ''running'' one can check the status on regarding on the <code>rsl.error.0000</code> file, e.g.:
<pre>$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain 1: 3.86105 elapsed seconds
</pre>

= WRF: known errors =

== CFL ==
* But if something went ''wrong'':
** [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition CFL]: At <code>wrfout/[InitialDATE]-[EndDATE]/</code> there are the files <code>rsl.[error/out].[nnnn]</code> (two per cpu). Use to appear <code>SIGSEV segmentation fault</code>, and to look different sources, usually <code>cfl</code>. e.g. (usually look for the largest <code>rsl</code> file (after the <code>0000</code>); [<code>$ ls -rS rsl.error.*</code>])
<pre>$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...)
d01 2012-12-01_01:30:00 33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours
d01 2012-12-01_01:30:00 MAX AT i,j,k: 100 94 21 vert_cfl,w,d(eta)= 5.897325 17.00738 3.2057911E-02
(...)</pre>

== NO working restart ==
Since a given version there is a need to include a new namelist parameter (in <code>&time_control</code> section) [[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm#restart WRF restart]] in order to make available the option to continue a simulation from a given restart
<PRE>
override_restart_timers = .true.
</PRE>

If we want to get values at the output files at the time of the restart, you need to add at the (<code>&time_control</code> section) the parameter
<PRE>
write_hist_at_0h_rst = .true.
</PRE>

== Setting e_we and e_sn for nested domains ==
<PRE>
ERROR: For nest 2, (e_we-s_we+1) must be one greater than an integer multiple of the parent_grid_ratio of "X".
</PRE>

This error indicates that condition of relation between e_we (it could be also e_sn) related to the parent_grid_ratio is not being accomplished.

For nested domains, e_we and e_sn must be one greater than an integer multiple of the nest's parent_grid_ratio. This is to ensure that a nest always starts and ends on a parent grid point. (i.e., e_we=n*parent_grid_ratio+1 for some integer n).

For example if parent_grid_ratio = 3, e_we could be 10 (or 13, 16, 19, etc).
This is because 10 - 1 = 9
And 9 % 3 = 0 (nine is multiple of 3)

== Unavailable variables when running metgrid ==
<PRE>
2025-06-24 12:03:22.797 --- INFORM: Couldn't find ST010040 at level 200100.000000 to fill level 40.000000 of ST.
2025-06-24 12:03:22.797 --- INFORM: Couldn't find [VARIABLE] at level 200100.000000 to fill level 10.000000 of [VARIABLE].
</PRE>

This error indicates that condition not all of the necessary variables where found in the files generated by ungrib. A possible cause for this is that when generating the links of the ungrib input, not all the files (and so, variables) where selected. Be aware that running ungrib will still figure as "succesfull", but only a few variables will have been processed. It is suggested taking a look in the ungrib.exe log where a table showing variables found are marked with an "X".

= Additional information =
For more additional information and further details visit [[WRFextras]]

= WRF4L: Lluís' WRF work-flow management =
For information about Lluí's WRF work-flow management visit here [[WRF4L]]

= WRF URBAN =
For information about urban simulation using WRF 4.5.0 or higher visit here [[WRF-URBAN]]

= CDXWRF: WRF for CORDEX =
A new module developed in CIMA to attain CORDEX variable demands visit here [[CDXWRF]]

= WRFles =
LES simulations with WRF model here [[WRFles]]

WRF

2025-06-10T17:02:55Z

Guido.tiscornia: /* Relación entre valores celdas y dominions */

= Model description =
Weather Research and Forecast model ([http://www2.mmm.ucar.edu/wrf/users/ WRF], Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this [http://www2.mmm.ucar.edu/wrf/users/docs/technote/v4_technote.pdf pdf]

= WRF and CIMA =
CIMA has an HPC called <code>hydra</code> to perform high intensive modelling efforts with WRF.

== hydra ==
The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

=== Selection of compilers & libraries ===

It does not have the <code>module</code> software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script <code>source /opt/load-libs.sh</code>

Example selecting intel 2021.4.0, mpich and netcdf 4

<pre style="shell">
$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
</pre>

== Compilation ==
Multiple different versions are available and pre-compiled in <code>hydra</code>, with the folloing structure:

<pre style="shell">
/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}
</pre>



== Forcings ==

=== Atmospheric forcings ===
This will change once <code>papa-deimos</code> is fully operational

Provide the atmospheric conditions to the model at a given date.

* There is a shared space called <code>/share</code>
* At <code>hydra</code> all forcings are at:
<pre>/share/DATA/</pre>
* ''ERA-Interim''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim ERA-Interim] forcings are:
<pre>/share/DATA/re-analysis/ERA-Interim/</pre>
** Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
*** <code>ERAI_pl[YYYY][MM]_[var1]-[var2].grib</code>: pressure levels variables (all levels). <code>GRIB</code> codes as:
**** 129: geopotential
**** 157: relative humidty
**** 130: Temperature
**** 131: u-wind
**** 132: v-wind
*** <code>ERAI_sfc[YYYY][MM].grib</code>: all surface levels variables (step 0)
** To download data:
*** Generate files from ECMWF web-page [https://apps.ecmwf.int/auth/login/?back=http%3A%2F%2Fapps.ecmwf.int%2Fdatasets%2Fdata%2Finterim-full-daily%2Flevtype%253Dsfc%2F ERA-Interim]
*** Go to the folder in <code>hydra</code>
<pre>cd /share/DATA/re-analysis/ERA-Interim/</pre>
*** get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:
<pre>$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib</pre>
*** Rename file according to its content
<pre>$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib</pre>

* ''ERA5''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5 ERA5] forcings are:
<pre>/share/DATA/re-analysis/ERA5/</pre>

* 'NCEP-NNRP1'
** Thus, full [https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html NCEP-NNRP1] forcings are ('''NOTE: no land data!!!'''):
<pre>/share/DATA/re-analysis/NCEP_NNRP/</pre>

=== Morphological forcings ===
Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

* In WRF there is a huge amount of data ans sources at different resolutions. At <code>hydra</code> everything is already there at:
<pre>/share/GEOG/</pre>
* They are ready to be use

= GENERIC Model use =
WRF has two main parts:
* ''WPS'': Generation of the domain, initial and boundary condition: Runs 4 programs:
** <code>geogrid</code>: domain generation
** <code>ungrib</code>: unpack atmospheric forcing \verb+grib+ files
** <code>metgrid</code>: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
** <code>real</code>: generation of the initial and boundary conditions using \verb+metgrid+ output
* ''WRF'': model it self: <code>wrf.exe</code>

At <code>hydra</code> all the code is already compiled at <code>/opt/wrf</code> (with folders <code>WPS</code> and <code>WRFV3</code>)
As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory
<pre>/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF</pre>

{| class="wikitable" style="margin:auto"
|-
! IMPORTANT NOTES
|-
| * No simulation has to be run in the <CODE>$HOME</CODE> directory, use <CODE>salidas/</CODE> instead
|-
| * Always run all the components of the model using the queue
|}

'''Recommendations:'''
* Do not run the WPS/WRF components at the same folder where the code resides. Always run it on another place and link the exectuables and the necessary files
* Split the execution of the model in 2 different places:
** Where the model components run
** Where the outputs of the model components are to be stored and kept
** For example
{| class="wikitable" style="margin:auto"
|-
| '''WRFrunning:''' || <CODE>salidas/estudios/[ExperimentName]/[Simulation]</CODE>
|-
| '''WRFstorage:''' || <CODE>salidas/EXPS/[ExperimentName]/[Simulation]</CODE>
|-
| Inside <CODE>WRFrunning</CODE>, <CODE>WRFstorage</CODE> || <CODE>geogrid/, ungrib/, metgrid/, run/, ...</CODE>
|}

* Dimension the amount of nodes/cores/walltime accordingly to the needs of your simulations
* Make use of restarts, try to avoid to run simulations for more than a week

== WPS ==
Let's assume that we work in a folder called <code>$WORKDIR</code> (at users' <code>${HOME}</code> at <code>hydra</code>) and we will work with a given WRF version located in <code>$WRFversion</code>. As example, let's create a two nested domain of ''30 km'' for the entire South America and a second one of ''4.285 km'' for Córdoba mountain ranges (see the [[WRF/namelist.wps]])



=== geogrid ===

It is used to generate the domain

* Create a folder for the geogrid section
<pre style="shell">
$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid
</pre>

* Link and copy all the necessary files (do not forget to add <code>opt_geogrid_tbl_path = './'</code> into <code>&geogrid</code> section) in the namelist!!):
<pre style="shell">
$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./
</pre>

* Domain configuration is done via <code>namelist.wps</code> (more information at: [https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/user_guide_V3.9/users_guide_chap3.html WPS user guide])
<pre style="shell">
$ vim namelist.wps
</pre>

In hydra, the path to the <code>geographic data</code>a has to be defined in the <code>namelist.wps</code> as follows:
<pre style="shell">
geog_data_path = '/share/GEOG/'
</pre>

Once it is defined, run it (all the necessary <code>PBS script files</code> are available in <code>/share/WRF</code>, you only need to change amount of processes and users's email):
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs
</pre>

* This will create the domain files, one for each domain.
<pre style="shell">
geo_em.d[nn].nc
</pre>

* Some variables from the geogrid files:
** <code>LANDMASK</code>: sea-land mask
** <code>XLAT_M</code>: latitude on mass point
** <code>XLONG_M</code>: longitude on mass point
** <code>HGT_M</code>: orographical height

The definition of the domain it is not a simple task and one needs to take into account a large amount of details, [[/DomainDefinition|here]] you can find a page with some indications

=== ungrib ===
Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the [https://gitlab.in2p3.fr/ipsl/lmd/intro/regipsl/regipsl/-/wikis/Tools/nc2ps nc2wps] tool. Which is already compiled in: <code>/share/tools/RegIPSL/tools/nc2wps</code>.

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir ungrib
$ cd ungrib
</pre>

* Linking necessary files from compiled source
<pre stye="shell">
$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./
</pre>

* Edit the namelist <code>namelist.wps</code> to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)
<pre style="shell">
$ vim namelist.wps
</pre>

* Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA
<pre style="shell">
$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./
$ cd ..
</pre>

* Re-link files for WRF with its own script
<pre style="shell">
./link_grib.csh GribDir/*
</pre>

* Should appear:
<pre style="shell">
$ ls GRIBFILE.AA*
GRIBFILE.AAA GRIBFILE.AAB GRIBFILE.AAC GRIBFILE.AAD
</pre>

* We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder <code>Variable_Tables</code>. In this example we use ECMWF ERA-Interim at pressure levels, thus we link
<pre style="shell">
$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable
</pre>

* We need to take the domain file and get the right dates
<pre style="shell">
$ cp ../geogrid/nameslist.wps ./
</pre>

* Files can be unpacked using the PBS job file (user's email has to be changed and the output folder called <CODE>odir</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_ungrib_intel.pbs ./
$ vim launch_ungrib_intel.pbs
$ qsub launch_ungrib_intel.pbs
</pre>

* If everything went fine, should appear:
<pre>FILE:[YYYY]-[MM]-[DD]_[HH]</pre>
* And...
<pre>$ tail run_ungrib.log
(...)
**********
Done deleting temporary files.
**********

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== metgrid ===
Horizontal interpolation of atmospheric forcing data at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir metgrid
$ cd metgrid
</pre>

* Getting necessary files (do not forget <code> opt_metgrid_tbl_path = './'</code>, into <code>&metgrid</code> section in namelist !! to avoid: <code>ERROR: Could not open file METGRID.TBL</code>)

<pre style="shell">
$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL
</pre>

* Getting the <code>ungrib</code> output
<pre style="shell">
$ ln -s ../ungrib/FILE* ./
</pre>

** Link the domains of simulation
<pre style="shell">
$ ln -s ../geogrid/geo_em.d* ./
</pre>

** Link the namelist from <code>ungrib</code> (to make sure we are using the same!)
<pre style="shell">
$ ln -s ../ungrib/namelist.wps ./
</pre>

* Get the PBS (job queue script) to run the <code>metrid.exe</code> (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>ungrib</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_metgrid_intel.pbs ./
$ vim launch_metgrid_intel.pbs
$ qsub launch_metgrid_intel.pbs
</pre>

* If everything went fine one should have
<pre>
met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc
</pre>

* And...
<pre style="shell">
$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of metgrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== real ===
Vertical interpolation of atmospheric forcing at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)
<pre style="shell">
$ mkdir run
$ cd run
</pre>

* Link all the necessary files from WRF (it already links all the necessary to run the model)
<pre style="shell">
$ ln -s $WRFversion/WRF/run/* ./
</pre>

* Remove and copy the configuration file (<code>namelist.input</code>)
<pre style="shell">
$ rm namelist.input
$ cp $WRFversion/WRF/run/namelist.input ./
</pre>

* Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here [[WRF/namelist.input]]

<pre style="shell">
$ vim namelist.input
</pre>

NOTE: If you are using the LCZ classification scheme, the following changes must be done in the <code>namelist.input</code> to properly simulate the domains with the data.
<pre style="shell">
&physics
use_wudapt_lcz = 1
num_land_cat = 61 #If you chose to run geogrid directly from WPS v4.5
num_land_cat = 41 #If you chose to use w2w
</pre>
When using LCZs, it is recommended to use the urban parameterization options BEP or BEP+BEM (<code>sf_urban_physics = 2 or 3</code>, respectively). In case you use SLUCM model (<code>sf_urban_physics = 1</code>) the lowest model level has to be above the highest building height.

* Linking the <code>metgrid</code> generated files
<pre style="shell">
$ ln -s ../metgrid/met_em.d*.nc ./
</pre>

* Getting the PBS job script for <code>real.exe</code>

<pre>$ cp /share/WRF/run_real.pbs ./</pre>
* Getting a bash script to run executables in hydra (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>metgrid</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_real_intel.pbs ./
$ vim launch_real_intel.pbs
$ qsub /share/WRF/launch_pbs.bash
</pre>

* If everything went fine one should have (the basic ones):
<pre>
wrfbdy_d[nn] wrfinput_d[nn] ...
</pre>

** <code>wrfbdy_d01</code>: Boundary conditions file (only for the first domain)
** <code>wrfinput_d[nn]</code>: Initial conditions file for each domain
** <code>wrffdda_d[nn]</code>: Nudging file [optional]
** <code>wrflowinp_d[nn]</code>: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
** And...
<pre style="shell">$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT
</pre>

== WRF ==
* Simulation (look on [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm\#_Description_of_Namelist description of namelist] for namelist configuration/specifications).

* Getting the necessary PBS job (same folder for <code>real</code>)
<pre style="shell">$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs
</pre>

* If everything went fine one should have (the basic ones):

<pre>outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] ...
</pre>

** <code>wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: simulation output (at η=(p-ptop)/(psfc-ptop) levels)
** <code>wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: restart file (to continue simulation)
** <code>wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file with extremes from internal integration [optional]
** <code>wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file at vertical pressure levels [optional]
** <code>namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: All the parameters used for the simulation
** <code>stations</code>: folder with the time-series files (<code>tslist</code>)
* And...
<pre style="shell">
tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF
</pre>

* While ''running'' one can check the status on regarding on the <code>rsl.error.0000</code> file, e.g.:
<pre>$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain 1: 3.86105 elapsed seconds
</pre>

= WRF: known errors =

== CFL ==
* But if something went ''wrong'':
** [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition CFL]: At <code>wrfout/[InitialDATE]-[EndDATE]/</code> there are the files <code>rsl.[error/out].[nnnn]</code> (two per cpu). Use to appear <code>SIGSEV segmentation fault</code>, and to look different sources, usually <code>cfl</code>. e.g. (usually look for the largest <code>rsl</code> file (after the <code>0000</code>); [<code>$ ls -rS rsl.error.*</code>])
<pre>$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...)
d01 2012-12-01_01:30:00 33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours
d01 2012-12-01_01:30:00 MAX AT i,j,k: 100 94 21 vert_cfl,w,d(eta)= 5.897325 17.00738 3.2057911E-02
(...)</pre>

== NO working restart ==
Since a given version there is a need to include a new namelist parameter (in <code>&time_control</code> section) [[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm#restart WRF restart]] in order to make available the option to continue a simulation from a given restart
<PRE>
override_restart_timers = .true.
</PRE>

If we want to get values at the output files at the time of the restart, you need to add at the (<code>&time_control</code> section) the parameter
<PRE>
write_hist_at_0h_rst = .true.
</PRE>

== Setting e_we and e_sn for nested domains ==
<PRE>
ERROR: For nest 2, (e_we-s_we+1) must be one greater than an integer multiple of the parent_grid_ratio of "X".
</PRE>

This error indicates that condition of relation between e_we (it could be also e_sn) related to the parent_grid_ratio is not being accomplished.

For nested domains, e_we and e_sn must be one greater than an integer multiple of the nest's parent_grid_ratio. This is to ensure that a nest always starts and ends on a parent grid point. (i.e., e_we=n*parent_grid_ratio+1 for some integer n).

For example if parent_grid_ratio = 3, e_we could be 10 (or 13, 16, 19, etc).
This is because 10 - 1 = 9
And 9 % 3 = 0 (nine is multiple of 3)

= Additional information =
For more additional information and further details visit [[WRFextras]]

= WRF4L: Lluís' WRF work-flow management =
For information about Lluí's WRF work-flow management visit here [[WRF4L]]

= WRF URBAN =
For information about urban simulation using WRF 4.5.0 or higher visit here [[WRF-URBAN]]

= CDXWRF: WRF for CORDEX =
A new module developed in CIMA to attain CORDEX variable demands visit here [[CDXWRF]]

= WRFles =
LES simulations with WRF model here [[WRFles]]

WRF

2025-06-10T16:03:22Z

Guido.tiscornia: /* Relación entre valores celdas y dominions */

= Model description =
Weather Research and Forecast model ([http://www2.mmm.ucar.edu/wrf/users/ WRF], Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this [http://www2.mmm.ucar.edu/wrf/users/docs/technote/v4_technote.pdf pdf]

= WRF and CIMA =
CIMA has an HPC called <code>hydra</code> to perform high intensive modelling efforts with WRF.

== hydra ==
The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

=== Selection of compilers & libraries ===

It does not have the <code>module</code> software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script <code>source /opt/load-libs.sh</code>

Example selecting intel 2021.4.0, mpich and netcdf 4

<pre style="shell">
$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
</pre>

== Compilation ==
Multiple different versions are available and pre-compiled in <code>hydra</code>, with the folloing structure:

<pre style="shell">
/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}
</pre>



== Forcings ==

=== Atmospheric forcings ===
This will change once <code>papa-deimos</code> is fully operational

Provide the atmospheric conditions to the model at a given date.

* There is a shared space called <code>/share</code>
* At <code>hydra</code> all forcings are at:
<pre>/share/DATA/</pre>
* ''ERA-Interim''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim ERA-Interim] forcings are:
<pre>/share/DATA/re-analysis/ERA-Interim/</pre>
** Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
*** <code>ERAI_pl[YYYY][MM]_[var1]-[var2].grib</code>: pressure levels variables (all levels). <code>GRIB</code> codes as:
**** 129: geopotential
**** 157: relative humidty
**** 130: Temperature
**** 131: u-wind
**** 132: v-wind
*** <code>ERAI_sfc[YYYY][MM].grib</code>: all surface levels variables (step 0)
** To download data:
*** Generate files from ECMWF web-page [https://apps.ecmwf.int/auth/login/?back=http%3A%2F%2Fapps.ecmwf.int%2Fdatasets%2Fdata%2Finterim-full-daily%2Flevtype%253Dsfc%2F ERA-Interim]
*** Go to the folder in <code>hydra</code>
<pre>cd /share/DATA/re-analysis/ERA-Interim/</pre>
*** get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:
<pre>$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib</pre>
*** Rename file according to its content
<pre>$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib</pre>

* ''ERA5''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5 ERA5] forcings are:
<pre>/share/DATA/re-analysis/ERA5/</pre>

* 'NCEP-NNRP1'
** Thus, full [https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html NCEP-NNRP1] forcings are ('''NOTE: no land data!!!'''):
<pre>/share/DATA/re-analysis/NCEP_NNRP/</pre>

=== Morphological forcings ===
Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

* In WRF there is a huge amount of data ans sources at different resolutions. At <code>hydra</code> everything is already there at:
<pre>/share/GEOG/</pre>
* They are ready to be use

= GENERIC Model use =
WRF has two main parts:
* ''WPS'': Generation of the domain, initial and boundary condition: Runs 4 programs:
** <code>geogrid</code>: domain generation
** <code>ungrib</code>: unpack atmospheric forcing \verb+grib+ files
** <code>metgrid</code>: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
** <code>real</code>: generation of the initial and boundary conditions using \verb+metgrid+ output
* ''WRF'': model it self: <code>wrf.exe</code>

At <code>hydra</code> all the code is already compiled at <code>/opt/wrf</code> (with folders <code>WPS</code> and <code>WRFV3</code>)
As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory
<pre>/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF</pre>

{| class="wikitable" style="margin:auto"
|-
! IMPORTANT NOTES
|-
| * No simulation has to be run in the <CODE>$HOME</CODE> directory, use <CODE>salidas/</CODE> instead
|-
| * Always run all the components of the model using the queue
|}

'''Recommendations:'''
* Do not run the WPS/WRF components at the same folder where the code resides. Always run it on another place and link the exectuables and the necessary files
* Split the execution of the model in 2 different places:
** Where the model components run
** Where the outputs of the model components are to be stored and kept
** For example
{| class="wikitable" style="margin:auto"
|-
| '''WRFrunning:''' || <CODE>salidas/estudios/[ExperimentName]/[Simulation]</CODE>
|-
| '''WRFstorage:''' || <CODE>salidas/EXPS/[ExperimentName]/[Simulation]</CODE>
|-
| Inside <CODE>WRFrunning</CODE>, <CODE>WRFstorage</CODE> || <CODE>geogrid/, ungrib/, metgrid/, run/, ...</CODE>
|}

* Dimension the amount of nodes/cores/walltime accordingly to the needs of your simulations
* Make use of restarts, try to avoid to run simulations for more than a week

== WPS ==
Let's assume that we work in a folder called <code>$WORKDIR</code> (at users' <code>${HOME}</code> at <code>hydra</code>) and we will work with a given WRF version located in <code>$WRFversion</code>. As example, let's create a two nested domain of ''30 km'' for the entire South America and a second one of ''4.285 km'' for Córdoba mountain ranges (see the [[WRF/namelist.wps]])



=== geogrid ===

It is used to generate the domain

* Create a folder for the geogrid section
<pre style="shell">
$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid
</pre>

* Link and copy all the necessary files (do not forget to add <code>opt_geogrid_tbl_path = './'</code> into <code>&geogrid</code> section) in the namelist!!):
<pre style="shell">
$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./
</pre>

* Domain configuration is done via <code>namelist.wps</code> (more information at: [https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/user_guide_V3.9/users_guide_chap3.html WPS user guide])
<pre style="shell">
$ vim namelist.wps
</pre>

In hydra, the path to the <code>geographic data</code>a has to be defined in the <code>namelist.wps</code> as follows:
<pre style="shell">
geog_data_path = '/share/GEOG/'
</pre>

Once it is defined, run it (all the necessary <code>PBS script files</code> are available in <code>/share/WRF</code>, you only need to change amount of processes and users's email):
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs
</pre>

* This will create the domain files, one for each domain.
<pre style="shell">
geo_em.d[nn].nc
</pre>

* Some variables from the geogrid files:
** <code>LANDMASK</code>: sea-land mask
** <code>XLAT_M</code>: latitude on mass point
** <code>XLONG_M</code>: longitude on mass point
** <code>HGT_M</code>: orographical height

The definition of the domain it is not a simple task and one needs to take into account a large amount of details, [[/DomainDefinition|here]] you can find a page with some indications

=== ungrib ===
Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the [https://gitlab.in2p3.fr/ipsl/lmd/intro/regipsl/regipsl/-/wikis/Tools/nc2ps nc2wps] tool. Which is already compiled in: <code>/share/tools/RegIPSL/tools/nc2wps</code>.

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir ungrib
$ cd ungrib
</pre>

* Linking necessary files from compiled source
<pre stye="shell">
$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./
</pre>

* Edit the namelist <code>namelist.wps</code> to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)
<pre style="shell">
$ vim namelist.wps
</pre>

* Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA
<pre style="shell">
$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./
$ cd ..
</pre>

* Re-link files for WRF with its own script
<pre style="shell">
./link_grib.csh GribDir/*
</pre>

* Should appear:
<pre style="shell">
$ ls GRIBFILE.AA*
GRIBFILE.AAA GRIBFILE.AAB GRIBFILE.AAC GRIBFILE.AAD
</pre>

* We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder <code>Variable_Tables</code>. In this example we use ECMWF ERA-Interim at pressure levels, thus we link
<pre style="shell">
$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable
</pre>

* We need to take the domain file and get the right dates
<pre style="shell">
$ cp ../geogrid/nameslist.wps ./
</pre>

* Files can be unpacked using the PBS job file (user's email has to be changed and the output folder called <CODE>odir</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_ungrib_intel.pbs ./
$ vim launch_ungrib_intel.pbs
$ qsub launch_ungrib_intel.pbs
</pre>

* If everything went fine, should appear:
<pre>FILE:[YYYY]-[MM]-[DD]_[HH]</pre>
* And...
<pre>$ tail run_ungrib.log
(...)
**********
Done deleting temporary files.
**********

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== metgrid ===
Horizontal interpolation of atmospheric forcing data at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir metgrid
$ cd metgrid
</pre>

* Getting necessary files (do not forget <code> opt_metgrid_tbl_path = './'</code>, into <code>&metgrid</code> section in namelist !! to avoid: <code>ERROR: Could not open file METGRID.TBL</code>)

<pre style="shell">
$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL
</pre>

* Getting the <code>ungrib</code> output
<pre style="shell">
$ ln -s ../ungrib/FILE* ./
</pre>

** Link the domains of simulation
<pre style="shell">
$ ln -s ../geogrid/geo_em.d* ./
</pre>

** Link the namelist from <code>ungrib</code> (to make sure we are using the same!)
<pre style="shell">
$ ln -s ../ungrib/namelist.wps ./
</pre>

* Get the PBS (job queue script) to run the <code>metrid.exe</code> (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>ungrib</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_metgrid_intel.pbs ./
$ vim launch_metgrid_intel.pbs
$ qsub launch_metgrid_intel.pbs
</pre>

* If everything went fine one should have
<pre>
met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc
</pre>

* And...
<pre style="shell">
$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of metgrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== real ===
Vertical interpolation of atmospheric forcing at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)
<pre style="shell">
$ mkdir run
$ cd run
</pre>

* Link all the necessary files from WRF (it already links all the necessary to run the model)
<pre style="shell">
$ ln -s $WRFversion/WRF/run/* ./
</pre>

* Remove and copy the configuration file (<code>namelist.input</code>)
<pre style="shell">
$ rm namelist.input
$ cp $WRFversion/WRF/run/namelist.input ./
</pre>

* Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here [[WRF/namelist.input]]

<pre style="shell">
$ vim namelist.input
</pre>

NOTE: If you are using the LCZ classification scheme, the following changes must be done in the <code>namelist.input</code> to properly simulate the domains with the data.
<pre style="shell">
&physics
use_wudapt_lcz = 1
num_land_cat = 61 #If you chose to run geogrid directly from WPS v4.5
num_land_cat = 41 #If you chose to use w2w
</pre>
When using LCZs, it is recommended to use the urban parameterization options BEP or BEP+BEM (<code>sf_urban_physics = 2 or 3</code>, respectively). In case you use SLUCM model (<code>sf_urban_physics = 1</code>) the lowest model level has to be above the highest building height.

* Linking the <code>metgrid</code> generated files
<pre style="shell">
$ ln -s ../metgrid/met_em.d*.nc ./
</pre>

* Getting the PBS job script for <code>real.exe</code>

<pre>$ cp /share/WRF/run_real.pbs ./</pre>
* Getting a bash script to run executables in hydra (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>metgrid</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_real_intel.pbs ./
$ vim launch_real_intel.pbs
$ qsub /share/WRF/launch_pbs.bash
</pre>

* If everything went fine one should have (the basic ones):
<pre>
wrfbdy_d[nn] wrfinput_d[nn] ...
</pre>

** <code>wrfbdy_d01</code>: Boundary conditions file (only for the first domain)
** <code>wrfinput_d[nn]</code>: Initial conditions file for each domain
** <code>wrffdda_d[nn]</code>: Nudging file [optional]
** <code>wrflowinp_d[nn]</code>: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
** And...
<pre style="shell">$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT
</pre>

== WRF ==
* Simulation (look on [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm\#_Description_of_Namelist description of namelist] for namelist configuration/specifications).

* Getting the necessary PBS job (same folder for <code>real</code>)
<pre style="shell">$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs
</pre>

* If everything went fine one should have (the basic ones):

<pre>outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] ...
</pre>

** <code>wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: simulation output (at η=(p-ptop)/(psfc-ptop) levels)
** <code>wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: restart file (to continue simulation)
** <code>wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file with extremes from internal integration [optional]
** <code>wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file at vertical pressure levels [optional]
** <code>namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: All the parameters used for the simulation
** <code>stations</code>: folder with the time-series files (<code>tslist</code>)
* And...
<pre style="shell">
tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF
</pre>

* While ''running'' one can check the status on regarding on the <code>rsl.error.0000</code> file, e.g.:
<pre>$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain 1: 3.86105 elapsed seconds
</pre>

= WRF: known errors =

== CFL ==
* But if something went ''wrong'':
** [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition CFL]: At <code>wrfout/[InitialDATE]-[EndDATE]/</code> there are the files <code>rsl.[error/out].[nnnn]</code> (two per cpu). Use to appear <code>SIGSEV segmentation fault</code>, and to look different sources, usually <code>cfl</code>. e.g. (usually look for the largest <code>rsl</code> file (after the <code>0000</code>); [<code>$ ls -rS rsl.error.*</code>])
<pre>$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...)
d01 2012-12-01_01:30:00 33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours
d01 2012-12-01_01:30:00 MAX AT i,j,k: 100 94 21 vert_cfl,w,d(eta)= 5.897325 17.00738 3.2057911E-02
(...)</pre>

== NO working restart ==
Since a given version there is a need to include a new namelist parameter (in <code>&time_control</code> section) [[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm#restart WRF restart]] in order to make available the option to continue a simulation from a given restart
<PRE>
override_restart_timers = .true.
</PRE>

If we want to get values at the output files at the time of the restart, you need to add at the (<code>&time_control</code> section) the parameter
<PRE>
write_hist_at_0h_rst = .true.
</PRE>

== Relación entre valores celdas y dominions ==
<PRE>
ERROR: For nest 2, (e_we-s_we+1) must be one greater than an integer multiple of the parent_grid_ratio of "X".
</PRE>

This error indicates that condition of relation between e_we (it could be also e_sn) related to the parent_grid_ratio is not being accomplished.

For nested domains, e_we and e_sn must be one greater than an integer multiple of the nest's parent_grid_ratio. This is to ensure that a nest always starts and ends on a parent grid point. (i.e., e_we=n*parent_grid_ratio+1 for some integer n).

For example if parent_grid_ratio = 3, e_we could be 10 (or 13, 16, 19, etc).
This is because 10 - 1 = 9
And 9 % 3 = 0 (nine is multiple of 3)

= Additional information =
For more additional information and further details visit [[WRFextras]]

= WRF4L: Lluís' WRF work-flow management =
For information about Lluí's WRF work-flow management visit here [[WRF4L]]

= WRF URBAN =
For information about urban simulation using WRF 4.5.0 or higher visit here [[WRF-URBAN]]

= CDXWRF: WRF for CORDEX =
A new module developed in CIMA to attain CORDEX variable demands visit here [[CDXWRF]]

= WRFles =
LES simulations with WRF model here [[WRFles]]

WRF

2025-06-10T15:44:35Z

Guido.tiscornia: /* WRF: known errors */

= Model description =
Weather Research and Forecast model ([http://www2.mmm.ucar.edu/wrf/users/ WRF], Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this [http://www2.mmm.ucar.edu/wrf/users/docs/technote/v4_technote.pdf pdf]

= WRF and CIMA =
CIMA has an HPC called <code>hydra</code> to perform high intensive modelling efforts with WRF.

== hydra ==
The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

=== Selection of compilers & libraries ===

It does not have the <code>module</code> software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script <code>source /opt/load-libs.sh</code>

Example selecting intel 2021.4.0, mpich and netcdf 4

<pre style="shell">
$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1
</pre>

== Compilation ==
Multiple different versions are available and pre-compiled in <code>hydra</code>, with the folloing structure:

<pre style="shell">
/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}
</pre>



== Forcings ==

=== Atmospheric forcings ===
This will change once <code>papa-deimos</code> is fully operational

Provide the atmospheric conditions to the model at a given date.

* There is a shared space called <code>/share</code>
* At <code>hydra</code> all forcings are at:
<pre>/share/DATA/</pre>
* ''ERA-Interim''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era-interim ERA-Interim] forcings are:
<pre>/share/DATA/re-analysis/ERA-Interim/</pre>
** Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
*** <code>ERAI_pl[YYYY][MM]_[var1]-[var2].grib</code>: pressure levels variables (all levels). <code>GRIB</code> codes as:
**** 129: geopotential
**** 157: relative humidty
**** 130: Temperature
**** 131: u-wind
**** 132: v-wind
*** <code>ERAI_sfc[YYYY][MM].grib</code>: all surface levels variables (step 0)
** To download data:
*** Generate files from ECMWF web-page [https://apps.ecmwf.int/auth/login/?back=http%3A%2F%2Fapps.ecmwf.int%2Fdatasets%2Fdata%2Finterim-full-daily%2Flevtype%253Dsfc%2F ERA-Interim]
*** Go to the folder in <code>hydra</code>
<pre>cd /share/DATA/re-analysis/ERA-Interim/</pre>
*** get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:
<pre>$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib</pre>
*** Rename file according to its content
<pre>$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib</pre>

* ''ERA5''
** Thus, part of [https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5 ERA5] forcings are:
<pre>/share/DATA/re-analysis/ERA5/</pre>

* 'NCEP-NNRP1'
** Thus, full [https://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html NCEP-NNRP1] forcings are ('''NOTE: no land data!!!'''):
<pre>/share/DATA/re-analysis/NCEP_NNRP/</pre>

=== Morphological forcings ===
Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

* In WRF there is a huge amount of data ans sources at different resolutions. At <code>hydra</code> everything is already there at:
<pre>/share/GEOG/</pre>
* They are ready to be use

= GENERIC Model use =
WRF has two main parts:
* ''WPS'': Generation of the domain, initial and boundary condition: Runs 4 programs:
** <code>geogrid</code>: domain generation
** <code>ungrib</code>: unpack atmospheric forcing \verb+grib+ files
** <code>metgrid</code>: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
** <code>real</code>: generation of the initial and boundary conditions using \verb+metgrid+ output
* ''WRF'': model it self: <code>wrf.exe</code>

At <code>hydra</code> all the code is already compiled at <code>/opt/wrf</code> (with folders <code>WPS</code> and <code>WRFV3</code>)
As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory
<pre>/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF</pre>

{| class="wikitable" style="margin:auto"
|-
! IMPORTANT NOTES
|-
| * No simulation has to be run in the <CODE>$HOME</CODE> directory, use <CODE>salidas/</CODE> instead
|-
| * Always run all the components of the model using the queue
|}

'''Recommendations:'''
* Do not run the WPS/WRF components at the same folder where the code resides. Always run it on another place and link the exectuables and the necessary files
* Split the execution of the model in 2 different places:
** Where the model components run
** Where the outputs of the model components are to be stored and kept
** For example
{| class="wikitable" style="margin:auto"
|-
| '''WRFrunning:''' || <CODE>salidas/estudios/[ExperimentName]/[Simulation]</CODE>
|-
| '''WRFstorage:''' || <CODE>salidas/EXPS/[ExperimentName]/[Simulation]</CODE>
|-
| Inside <CODE>WRFrunning</CODE>, <CODE>WRFstorage</CODE> || <CODE>geogrid/, ungrib/, metgrid/, run/, ...</CODE>
|}

* Dimension the amount of nodes/cores/walltime accordingly to the needs of your simulations
* Make use of restarts, try to avoid to run simulations for more than a week

== WPS ==
Let's assume that we work in a folder called <code>$WORKDIR</code> (at users' <code>${HOME}</code> at <code>hydra</code>) and we will work with a given WRF version located in <code>$WRFversion</code>. As example, let's create a two nested domain of ''30 km'' for the entire South America and a second one of ''4.285 km'' for Córdoba mountain ranges (see the [[WRF/namelist.wps]])



=== geogrid ===

It is used to generate the domain

* Create a folder for the geogrid section
<pre style="shell">
$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid
</pre>

* Link and copy all the necessary files (do not forget to add <code>opt_geogrid_tbl_path = './'</code> into <code>&geogrid</code> section) in the namelist!!):
<pre style="shell">
$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./
</pre>

* Domain configuration is done via <code>namelist.wps</code> (more information at: [https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/user_guide_V3.9/users_guide_chap3.html WPS user guide])
<pre style="shell">
$ vim namelist.wps
</pre>

In hydra, the path to the <code>geographic data</code>a has to be defined in the <code>namelist.wps</code> as follows:
<pre style="shell">
geog_data_path = '/share/GEOG/'
</pre>

Once it is defined, run it (all the necessary <code>PBS script files</code> are available in <code>/share/WRF</code>, you only need to change amount of processes and users's email):
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs
</pre>

* This will create the domain files, one for each domain.
<pre style="shell">
geo_em.d[nn].nc
</pre>

* Some variables from the geogrid files:
** <code>LANDMASK</code>: sea-land mask
** <code>XLAT_M</code>: latitude on mass point
** <code>XLONG_M</code>: longitude on mass point
** <code>HGT_M</code>: orographical height

The definition of the domain it is not a simple task and one needs to take into account a large amount of details, [[/DomainDefinition|here]] you can find a page with some indications

=== ungrib ===
Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the [https://gitlab.in2p3.fr/ipsl/lmd/intro/regipsl/regipsl/-/wikis/Tools/nc2ps nc2wps] tool. Which is already compiled in: <code>/share/tools/RegIPSL/tools/nc2wps</code>.

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir ungrib
$ cd ungrib
</pre>

* Linking necessary files from compiled source
<pre stye="shell">
$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./
</pre>

* Edit the namelist <code>namelist.wps</code> to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)
<pre style="shell">
$ vim namelist.wps
</pre>

* Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA
<pre style="shell">
$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./
$ cd ..
</pre>

* Re-link files for WRF with its own script
<pre style="shell">
./link_grib.csh GribDir/*
</pre>

* Should appear:
<pre style="shell">
$ ls GRIBFILE.AA*
GRIBFILE.AAA GRIBFILE.AAB GRIBFILE.AAC GRIBFILE.AAD
</pre>

* We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder <code>Variable_Tables</code>. In this example we use ECMWF ERA-Interim at pressure levels, thus we link
<pre style="shell">
$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable
</pre>

* We need to take the domain file and get the right dates
<pre style="shell">
$ cp ../geogrid/nameslist.wps ./
</pre>

* Files can be unpacked using the PBS job file (user's email has to be changed and the output folder called <CODE>odir</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_ungrib_intel.pbs ./
$ vim launch_ungrib_intel.pbs
$ qsub launch_ungrib_intel.pbs
</pre>

* If everything went fine, should appear:
<pre>FILE:[YYYY]-[MM]-[DD]_[HH]</pre>
* And...
<pre>$ tail run_ungrib.log
(...)
**********
Done deleting temporary files.
**********

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== metgrid ===
Horizontal interpolation of atmospheric forcing data at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)

<pre style="shell">
$ mkdir metgrid
$ cd metgrid
</pre>

* Getting necessary files (do not forget <code> opt_metgrid_tbl_path = './'</code>, into <code>&metgrid</code> section in namelist !! to avoid: <code>ERROR: Could not open file METGRID.TBL</code>)

<pre style="shell">
$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL
</pre>

* Getting the <code>ungrib</code> output
<pre style="shell">
$ ln -s ../ungrib/FILE* ./
</pre>

** Link the domains of simulation
<pre style="shell">
$ ln -s ../geogrid/geo_em.d* ./
</pre>

** Link the namelist from <code>ungrib</code> (to make sure we are using the same!)
<pre style="shell">
$ ln -s ../ungrib/namelist.wps ./
</pre>

* Get the PBS (job queue script) to run the <code>metrid.exe</code> (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>ungrib</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_metgrid_intel.pbs ./
$ vim launch_metgrid_intel.pbs
$ qsub launch_metgrid_intel.pbs
</pre>

* If everything went fine one should have
<pre>
met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc
</pre>

* And...
<pre style="shell">
$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of metgrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
</pre>

=== real ===
Vertical interpolation of atmospheric forcing at the domain of simulation

* Creation of the folder (from <code>$WORKDIR</code>)
<pre style="shell">
$ mkdir run
$ cd run
</pre>

* Link all the necessary files from WRF (it already links all the necessary to run the model)
<pre style="shell">
$ ln -s $WRFversion/WRF/run/* ./
</pre>

* Remove and copy the configuration file (<code>namelist.input</code>)
<pre style="shell">
$ rm namelist.input
$ cp $WRFversion/WRF/run/namelist.input ./
</pre>

* Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here [[WRF/namelist.input]]

<pre style="shell">
$ vim namelist.input
</pre>

NOTE: If you are using the LCZ classification scheme, the following changes must be done in the <code>namelist.input</code> to properly simulate the domains with the data.
<pre style="shell">
&physics
use_wudapt_lcz = 1
num_land_cat = 61 #If you chose to run geogrid directly from WPS v4.5
num_land_cat = 41 #If you chose to use w2w
</pre>
When using LCZs, it is recommended to use the urban parameterization options BEP or BEP+BEM (<code>sf_urban_physics = 2 or 3</code>, respectively). In case you use SLUCM model (<code>sf_urban_physics = 1</code>) the lowest model level has to be above the highest building height.

* Linking the <code>metgrid</code> generated files
<pre style="shell">
$ ln -s ../metgrid/met_em.d*.nc ./
</pre>

* Getting the PBS job script for <code>real.exe</code>

<pre>$ cp /share/WRF/run_real.pbs ./</pre>
* Getting a bash script to run executables in hydra (remember to edit the user's email in the pbs job and the input/output folders called <CODE>idir, odir</CODE>; idir: output of <CODE>metgrid</CODE>)
<pre style="shell">
$ cp /share/tools/workflows/direct/WRF/hydra/launch_real_intel.pbs ./
$ vim launch_real_intel.pbs
$ qsub /share/WRF/launch_pbs.bash
</pre>

* If everything went fine one should have (the basic ones):
<pre>
wrfbdy_d[nn] wrfinput_d[nn] ...
</pre>

** <code>wrfbdy_d01</code>: Boundary conditions file (only for the first domain)
** <code>wrfinput_d[nn]</code>: Initial conditions file for each domain
** <code>wrffdda_d[nn]</code>: Nudging file [optional]
** <code>wrflowinp_d[nn]</code>: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
** And...
<pre style="shell">$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT
</pre>

== WRF ==
* Simulation (look on [http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm\#_Description_of_Namelist description of namelist] for namelist configuration/specifications).

* Getting the necessary PBS job (same folder for <code>real</code>)
<pre style="shell">$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs
</pre>

* If everything went fine one should have (the basic ones):

<pre>outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] ...
</pre>

** <code>wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: simulation output (at η=(p-ptop)/(psfc-ptop) levels)
** <code>wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: restart file (to continue simulation)
** <code>wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file with extremes from internal integration [optional]
** <code>wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: file at vertical pressure levels [optional]
** <code>namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]</code>: All the parameters used for the simulation
** <code>stations</code>: folder with the time-series files (<code>tslist</code>)
* And...
<pre style="shell">
tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF
</pre>

* While ''running'' one can check the status on regarding on the <code>rsl.error.0000</code> file, e.g.:
<pre>$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain 1: 3.86105 elapsed seconds
</pre>

= WRF: known errors =

== CFL ==
* But if something went ''wrong'':
** [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition CFL]: At <code>wrfout/[InitialDATE]-[EndDATE]/</code> there are the files <code>rsl.[error/out].[nnnn]</code> (two per cpu). Use to appear <code>SIGSEV segmentation fault</code>, and to look different sources, usually <code>cfl</code>. e.g. (usually look for the largest <code>rsl</code> file (after the <code>0000</code>); [<code>$ ls -rS rsl.error.*</code>])
<pre>$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...)
d01 2012-12-01_01:30:00 33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours
d01 2012-12-01_01:30:00 MAX AT i,j,k: 100 94 21 vert_cfl,w,d(eta)= 5.897325 17.00738 3.2057911E-02
(...)</pre>

== NO working restart ==
Since a given version there is a need to include a new namelist parameter (in <code>&time_control</code> section) [[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/users_guide_chap5.htm#restart WRF restart]] in order to make available the option to continue a simulation from a given restart
<PRE>
override_restart_timers = .true.
</PRE>

If we want to get values at the output files at the time of the restart, you need to add at the (<code>&time_control</code> section) the parameter
<PRE>
write_hist_at_0h_rst = .true.
</PRE>

== Relación entre valores celdas y dominions ==
<PRE>
ERROR: For nest 2, (e_we-s_we+1) must be one greater than an integer multiple of the parent_grid_ratio of 5.
</PRE>

= Additional information =
For more additional information and further details visit [[WRFextras]]

= WRF4L: Lluís' WRF work-flow management =
For information about Lluí's WRF work-flow management visit here [[WRF4L]]

= WRF URBAN =
For information about urban simulation using WRF 4.5.0 or higher visit here [[WRF-URBAN]]

= CDXWRF: WRF for CORDEX =
A new module developed in CIMA to attain CORDEX variable demands visit here [[CDXWRF]]

= WRFles =
LES simulations with WRF model here [[WRFles]]