Ir al contenido

PALM

De Wikicima

PALM is a large-eddy simulation (LES) model for atmospheric and oceanic boundary-layer flows, developed mainly at the Institute of Meteorology and Climatology of Leibniz Universität Hannover. It is written in Fortran, parallelized with MPI, uses NetCDF for input and output, and is distributed as free software (GPLv3). Instead of parametrizing boundary-layer turbulence as a mesoscale model such as WRF does, PALM resolves the large eddies explicitly, which makes it suitable for simulations at the scale of meters. PALM-4U (PALM for urban applications) is the set of components that extend the model to cities: buildings are resolved explicitly on the grid, and dedicated modules handle the urban surface energy balance (urban surface model), soil and vegetation (land surface model), 3D shading and reflections between buildings (radiative transfer model), trees (plant canopy model) and human thermal comfort indices (biometeorology), among others. A PALM-4U simulation takes a static driver (a NetCDF file describing terrain, buildings, vegetation and surface types) and, optionally, a dynamic driver with boundary conditions taken from a larger-scale model, typically WRF. For a general description of the model system see Maronga et al. (2020), linked at the end.

This page describes how PALM 24.04 was compiled and run on Hydra (PBS, queue larga), including the problems found along the way. It was tested between late June and early July 2026. Wherever you see <user>, use your own Hydra username.

Author: Guido Tiscornia (CIMA).

Starting point

PALM was built with the system gfortran, the system NetCDF (4.9.0 C, 4.5.4 Fortran) and Intel MPI 2021.18, which in this build wraps gfortran. It was validated with example_cbl (clean diff against the reference output), and urban cases were run in batch from 1 node up to 2 nodes (256 processes).

The Intel compiler was not used. Hydra’s Intel environment has two oneAPI installations (2021.4.0 with ifort and 2026.0 with ifx), and the MPICH mpif90 calls an ifort that is not in the PATH. With a couple of patches a simple MPI “hello world” worked, but the PALM installer ended up producing a mixed build that failed at runtime with undefined symbol: __netcdf_MOD_nf90_put_var.... After two attempts we switched to a pure gfortran build.

Hydra, for this purpose

There is a single queue, larga. The nodes (as of July 2026; check with pbsnodes):

  • 128 cores, nodes 90 to 97, about 132 GB: production. Usually 2 to 4 may be free.
  • 48 cores, nodes 61 to 76, about 132 GB: almost always free, good for tests.
  • 24 and 40 cores, nodes 51 and 52, about 32 GB: used for undergraduate courses mainly

Memory is not requested from PBS: the whole node is allocated. Walltimes of up to 7 days are accepted without problems.

Installation

Download (at the moment git clone asks for credentials inside of PALM website, but the tarball does not): 

mkdir -p ~/salidas/palm/current_version && cd ~/salidas/palm/current_version
wget https://gitlab.palm-model.org/releases/palm_model_system/-/archive/v24.04/palm_model_system-v24.04.tar.gz
tar -xzf palm_model_system-v24.04.tar.gz
mv palm_model_system-v24.04 palm_model_system

Environment script

Hydra’s default environment breaks PALM, so it has to be cleaned up. Save this, for example, as ~/cargar_entorno_palm.sh: 

#!/bin/bash
# palm environment on hydra: system gfortran + system netcdf + intel mpi 2021.18
# use with: source ~/cargar_entorno_palm.sh  (source it, do not execute it)

# remove intel's netcdf and mpich from ld_library_path
export LD_LIBRARY_PATH=$(echo $LD_LIBRARY_PATH | tr ':' '\n' \
    | grep -v "netcdf/netcdf-4/intel" \
    | grep -v "mpich" \
    | tr '\n' ':' | sed 's/:$//')

# intel mpi runtime: needed on compute nodes, which start with an empty environment
export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.18/lib:$LD_LIBRARY_PATH

# intel mpi 2021.18 first in the path, mpich removed
export PATH=/opt/intel/oneapi/mpi/2021.18/bin:$(echo $PATH | tr ':' '\n' \
    | grep -v "mpich" \
    | tr '\n' ':' | sed 's/:$//')

# palm executables
export PATH=/nfsmounts/storage/scratch/<user>/palm/bin:$PATH

echo "mpif90 -> $(which mpif90)"
echo "wraps -> $(mpif90 -show 2>/dev/null | awk '{print $1}')"

It has to be sourced in every new terminal. After loading it, the second line must say gfortran; if it does not, do not continue.

Build

With the environment loaded: 

cd ~/salidas/palm/current_version/palm_model_system
bash install -p ~/salidas/palm -s /usr -t /usr -x

-p is the install prefix, -s and -t force the system NetCDF (otherwise autodetection picks Intel’s), and -x cleans the previous build. When it finishes, the executables are in ~/salidas/palm/bin and the default configuration is ~/salidas/palm/.palm.config.default. RRTMG is linked in.

Interactive test

cd ~/salidas/palm
mkdir -p JOBS/example_cbl/INPUT
cp current_version/palm_model_system/packages/palm/model/tests/cases/example_cbl/INPUT/example_cbl_p3d JOBS/example_cbl/INPUT/
palmrun -r example_cbl -c default -a "d3#" -X 4 -v -z

It should end with --> palmrun finished. To validate, compare the RUN_CONTROL file against the case’s reference. On the login node, use few processes.

Batch mode

PALM uses one configuration file per suffix, selected with -c. We work on a copy so the default one stays untouched: 

cp ~/salidas/palm/.palm.config.default ~/salidas/palm/.palm.config.hydra
CFG=~/salidas/palm/.palm.config.hydra
sed -i 's|^#%submit_command.*|%submit_command      qsub|' $CFG
sed -i 's|^#%defaultqueue.*|%defaultqueue        larga|' $CFG
sed -i 's|^#%memory.*|%memory              2000|' $CFG
sed -i 's|^#%login_init_cmd.*|%login_init_cmd       source /home/<user>/cargar_entorno_palm.sh|' $CFG
grep -nE '%submit_command|%defaultqueue|%memory|%login_init_cmd' $CFG

The block of PBS directives in the template already works on Hydra, and the BDT: block (queue dataq) is only used to transfer files from remote hosts, so it is left alone. %memory is an internal PALM variable and never reaches PBS.

The most important problem of this stage: compute nodes start with an empty environment, and without the Intel MPI runtime in LD_LIBRARY_PATH the job fails with libmpi.so.12 => not found. That is why the script includes the runtime line and why it is loaded through %login_init_cmd. To check this without wasting a real run, a small job that runs ldd on the executable is enough: 

#!/bin/bash
#PBS -N diag_entorno
#PBS -l walltime=00:05:00
#PBS -l nodes=1:ppn=48
#PBS -o /nfsmounts/storage/scratch/<user>/palm/diag_entorno.log
#PBS -j oe
#PBS -q larga

source /home/<user>/cargar_entorno_palm.sh
EXE=/home/<user>/salidas/palm/MAKE_DEPOSITORY_hydra/palm
if ldd $EXE | grep -q "not found"; then
    echo "PROBLEM:"; ldd $EXE | grep "not found"
else
    echo "OK: all libraries resolve"
fi

Submit it with qsub and read the log. The executable in MAKE_DEPOSITORY_hydra only exists after the first run with -c hydra.

Running

source ~/cargar_entorno_palm.sh
cd ~/salidas/palm
palmrun -r <case> -c hydra -a "d3#" -X 48 -T 48 -t 1800 -q larga -b -v -z

-r is the case, -X the total number of processes, -T the processes per node (nodes = X/T), -t the walltime in seconds, -b submits to the queue, and -F generates the job script without submitting it, which is handy for inspecting it. -a "d3#" is mandatory: without it palmrun aborts with no activation string list given. For two 128-core nodes use -X 256 -T 128. To give each process more memory you can reserve whole nodes and use fewer cores per node, for example -X 128 -T 64.

Each new configuration recompiles once into its own MAKE_DEPOSITORY. The number PALM gives to its own runs (for example case.650) is not the PBS job ID.

Setting up a case

Each case lives in ~/salidas/palm/JOBS/<case>/INPUT/, with the namelist <case>_p3d and, for urban cases, the static driver <case>_static (without .nc). palmrun is always called from ~/salidas/palm. Rules that caused problems:

  • Only _p3d and _static files go in INPUT/. A backup there (for example case_p3d.bak) makes the run abort with wrong ending.
  • nx+1 and ny+1 must factorize into 2, 3 and 5. That is why 384 was used and not 400.
  • PALM counts the maximum index (nx = 383 means 384 points), while GEO4PALM counts points. If they do not match, the error is static driver: horizontal dimension ... does not match model dimension.
  • Always set data_output_2d_on_each_pe = .FALSE. in &runtime_parameters. With the default value, combine_plot_fields breaks on NFS and outputs are lost.
  • _xy output needs section_xy, for example section_xy = 0.
  • The static driver must declare all five surface type variables, including water_type, even if one is entirely fill values (otherwise error DRV0011).

Minimal namelist used for the benchmark (derived from example_cbl): 

&initialization_parameters
    nx = 383, ny = 383, nz = 95,
    dx = 10.0, dy = 10.0, dz = 10.0,
    dz_stretch_level = 1225.0, dz_stretch_factor = 1.08,
    initializing_actions = 'set_constant_profiles',
    ug_surface = 0.0, vg_surface = 0.0, pt_surface = 300.0,
    pt_vertical_gradient = 0.0, 1.0,
    pt_vertical_gradient_level = 0.0, 800.0,
    surface_heatflux = 0.1, bc_pt_b = 'neumann',
    fft_method = 'temperton-algorithm',
/
&runtime_parameters
    end_time = 120.0,
    create_disturbances = .TRUE., dt_disturb = 150.0,
    disturbance_energy_limit = 0.01,
    data_output_2d_on_each_pe = .FALSE.,
    netcdf_data_format = 2, restart_data_format = 'mpi',
    dt_run_control = 0.0,
/

Monitoring and results

qstat -u <user>                          # Q queued, R running, C completed
pbsnodes -l free                         # free nodes
env LD_LIBRARY_PATH= tail -15 ~/salidas/palm/tmp/<case>.<id>/RUN_CONTROL   # live progress
ls -lt ~/salidas/palm/JOBS/<case>/MONITORING/
env LD_LIBRARY_PATH= ncdump -h ~/salidas/palm/JOBS/<case>/OUTPUT/<case>_xy.000.nc

The env LD_LIBRARY_PATH= prefix is needed on Hydra because the system ncdump clashes with the NetCDF library that PALM’s environment puts first (undefined symbol: nc_reclaim_data). Output files get a suffix .000, .001, … per successful run. The _cpu.NNN file is only written if the run finished normally, so it is a good way to confirm it reached the end. Always look at the actual contents (ls, ncdump -h) before considering a run good.

Tools and links

Documentation and downloads:

Obtenido de «http://wiki.cima.fcen.uba.ar/index.php?title=PALM&oldid=3852»