WRF

Model description

Weather Research and Forecast model (WRF, Skamarock, 2008) is a limited area atmospheric model developed by a consortium of american institutions with contributions from the whole community. It is a non-hydrostatic primitive equations model used in a large variety of research areas.

Techincal aspects of the model (v 4.0) are provided in this pdf

WRF and CIMA

CIMA has an HPC called hydra to perform high intensive modelling efforts with WRF.

hydra

The HPC is part of the Servicio Nacional de Cómputo de Alta Capacidad (SNCAD).

It has been re-installed and re-sorted in Jan 2022 and it has been adapted to common configurations in other international research centers.

Selection of compilers & libraries

It does not have the module software to manage compilers and libraries, but it does have an script which allows users to select the compiler. This is done at the terminal with the script source /opt/load-libs.sh

Example selecting intel 2021.4.0, mpich and netcdf 4

$ source /opt/load-libs.sh

Available libraries:
1) INTEL 2021.4.0: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
2) INTEL 2021.4.0: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
3) GNU 10.2.1: MPICH 3.4.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
4) GNU 10.2.1: OpenMPI 4.1.2, NetCDF 4, HDF5 1.10.5, JASPER 2.0.33
0) Exit
Choose an option: 1

The following libraries, compiled with Intel 2021.4.0 compilers, were loaded:
* MPICH 3.4.2
* NetCDF 4
* HDF5 1.10.5
* JASPER 2.0.33

To change it please logout and login again.

To load this libraries from within a script add this line to script:
source /opt/load-libs.sh 1

Compilation

Multiple different versions are available and pre-compiled in hydra, with the folloing structure:

/opt/wrf/WRF-{version}/{compiler}/{compiler-version}/{architecture}/{WRF/WPS}

Forcings

Atmospheric forcings

This will change once papa-deimos is fully operational

Provide the atmospheric conditions to the model at a given date.

• There is a shared space called /share
• At hydra all forcings are at:

/share/DATA/

• ERA-Interim
  • Thus, part of ERA-Interim forcings are:

/share/DATA/re-analysis/ERA-Interim/

• • Global monthly files at 0.75° horizontal resolution and all time-steps: 00, 06, 12, 18 are labelled as:
    • ERAI_pl[YYYY][MM]_[var1]-[var2].grib: pressure levels variables (all levels). GRIB codes as:
      • 129: geopotential
      • 157: relative humidty
      • 130: Temperature
      • 131: u-wind
      • 132: v-wind
    • ERAI_sfc[YYYY][MM].grib: all surface levels variables (step 0)
  • To download data:
    • Generate files from ECMWF web-page ERA-Interim
    • Go to the folder in hydra

cd /share/DATA/re-analysis/ERA-Interim/

• • • get the file (as link from ECMWF web-page (right bottom on `Download grib'), as e.g.:

$ wget https://stream.ecmwf.int/data/atls05/data/data02/scratch/_mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib

• • • Rename file according to its content

$ mv _mars-atls05-a82bacafb5c306db76464bc7e824bb75-zn7P44.grib ERAI_sfc201302.grib

• ERA5
  • Thus, part of ERA5 forcings are:

/share/DATA/re-analysis/ERA5/

• 'NCEP-NNRP1'
  • Thus, full NCEP-NNRP1 forcings are (NOTE: no land data!!!):

/share/DATA/re-analysis/NCEP_NNRP/

Morphological forcings

Provide the geomorphological information for the domain of simulation: topography, land-use, vegetation-types, etc...

• In WRF there is a huge amount of data ans sources at different resolutions. At hydra everything is already there at:

/share/GEOG/

• They are ready to be use

GENERIC Model use

WRF has two main parts:

• WPS: Generation of the domain, initial and boundary condition: Runs 4 programs:
  • geogrid: domain generation
  • ungrib: unpack atmospheric forcing \verb+grib+ files
  • metgrid: horizontal interpolation of the unpacked atmospheric forcing files at the domain of simulation
  • real: generation of the initial and boundary conditions using \verb+metgrid+ output
• WRF: model it self: wrf.exe

At hydra all the code is already compiled at /opt/wrf (with folders WPS and WRFV3) As example WRFv4.3.3 compiled with intel-compilers with distributed and shared memory

/opt/wrf/WRF-4.3.3/intel/2021.4.0/dm+sm/WRF

WPS

Let's assume that we work in a folder called $WORKDIR (at users' ${HOME} at hydra) and we will work with a given WRF version located in $WRFversion. As example, let's create a two nested domain of 30 km for the entire South America and a second one of 4.285 km for Córdoba mountain ranges (see the WRF/namelist.wps)

WRF two domain configuration 30 and 4.285 km

geogrid

It is used to generate the domain

• Create a folder for the geogrid section

$ cd $WORKDIR
$ mkdir geogrid
$ cd geogrid

• Link and copy all the necessary files (do not forget to add opt_geogrid_tbl_path = './' into &geogrid section) in the namelist!!):

$ ln -s $WRFversion/WPS/geogrid/geogrid.exe ./
$ ln -s $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW ./GEOGRID.TBL
$ cp $WRFversion/WPS/geogrid/namelist.wps ./

• Domain configuration is done via namelist.wps (more information at: WPS user guide)

$ vim namelist.wps

In hydra, the path to the geographic dataa has to be defined in the namelist.wps as follows:

 geog_data_path = '/share/GEOG/'

Once it is defined, run it (all the necessary PBS script files are available in /share/WRF, you only need to change amount of processes and users's email):

$ cp /share/WRF/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs

• This will create the domain files, one for each domain.

geo_em.d[nn].nc

• Some variables from the geogrid files:
  • LANDMASK: sea-land mask
  • XLAT_M: latitude on mass point
  • XLONG_M: longitude on mass point
  • HGT_M: orographical height

Local Climate Zones (LCZs)

The LCZ classification scheme describes and characterizes the surface structure and surface cover of cities in 17 types of zones for urban studies in weather simulations (more information at: LCZs (Stewart & Oke, 2012). To use the LCZ data there are two options.

Using geogrid directly from WPS v4.5

The release of WRF v4.5 included a new geogrid table, GEOGRID.TBL.ARW_LCZ, that includes an entry for a global 100-m LCZ dataset. The WPS Geographical Input Data Mandatory for Specific Applications is already dowloaded in hydra in the same geog_data_path defined before.

NOTE: If you are not using hydra, you can dowload the data directly from the following link: CGLC-MODIS-LCZ_100m. Once the data is downloaded and placed into the geog folder, untar the file and it is ready for use.

• Go to the geogrid folder created before and copy the GEOGRID.TBL.ARW_LCZ.

$ cd $WORKDIR/WPS/geogrid
$ cp $WRFversion/WPS/geogrid/GEOGRID.TBL.ARW_LCZ ./GEOGRID.TBL

• Run geogrid.exe and you are ready to use the data.

NOTE: The use of LCZ data require the addition and change of some parameters in the WRF namelist, these changes will be descibed in the re

$ cp /share/WRF/launch_geogrid_intel.pbs ./
$ qsub launch_geogrid_intel.pbs

Using w2w

If you are not using WRF v4.5 or higher, or you want to use data from the World Urban Dataset and Access Portal Tools WUDAPT. There is an additional step you need to follow after running geogrid.exe without linking the GEOGRID.TBL.ARW_LCZ. w2w is a python tool to force the LCZ categorization into the geogrid resulting file. To compile w2w you need two input files, the geogrid resulting file geo_em.d[nn].nc and the LCZ Global Map from WUDAPT.

The LCZ Global Map covers the globe with 100m resolution and it is published in the open-access data through the following link in TIFF format. NOTE: The size of the file may cause some problems when running w2w, so it is recommended to crop the file around your study area before using w2w tool.

• Install the w2w tool:

$ pip install git+https://github.com/matthiasdemuzere/w2w

• Run w2w with the input files detailed above.

NOTE: Both the LCZ file and the geo_em file must be in the same input_directory

$ w2w input_directory LCZ_cropped_file.tif geo_em.d[nn].nc

• w2w creates four files saving the original geo_em.dXX.nc file:
  • geo_em.d[nn].nc: Original geo_em file
  • geo_em.d[nn]_NoUrban.nc: Urban areas are replaced with the dominant surrounding rural land use.
  • geo_em.d[nn]_LCZ_extent.nc: The extension of urban areas is updated using LCZ.
  • geo_em.d[nn]_LCZ_params.nc: Urban areas are described using LCZ categories.

NOTE: To run metgrid.exe using the geo_em.dXX_LCZ_params.nc file, you have to change the file name to the name of the original file, and continue with the following steps.

ungrib

Prepare and unpack grib files from the GCM forcing

NOTE: In case atmospheric forcing data is not in GRIB format, and it is in standard netCDF format, user will need to use the nc2wps tool. Which is already compiled in: /share/tools/RegIPSL/tools/nc2wps.

• Creation of the folder (from $WORKDIR)

$ mkdir ungrib 
$ cd ungrib

• Linking necessary files from compiled source

$ ln -s $WRFversion/WPS/ungrib/ungrib.exe ./
$ ln -s $WRFversion/WPS/link_grib.csh ./
$ cp ../geogrid/namelist.wps ./

• Edit the namelist namelist.wps to set up the right period of simulation and frequency of forcing (you can also use nano, emacs, ...)

$ vim namelist.wps

• Creation of a folder for the necessary GRIB files and linking the necessary files (eg. 4 files per month) from a folder with all the necesary data called $inDATA

$ mkdir GribDir
$ cd GribDir
$ ln -s $inDATA/*201212*.grib ./ 
$ cd ..

• Re-link files for WRF with its own script

./link_grib.csh GribDir/*

• Should appear:

$ ls GRIBFILE.AA*
GRIBFILE.AAA\  GRIBFILE.AAB  GRIBFILE.AAC  GRIBFILE.AAD

• We need to provide equivalences of the GRIB codes to the real variables. WRF comes with already defined GRIB equivalencies from different sources in folder Variable_Tables. In this example we use ECMWF ERA-Interim at pressure levels, thus we link

$ ln -s $WRFversion/WPS/ungrib/Variable_Tables/Vtable.ERA-interim.pl ./Vtable

• We need to take the domain file and get the right dates

$ cp ../geogrid/nameslist.wps ./

• Files can be unpacked using the PBS job file (user's email has to be changed)

$ cp /share/WRF/launch_ungrib_intel.pbs ./
$ qsub launch_ungrib_intel.pbs

• If everything went fine, should appear:

FILE:[YYYY]-[MM]-[DD]_[HH]

• And...

$ tail run_ungrib.log 
(...) 
********** 
Done deleting temporary files. 
********** 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
!  Successful completion of ungrib.   ! 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

metgrid

Horizontal interpolation of atmospheric forcing data at the domain of simulation

• Creation of the folder (from $WORKDIR)

$ mkdir metgrid 
$ cd metgrid

• Getting necessary files (do not forget opt_metgrid_tbl_path = './', into &metgrid section in namelist !! to avoid: ERROR: Could not open file METGRID.TBL)

$ ln -s $WRFversion/WPS/metgrid/metgrid.exe ./ 
$ ln -s $WRFversion/WPS/metgrid/METGRID.TBL.ARW ./METGRID.TBL

• Getting the ungrib output

$ ln -s ../ungrib/FILE* ./

• • Link the domains of simulation

$ ln -s ../geogrid/geo_em.d* ./

• • Link the namelist from ungrib (to make sure we are using the same!)

$ ln -s ../ungrib/namelist.wps ./

• Get the PBS (job queue script) to run the metrid.exe (remember to edit the user's email in the pbs job)

$ cp /share/WRF/launch_metgrid_intel.pbs ./
$ qsub launch_metgrid_intel.pbs

• If everything went fine one should have

met_em.d[nn].[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS].nc

• And...

$ tail run_metgrid.log
(...)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 
!  Successful completion of metgrid.  ! 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

real

Vertical interpolation of atmospheric forcing at the domain of simulation

• Creation of the folder (from $WORKDIR)

$ mkdir run 
$ cd run

• Link all the necessary files from WRF (it already links all the necessary to run the model)

$ ln -s $WRFversion/WRF/run/* ./

• Remove and copy the configuration file (namelist.input)

$ rm namelist.input 
$ cp $WRFversion/WRF/run/namelist.input ./

• Edit the file and prepare the configuration for the run (re-adapt domain, physics, dates, output....). See an example for the two nested domain here WRF/namelist.input

$ vim namelist.input

• Linking the metgrid generated files

$ ln -s ../metgrid/met_em.d*.nc ./

• Getting the PBS job script for real.exe

$ cp /share/WRF/run_real.pbs ./

• Getting a bash script to run executables in hydra (there is an issue with ulimit which does not allow to run with more than one node)

$ cp /share/WRF/launch_real_intel.pbs ./
$ qsub /share/WRF/launch_pbs.bash

• If everything went fine one should have (the basic ones):

wrfbdy_d[nn]  wrfinput_d[nn]  ...

• • wrfbdy_d01: Boundary conditions file (only for the first domain)
  • wrfinput_d[nn]: Initial conditions file for each domain
  • wrffdda_d[nn]: Nudging file [optional]
  • wrflowinp_d[nn]: File with updating (every time-step of the atmospheric forcing) surface characteristics [optional]
  • And...

$ tail outreal/rsl.error.0000
(...)
real_em: SUCCESS COMPLETE REAL_EM INIT

WRF

• Simulation (look on description of namelist for namelist configuration/specifications).

• Getting the necessary PBS job (same folder for real)

$ cp /share/WRF/launch_wrf_intel.pbs ./
qsub launch_wrf_intel.pbs

• If everything went fine one should have (the basic ones):

outwrf/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] outwrf/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]  ...

• • wrfout/wrfout_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]: simulation output (at η=(p-p_top)/(p_sfc-p_top) levels)
  • wrfout/wrfrst_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]: restart file (to continue simulation)
  • wrfout/wrfxtrm_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]: file with extremes from internal integration [optional]
  • wrfout/wrfpress_d[nn]_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]: file at vertical pressure levels [optional]
  • namelist.output_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]: All the parameters used for the simulation
  • stations: folder with the time-series files (tslist)
• And...

tail outwrf/rsl.error.0000
(...)
d01 [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] wrf: SUCCESS COMPLETE WRF

• While running one can check the status on regarding on the rsl.error.0000 file, e.g.:

$ tail rsl.error.0000
(...)
Timing for main: time [YYYY]-[MM]-[DD]_[HH]:[MI]:[SS] on domain  1:  3.86105 elapsed seconds

WRF: known errors

CFL

• But if something went wrong:
  • CFL: At wrfout/[InitialDATE]-[EndDATE]/ there are the files rsl.[error/out].[nnnn] (two per cpu). Use to appear SIGSEV segmentation fault, and to look different sources, usually cfl. e.g. (usually look for the largest rsl file (after the 0000); [$ ls -rS rsl.error.*])

$ wrfout/20121201000000-20121210000000/rsl.error.0009
(...) 
d01 2012-12-01_01:30:00  33 points exceeded cfl=2 in domain d01 at time 2012-12-01_01:30:00 hours 
d01 2012-12-01_01:30:00  MAX AT i,j,k:    100    94    21 vert_cfl,w,d(eta)=    5.897325    17.00738    3.2057911E-02
(...)

NO working restart

Since a given version there is a need to include a new namelist parameter (in &time_control section) [WRF restart] in order to make available the option to continue a simulation from a given restart

override_restart_timers                = .true.

If we want to get values at the output files at the time of the restart, you need to add at the (&time_control section) the parameter

write_hist_at_0h_rst                   = .true.

Additional information

For more additional information and further details visit WRFextras

WRF4L: Lluís' WRF work-flow management

For information about Lluí's WRF work-flow management visit here WRF4L

CDXWRF: WRF for CORDEX

A new module developed in CIMA to attain CORDEX variable demands visit here CDXWRF

WRFles

LES simulations with WRF model here WRFles