Lluis.fita: Página creada con «There is a far more powerful tool to manage work-flow of [http://www.meteo.unican.es/software/wrf4g/ WRF4G]. Lluís developed a less powerful one which is here desc...»

2025-07-31T08:53:47Z

Página creada con «There is a far more powerful tool to manage work-flow of [http://www.meteo.unican.es/software/wrf4g/ WRF4G]. Lluís developed a less powerful one which is here desc...»

Página nueva

There is a far more powerful tool to manage work-flow of [http://www.meteo.unican.es/software/wrf4g/ WRF4G].

Lluís developed a less powerful one which is here described

WRF work-flow management is done via 5 scripts (these are the specifics for hydra [CIMA cluster]):
* <code>EXPERIMENTparameters.txt</code>: General ASCII file which configures the experiment and chain of simulations (chunks). This is the unique file to modify
* <code>wrf4l_experiment.pbs</code>: PBS-queue job which prepares the experiment of the environment
* <code>wrf4l_WPS.pbs</code>: PBS-queue job which launch the WPS section of the model: <code>ungrib.exe</code>, <code>metgrid.exe</code>, <code>real.exe</code>
* <code>wrf4l_WRF.pbs</code>: PBS-queue job which launch the <code>wrf.exe</code>

* There is a folder called <code>components</code> with shell and python scripts necessary for the work-flow management

An experiment which contains a period of simulation is divided by '''chunks''' small pieces of times which are manageable by the model. The work-flow follows these steps using <code>run_experiments.pbs</code>:
# Copy and link all the required files for a given '''chunk''' of the whole period of simulation following the content of <code>EXPERIMENTparameters.txt</code>
# Launches <code>wrf4l_WPS.pbs</code> which will produce the necessary files for the period of the given '''chunk'''
# Launches <code>wrf4l_WRF.pbs</code> which will simulated the period of the given '''chunk''' (which waits until the end of <code>run_WPS.pbs</code>)
# Launches the next <code>wrf4l_experiments.pbs</code> (which waits until the end of <code>wrf4l_WRF.pbs</code>)

[[File:WRF4L_resize.png]]

All the scripts are located in <code>hydra</code> at:
<pre>
/share/tools/workflows/WRF4L/hydra
</pre>

== How to simulate ==
# Assuming that we have already defined the domains of simulation and that we have a suited <CODE>namelist.wps</CODE> and <CODE>namelist.input</CODE> WRF namelist files in a folder called <CODE>$GEOGRID</CODE>. We will gather all this information in the storage folder for the experiment (let's assume called $STORAGEdir) and we create there a folder for the information of the domains.
<PRE style"shell">
mkdir ${STORAGEdir}/domains
cp ${GEOGRID}/geo_em.d* ${STORAGEdir}/domains
cp ${GEOGRID}/namelist.wps ${STORAGEdir}/domains
cp ${GEOGRID}/namelist.input ${STORAGEdir}/domains
</PRE>

# We then move to the <CODE>${STORAGEdir}/domains</CODE> and create a template file for the <CODE>namelist.wps</CODE>
<PRE style="shell">
cd ${STORAGEdir}/domains
cp namelist.wps namelist.wps_template
diff namelist.wps namelist.wps_template
4,6c4,6
< start_date = '2010-06-01_00:00:00', '2010-06-01_00:00:00'
< end_date = '2010-06-16_00:00:00', '2010-06-16_00:00:00'
< interval_seconds = 10800,
---
> start_date = 'iyr-imo-ida_iho:imi:ise', 'iyr-imo-ida_iho:imi:ise',
> end_date = 'eyr-emo-eda_eho:emi:ese', 'eyr-emo-eda_eho:emi:ese',
> interval_seconds = infreq
</PRE>
# In case we are going to use <CODE>nc2wps</CODE> to generate the intermediate files (<CODE>ungrib.exe</CODE> output directly generated from netCDF files), we need to generate a template for the <CODE>namelist.nc2wps</CODE>. There are different options available in <CODE>/share/tools/workflows/WRF4L/hydra</CODE>. In this example we use the one prepared for ERA5 data as it is kept in Papa-Deimos:
<PRE style="shell">
cp /share/tools/workflows/WRF4L/hydra/namelist.nc2wps_template_ERA5 ${STORAGEdir}/domains/namelist.nc2wps_template
</PRE>

# Creation of a new folder from where launch the experiment [ExperimentName] (g.e. somewhere at <CODE>salidas/<CODE> folder)
<pre>
$ mkdir salidas/[ExperimentName]
cd salidas/[ExperimentName]
</pre>
# copy basic WRF4L files to this folder
<pre>
cp /share/tools/workflows/WRF4L/hydra_old/wrf4l_experiment.pbs ./
cp /share/tools/workflows/WRF4L/hydra_old/wrf4l_WPS.pbs ./
cp /share/tools/workflows/WRF4L/hydra_old/wrf4l_WRF.pbs ./
</pre>
# Edit the configuration/set-up of the simulation of the experiment (e.g. period of simulation, version of WPS and WRF to use, additional <CODE>namelist.input</CODE> parameters, multiple folders: <CODE>storageHOME</CODE>, <CODE>runHOME</CODE>, MPI configuration, ...)
<pre>
$ vim EXPERIMENTparameters.txt
</pre>
# Change the name of the user in the <CODE>wrf4l_experiment.pbs</CODE>
<PRE style="shell">
diff wrf4l_experiment.pbs /share/tools/workflows/WRF4L/hydra_old/wrf4l_experiment.pbs
11c11
< #PBS -M lluis.fita@cima.fcen.uba.ar
---
> #PBS -M [user]@cima.fcen.uba.ar
</PRE>

# Launch the experiment
<pre>
$ qsub wrf4l_experiment.pbs
</pre>

When it is running one would have (runnig WRF job <code>wrf_[SimName]</code> `R', and <code>exp_[SimName]</code> in hold `H'):
<pre>
$ qstat -u $USER

hydra:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
-------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
397.hydra lluis.fi larga wps_ 27567 1 16 20gb 168:0 R --
398.hydra lluis.fi larga wrf_ -- 1 1 20gb 168:0 H --
399.hydra lluis.fi larga exp_ -- 1 1 2gb 168:0 H --
</pre>

In case of crash of the simulation, after fixing the issue, go to <code>[runHOME]/[ExpName]/[SimName]</code> and re-launch the experiment (after the first run the <code>scratch</code> is switched automatically to `false')
<pre>
$ qsub wrf4l_experiment.pbs
</pre>

== Checking the experiment ==
Once the experiment runs, one needs to look on (following name of the variables from <code>EXPERIMENTparameters.txt</code>
* <code>[runHOME]/[ExpName]/[SimName]</code>: Will content the copies of the templates <code>namelist.wps</code>, <code>namelist.input</code> and a file <code>chunk_attemps.inf</code> which counts how many times a '''chunk''' has been attempted to be run (if it reached 4 times, the <code>WRF4L</code> is stopped)
* <code>[runHOME]/[ExpName]/[SimName]/run</code>: actual folder where the computing nodes run the model. In a folder called <code>wrfout</code> there is a folder for each '''chunk''' with the standard output of the model
* <code>[runHOME]/[ExpName]/[SimName]/run/outwrf/[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf]</code>: folder with the standard output and all the required files to run a given '''chunk'''. The content of all this folder is compressed and kept in <code>[storageHOME]/[ExpName]/[SimName]/config_[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf].tar.gz</code>
* <code>[storageHOME]/[ExpName]/[SimName]</code> (in [storageHOST]): output of the already ran '''chunks''' as <code>[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf]</code> for a chunk from <code>[YYYYi]/[MMi]/[DDi] [HHi]:[MIi]:[SSi]</code> to <code>[YYYYf]/[MMf]/[DDf] [HHf]:[MIf]:[SSf]</code>

=== When something went wrong ===
If there has been any problem check the last chunk (in outwrf/[PERIODchunk]) to try to understand what happens and where the problem comes from:
* <code>rsl.[error/out].[nnnn]</code>: These are the files which content the standard output while running the model. One file for each process. If the problem was something related to model execution and it has been prepared for the error, a correct message must appear. (look first for the largest files...
<pre>
$ ls -lrS rsl.error.*
</pre>
* <code>run_wrf.log</code>: These are the files which content the standard output of the model. Search for `segmentation faults' in form of (it might differ):
<pre>forrtl: error (63): output conversion error, unit -5, file Internal Formatted Write
Image PC Routine Line Source
wrf.exe 00000000032B736A Unknown Unknown Unknown
wrf.exe 00000000032B5EE5 Unknown Unknown Unknown
wrf.exe 0000000003265966 Unknown Unknown Unknown
wrf.exe 0000000003226EB5 Unknown Unknown Unknown
wrf.exe 0000000003226671 Unknown Unknown Unknown
wrf.exe 000000000324BC3C Unknown Unknown Unknown
wrf.exe 0000000003249C94 Unknown Unknown Unknown
wrf.exe 00000000004184DC Unknown Unknown Unknown
libc.so.6 000000319021ECDD Unknown Unknown Unknown
wrf.exe 00000000004183D9 Unknown Unknown Unknown
</pre>
* on <code>[runHOME]/[ExpName]/[SimName]</code>, check the output of the PBS jobs. Which are called:
** <code>exp-[SimName].o[nnnn]</code>: output of the <code>wrf4l_experiment.pbs</code>
** <code>wps-[SimName].o[nnnn]</code>: output of the <code>wrf4l_WPS.pbs</code>
** <code>wrf-[SimName].o[nnnn]</code>: output of the <code>wrf4l_WRF.pbs</code>
* Check <code>[runHOME]/[ExpName]/[SimName]/run/namelist.output</code> which holds all the parameters (even the default ones) used in the simulation

== EXPERIMENTSparameters.txt ==
This ASCII file configures all the simulation. It assumes:
* Required files, forcings, storage, compiled version of the code might be at different machines.
* There is a folder with a given template version of the <code>namelist.input</code> which will be used and changed accordingly to the requirement of the experiments

Location of the WRF4L main folder (example for <code>hydra</code>)
<pre>
# Home of the WRF4L
wrf4lHOME=/share/workflows/WRF4L
</pre>

Name of the machine where the experiment is running ('''NOTE:''' a folder with specific work-flow must exist as <code>$wrf4lHOME/${HPC}</code>)
<pre>
# Machine specific work-flow files for $HPC (a folder with specific work-flow must exist as $wrf4lHOME/${HPC})
HPC=hydra
</pre>

Name of the compiler used to compile the model ('''NOTE:''' a file called <code>$wrf4lHOME/arch/${HPC}_${compiler}.env</code> must exist)
<pre>
# Compilation (a file called $wrf4lHOME/arch/${HPC}_${compiler}.env must exist)
compiler=intel
</pre>

Name of the experiment
<pre>
# Experiment name
ExpName = WRFsensSFC
</pre>

Name of the simulation. Here is understood that a given experiment could have the model configured with different set-ups (here identified with a different name of simulation)
<pre>
# Simulation name
SimName = control
</pre>

Which binary of <code>python</code> 2.x to be used
<pre>
# python binary
pyBIN=/home/lluis.fita/bin/anaconda2/bin/python2.7
</pre>

Should this simulation be run from the beginning or not. If it is set to `true', it will remove all the pre-existing content of the folder [ExpName]/[SimName] in the running and in the storage spaces. '''Be careful'''. In case of `false' simulation will continue from the last successful ran '''chunk''' (checking the restart files).
<pre>
# Start from the beginning (keeping folder structure)
scratch = false
</pre>

Period of the simulation of the simulation (In this example from 1958 Jan 1st to 2015 Dec 31)
<pre>
# Experiment starting date
exp_start_date = 19790101000000
# Experiment ending date
exp_end_date = 20150101000000
</pre>

Length of the chunks (do not make chunks larger than 1-month!!)
<pre>
# Chunk Length [N]@[unit]
# [unit]=[year, month, week, day, hour, minute, second]
chunk_length = 1@month
</pre>

Selection of the machines and users to each machine where the different requirement files are located and the output should be placed.
* '''NOTE:''' this will only work if one set-up the <code>.ssh</code> public/private keys in each involved USER/HOST.
* '''NOTE 2:''' All the forcings, compiled code, ... are already at <code>hydra</code> at the common space called <code>share</code>
* '''NOTE 3:''' From the computing nodes, one can not access to the <code>/share</code> folder and to any of the CIMA's storage machines: skogul, freyja, ... For that reason, one need to use these system of <code>[USER]@[HOST]</code> accounts. <code>*.pbs</code> scripts uses a series of wrappers of the standard functions: <code>cp, ln, ls, mv, ....</code> which manage them `from' and `to' different pairs of <code>[USER]@[HOST]</code>. '''NOTE:''' This will only work if the public/private ssh key pairs have been set-up (see more details at [[llaves_ssh]])
<pre>
# Hosts
# list of different hosts and specific user
# [USER]@[HOST]
# NOTE: this will only work if public keys have been set-up
##
# Host with compiled code, namelist templates
codeHOST=lluis.fita@hydra
# forcing Host with forcings (atmospherics and morphologicals)
forcingHOST=lluis.fita@hydra
# output Host with storage of output (including restarts)
outHOST=lluis.fita@hydra
</pre>

Templates of the configuration of WRF: <code>namelist.wps</code>, <code>namelist.input</code> files. '''NOTE:''' they will be changed according to the content of <code>EXPERIMENTparameters.txt</code> like period of the '''chunk''', atmospheric forcing, differences of the set-up, ... (located in the <code>[codeHOST]</code>
<pre>
# Folder with the `namelist.wps', `namelist.input' and `geo_em.d[nn].nc' of the experiment
domainHOME = /home/lluis.fita/salidas/estudios/dominmios/SA50k
</pre>

Folder where the WRF model will run in the computing nodes (on top of that there will be two more folders [ExpName]/[SimName]). WRF will run at the folder [ExpName]/[SimName]/run
<pre>
# Running folder
runHOME = /home/lluis.fita/estudios/WRFsensSFC/sims
</pre>

Folder with the compiled version of the WPS (located at <code>[codeHOST]</code>)
<pre>
# Folder with the compiled source of WPS
wpsHOME = /share/WRF/WRFV3.9.1/ifort/dmpar/WPS
</pre>

Folder with the compiled version of the WRF (located at <code>[codeHOST]</code>)
<pre>
# Folder with the compiled source of WRF
wrfHOME = /share/WRF/WRFV3.9.1/ifort/dmpar/WRFV3
</pre>

Folder to storage all the output of the model (history files, restarts and compressed file with content of the configuration and the standard output of the given run). The content of the folder will be organized by chunks (located at <code>[storageHOST]</code>)
<pre>
# Storage folder of the output
storageHOME = /home/lluis.fita/salidas/estudios/WRFsensSFC/sims/output
</pre>

Wether modules should be load (not used for <code>hydra</code>)
<pre>
# Modules to load ('None' for any)
modulesLOAD = None
</pre>

Names of the files used to check that the '''chunk''' has properly ran
<pre>
# Model reference output names (to be used as checking file names)
nameLISTfile = namelist.input # namelist
nameRSTfile = wrfrst_d01_ # restart file
nameOUTfile = wfrout_d01_ # output file
</pre>

Extensions of the files with the configuration of WRF (to be retrieved from <code>codeHOST</code> and <code>domainHOME</code>)
<pre>
# Extensions of the files with the configuration of the model
configEXTS = wps:input
</pre>

To continue from a previous '''chunk''' one needs to use the `restart' files. But they need to be renamed, because otherwise they will be re-written. Here one specifies the original name of the file <code>[origFile]</code> and the name to be used to avoid the re-writting <code>[destFile]</code>. It uses a complex bash script which even can deal with the change of dates according to the period of the '''chunk''' (':' list of <code>[origFile]@[destFile]</code>). They will located at the <code>[storageHOST]</code>
<pre>
# restart file names
# ':' list of [tmplrstfilen|[NNNNN1]?[val1]#[...[NNNNNn]?[valn]]@[tmpllinkname]|[NNNNN1]?[val1]#[...[NNNNNn]?[valn]]
# [tmplrstfilen]: template name of the restart file (if necessary with [NNNNN] variables to be substituted)
# [NNNNN]: section of the file name to be automatically substituted
# `[YYYY]': year in 4 digits
# `[YY]': year in 2 digits
# `[MM]': month in 2 digits
# `[DD]': day in 2 digits
# `[HH]': hour in 2 digits
# `[SS]': second in 2 digits
# `[JJJ]': julian day in 3 digits
# [val]: value to use (which is systematically defined in `run_OR.pbs')
# `%Y%': year in 4 digits
# `%y%': year in 2 digits
# `%m%': month in 2 digits
# `%d%': day in 2 digits
# `%h%': hour in 2 digits
# `%s%': second in 2 digits
# `%j%': julian day in 3 digits
# [tmpllinkname]: template name of the link of the restart file (if necessary with [NNNNN] variables to be substituted)
rstFILES=wrfrst_d01_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]|YYYY?%Y#MM?%m#DD?%d#HH?%H#MI?%M#SS?%S@wrfrst_d01_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]|YYYY?%Y#MM?%m#DD?%d#HH?%H#MI?%M#SS?%S
</pre>

Folder with the input data (located at <code>[forcingHOST]</code>).
<pre>
# Folder with the input morphological forcing data
indataHOME = /share/DATA/re-analysis/ERA-Interim
</pre>

Format of the input data and name of files
<pre>
# Data format (grib, nc)
indataFMT= grib
# For `grib' format
# Head and tail of indata files names.
# Assuming ${indataFheader}*[YYYY][MM]*${indataFtail}.[grib/nc]
indataFheader=ERAI_
indataFtail=
</pre>

In case of netCDF input data, there is a bash script which transforms the data to grib, to be used later by <code>ungrib</code>

Variable table to use in <code>ungrib</code>
<pre>
# Type of Vtable for ungrib as Vtable.[VtableType]
VtableType=ERA-interim.pl
</pre>

Folder with the atmospheric forcing data (located at <code>[forcingHOST]</code>).
<pre>
# For `nc' format
# Folder which contents the atmospheric data to generate the initial state
iniatmosHOME = ./
# Type of atmospheric data to generate the initial state
# `ECMWFstd': ECMWF 'standard' way ERAI_[pl/sfc][YYYY][MM]_[var1]-[var2].grib
# `ERAI-IPSL': ECMWF ERA-INTERIM stored in the common IPSL way (.../4xdaily/[AN\_PL/AN\_SF])
iniatmosTYPE = 'ECMWFstd'
</pre>

Here on can change values on the template <code>namelist.input</code>. It will change the values of the provided parameters with a new value. If the given parameter is not in the template of the <code>namelist.input</code> it will be automatically added.
<pre>
## Namelist changes
nlparameters = ra_sw_physics;4,ra_lw_physics;4,time_step;180
</pre>

Name of WRF's executable (to be localized at <code>[orHOME]</code> folder from <code>[codeHOST]</code>)
<pre>
# Name of the exectuable
nameEXEC=wrf.exe
</pre>

':' separated list of netCDF file names from WRF's output which do not need to be kept
<pre>
# netCDF Files which will not be kept anywhere
NokeptfileNAMES=''
</pre>

':' separated list of headers of netCDF file names from WRF's output which need to be kept
<pre>
# Headers of netCDF files need to be kept
HkeptfileNAMES=wrfout_d:wrfxtrm_d:wrfpress_d:wrfcdx_d
</pre>

':' separated list of headers of restarts netCDF file names from WRF's output which need to be kept
<pre>
# Headers of netCDF restart files need to be kept
HrstfileNAMES=wrfrst_d
</pre>

Parallel configuration of the run.
<pre>
# WRF parallel run configuration
## Number of nodes
Nnodes = 1
## Number of mpi procs
Nmpiprocs = 16
## Number of shared memory threads ('None' for no openMP threads)
Nopenthreads = None
## Memory size of shared memory threads
SIZEopenthreads = 200M
## Memory for PBS jobs
MEMjobs = 30gb
</pre>

Generic definitions
<pre>
## Generic
errormsg=ERROR -- error -- ERROR -- error
warnmsg=WARNING -- warning -- WARNING -- warning
</pre>

WRF4Lold - Historial de revisiones

Lluis.fita: Página creada con «There is a far more powerful tool to manage work-flow of [http://www.meteo.unican.es/software/wrf4g/ WRF4G]. Lluís developed a less powerful one which is here desc...»