Ver el código fuente de «WRF4L»

There is a far more powerful tool to manage work-flow of [http://www.meteo.unican.es/software/wrf4g/ WRF4G]. 

Llu&iacute;s developed a less powerful one which is here described

WRF work-flow management is done via 5 scripts (these are the specifics for hydra [CIMA cluster]):
* <code>EXPERIMENTparameters.txt</code>: General ASCII file which configures the experiment and chain of simulations (chunks). This is the unique file to modify
* <code>wrf4l_experiment.pbs</code>: PBS-queue job which prepares the experiment of the environment
* <code>wrf4l_WPS.pbs</code>: PBS-queue job which launch the WPS section of the model: <code>ungrib.exe</code>, <code>metgrid.exe</code>, <code>real.exe</code>
* <code>wrf4l_WRF.pbs</code>: PBS-queue job which launch the <code>wrf.exe</code>
<!---* <code>launch_pbs.bash</code>: Necessary shell script to launch jobs which use more than one node in CIMA's <code>hydra</code> cluster-->
* There is a folder called <code>components</code> with shell and python scripts necessary for the work-flow management

An experiment which contains a period of simulation is divided by '''chunks''' small pieces of times which are manageable by the model. The work-flow follows these steps using <code>run_experiments.pbs</code>:
# Copy and link all the required files for a given '''chunk''' of the whole period of simulation following the content of <code>EXPERIMENTparameters.txt</code>
# Launches <code>wrf4l_WPS.pbs</code> which will produce the necessary files for the period of the given '''chunk'''
# Launches <code>wrf4l_WRF.pbs</code> which will simulated the period of the given '''chunk''' (which waits until the end of <code>run_WPS.pbs</code>)
# Launches the next <code>wrf4l_experiments.pbs</code> (which waits until the end of <code>wrf4l_WRF.pbs</code>)

[[File:WRF4L.png]]

All the scripts are located in <code>hydra</code> at:
<pre>
/share/tools/workflows/WRF4L/hydra
</pre>

== How to simulate ==
# Creation of a new folder from where launch the experiment [ExperimentName] (g.e. somewhere at $HOME)
<pre>
$ mkdir [ExperimentName]
cd [ExperimentName]
</pre>
# copy the WRF4L files to this folder
<pre>
$ cp /share/tools/workflows/WRF4L/hydra/EXPERIMENTparameters.txt ./
$ cp /share/tools/workflows/WRF4L/hydra/wrf4l_experiment.pbs ./
$ cp /share/tools/workflows/WRF4L/hydra/wrf4l_WPS.pbs ./
$ cp /share/tools/workflows/WRF4L/hydra/wrf4l_WRF.pbs ./
</pre>
# Edit the configuration/set-up of the simulation of the experiment
<pre>
$ vim EXPERIMENTparameters.txt
</pre>
# Launch the simulation of the experiment
<pre>
$ qsub wrf4l_experiment.pbs
</pre>
When it is running one would have (runnig WRF job <code>wrf_[SimName]</code> `R', and <code>exp_[SimName]</code> in hold `H'):
<pre>
$ qstat -u $USER

hydra: 
                                                                         Req'd  Req'd   Elap
Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time
-------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
397.hydra            lluis.fi larga    wps_              27567     1  16   20gb 168:0 R   -- 
398.hydra            lluis.fi larga    wrf_                --      1   1   20gb 168:0 H   -- 
399.hydra            lluis.fi larga    exp_                --      1   1    2gb 168:0 H   -- 
</pre>

In case of crash of the simulation, after fixing the issue, go to <code>[runHOME]/[ExpName]/[SimName]</code> and re-launch the experiment (after the first run the <code>scratch</code> is switched automatically to `false')
<pre>
$ qsub wrf4l_experiment.pbs
</pre>

== Checking the experiment ==
Once the experiment runs, one needs to look on (following name of the variables from <code>EXPERIMENTparameters.txt</code>
* <code>[runHOME]/[ExpName]/[SimName]</code>: Will content the copies of the templates <code>namelist.wps</code>, <code>namelist.input</code> and a file <code>chunk_attemps.inf</code> which counts how many times a '''chunk''' has been attempted to be run (if it reached 4 times, the <code>WRF4L</code> is stopped)
* <code>[runHOME]/[ExpName]/[SimName]/run</code>: actual folder where the computing nodes run the model. In a folder called <code>wrfout</code> there is a folder for each '''chunk''' with the standard output of the model
* <code>[runHOME]/[ExpName]/[SimName]/run/outwrf/[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf]</code>: folder with the standard output and all the required files to run a given '''chunk'''. The content of all this folder is compressed and kept in  <code>[storageHOME]/[ExpName]/[SimName]/config_[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf].tar.gz</code>
* <code>[storageHOME]/[ExpName]/[SimName]</code> (in [storageHOST]): output of the already ran '''chunks''' as <code>[YYYYi][MMi][DDi][HHi][MIi][SSi]-[YYYYf][MMf][DDf][HHf][MIf][SSf]</code> for a chunk from <code>[YYYYi]/[MMi]/[DDi] [HHi]:[MIi]:[SSi]</code> to <code>[YYYYf]/[MMf]/[DDf] [HHf]:[MIf]:[SSf]</code>

=== When something went wrong ===
If there has been any problem check the last chunk (in outwrf/[PERIODchunk]) to try to understand what happens and where the problem comes from:
* <code>rsl.[error/out].[nnnn]</code>: These are the files which content the standard output while running the model. One file for each process. If the problem was something related to model execution and it has been prepared for the error, a correct message must appear. (look first for the largest files... 
<pre>
$ ls -lrS rsl.error.*
</pre>
* <code>run_wrf.log</code>: These are the files which content the standard output of the model. Search for `segmentation faults' in form of (it might differ):
<pre>forrtl: error (63): output conversion error, unit -5, file Internal Formatted Write
Image              PC                Routine            Line        Source
wrf.exe            00000000032B736A  Unknown               Unknown  Unknown
wrf.exe            00000000032B5EE5  Unknown               Unknown  Unknown
wrf.exe            0000000003265966  Unknown               Unknown  Unknown
wrf.exe            0000000003226EB5  Unknown               Unknown  Unknown
wrf.exe            0000000003226671  Unknown               Unknown  Unknown
wrf.exe            000000000324BC3C  Unknown               Unknown  Unknown
wrf.exe            0000000003249C94  Unknown               Unknown  Unknown
wrf.exe            00000000004184DC  Unknown               Unknown  Unknown
libc.so.6          000000319021ECDD  Unknown               Unknown  Unknown
wrf.exe            00000000004183D9  Unknown               Unknown  Unknown
</pre>
* on <code>[runHOME]/[ExpName]/[SimName]</code>, check the output of the PBS jobs. Which are called:
** <code>exp-[SimName].o[nnnn]</code>: output of the <code>wrf4l_experiment.pbs</code>
** <code>wps-[SimName].o[nnnn]</code>: output of the <code>wrf4l_WPS.pbs</code>
** <code>wrf-[SimName].o[nnnn]</code>: output of the <code>wrf4l_WRF.pbs</code>
* Check <code>[runHOME]/[ExpName]/[SimName]/run/namelist.output</code> which holds all the parameters (even the default ones) used in the simulation

== EXPERIMENTSparameters.txt ==
This ASCII file configures all the simulation. It assumes:
* Required files, forcings, storage, compiled version of the code might be at different machines.
* There is a folder with a given template version of the <code>namelist.input</code> which will be used and changed accordingly to the requirement of the experiments

Location of the WRF4L main folder (example for <code>hydra</code>)
<pre>
# Home of the WRF4L
wrf4lHOME=/share/workflows/WRF4L
</pre>

Name of the machine where the experiment is running ('''NOTE:''' a folder with specific work-flow must exist as <code>$wrf4lHOME/${HPC}</code>)
<pre>
# Machine specific work-flow files for $HPC (a folder with specific work-flow must exist as $wrf4lHOME/${HPC})
HPC=hydra
</pre>

Name of the compiler used to compile the model ('''NOTE:''' a file called <code>$wrf4lHOME/arch/${HPC}_${compiler}.env</code> must exist)
<pre>
# Compilation (a file called $wrf4lHOME/arch/${HPC}_${compiler}.env must exist)
compiler=intel
</pre>

Name of the experiment
<pre>
# Experiment name
ExpName = WRFsensSFC
</pre>

Name of the simulation. Here is understood that a given experiment could have the model configured with different set-ups (here identified with a different name of simulation)
<pre>
# Simulation name
SimName = control
</pre>

Which binary of <code>python</code> 2.x to be used
<pre>
# python binary
pyBIN=/home/lluis.fita/bin/anaconda2/bin/python2.7
</pre>

Should this simulation be run from the beginning or not. If it is set to `true', it will remove all the pre-existing content of the folder [ExpName]/[SimName] in the running and in the storage spaces. '''Be careful'''. In case of `false' simulation will continue from the last successful ran '''chunk''' (checking the restart files).
<pre>
# Start from the beginning (keeping folder structure)
scratch = false
</pre>

Period of the simulation of the simulation (In this example from 1958 Jan 1st to 2015 Dec 31)
<pre>
# Experiment starting date
exp_start_date = 19790101000000
# Experiment ending date
exp_end_date = 20150101000000
</pre>

Length of the chunks (do not make chunks larger than 1-month!!)
<pre>
# Chunk Length [N]@[unit]
#  [unit]=[year, month, week, day, hour, minute, second]
chunk_length = 1@month
</pre>

Selection of the machines and users to each machine where the different requirement files are located and the output should be placed. 
* '''NOTE:''' this will only work if one set-up the <code>.ssh</code> public/private keys in each involved USER/HOST. 
* '''NOTE 2:''' All the forcings, compiled code, ... are already at <code>hydra</code> at the common space called <code>share</code>
* '''NOTE 3:''' From the computing nodes, one can not access to the <code>/share</code> folder and to any of the CIMA's storage machines: skogul, freyja, ... For that reason, one need to use these system of <code>[USER]@[HOST]</code> accounts. <code>*.pbs</code> scripts uses a series of wrappers of the standard functions: <code>cp, ln, ls, mv, ....</code> which manage them `from' and `to' different pairs of <code>[USER]@[HOST]</code>. '''NOTE:''' This will only work if the public/private ssh key pairs have been set-up (see more details at [[llaves_ssh]])
<pre>
# Hosts
#   list of different hosts and specific user
#     [USER]@[HOST]
#   NOTE: this will only work if public keys have been set-up
##
# Host with compiled code, namelist templates
codeHOST=lluis.fita@hydra
# forcing Host with forcings (atmospherics and morphologicals)
forcingHOST=lluis.fita@hydra
# output Host with storage of output (including restarts)
outHOST=lluis.fita@hydra
</pre>

Templates of the configuration of WRF: <code>namelist.wps</code>, <code>namelist.input</code> files. '''NOTE:''' they will be changed according to the content of <code>EXPERIMENTparameters.txt</code> like period of the '''chunk''', atmospheric forcing, differences of the set-up, ... (located in the <code>[codeHOST]</code>
<pre>
# Folder with the `namelist.wps', `namelist.input' and `geo_em.d[nn].nc' of the experiment
domainHOME = /home/lluis.fita/salidas/estudios/dominmios/SA50k
</pre>

Folder where the WRF model will run in the computing nodes (on top of that there will be two more folders [ExpName]/[SimName]). WRF will run at the folder [ExpName]/[SimName]/run
<pre>
# Running folder
runHOME = /home/lluis.fita/estudios/WRFsensSFC/sims
</pre>

Folder with the compiled version of the WPS (located at <code>[codeHOST]</code>)
<pre>
# Folder with the compiled source of WPS
wpsHOME = /share/WRF/WRFV3.9.1/ifort/dmpar/WPS
</pre>

Folder with the compiled version of the WRF (located at <code>[codeHOST]</code>)
<pre>
# Folder with the compiled source of WRF
wrfHOME = /share/WRF/WRFV3.9.1/ifort/dmpar/WRFV3
</pre>

Folder to storage all the output of the model (history files, restarts and compressed file with content of the configuration and the standard output of the given run). The content of the folder will be organized by chunks (located at <code>[storageHOST]</code>)
<pre>
# Storage folder of the output
storageHOME = /home/lluis.fita/salidas/estudios/WRFsensSFC/sims/output
</pre>

Wether modules should be load (not used for <code>hydra</code>)
<pre>
# Modules to load ('None' for any)
modulesLOAD = None
</pre>

Names of the files used to check that the '''chunk''' has properly ran 
<pre>
# Model reference output names (to be used as checking file names)
nameLISTfile = namelist.input # namelist
nameRSTfile = wrfrst_d01_ # restart file
nameOUTfile = wfrout_d01_ # output file
</pre>

Extensions of the files with the configuration of WRF (to be retrieved from <code>codeHOST</code> and <code>domainHOME</code>)
<pre>
# Extensions of the files with the configuration of the model
configEXTS = wps:input
</pre>

To continue from a previous '''chunk''' one needs to use the `restart' files. But they need to be renamed, because otherwise they will be re-written. Here one specifies the original name of the file <code>[origFile]</code> and the name to be used to avoid the re-writting <code>[destFile]</code>. It uses a complex bash script which even can deal with the change of dates according to the period of the '''chunk''' (':' list of <code>[origFile]@[destFile]</code>). They will located at the <code>[storageHOST]</code>
<pre>
# restart file names
# ':' list of [tmplrstfilen|[NNNNN1]?[val1]#[...[NNNNNn]?[valn]]@[tmpllinkname]|[NNNNN1]?[val1]#[...[NNNNNn]?[valn]]
#    [tmplrstfilen]: template name of the restart file (if necessary with [NNNNN] variables to be substituted)
#      [NNNNN]: section of the file name to be automatically substituted
#        `[YYYY]': year in 4 digits
#        `[YY]': year in 2 digits
#        `[MM]': month in 2 digits
#        `[DD]': day in 2 digits
#        `[HH]': hour in 2 digits
#        `[SS]': second in 2 digits
#        `[JJJ]': julian day in 3 digits
#      [val]: value to use (which is systematically defined in `run_OR.pbs')
#        `%Y%': year in 4 digits
#        `%y%': year in 2 digits
#        `%m%': month in 2 digits
#        `%d%': day in 2 digits
#        `%h%': hour in 2 digits
#        `%s%': second in 2 digits
#        `%j%': julian day in 3 digits
#    [tmpllinkname]: template name of the link of the restart file (if necessary with [NNNNN] variables to be substituted)
rstFILES=wrfrst_d01_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]|YYYY?%Y#MM?%m#DD?%d#HH?%H#MI?%M#SS?%S@wrfrst_d01_[YYYY]-[MM]-[DD]_[HH]:[MI]:[SS]|YYYY?%Y#MM?%m#DD?%d#HH?%H#MI?%M#SS?%S
</pre>

Folder with the input data (located at <code>[forcingHOST]</code>). 
<pre>
# Folder with the input morphological forcing data
indataHOME = /share/DATA/re-analysis/ERA-Interim
</pre>

Format of the input data and name of files
<pre>
# Data format (grib, nc)
indataFMT= grib
# For `grib' format
#   Head and tail of indata files names.
#     Assuming ${indataFheader}*[YYYY][MM]*${indataFtail}.[grib/nc]
indataFheader=ERAI_
indataFtail=
</pre>

In case of netCDF input data, there is a bash script which transforms the data to grib, to be used later by <code>ungrib</code>

Variable table to use in <code>ungrib</code>
<pre>
#   Type of Vtable for ungrib as Vtable.[VtableType]
VtableType=ERA-interim.pl
</pre>

Folder with the atmospheric forcing data (located at <code>[forcingHOST]</code>). 
<pre>
# For `nc' format
#   Folder which contents the atmospheric data to generate the initial state
iniatmosHOME = ./
#   Type of atmospheric data to generate the initial state
#     `ECMWFstd': ECMWF 'standard' way ERAI_[pl/sfc][YYYY][MM]_[var1]-[var2].grib
#     `ERAI-IPSL': ECMWF ERA-INTERIM stored in the common IPSL way (.../4xdaily/[AN\_PL/AN\_SF])
iniatmosTYPE = 'ECMWFstd'
</pre>

Here on can change values on the template <code>namelist.input</code>. It will change the values of the provided parameters with a new value. If the given parameter is not in the template of the <code>namelist.input</code> it will be automatically added.
<pre>
## Namelist changes
nlparameters = ra_sw_physics;4,ra_lw_physics;4,time_step;180
</pre>

Name of WRF's executable (to be localized at <code>[orHOME]</code> folder from <code>[codeHOST]</code>)
<pre>
# Name of the exectuable
nameEXEC=wrf.exe
</pre>

':' separated list of netCDF file names from WRF's output which do not need to be kept
<pre>
# netCDF Files which will not be kept anywhere
NokeptfileNAMES=''
</pre>

':' separated list of headers of netCDF file names from WRF's output which need to be kept
<pre>
# Headers of netCDF files need to be kept
HkeptfileNAMES=wrfout_d:wrfxtrm_d:wrfpress_d:wrfcdx_d
</pre>

':' separated list of headers of restarts netCDF file names from WRF's output which need to be kept
<pre>
# Headers of netCDF restart files need to be kept
HrstfileNAMES=wrfrst_d
</pre>

Parallel configuration of the run.
<pre>
# WRF parallel run configuration
## Number of nodes
Nnodes = 1
## Number of mpi procs
Nmpiprocs = 16
## Number of shared memory threads ('None' for no openMP threads)
Nopenthreads = None
## Memory size of shared memory threads
SIZEopenthreads = 200M
## Memory for PBS jobs
MEMjobs = 30gb
</pre>

Generic definitions
<pre>
## Generic
errormsg=ERROR -- error -- ERROR -- error
warnmsg=WARNING -- warning -- WARNING -- warning
</pre>