WRFles

Here are described the steps to perform a LES test simulation using WRF in CIMA’s HPC resource called `hydra'

Initial similar pdf version of the same information is available here Archivo:WRFles.pdf

Compilation

Here are described the steps followed to compile LES test WRF code in hydra

NOTE: There is no need to re-compile the model. It is only provided for completeness !

1. First we need to specify a location for the compilation of the code:

/share/WRF/WRFgit/WRFlesideal/ifort/dmpar

1. From that location in hydra, download the source code of the WRF model (since V4.0 there is not provided anymore a compressed file with the code)

$ git clone https://github.com/wrf-model/WRF

1. In order to compile, first it is required to define some environmental variables into the system

$ export NETCDF=’/usr/local’
$ export WRFIO_NCD_LARGE_FILE_SUPPORT=1

1. Then prepare the file with the specific configuration for the compilation (in this case using intel compiler in `dmpar' mode [distributed memory only] and no nesting 1 )

configure.wrf (13. (serial) 14. (smpar) 15. (dmpar) 16. (dm+sm) INTEL (ifort/icc)) [no nesting]

1. Compile te LES test case

$ ./compile em_les >& compile.log

1. Verify that there was not any error during compilation (should be empty)

$ cat -n compile.log | grep Error

Use

Here are described the different steps to perform the WRF’s LES test simulation. This run test is done with standard original WRF’s LES ideal simulation (see details in table 1).

1. Simulation will be done in a different folder than the one where the compilation was made:

/home/lluis.fita/estudios/LES/test

1. Link the compiled and necessary files

$ ln -s /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/run/* ./

1. Remove the namelist file in order to do not modify WRF's original namelist in case we want to modify something

$ rm namelist.input
$ cp /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/namelist.input ./

1. Get the PBS jobs and scripts to run the simulation

$ cp /share/WRF/run_ideal.pbs ./
$ cp /share/WRF/launch_pbs.bash ./

1. Generate the initial conditions specific for the case (in this case using the default input_sounding (see /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/README.les for more details and options)

$ qsub run_ideal.pbs

1. Verify job is running propertly (it is very short, it last 18 seconds!)

$ qstat
Job id Name User Time Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8498.hydra WRF4L_ideal lluis.fita 00:00:12 R larga

1. At the end one should have inside realout/ folder:

$ ls -lrt realout/
total 3744
-rw-r–r– 1 lluis.fita cima 85706 Apr 12 13:27 namelist.output_00010101000000-00010101010000
-rw-r–r– 1 lluis.fita cima  5020 Apr 12 13:27 namelist.input_00010101000000-00010101010000
drwxr-xr-x 2 lluis.fita cima 4096 Apr 12 13:27 00010101000000-00010101010000
-rw-r–r– 1 lluis.fita cima 3733932 Apr 12 13:27 wrfinput_d01

1. And the rsl file should content:

$ cat realout/00010101000000-00010101010000/rsl.error.0000
Ntasks in X 4, ntasks in Y 4
IDEAL V4.0.3 PREPROCESSOR
************************************
Parent domain
ids,ide,jds,jde 1 40 1 40
ims,ime,jms,jme -4 17 -4 17
ips,ipe,jps,jpe 1 10 1 10
************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
   alloc_space_field: domain 1, 11151368 bytes allocated
  wrf: SUCCESS COMPLETE IDEAL INIT

Table 1: WRF’s LES test simulation specifications (from WRF namelist.input file from test/em_les folder specification ([after WRF user’s web page instructions])

value	specification	value	specification	value	specification
dimx	40	dimy	40	dimz	40
dx (m)	100.	dy (m)	100.	time_step (s)	1
ztop (m)	2000	periodic_bx	true	periodic_by	true
c_s	0.18	c_k	0.10	tke_heat_flux	0.24

1. Once initial conditions are generated (realout/wrfinput_d01), one can run the simulation. First the WRF

PBS job script is required

$ cp /share/WRF/run_wrf.pbs ./

1. Simulation can be performed (by default 24 nodes)

$ qsub ru_wrf.pbs
8499.hydra

1. Verify simulation is propertly running

$ qstat
Job id Name UserT ime Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8498.hydra WRF4L_ideal lluis.fita 00:00:18 C larga
8499.hydra WRF4L_wrf lluis.fita          0 R larga

1. At the end of job

$ cat wrfout/00010101000000-00010101010000/rsl.error.0000
taskid: 0 hostname: node44 
module_io_quilt_old.F 2931 F
Quilting with 1 groups of 0 I/O tasks.
Ntasks in X 4, ntasks in Y 6
  For domain 1, the domain size is too small for this many processors,
or the decomposition aspect ratio is poor.
  Minimum decomposed computational patch size, either x-dir or y-dir, is 10 grid
cells.
  e_we =40, nproc_x =4, with cell width in x-direction =10
  e_sn =40, nproc_y =6, with cell width in y-direction =6
  –- ERROR: Reduce the MPI rank count, or redistribute the tasks.
––––––– FATAL CALLED –––––––-
FATAL CALLED FROM FILE: <stdin> LINE:1745
NOTE:1 namelist settings are wrong. Please check and reset these options
–––––––––––––––––––––-
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

1. An error is found related to an excess of cpu resources used. Thus there is a need to reduce it. Current configuration

$ cat run_wrf.pbs
(...)
## Queue
#PBS -q large
### Max run time
#PBS -l walltime=168:00:00
### Max memory
#PBS -l vmem=20gb
#PBS -l mem=20gb
## resources
#PBS -l nodes=1:ppn=24
(...)
mpiexec -n 24 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log

1. There is a need to reduce the amount of resources. Let’s try with 16 processes

$ vim run_wrf.pbs
(...)
## resources
#PBS -l nodes=1:ppn=16
(...)
mpiexec -n 16 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log

1. re-launching job

$ qsub ru_wrf.pbs
8501.hydra

1. Checking job running

$ qstat
Job id Name User Time Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8500.hydra WRF4L_wrf lluis.fita 00:01:03 R larga

1. At the end of simulation (note that with 16 process 1 time-step (1 s) is simulated with 0.0738 seconds of real time. Meaning that there is a simulation speed of around 13.6 times faster than simulation time)

$ tail wrfout/00010101000000-00010101010000/rsl.error.0000
Timing for main: time 0001-01-01_00:59:54 on domain 1: 0.07378 elapsed seconds
Timing for main: time 0001-01-01_00:59:55 on domain 1: 0.07362 elapsed seconds
Timing for main: time 0001-01-01_00:59:56 on domain 1: 0.07385 elapsed seconds
Timing for main: time 0001-01-01_00:59:57 on domain 1: 0.07380 elapsed seconds
Timing for main: time 0001-01-01_00:59:58 on domain 1: 0.07392 elapsed seconds
Timing for main: time 0001-01-01_00:59:59 on domain 1: 0.07399 elapsed seconds
Timing for main: time 0001-01-01_01:00:00 on domain 1: 0.07406 elapsed seconds
Timing for Writing wrfout_d01_0001-01-01_01:00:00 for domain 1: 0.31056 elapsed seconds
Timing for Writing restart for domain 1: 2.09555 elapsed seconds 
d01 0001-01-01_01:00:00 wrf: SUCCESS COMPLETE WRF

1. One should have inside wrfout/ folder

$ ls wrfout
00010101000000-00010101010000
namelist.input_00010101000000-00010101010000
namelist.output_00010101000000-00010101010000
run_wrf-00010101000000-00010101010000.log
stations
wrfout_d01_0001-01-01_00:00:00
wrfrst_d01_0001-01-01_01:00:00

1. Standard output of 2 time-steps is about 6.7M

$ ls -lh wrfout/wrfout_d01_0001-01-01_00:00:00
-rw-r–r– 1 lluis lluis 6.7M Apr 12 13:38 wrfout/wrfout_d01_0001-01-01_00:00:00

1. See figures from 1 to 4 with some preliminarily results

Figure 3: Temporal evolution of of the 2-m mixing ratio (kgkg^-1) every 15 minutes of simulation

Figure 4: Temporal evolution of vertical cross section at the center of the domain of mixing ratio (kgkg^-1) every 15 minutes of simulation

Figure 3: Temporal evolution of of the 10-m wind-speed (ms^-1) every 15 minutes of simulation

Figure 4: Temporal evolution of vertical cross section at the center of the domain of the wind-speed (ms^-1) every 15 minutes of simulation