WRFles

Here are derscribed the steps to perform a LES test simulation using WRF in CIMA’s HPC resource called `hydra'

Compilation

Here are described the steps followed to compile LES test WRF code in hydra

1. First we need to specify a location for the compilation of the code:

/share/WRF/WRFgit/WRFlesideal/ifort/dmpar

1. From that location in hydra, download the source code of the WRF model (since V4.0 there is not provided anymore a compressed file with the code)

$ git clone https://github.com/wrf-model/WRF

1. In order to compile, first it is required to define some environmental variables into the system

$ export NETCDF=’/usr/local’
$ export WRFIO_NCD_LARGE_FILE_SUPPORT=1

1. Then prepare the file with the specific configuration for the compilation (in this case using intel compiler in `dmpar' mode [distributed memory only] and no nesting 1 )

configure.wrf (13. (serial) 14. (smpar) 15. (dmpar) 16. (dm+sm) INTEL (ifort/icc)) [no nesting]

1. Compile te LES test case

$ ./compile em_les >& compile.log

1. Verify that there was not any error during compilation (should be empty)

$ cat -n compile.log | grep Error

Table 1: WRF’s LES test simulation specificications (from WRF namelist.input file from test/em_les folder specification

value	specification	value	specification	value	specification
dimx	40	dimy	40	dimz	40
dx (m)	100.	dy (m)	100.	time_step (s)	1
ztop (m)	2000	periodic_bx	true	periodic_by	true
c_s	0.18	c_k	0.10	tke_heat_flux	0.24

Use

Here are describerd the different steps to perform the WRF’s LES test simulation. This run test is done with standard original WRF’s LES ideal simulation (see details in table 1).

1. Simulation will be done in a different folder than the one where the compilation was made:

/home/lluis.fita/estudios/LES/test

1. Link the compiled and necessary files

$ ln -s /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/run/* ./

1. Remove the namelist file in order to do not modify WRF’s original namelist in case we want to modify something

$ rm namelist.input $ cp /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/namelist.input ./

1. Get the PBS jobs and scripts to run the simulation

$ cp /share/WRF/run_ideal.pbs ./ $ cp /share/WRF/launch_pbs.bash ./

1. Generate the initial conditions specific for the case (in this case using the defailt input_sounding 2 )

$ qsub run_ideal.pbs

1. Verify job is running propertly (it is very short, it last 18 seconds!)

$ qstat Job id Name User ––––––––––––- –––––––– –––––––- –––– - ––- (...) 8498.hydra WRF4L_ideal lluis.fita Time Use S Queue 00:00:12 R larga 7. At the end one should have $ ls -lrt realout/ total 3744 -rw-r–r– 1 lluis.fita cima 85706 Apr 12 13:27 namelist.output_00010101000000-00010101010000 -rw-r–r– 1 lluis.fita cima 5020 Apr 12 13:27 namelist.input_00010101000000-00010101010000 drwxr-xr-x 2 lluis.fita cima 4096 Apr 12 13:27 00010101000000-00010101010000 -rw-r–r– 1 lluis.fita cima 3733932 Apr 12 13:27 wrfinput_d01 1 after WRF user’s web page instructions http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.0/users_guide_chap5.html# _Installing_WRF 2 see /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/README.les for more details and optionsWRF-LES test at CIMA 3

1. And the rsl file shoult content:

$ cat realout/00010101000000-00010101010000/rsl.error.0000 Ntasks in X 4, ntasks in Y 4 IDEAL V4.0.3 PREPROCESSOR

Parent domain ids,ide,jds,jde 1 40 1 40 ims,ime,jms,jme -4 17 -4 17 ips,ipe,jps,jpe 1 10 1 10

DYNAMICS OPTION: Eulerian Mass Coordinate alloc_space_field: domain 1, 11151368 bytes allocated wrf: SUCCESS COMPLETE IDEAL INIT

1. Once initial conditions are generated (realout/wrfinput_d01), one can run the simulation. First the WRF

PBS job script is required $ cp /share/WRF/run_wrf.pbs ./

1. Simulation can be performed (by default 24 nodes)

$ qsub ru_wrf.pbs 8499.hydra

1. Verify simulation is propertly running

$ qstat Job id Name User ––––––––––––- –––––––– –––––––- –––– - ––- (...) 8498.hydra WRF4L_ideal lluis.fita 8499.hydra WRF4L_wrf lluis.fita Time Use S Queue 00:00:18 C larga 0 R larga 12. At the end of job $ cat wrfout/00010101000000-00010101010000/rsl.error.0000 taskid: 0 hostname: node44 module_io_quilt_old.F 2931 F Quilting with 1 groups of 0 I/O tasks. Ntasks in X 4, ntasks in Y 6 For domain 1, the domain size is too small for this many processors, or the decomposition aspect ratio is poor. Minimum decomposed computational patch size, either x-dir or y-dir, is 10 grid cells. e_we = 40, nproc_x = 4, with cell width in x-direction = 10 e_sn = 40, nproc_y = 6, with cell width in y-direction = 6 –- ERROR: Reduce the MPI rank count, or redistribute the tasks. ––––––– FATAL CALLED –––––––- FATAL CALLED FROM FILE: <stdin> LINE: 1745 NOTE: 1 namelist settings are wrong. Please check and reset these options –––––––––––––––––––––- application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0WRF-LES test at CIMA 4

1. An error is found related to an exceess of cpu resources used. Thus there is a need to reduce it. Current

configuration $ cat run_wrf.pbs (...)

1. 1. Queue
   2. PBS -q large
      1. Max run time
2. PBS -l walltime=168:00:00
   1. 1. Max memory
3. PBS -l vmem=20gb
4. PBS -l mem=20gb
   1. resources
5. PBS -l nodes=1:ppn=24

(...) mpiexec -n 24 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log

1. There is a need to reduce the amount of resources. Let’s try with 16 processes

$ vim run_wrf.pbs (...)

1. 1. resources
2. PBS -l nodes=1:ppn=16

(...) mpiexec -n 16 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log

1. re-launching job

$ qsub ru_wrf.pbs 8501.hydra

1. Checking job running

$ qstat Job id Name User ––––––––––––- –––––––– –––––––- –––– - ––- (...) 8500.hydra WRF4L_wrf lluis.fita Time Use S Queue 00:01:03 R larga

1. At the end of simulation ( 3 )

3 note that with 16 process 1 time-step (1 s) is simulated with 0.0738 seconds of real time. Meaning that there is a simulation speed of around 13.6 times faster than simulation timeWRF-LES test at CIMA 5 $ tail wrfout/00010101000000-00010101010000/rsl.error.0000 Timing for main: time 0001-01-01_00:59:54 on domain 1: 0.07378 elapsed seconds Timing for main: time 0001-01-01_00:59:55 on domain 1: 0.07362 elapsed seconds Timing for main: time 0001-01-01_00:59:56 on domain 1: 0.07385 elapsed seconds Timing for main: time 0001-01-01_00:59:57 on domain 1: 0.07380 elapsed seconds Timing for main: time 0001-01-01_00:59:58 on domain 1: 0.07392 elapsed seconds Timing for main: time 0001-01-01_00:59:59 on domain 1: 0.07399 elapsed seconds Timing for main: time 0001-01-01_01:00:00 on domain 1: 0.07406 elapsed seconds Timing for Writing wrfout_d01_0001-01-01_01:00:00 for domain 1: 0.31056 elapsed seconds Timing for Writing restart for domain 1: 2.09555 elapsed seconds d01 0001-01-01_01:00:00 wrf: SUCCESS COMPLETE WRF

1. One should have inside wrfout folder

$ ls wrfout 00010101000000-00010101010000 namelist.input_00010101000000-00010101010000 namelist.output_00010101000000-00010101010000 run_wrf-00010101000000-00010101010000.log stations wrfout_d01_0001-01-01_00:00:00 wrfrst_d01_0001-01-01_01:00:00

1. Standard output of 2 time-steps is about 6.7M

$ ls -lh wrfout/wrfout_d01_0001-01-01_00:00:00 -rw-r–r– 1 lluis lluis 6.7M Apr 12 13:38 wrfout/wrfout_d01_0001-01-01_00:00:00

1. See figures from 1 to 4 with some preiliminary results