Ver el código de «WRFles»

Here are described the steps to perform a LES test simulation using WRF in CIMA’s HPC resource called <CODE>`hydra'</CODE>

Initial similar pdf version of the same information is available here [[Archivo:WRFles.pdf]]

= Compilation = 
Here are described the steps followed to compile LES test WRF code in hydra 

'''NOTE:''' There is no need to re-compile the model. It is only provided for completeness ! 

# First we need to specify a location for the compilation of the code:
<PRE>
/share/WRF/WRFgit/WRFlesideal/ifort/dmpar
</PRE>
# From that location in hydra, download the source code of the WRF model (since V4.0 there is not provided anymore a compressed file with the code)
<PRE>
$ git clone https://github.com/wrf-model/WRF
</PRE>
# In order to compile, first it is required to define some environmental variables into the system
<PRE>
$ export NETCDF=’/usr/local’
$ export WRFIO_NCD_LARGE_FILE_SUPPORT=1
</PRE>
# Then prepare the file with the specific configuration for the compilation (in this case using intel compiler in `dmpar' mode [distributed memory only] and no nesting 1 )
<PRE>
configure.wrf (13. (serial) 14. (smpar) 15. (dmpar) 16. (dm+sm) INTEL (ifort/icc)) [no nesting]
</PRE>
# Compile te LES test case
<PRE>
$ ./compile em_les >& compile.log
</PRE>
# Verify that there was not any error during compilation (should be empty)
<PRE>
$ cat -n compile.log | grep Error
</PRE>

= Use =
Here are described the different steps to perform the WRF’s LES test simulation. This run test is done with standard original WRF’s LES ideal simulation (see details in table 1).
# Simulation will be done in a different folder than the one where the compilation was made:
<PRE>
/home/lluis.fita/estudios/LES/test
</PRE>
# Link the compiled and necessary files
<PRE>
$ ln -s /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/run/* ./
</PRE>
# Remove the namelist file in order to do not modify WRF's original namelist in case we want to modify something
<PRE>
$ rm namelist.input
$ cp /share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/namelist.input ./
</PRE>
# Get the PBS jobs and scripts to run the simulation
<PRE>
$ cp /share/WRF/run_ideal.pbs ./
$ cp /share/WRF/launch_pbs.bash ./
</PRE>
# Generate the initial conditions specific for the case (in this case using the default <CODE>input_sounding</CODE> (see <CODE>/share/WRF/WRFgit/WRFlesideal/ifort/dmpar/WRF/test/em_les/README.les</CODE> for more details and options)
<PRE>
$ qsub run_ideal.pbs
</PRE>
# Verify job is running propertly (it is very short, it last 18 seconds!)
<PRE>
$ qstat
Job id Name User Time Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8498.hydra WRF4L_ideal lluis.fita 00:00:12 R larga
</PRE>
# At the end one should have inside <CODE>realout/</CODE> folder:
<PRE>
$ ls -lrt realout/
total 3744
-rw-r–r– 1 lluis.fita cima 85706 Apr 12 13:27 namelist.output_00010101000000-00010101010000
-rw-r–r– 1 lluis.fita cima  5020 Apr 12 13:27 namelist.input_00010101000000-00010101010000
drwxr-xr-x 2 lluis.fita cima 4096 Apr 12 13:27 00010101000000-00010101010000
-rw-r–r– 1 lluis.fita cima 3733932 Apr 12 13:27 wrfinput_d01
</PRE>
# And the <CODE>rsl</CODE> file should content:
<PRE>
$ cat realout/00010101000000-00010101010000/rsl.error.0000
Ntasks in X 4, ntasks in Y 4
IDEAL V4.0.3 PREPROCESSOR
************************************
Parent domain
ids,ide,jds,jde 1 40 1 40
ims,ime,jms,jme -4 17 -4 17
ips,ipe,jps,jpe 1 10 1 10
************************************
DYNAMICS OPTION: Eulerian Mass Coordinate
   alloc_space_field: domain 1, 11151368 bytes allocated
  wrf: SUCCESS COMPLETE IDEAL INIT
</PRE>
{| class="wikitable"
|+ Table 1: WRF’s LES test simulation specifications (from WRF namelist.input file from test/em_les folder specification ([[http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/v4.0/users_guide_chap5.html#_Installing_WRF after WRF user’s web page instructions]])
|-
! value
! specification
! value
! specification
! value
! specification
|-
| dimx
| 40
| dimy
| 40
| dimz
| 40
|-
| dx (m)
| 100.
| dy (m)
| 100.
| time_step (s)
| 1
|-
| ztop (m)
| 2000
| periodic_bx
| true
| periodic_by
| true
|-
| c_s
| 0.18
| c_k
| 0.10
| tke_heat_flux
| 0.24
|-
|}

# Once initial conditions are generated (<CODE>realout/wrfinput_d01</CODE>), one can run the simulation. First the WRF
PBS job script is required
<PRE>
$ cp /share/WRF/run_wrf.pbs ./
</PRE>
# Simulation can be performed (by default 24 nodes)
<PRE>
$ qsub ru_wrf.pbs
8499.hydra
</PRE>
# Verify simulation is propertly running
<PRE>
$ qstat
Job id Name UserT ime Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8498.hydra WRF4L_ideal lluis.fita 00:00:18 C larga
8499.hydra WRF4L_wrf lluis.fita          0 R larga
</PRE>
# At the end of job
<PRE>
$ cat wrfout/00010101000000-00010101010000/rsl.error.0000
taskid: 0 hostname: node44 
module_io_quilt_old.F 2931 F
Quilting with 1 groups of 0 I/O tasks.
Ntasks in X 4, ntasks in Y 6
  For domain 1, the domain size is too small for this many processors,
or the decomposition aspect ratio is poor.
  Minimum decomposed computational patch size, either x-dir or y-dir, is 10 grid
cells.
  e_we =40, nproc_x =4, with cell width in x-direction =10
  e_sn =40, nproc_y =6, with cell width in y-direction =6
  –- ERROR: Reduce the MPI rank count, or redistribute the tasks.
––––––– FATAL CALLED –––––––-
FATAL CALLED FROM FILE: <stdin> LINE:1745
NOTE:1 namelist settings are wrong. Please check and reset these options
–––––––––––––––––––––-
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
</PRE>
# An error is found related to an excess of cpu resources used. Thus there is a need to reduce it. Current configuration
<PRE>
$ cat run_wrf.pbs
(...)
## Queue
#PBS -q large
### Max run time
#PBS -l walltime=168:00:00
### Max memory
#PBS -l vmem=20gb
#PBS -l mem=20gb
## resources
#PBS -l nodes=1:ppn=24
(...)
mpiexec -n 24 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log
</PRE>
# There is a need to reduce the amount of resources. Let’s try with 16 processes
<PRE>
$ vim run_wrf.pbs
(...)
## resources
#PBS -l nodes=1:ppn=16
(...)
mpiexec -n 16 ./launch_pbs.bash ./${execn}.exe >& run_${execn}0.log
</PRE>
# re-launching job
<PRE>
$ qsub ru_wrf.pbs
8501.hydra
</PRE>
# Checking job running
<PRE>
$ qstat
Job id Name User Time Use S Queue
––––––––––––- –––––––– –––––––- –––– - ––-
(...)
8500.hydra WRF4L_wrf lluis.fita 00:01:03 R larga
</PRE>
# At the end of simulation (note that with 16 process 1 time-step (1 s) is simulated with 0.0738 seconds of real time. Meaning that there is a simulation speed of around 13.6 times faster than simulation time)
<PRE>
$ tail wrfout/00010101000000-00010101010000/rsl.error.0000
Timing for main: time 0001-01-01_00:59:54 on domain 1: 0.07378 elapsed seconds
Timing for main: time 0001-01-01_00:59:55 on domain 1: 0.07362 elapsed seconds
Timing for main: time 0001-01-01_00:59:56 on domain 1: 0.07385 elapsed seconds
Timing for main: time 0001-01-01_00:59:57 on domain 1: 0.07380 elapsed seconds
Timing for main: time 0001-01-01_00:59:58 on domain 1: 0.07392 elapsed seconds
Timing for main: time 0001-01-01_00:59:59 on domain 1: 0.07399 elapsed seconds
Timing for main: time 0001-01-01_01:00:00 on domain 1: 0.07406 elapsed seconds
Timing for Writing wrfout_d01_0001-01-01_01:00:00 for domain 1: 0.31056 elapsed seconds
Timing for Writing restart for domain 1: 2.09555 elapsed seconds 
d01 0001-01-01_01:00:00 wrf: SUCCESS COMPLETE WRF
</PRE>
# One should have inside <CODE>wrfout/</CODE> folder
<PRE>
$ ls wrfout
00010101000000-00010101010000
namelist.input_00010101000000-00010101010000
namelist.output_00010101000000-00010101010000
run_wrf-00010101000000-00010101010000.log
stations
wrfout_d01_0001-01-01_00:00:00
wrfrst_d01_0001-01-01_01:00:00
</PRE>
# Standard output of 2 time-steps is about 6.7M
<PRE>
$ ls -lh wrfout/wrfout_d01_0001-01-01_00:00:00
-rw-r–r– 1 lluis lluis 6.7M Apr 12 13:38 wrfout/wrfout_d01_0001-01-01_00:00:00
</PRE>
# See figures from 1 to 4 with some preliminarily results

[[Archivo:WRFles_qv2.gif|frame|50px|Figure 3: Temporal evolution of of the 2-m mixing ratio (kgkg<SUP>-1</SUP>) every 15 minutes of simulation]]

[[Archivo:WRFles_qv.gif|frame|50px|Figure 4: Temporal evolution of vertical cross section at the center of the domain of mixing ratio (kgkg<SUP>-1</SUP>) every 15 minutes of simulation]]

[[Archivo:WRFles_wss.gif|frame|50px|Figure 3: Temporal evolution of of the 10-m wind-speed (ms<SUP>-1</SUP>) every 15 minutes of simulation]]

[[Archivo:WRFles_ws.gif|frame|50px|Figure 4: Temporal evolution of vertical cross section at the center of the domain of the wind-speed  (ms<SUP>-1</SUP>) every 15 minutes of simulation]]