Revisión de 11:50 7 dic 2022

Instalando y usando SMOKE en hydra

Página de SMOKE: https://www.cmascenter.org/smoke/

Manual de usuarie User's manual v4.5

Compilar

Archivo comprimido desde la página web (bajo registro previo)

smoke_v49.Linux2_x86_64ifort.tar.gz

intel

En del directorio

tar xvfz smoke_v49.Linux2_x86_64ifort.tar.gz
cd subsys/smoke
ls
Linux2_x86_64ifort  src
ls Linux2_x86_64ifort/
aggwndw      cntlmat     gentpro   inlineto2d  met4moves   mrgelev    normbeis4  smkinven   surgtool  uam2ncf
beld3to2     elevpoint   geofac    invsplit    metcombine  mrggrid    pktreduc   smkmerge   temporal
bluesky2inv  extractida  grdmat    layalloc    metscan     mrgpt      saregroup  smkreport  tmpbeis3
cemscan      gcntl4carb  grwinven  laypoint    movesmrg    normbeis3  smk2emis   spcmat     tmpbeis4

gcc

Directorio de instalación /home/solange.luque/MODELOS/SMOKE

Desplegando código

mv ~/smoke_v49.Linux2_x86_64ifort.tar.gz ./
mkdir gcc/
tar xvfz ../smoke_v49.Linux2_x86_64ifort.tar.gz 
mv subsys/smoke/src ./
mv subsys/smoke/Linux2_x86_64ifort ./
rmdir subsys/smoke/
rmdir subsys
mkdir Linux2_x86_64

Compilando

cd src
cp Makeinclude Makeinclude.old
cp Makeinclude Makeinclude.Linux2_x86_64
diff Makeinclude.Linux2_x86_64 Makeinclude.old
31c31
< BASEDIR = /home/solange.luque/MODELOS/SMOKE/gcc/src
---
> BASEDIR = ${SMK_HOME}/subsys/smoke/src
35c35
< IOBASE  = /home/solange.luque/libraries/ioapi/v3.2-20200828/gnu
---
> IOBASE  = ${SMK_HOME}/subsys/ioapi
40c40
< INSTDIR = /home/solange.luque/MODELOS/SMOKE/gcc/${BIN}
---
> INSTDIR = /somewhere/apps/${BIN}
48,49c48,49
< # IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  Intel, Absoft, Pathscale, UNIX Fortran
<  IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  GNU   Fortran
---
>  IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  Intel, Absoft, Pathscale, UNIX Fortran
> # IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  GNU   Fortran
53,54c53,54
< # EFLAG = -extend-source 132 -zero                       #  Intel Fortran
<  EFLAG = -ffixed-line-length-132  -fno-backslash       #  GNU   Fortran
---
>  EFLAG = -extend-source 132 -zero                       #  Intel Fortran
> # EFLAG = -ffixed-line-length-132  -fno-backslash       #  GNU   Fortran
59c59
< MFLAGS  = -fbacktrace
---
> MFLAGS  = -traceback

Hay un error de sintaxis Fortran en /home/solange.luque/libraries/ioapi/v3.2-20200828/gnu/ioapi/fixed_src/PARMS3.EXT, en la línea #104, queda:

!        INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) =                             &
!     &        (/ M3INT, M3REAL, M3DBLE, M3INT8 /)
        INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) = (/ M3INT, M3REAL, M3DBLE, M3INT8 /)

Copilamos

/opt/load-libs.sh
source /opt/env_scripts/load_gnu-10.2.1_openmpi-4.1.2.sh
export BIN=Linux2_x86_64
ln -s Makeinclude.Linux2_x86_64 ./Makeinclude
make >& run_make.log
ls ../Linux2_x86_64 | grep -v .o | grep -v .mod
aggwndw        bluesky2inv        cemscan        cntlmat        extractida        gcntl4carb
grdmat         grwinven           invsplit       libfileset.a   metscan           mrgelev
mrggrid        mrgpt              pktreduc       smk2emis       smkinven          smkmerge
spcmat         tmpbeis3           tmpbeis4       uam2ncf

gcc - debug

Directorio de instalación /home/solange.luque/MODELOS/SMOKE_dbg

NOTA: Todo se tendría que llamar Linux2_x86_64_dbg, pero después no encuentra en la compilación de I/O - API, el ioapi/Makinclude.Linux2_x86_64_dbg, puiesto que sí existe un ioapi/Makinclude.Linux2_x86_64dbg (que rompimos, ver CMAQ gnu - debug). Hacemos trampas, poniéndole el nombre Linux2_x86_64

Desplegando código

mv ~/smoke_v49.Linux2_x86_64ifort.tar.gz ./
mkdir gcc/
cd gcc
tar xvfz /home/solange.luque/MODELOS/SMOKE/smoke_v49.Linux2_x86_64ifort.tar.gz
mv subsys/smoke/src ./
mv subsys/smoke/Linux2_x86_64ifort ./
rmdir subsys/smoke/
rmdir subsys
mkdir Linux2_x86_64

Compilando

cd src
cp Makeinclude Makeinclude.old
cp Makeinclude Makeinclude.Linux2_x86_64
rm Makeinclude
diff Makeinclude.Linux2_x86_64 Makeinclude.old
30d29
< SMK_HOME = /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src
32c31
< BASEDIR = /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src
---
> BASEDIR = ${SMK_HOME}/subsys/smoke/src
36c35
< IOBASE  = /home/solange.luque/libraries/ioapi/v3.2-20200828/gnu
---
> IOBASE  = ${SMK_HOME}/subsys/ioapi
41c40
< INSTDIR = /home/solange.luque/MODELOS/SMOKE/gcc/${BIN}
---
> INSTDIR = /somewhere/apps/${BIN}
49,50c48,49
< # IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  Intel, Absoft, Pathscale, UNIX Fortran
<  IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  GNU   Fortran
---
>  IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  Intel, Absoft, Pathscale, UNIX Fortran
> # IFLAGS = -I${IOINC} -I${INCDIR} -I${IOBIN}                #  GNU   Fortran
54,55c53,54
< # EFLAG = -extend-source 132 -zero                       #  Intel Fortran
<  EFLAG = -ffixed-line-length-132  -fno-backslash       #  GNU   Fortran
---
>  EFLAG = -extend-source 132 -zero                       #  Intel Fortran
> # EFLAG = -ffixed-line-length-132  -fno-backslash       #  GNU   Fortran
60c59
< MFLAGS = -g -Wall -Wextra -Warray-temporaries -Wconversion -fimplicit-none -fbacktrace -fcheck=all 
  -ffpe-trap=zero,overflow,underflow -finit-real=nan
---
> MFLAGS  = -traceback

Hay un error de sintaxis Fortran en /home/solange.luque/libraries/ioapi/v3.2-20200828/gnu/ioapi/fixed_src/PARMS3.EXT, en la línea #104, queda:

!        INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) =                             &
!     &        (/ M3INT, M3REAL, M3DBLE, M3INT8 /)
        INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) = (/ M3INT, M3REAL, M3DBLE, M3INT8 /)

Copilamos

/opt/load-libs.sh
source /opt/env_scripts/load_gnu-10.2.1_openmpi-4.1.2.sh
export BIN=Linux2_x86_64
ln -s Makeinclude.Linux2_x86_64 ./Makeinclude
make >& run_make.log
ls ../Linux2_x86_64 | grep -v .o | grep -v .mod
aggwndw        bluesky2inv        cemscan        cntlmat        extractida        gcntl4carb
grdmat         grwinven           invsplit       libfileset.a   metscan           mrgelev
mrggrid        mrgpt              pktreduc       smk2emis       smkinven          smkmerge
spcmat         tmpbeis3           tmpbeis4       uam2ncf

Usar

Empecemos por el test-case

NOTA: Los shells son todos #!/bin/csh, pero de verdad se utiliza #!/bin/tcsh CMAS Source Too many arguments is not csh!

test-case

Post en: foro CMAS

Sacado de la página de SMOKE https://github.com/CEMPD/SMOKE-TestCase/ (previo registro)

Directorio de trabajo: /home/solange.luque/MODELOS/SMOKE/DATA

googleDrive

SMOKE-TestCase-BEIS4-inputs.tar.gz-20221026T133036Z-001.zip       SMOKE-TestCase-inputs.tar.gz-20221026T133039Z-001.zip
SMOKE-TestCase-BEIS4-output.tar.gz-20221026T133038Z-001.zip       SMOKE-TestCase-output.tar.gz-20221026T133041Z-001.zip

unzip SMOKE-TestCase-BEIS4-inputs.tar.gz-20221026T133036Z-001.zip
unzip SMOKE-TestCase-BEIS4-output.tar.gz-20221026T133038Z-001.zip
unzip SMOKE-TestCase-inputs.tar.gz-20221026T133039Z-001.zip
unzip SMOKE-TestCase-output.tar.gz-20221026T133041Z-001.zip
tar xvfz SMOKE-TestCase-BEIS4-inputs.tar.gz  
tar xvfz SMOKE-TestCase-BEIS4-output.tar.gz  
tar xvfz SMOKE-TestCase-inputs.tar.gz  
tar xvfz SMOKE-TestCase-output.tar.gz

Nos queda:

ls
biogenics_output  SMOKE-TestCase-BEIS4-inputs.tar.gz                           
SMOKE-TestCase-inputs.tar.gz-20221026T133039Z-001.zip
ge_dat            SMOKE-TestCase-BEIS4-inputs.tar.gz-20221026T133036Z-001.zip  
SMOKE-TestCase-output.tar.gz
output_forQA      SMOKE-TestCase-BEIS4-output.tar.gz                           
SMOKE-TestCase-output.tar.gz-20221026T133041Z-001.zip
smoke4.7          SMOKE-TestCase-BEIS4-output.tar.gz-20221026T133038Z-001.zip
SMOKE-TestCase    SMOKE-TestCase-inputs.tar.gz

Descargamos archivos para simular en /home/solange.luque/MODELOS/SMOKE/DATA

mkdir scripts
git clone https://github.com/CEMPD/SMOKE-TestCase.git
ls 
SMOKE-TestCase

gcc

Seteando entorno

/opt/load-libs.sh 3
source /opt/env_scripts/load_gnu-10.2.1_mpich-3.4.2.sh

Preparando directorio para simular /home/solange.luque/estudios/SMOKE-TestCase

cp -R ~/MODELOS/SMOKE/DATA/SMOKE-TestCase/scripts ./
cd scripts 
diff directory_definitions.csh ~/MODELOS/SMOKE/scripts/SMOKE-TestCase/scripts/directory_definitions.csh 
13c13
< setenv INSTALL_DIR "/home/solange.luque/MODELOS/SMOKE/DATA"
---
> setenv INSTALL_DIR "/proj/ie/proj/SMOKE/chef"
19c19
< setenv MET_ROOT "/home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/inputs/biogenics"
---
> setenv MET_ROOT "/proj/ie/proj/TRECH/emissions/mcip"
24c24
< setenv MET_ROOT_3D "/home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/inputs/biogenics"
---
> setenv MET_ROOT_3D "/proj/ie/proj/TRECH/emissions/mcip"
46c46
< setenv SMOKE_LOCATION "/home/solange.luque/MODELOS/SMOKE/gcc/Linux2_x86_64"
---
> setenv SMOKE_LOCATION "/proj/ie/proj/CMAS/CMAQ/christos/SMOKE_REPO/exec/Linux2_x86_64ifort_medium"
53c53
< setenv IOAPI_LOCATION "/home/solange.luque/libraries/ioapi/v3.2-20200828/gnu/Linux2_x86_64"
---
> setenv IOAPI_LOCATION "/proj/ie/proj/CMAS/CMAQ/christos/SMOKE_test/SMOKE_4.8.1/subsys/ioapi-3.2/Linux2_x86_64ifort_medium"

Ajustando scripts al cluster:

$ cat /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/emf/smk_ar_annual_emf.csh | grep py
set log_analyzer = $SCRIPTS/log_analyzer/log_analyzer.py
set path_parser  = $SCRIPTS/run/path_parser.py

• línea #1 de /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/log_analyzer/log_analyzer.py, /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/run/path_parser.py, /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/annual_report/annual_report_v2.py

#!/usr/bin/env python3

Error de c-shell en el script /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/run/set_months_v4.csh, sintaxis errónea en dar la cantidad de argumentos: está $# y tiene que ser $#argv web, en línea # 30:

if ( $#argv == 3 ) then

En /home/solange.luque/MODELOS/SMOKE/DATA/smoke4.7/scripts/platform está seteado por default la abreviación de intel, así que hay que cambiarlo.

#set comp_abbrv = ifort
set comp_abbrv = gcc

Desde /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/scripts/

mkdir /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/scripts/logs
cd nonpoint/
./Annual_np_oilgas_12US1_2017gb_17j_TestCase.csh >& run_nonpoint.log

Nota: Antes de correr, si no es la primera vez que se lo hace, hay que borrar el log timelog_np_oilgas_SMOKE-TestCase_12US1_all.txt que está en /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/intermed/np_oilgas/logs/. Sino, el programa timetracker tira error de que tiene multiple primary keys disponibles.

cd /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/intermed/np_oilgas/logs
rm timelog_np_oilgas_SMOKE-TestCase_12US1_all.txt

Correr el siguiente sector:

cd ../point
./Annual_pt_oilgas_onetime_12US1_2017gb_17j_TestCase.csh >& run_Annual_onetime.log

intel

Trabajanod con la compilación de intel

cd /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/scripts
cp directory_definitions.csh directory_definitions.csh.gcc
diff directory_definitions.csh directory_definitions.csh.gcc
46c46
< setenv SMOKE_LOCATION "/home/solange.luque/MODELOS/SMOKE/intel/subsys/smoke/Linux2_x86_64ifort"
---
> setenv SMOKE_LOCATION "/home/solange.luque/MODELOS/SMOKE/gcc/Linux2_x86_64"
53c53
< setenv IOAPI_LOCATION "/home/solange.luque/libraries/ioapi/v3.2-20200828/intel/Linux2_x86_64ifort"
---
> setenv IOAPI_LOCATION "/home/solange.luque/libraries/ioapi/v3.2-20200828/gnu/Linux2_x86_64"

Ejexcutando:

cd nonpoint/
cp run_nonpoint.log run_nonpoint.log.gcc 
./Annual_np_oilgas_12US1_2017gb_17j_TestCase.csh >& run_nonpoint.log
ls /home/solange.luque/estudios/CMAQ-ideal/DATA/mcip/AMBA/
GRIDBDY2D_160702.nc  GRIDDOT2D_160702.nc   METCRO2D_160702.nc  namelist.mcip
GRIDCRO2D_160702.nc  LUFRAC_CRO_160702.nc  METCRO3D_160702.nc  SOI_CRO_160702.nc
GRIDDESC             METBDY3D_160702.nc    METDOT3D_160702.nc

inventarios

SMOKE parte de unos ficheros de inventarios de emisiones que están en ASCII. SMOKE lo que hace, es procesarlos para generar los mapas de emisiones para ingestar a CMAQ.

Encontramos un pdf con mucha información del significado de cada elemento de los inventarios EPA-Emissions

nopoint

Agarramos estos inventarios como ejemplo:

Fichero base del TestCase:

/home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/inputs/np_oilgas/np_oilgas_2017NEI_NONPOINT_20200501_04may2020_nf_v1.csv
../read_table.bash np_oilgas_2017NEI_NONPOINT_20200501_04may2020_nf_v1.csv country 0.6598465 ,
1: country_cd= "US"
2: region_cd= "17001"
3: tribal_code= 
4: census_tract_cd= 
5: shape_id= 
6: scc= "2310021501"
7: emis_type= 
8: poll= "VOC"
9: ann_value= 0.6598465
10: ann_pct_red= 
11: control_ids= 
12: control_measures= 
13: current_cost= 
14: cumulative_cost= 
15: projection_factor= 
16: reg_codes= 
17: calc_method= 6
18: calc_year= 2017
19: date_updated= 20200430
20: data_set_id= "2017EPA_NONPOINT"
21: jan_value= 
22: feb_value= 
23: mar_value= 
24: apr_value= 
25: may_value= 
26: jun_value= 
27: jul_value= 
28: aug_value= 
29: sep_value= 
30: oct_value= 
31: nov_value= 
32: dec_value= 
33: jan_pctred= 
34: feb_pctred= 
35: mar_pctred= 
36: apr_pctred= 
37: may_pctred= 
38: jun_pctred= 
39: jul_pctred= 
40: aug_pctred= 
41: sep_pctred= 
42: oct_pctred= 
43: nov_pctred= 
44: dec_pctred= 
45: comment= 

Archivo con el significado de cada pixel/county de los US:

/home/solange.luque/MODELOS/SMOKE/DATA/ge_dat/costcy_for_2017platform_24apr2020_nf_v1.txt

point

En: /home/solange.luque/MODELOS/SMOKE/DATA/SMOKE-TestCase/inputs/pt_oilgas

 ../read_table.bash pt_oilgas_2017NEIpost_POINT_20200618_09feb2021_v0.csv country 0.0757 ,
1: country_cd= "US"
2: region_cd= "01073"
3: tribal_code= 
4: facility_id= "1001411"
5: unit_id= "63024513"
6: rel_point_id= "57750712"
7: process_id= "86737214"
8: agy_facility_id= "010730267"
9: agy_unit_id= "9"
10: agy_rel_point_id= "9"
11: agy_process_id= "1"
12: scc= "20200252"
13: poll= "540841"
14: ann_value= 0.0757
15: ann_pct_red= 
16: facility_name= "SOUTHERN NATURAL GAS CORP."
17: erptype= "2"
18: stkhgt= 38
19: stkdiam= 2.5
20: stktemp= 615
21: stkflow= 220.7666
22: stkvel= 45
23: naics= "486210"
24: longitude= -86.76774
25: latitude= 33.59422
26: ll_datum= 
27: horiz_coll_mthd= 
28: design_capacity= 
29: design_capacity_units= 
30: reg_codes= 
31: fac_source_type= "110"
32: unit_type_code= "999"
33: control_ids= 
34: control_measures= 
35: current_cost= 
36: cumulative_cost= 
37: projection_factor= 
38: submitter_id= "ALJCBOH"
39: calc_method= 8
40: data_set_id= "2017JeffCty"
41: facil_category_code= "HAP"
42: oris_facility_code= " "
43: oris_boiler_id= 
44: ipm_yn= 
45: calc_year= "2017"
46: date_updated= 20200617
47: fug_height= 
48: fug_width_xdim= 
49: fug_length_ydim= 
50: fug_angle= 
51: zipcode= "35217"
52: annual_avg_hours_per_year= 
53: jan_value= 
54: feb_value= 
55: mar_value= 
56: apr_value= 
57: may_value= 
58: jun_value= 
59: jul_value= 
60: aug_value= 
61: sep_value= 
62: oct_value= 
63: nov_value= 
64: dec_value= 
65: jan_pctred= 
66: feb_pctred= 
67: mar_pctred= 
68: apr_pctred= 
69: may_pctred= 
70: jun_pctred= 
71: jul_pctred= 
72: aug_pctred= 
73: sep_pctred= 
74: oct_pctred= 
75: nov_pctred= 
76: dec_pctred= 
77: comment=  "US"