Skip to content

A Quick Start Guide for Running the MPASJEDI 2024052700 case on Jet or Hera

Jump to bottom
Guoqing Ge edited this page Oct 11, 2024 · 6 revisions

NOTE: This quick start guide only applies to the MPASJEDI 2024052700 case (including both conus12km and atl12km cases)

If you want to run the FV3JEDI cases or the hofx validation, please check the following two wiki's:
*Running FV3‐JEDI or MPAS‐JEDI ctest and setting up an experiment (You can also follow the left link to set up the MPASJEDI 2024052700 conus12km case if you choose to do so)
*Running FV3‐JEDI Hofx Validation

1. Clone and build RDASApp

Follow this instruction: https://github.com/NOAA-EMC/RDASApp/blob/develop/docs/build_and_test.md

2. A quick start guide using link_mpasjedi_expr.sh 

cd RDASApp # make sure you are under the top directory of RDASApp
cd rrfs-test/scripts
./link_mpasjedi_expr.sh

This will create an experiment directory at RDASApp/expr/mpas_2024052700, change to that directory and you are ready to run a single observation 3DEnVar MPASJEDI experiment.

cd ../../expr/mpas_2024052700
vi run_jedi.sh  # make sure you use the correct account, qos, partition and reservation)
sbatch run_jedi.sh

NOTE:
The first few lines of run_jedi.sh are as follows:

  1 #! /bin/sh
  2 #SBATCH --account=rtrr
  3 #SBATCH --qos=batch
  4 ###SBATCH --partition=kjet
  5 ###SBATCH --reservation=rrfsens
  6 #SBATCH --ntasks=120
  1. You don't need to specify partition and reservation on Hera, so leave line 3 and line 4 as is.
  2. You may need to specify #SBATCH --partition=kjet on Jet (remove ## in line 4)
  3. If you have reservation allocation on Jet, you can set it up, such as #SBATCH --reservation=rrfsens in line 5.
    If you don't know what a "reservation" is, just ignore it and don't need to make any changes.

3. Make some plots

The following Python plotting scripts are available in the experiment directory:

mpasjedi_increment_singleob.py
mpasjedi_spread.py
# mpasjedi_increment_fulldom.py

First load the EVA python environment from the experiment directory as follows:

source ../../ush/load_eva.sh

You should be able to see the (eva) text at the beginning of your command lines now. Then you can run the following commands to generate figures:

./mpasjedi_increment_singleob.py
./python mpasjedi_spread.py

4. Running the atl12km configuration

Follow the above step, but replace the following command:
./link_mpasjedi_expr.sh
with
./link_mpasjedi_expr.sh atl12km

If you want to learn more about what atl12km is, you may want to check this link.

5. Using pre-built executables from other RDASApp clones and/or create a customized experiment directory

The first few lines of link_mpasjedi_expr.sh are as follows:

  1 #!/bin/bash
  2 RDASApp=$( git rev-parse --show-toplevel 2>/dev/null )
  3 if [[ -z ${RDASApp} ]]; then
  4   echo "Not under a clone of RDASApp!"
  5   echo "Please delete line 2-7 and set RDASApp variable mannually"
  6   exit
  7 fi
  8 
  9 #RDASApp="/path/to/RDASApp"  # set this variable if line2-7 was removed
 10 BUMPLOC="conus12km-401km11levels"
 11 exprname="mpas_2024052700"
 12 if [[ "$1" == "atl12km" ]]; then
 13   BUMPLOC="atl12km-401km11levels"
 14   exprname="atl_2024052700"
 15 fi
 16 expdir=${RDASApp}/expr/${exprname}  # can be set to any directory

You can remove lines 2-8, and uncomment line 9 to set the RDASApp variable to use another clone.

For example, if I want to test Ming's updates which sync all submodules to the latest JCSDA ones, I can set RDASApp in line 9 to: RDASApp=/scratch1/BMC/wrfruc/mhu/jedi/sync/RDASApp

And then because I cannot write to Ming's directory, I set expdir in Line 16 to:
expdir=/scratch1/BMC/wrfruc/gge/RDASApp/expr/mpas_2024052700_sync

This will set up an experiment directory for me to test Ming's recent updates.

NOTE: DON'T modify exprname as it is a reserved keyword to determine which set of fix files should be linked to expdir.

6. Generating new BUMP localization files

At the experiment directory, you can edit the run_bump.sh files as in step 2 and then run it using sbatch run_bump.sh

7. Assimilating other observation types

The following ioda observation files are available under the data/obs subdirectory of the experiment directory:

ioda_adpsfc.nc
ioda_adpupa.nc
ioda_aircraft.nc
ioda_ascatw.nc
ioda_gpsipw.nc
ioda_mesonet.nc
ioda_profiler.nc
ioda_rassda.nc
ioda_satwnd.nc

One can try to assimilate these observations. We have not tested them so no assimilation yaml files are available for these at the moment.
Example yaml files may be find under RDASApp/sorc/mpas-jedi/test/testinput

8. Running LETKF or GETKF

By default, the run_jedi.sh script runs EnVar using the input file rrfs_mpasjedi_2024052700_Ens3Dvar.yaml. YAML files are also available to run local ensemble DA applications including LETKF and GETKF for this MPAS-JEDI test case:

rrfs_mpasjedi_2024052700_letkf.yaml
rrfs_mpasjedi_2024052700_getkf.yaml

These ensemble DA solvers can be run in the same experiment directory configured by link_mpasjedi_expr.sh. To change from running EnVar to LETKF/GETKF, you will need to edit run_jedi.sh script to 1) select the inputfile corresponding to the solver of preference, and 2) change the srun command at the bottom to now run mpasjedi_enkf.x instead of mpasjedi_variational.x.

Then simply submit the job using:

sbatch run_jedi.sh

If successful, two files will be created in the run directory including bkg.2024-05-27_00.00.00.nc and ana.2024-05-27_00.00.00.nc containing the ensemble mean background and analysis files, respectively.

The current yaml configurations only output the ensemble mean analyses. If you want to update the ensemble members, you must set the following line in the yaml:

driver:
  save posterior ensemble: true

Additionally, you must also modify the output section as follows:

output: 
  filename: ./ana.%{member}%.2024-05-27_00.00.00.nc
  stream name: analysis

The above setting will output the ensemble member analyses to the main run directory with names like ana.001.2024-05-27_00.00.00.nc, ana.002.2024-05-27_00.00.00.nc, etc. The location and output names of the analyses can be changed by further modifying filename above.

Furthermore, the mpasjedi_increment_singleob.py and mpasjedi_increment_fulldom.py python scripts also have options to plot increments between the analysis and ensemble mean background (instead of between the analysis and background control member). To plot these increments for LETKF/GETKF, you must open these scripts and then uncomment the lines that say FOR LETKF on them.

*If you have any questions on this MPASJEDI test case, you may create an issue at the RDASApp repository or send emails to Guoqing.Ge@noaa.gov or Samuel.Degelia@noaa.gov.

Clone this wiki locally