Refactor gridded wave post

[AntonMFernando-NOAA]

Description

The gridded wave post (wavepostsbs) script currently loops over all forecast hours and acts as its own post manager. Other component post and product scripts operate on one forecast time. This PR will update the wave post to match other components, that creates a bunch of short jobs instead of one long job (per member).
Resolves #2290

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• Run CI tests in HERA, ORION, and HERCULES

Checklist

• Gridded wave post scripts that process only one forecast hour specified by the workflow manager (rocoto/ecflow)
• Rocoto workflow updated to create the corresponding tasks for the new paradigm for both GFS and GEFS
• Output identical to current system for both GFS and GEFS

[AntonMFernando-NOAA]

@WalterKolczynski-NOAA I think this PR is ready to review. However, I don't know much about editing the exgfs_wave_post_gridded_sbs.sh script that produces various files. Need some help with that. CI test in HERCULES passed.

[JessicaMeixner-NOAA]

@AntonMFernando-NOAA do you have a test case with develop and then a test case with this branch?

[sbanihash]

@MatthewMasarik-NOAA , Since you are working on this script (scripts/exgfs_wave_post_gridded_sbs.sh) related to the GFS ops support, would you please take a look at the changes and check if this change would possibly solve the ops failure issue as well?

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA , Since you are working on this script (scripts/exgfs_wave_post_gridded_sbs.sh) related to the GFS ops support, would you please take a look at the changes and check if this change would possibly solve the ops failure issue as well?

Here is the branch started: https://github.com/MatthewMasarik-NOAA/global-workflow/tree/fix/wave-post. The last commit on it may be critical in addressing the post failure. Refactoring this loop was the longer term goal.

[AntonMFernando-NOAA]

@AntonMFernando-NOAA do you have a test case with develop and then a test case with this branch?
@JessicaMeixner-NOAA Yes. Did not run a test case with develop recently. The one I have is one week old. Tests are run on HERCULES.
with develop - /work2/noaa/global/antonf/RUNTESTS/COMROOT/gefs_ht_dis
with current branch - /work2/noaa/global/antonf/RUNTESTS/COMROOT/gefs1

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA , Since you are working on this script (scripts/exgfs_wave_post_gridded_sbs.sh) related to the GFS ops support, would you please take a look at the changes and check if this change would possibly solve the ops failure issue as well?

@sbanihash I can't say how the changes to that script made here would change the overall program execution. I think the length and complexity of the main loop make it difficult to reason about it by inspection. That is why I thought refactoring was a good direction.

[JessicaMeixner-NOAA]

@AntonMFernando-NOAA - Okay I can't go into your directory b/c of permissions, but essentially you are going to want to make sure you still have the same number of files in the gridded wave post output and that the files are the same before and after your changes.

Also, while this breaks up the job into multiple parts - which is awesome! - we'll likely have other parts of this code that have "waiting" that likely need to be removed. That will require additional time that I do not have - but hopefully @sbanihash and @MatthewMasarik-NOAA can review that sooner than I'll be able to get to it.

[KateFriedman-NOAA]

@AntonMFernando-NOAA please sync your branch with develop and resolve any remaining conflicts, thanks!

[NeilBarton-NOAA]

I'm concerned about rocoto not being able to handle the additional tasks for long forecasts. Maybe the merge should be delayed until a rocoto workaround is merged.

[AntonMFernando-NOAA]

@AntonMFernando-NOAA - Okay I can't go into your directory b/c of permissions, but essentially you are going to want to make sure you still have the same number of files in the gridded wave post output and that the files are the same before and after your changes.

Also, while this breaks up the job into multiple parts - which is awesome! - we'll likely have other parts of this code that have "waiting" that likely need to be removed. That will require additional time that I do not have - but hopefully @sbanihash and @MatthewMasarik-NOAA can review that sooner than I'll be able to get to it.

@JessicaMeixner-NOAA Did a diff check between develop and the current branch. The output can be found at /work/noaa/global/antonf/archive_rotdir/diff_wave_dev.txt.