Preparing for coupled model testing

[wwieder]

• Model Version: when we start coupled model testing next month (March 2023) I'd like to use a long-name compset that uses our latest CTSM5.1 dev tag.

• Surface Datasets: we also are interested in exercising new surface datasets we'll be using with CTSM6 in CESM3, but these have to be generated. What spatial resolution are you planning to run the land model in the next round of coupled model simulations?

  • @adamrher responded: the land grid is ne30pg3, and official mesh file is here:
    /glade/p/cesmdata/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc
  • @slevisconsulting is this something you can do?

• Initial conditions: I'm also assuming we should provide a restart file with land initial conditions? Is this OK to generate with an offline simulation, or do we typically do this using a modelled atmosphere data?

  • @adamrher responded: I would propose that we do an offline spinup forced by cycling over a 20 year long F-case, in order to create a land restart file for use in the coupled simulations. It's important that we don't shock the runoff fluxes with a snow pack that is not reflective of the climate produced.
  • @olyson your thoughts on this?

• Nitrogen Deposition: @adamrher asked, can you confirm whether we need to pass nitrogen deposition fluxes to the land model? It appears this functionality is not entirely finished yet: part of cam6_3_098: always pass NDEP from CAM and remove sst specs CAM#735. For reference, It looks like our 26 series uses BGC_CROP, so if nitrogen is needed is that going to be an issue?

  • @billsacks and @ekluzek, does this relate to stream data input now set in new config_component.xml variables #1929 and how we handle streams files?

• Are there other things the LMWG should be preparing for these test simulations? or other people that need to be involved in the discussion? @cecilehannay, @JulioTBacmeister, others...

[billsacks]

Nitrogen Deposition: @adamrher asked, can you confirm whether we need to pass nitrogen deposition fluxes to the land model? It appears this functionality is not entirely finished yet: ESCOMP/CAM#735. For reference, It looks like our 26 series uses BGC_CROP, so if nitrogen is needed is that going to be an issue?

• @billsacks and @ekluzek, does this relate to stream data input now set in new config_component.xml variables #1929 and how we handle streams files?

See #1868 (reply in thread) . I keep forgetting exactly what we do and don't need #1929 for... at one point I understood that to be needed in order to receive NDEP from ATM, but my latest recollection is that it isn't actually strictly needed. On the other hand, I think that ESCOMP/CAM#735 is needed, technically: Currently, we are in an in-between place where DATM always expects to read and send NDEP, but CTSM ignores NDEP from ATM and instead still reads NDEP itself. I believe that the current technical problem is that, since DATM expects to find NDEP on its forcing files, we need that CAM PR so that we can generate new cplhist forcing files with NDEP on them... at that point we should be able to run a cplhist-forced I compset, though CTSM wouldn't yet actually use NDEP from ATM.

I think we have 3 options for how to proceed: we can go backwards, slightly forwards or all the way forwards:

(1) Backwards: We could make a CDEPS branch where DATM doesn't expect NDEP on its forcing files, at least in CPLHIST mode. I believe that we could then generate data for and run a CPLHIST-forced case without needing ESCOMP/CAM#735.

(2) Slightly forwards: If we bring in ESCOMP/CAM#735 but do nothing else, I believe that we could create data for and run a CPLHIST-forced case. We wouldn't actually be using NDEP from CAM, but if the CAM forcings are from a non-chemistry run, this shouldn't make a difference.

(3) All the way forwards: In addition to (2), we could also make the changes in CTSM needed to get NDEP from ATM rather than reading NDEP itself.

Major caveat with all of this that I have never had my head completely around all of this, so I could be getting some of this wrong :-)

[adamrher]

Hi Bill, I can provide some additional info here as I worked with Mariana in October to run cplhist compset in cesm2_3_alpha10a. I'd have to look it up but I did have to swap in some mediator files with fixes from Mariana for it to work (Im almost certain these are on the trunk now). But in order to deal with the ATM not producing ndep fluxes but CPL expecting it, we set DATM_PRESNDEP=none and that way it wasn't required by the CPL. Seems like this might be the easiest solution as it sounds like CTSM wouldn't have used the ndep fluxes even if they were provided.

[klindsay28]

I'm pretty sure that CTSM uses ndep fluxes if they are provided by the coupler. This was used in CESM2 WACCM runs, and is now implemented in CTSM's NUOPC cap in the following block of code:

CTSM/src/cpl/nuopc/lnd_import_export.F90

Lines 668 to 677 in 05fb17b

	! Atmosphere ndep
	if (fldchk(importState, Faxa_ndep)) then
	! The mediator is sending ndep in units of kgN/m2/s - and ctsm
	! uses units of gN/m2/sec so the following conversion needs to happen
	call state_getimport_2d(importState, Faxa_ndep, forc_ndep(begg:,:), rc=rc)
	if (ChkErr(rc,__LINE__,u_FILE_u)) return
	do g = begg, endg
	atm2lnd_inst%forc_ndep_grc(g) = (forc_ndep(g,1) + forc_ndep(g,2))*1000._r8
	end do
	end if

[mvertens]

@adamrher - I think there is more that is needed. I believe that you need to modify the mediator auxiliary file settings so that it does not try to write out ndep from the atm. I think the simplest way forwards is to merge the CAM PR such that ndep is sent from CAM and then no one offs are needed by either datm or cmeps. I have not had time yet to address all of the issues since we have been and travel and moving. I think to a large extend the PR is ready and there still seems to be some minor issues that I still need to look at.

[adamrher]

Thanks for clarification. I was advocating for a simple solution to Bill's proposed (1), which I'm sure we could make happen. And so you're saying that might be more work than it's worth, given ESCOMP/CAM#735 is nearing completion. Fair point. That would be Bill's option (2). And I think probably the direction we're headed for this next phase of coupled simulations, assuming that PR can get completed by the end of this month.

To the CTSM folks, what are the implications scientifically? Do we expect that if we complete Bill's proposed (3) -- have the land model read in ndep fluxes sent from the atmosphere (do I have that right?) --- would we expect the ndep fluxes to be very different if they come from the atm as opposed to CTSM reading them from a file? This would presumably effect growth rates of plants?

[billsacks]

To the CTSM folks, what are the implications scientifically? Do we expect that if we complete Bill's proposed (3) -- have the land model read in ndep fluxes sent from the atmosphere (do I have that right?) --- would we expect the ndep fluxes to be very different if they come from the atm as opposed to CTSM reading them from a file? This would presumably effect growth rates of plants?

My understanding is that, when running with a version of CAM without chemistry – i.e., what's enabled with ESCOMP/CAM#735 – the ndep fluxes should be essentially the same from CAM as what we've been getting internally in CTSM.

And I think @klindsay28 is right about the CTSM code already being set up to use ATM-provided NDEP if available. Looking into this a bit more, I think what is not set up yet is having CDEPS's datm set the appropriate flag(s) in drv_flds_in signaling that it is providing ndep. Without this, I think CTSM is setting ndep_from_cpl to .false., which means that the atm-provided ndep is being overwritten by CTSM's internal ndep. I could be wrong in my analysis here, though: I just looked through this for a few minutes to try to understand what's going on right now.

[wwieder]

My head is kind of swimming here. Do we need to have a meeting in advance of the Feb 28 co-chairs meeting to figure out what's needed for these coupled model tests, or is that part of the purpose of the Feb 28 meeting?

[olyson]

Initial conditions: I'm also assuming we should provide a restart file with land initial conditions? Is this OK to generate with an offline simulation, or do we typically do this using a modelled atmosphere data?

@adamrher responded: I would propose that we do an offline spinup forced by cycling over a 20 year long F-case, in order to create a land restart file for use in the coupled simulations. It's important that we don't shock the runoff fluxes with a snow pack that is not reflective of the climate produced.
@olyson your thoughts on this?

In the latest CESM development simulations we have been using an initial file from the 016 development simulation, because the default initial file used in the previous simulations had a great deal of snow in the Arctic that was melting and producing a large amount of runoff. That initial file seems to have solved that problem for our recent CESM development simulations. I don't have a feeling for how different the coupled model climate will be with updated CAM and updated CTSM and whether the current initial file will be adequate. Also depends on the goals of the next round of simulations and what amount of resources we want to expend to reduce the risk of something similar happening again.

[adamrher]

Thanks Keith. I think given the uncertainty on the ndep fluxes, that I shouldn't be so evangelical as to hold up the next coupled phase. Given the amount of time it will take to resolve the ndep issue AND generate a spun-up inic, that we should just begin the next coupled phase with whatever we have at the time. That is, just use the 016 restart if we haven't gotten to the spinup.

[slevis-lmwg]

I'm happy to run the mksurfdata_esmf tool to generate an fsurdat from this mesh file /glade/p/cesmdata/inputdata/share/meshes/ne30pg3_ESMFmesh_cdf5_c20211018.nc.

@adamrher is this a default grid (ne30np4.pg3)? If not, the mksurfdata_esmf tool asks for the nx and ny of the grid; could you send me the values?

[slevis-lmwg]

@adamrher just a heads up that I am generating new versions of the files that I provided before (ne30, ne16, ne5, ne3), now with the latest PFT rawdata. The new files are in /glade/work/slevis/git/mksurfdata_toolchain/tools/mksurfdata_esmf and they span 1850-2100, so their names are:

surfdata_ne30np4.pg3_SSP5-8.5_78pfts_CMIP6_1850-2100_c230227.nc
landuse.timeseries_ne30np4.pg3_SSP5-8.5_78_CMIP6_1850-2100_c230227.nc

and accordingly for the other grids.

[olyson]

Diagnostics for a historical using the new datasets @slevisconsulting supplied (c230227) compared to CLM5 are here:

https://webext.cgd.ucar.edu/I20TR/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_sfcdat230227_hist/lnd/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_sfcdat230227_hist.1994_2013-clm50_ctsm51d118_ne30pg3g17_GSWP3V1_hist.1994_2013/setsIndex.html

and compared to the historical with c230216 datasets are here:

https://webext.cgd.ucar.edu/I20TR/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_sfcdat230227_hist/lnd/ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_sfcdat230227_hist.1994_2013-ctsm51_ctsm51d118_ne30pg3g17_GSWP3V1_hist.1994_2013/setsIndex.html

[wwieder]

Just to confirm, @olyson, results from simulations with the newest surface datasets (c230227) don't look appreciably different than the first ones we were testing (c230216)

[olyson]

Right, maybe @lawrencepj1 or @slevisconsulting can comment on exactly what the differences are between these two datasets. I confirmed that the only difference between the two CTSM5.1 simulations is the surface/landuse dataset.

[slevis-lmwg]

Right, maybe @lawrencepj1 or @slevisconsulting can comment on exactly what the differences are between these two datasets. I confirmed that the only difference between the two CTSM5.1 simulations is the surface/landuse dataset.

Using Peter's very latest pft raw datasets, as he described them here.

[wwieder]

Thanks for all the discussion on this thread everyone. I'm still a little confused about where we should prioritize our efforts.

I spoke with Gokhan yesterday who emphasized that we want these new coupled simulations to be as advanced as we can get for individual component models, but not any farther. Towards this end, I think we should shoot for functional, but but not worry about perfection. As such, I suggest we try to generate initial conditions with a:

• CTSM5.1 dev tag on the
• ne30 grid with as close as we can to a final
• CTSM5.2 surface dataset, using
• a 20 year long F-case (or what ever @adamrher and @olyson deem appropriate)

From the thread above it seems like @olyson is close to being able to start this, aside from the PFTDATA_MASK error (and what ever crops up next ). As @adamrher noted, worse case we can start with the old 016 restart files previously used for testing, but given changes in CTSM model version and surface data generating new initial conditions seems preferable?

Under this plan I'd put the ndep issue as a lower priority for the March CESM3 testing efforts, recognizing it still needs to be resolved.

What do others think of these suggestions?

[adamrher]

That seems like a good plan. I will let you know when we have converged on a cam tag for use in the next coupled phase, so we can start the process. Hopefully next week.

Just an FYI, the current CTSM externals at the head of CAM are dev114, and at the head of CESM, dev115. Are these tags considered the "CTSM5.1 dev tag" or are we going to have to update the externals?

[olyson]

I would think the externals need to be updated. At least to ctsm5.1.dev116, which has the changes in mosart to send negative runoff to the river outlets, and a change to stability (zetamaxstable). dev117 and dev118 (the latest) are bfb changes involving NEON and a change to replace ncar_pylib, respectively. So might as well update to the latest? It looks like there are other science tags coming up but I don't know what the schedule for these are @wwieder .

[wwieder]

It sounds like @billsacks is going to update externals, which seems like the highest priority for @adamrher's needs. Science developments will keep coming in, but getting Adam running with the right externals seems most important.

[wwieder]

From our discussions at co-chairs it seems like we should try to get going on initialization? Given the CAM PR with N-dep in CPLHIST may be slow, are we best off to create a one-off branch that backs out these so we can get going with spinup?

[adamrher]

I verified that it is reading ndep from the streams file specified in lnd_in.

Ah that's great. I wasn't sure what was happening with ndep with DATM_PRESNDEP=none, so I'm glad it's still grabbing the streams from clm. Seems like a solid plan b.

I will get back to you this afternoon after I huddle with Julio about the CAM runs with cplhist output for use in the spinup.

[adamrher]

We are going to use this week to sort out the emissions and some other issues. Let's see where we are at at the next Tues co-chairs meeting and we can discuss the spinup during / after we meet.

[wwieder]

OK, I suggest that @adamrher and the CAM team run an F case and save the last 20 years of CPLHIST output that @olyson can use to generate new land initial conditions.

@olyson noted that he's tested this on the CTSM side using old CPLHIST files, but not with newer CESM2.3_alpha (or other) tags. @adamrher do you have more recent CPLHIST that we can test this out with to make sure our whole workflow works?

[billsacks]

@adamrher - Also, beyond doing an end-to-end software smoke test of this coupler history-forced case, do you know if this coupler history workflow with the new CMEPS/CDEPS has been through some level of scientific validation to ensure that the climate is effectively the same in the I case as in the original F case?

[adamrher]

I do actually have cplhist streams from a cesm2_3_alpha10a tag. 1 full year of streams generated here:

/glade/scratch/aherring/cesm2_3_alpha10a.streams_FHIST_ne30pg3_t061_L32dev_nuopc_1800pes_221114_genstreams_Nx1yrs/run/

I think all the files are there. I'm kind of surprised they didn't get scrubbed yet.

Here's a run forced by those streams, for multiple years. Pointing @olyson to the user_nl_datm_streams file since that's where the sausage gets made:

/glade/scratch/aherring/cesm2_3_alpha10a.streams_IHIST_ne30pg3_t061_nuopc_600pes_221115_runstreams_Nx2yr/user_nl_datm_streams

There's a lot of details -- maybe @olyson email me if you're having issues forcing a run with these streams.

I suggest that @adamrher and the CAM team run an F case and save the last 20 years of CPLHIST output

Agreed. We are still finalizing the CAM tag. Will provide an update at Tues co-chairs mtg.

[adamrher]

do you know if this coupler history workflow with the new CMEPS/CDEPS has been through some level of scientific validation to ensure that the climate is effectively the same in the I case as in the original F case?

No, I did not validate the climate of the cplhist forced I compset. I got it to work, and shut the book closed. I believe I'm the only one who has tried the cplhist workflow w/ CMEPS/CDEPS as it required working closely with Mariana and Jim at the end of last year.

[billsacks]

Okay, thanks. My feeling is that some level of scientific validation should probably be done by someone if it hasn't been (we could check with Mariana to see if she happened to do it at some point). I think there are two possible approaches:

(1) The rigorous approach: make some temporary changes that should make the I case identical to the F case, at least within roundoff. This would mean writing coupler history files every time step, and possibly some other changes. I have used this approach to validate T compsets (CISM forced by DLND) in the past. If I remember correctly, Keith Lindsay might have done something similar for CPLHIST-forced G cases. Then we can do short runs (possibly as short as a few time steps) and verify that the results are the same within roundoff.

(2) The eyeball approach: use the standard setup with somewhat longer runs, and verify that the land fields are climatologically the same in the I and F compsets.

I can see arguments for either: (1) gives me more confidence but would likely take more effort... though (2) can end up taking significant effort if there are ambiguities in the diagnostics.

[mvertens]

@billsacks - I did a lot of careful very short validation comparing mct versus nuopc cpl hist for the aux files that are generated. That was a while ago, however and did not encompass the type of longer validations that you are suggestions.

[billsacks]

@mvertens - thanks for pointing that out. That nuopc vs mct comparison feels like an equivalently good validation. What you described seems like half of the validation. The other half would be validating that the CDEPS DATM cplhist mode was set up correctly / equivalently to the mct version. Did you get a chance to confirm that as well? (Or is that maybe unnecessary?)

[mvertens]

@billsacks - the setup is not the same since the implementation is completely different. In NUOPC we are setting all the auxhist settings in xml files - whereas in mct it is set directly in the code. Since mct is now deprecated the comparison cannot be repeated. At a mininum a quick eyeball validation would be useful. It would be helpful to also talk to Keith Lindsay about his careful work. I think that it only used several time steps to validate the cplhist formulation.

[billsacks]

Thanks @mvertens . @adamrher or @olyson - would one or both of you be able to take the lead on this validation (possibly pulling in Keith Lindsay)? I'd be happy to talk more with you about it.

[olyson]

Yes, that is probably something I can do. That way I can get familiar with the new coupler history infrastructure in preparation for the spinup we want to do.

[billsacks]

@adamrher - I was just talking with @olyson about doing the test run that we'll use to validate the cplhist forcing mode. We were thinking of using the upcoming CESM2_3_alpha12c; current externals are here: https://github.com/ESCOMP/CESM/blob/master/Externals.cfg . Does this seem reasonable to you? In particular, cam is at cam6_3_086 -- so a bit behind current cam development... do you see any problems with that for the sake of this validation?

[adamrher]

I don't think it matters what cam tag you use for the validation. We are just trying to make sure the cplhist forcing actually resembles the F-case it was derived from, right? If so, your plans sounds good to me.

[billsacks]

From discussion with @olyson on this: He's going to start a multi-year validation of the cplhist case soon. We'll use the upcoming alpha12c if it's ready in time; otherwise, I'll look into whether alpha12b or even beta11 would be usable for this purpose.

The main thing he'll look at is a comparison of climatologies between the I case and the F case used to generate the cplhist forcing data. The biggest potential issues we can foresee are (1) some field(s) are not plugged in correctly (which should hopefully be obvious when looking at the runs); or (2) something is wrong with the time interpolation / offsets. For (2) he'll pay attention to things like solar radiation diurnal cycle and/or check the time metadata on the files together with the offsets and interpolation methods specified for the streams (e.g., compared with the old mct versions).

Our hope is that this test will show that the I case is close enough to the F case. If it seems like there may be significant differences, then we'll think about an additional validation approach, which could include one of:

• Rather than comparing an I and F nuopc case, use a code base that supports both mct and nuopc; from each of mct and nuopc, do this F-then-I case and compare the nuopc vs mct I cases
• Writing forcing fields at a higher temporal frequency, possibly even every coupling time step, which should allow the two cases to match more closely
• Reviewing the relevant code (especially the datm code) for differences between the mct and nuopc versions

@olyson please chime in if I missed anything important or got something wrong.

[olyson]

Yes, I agree. Other things need to be changed also I think. E.g., the file naming conventions, instead of .cpl.ha2x1d., the new convention for aerosol deposition files seems to be .cpl.hx.atm.24h.avrg.. And if I recall correctly, I'm not sure it's necessary anymore that the daily coupler history files be concatenated to monthly first (i.e., the %ym in the file name which I assume means year/month). I could be wrong about that part, but @adamrher 's I-case just used the original daily files (I guess that will be proven in the validation).

[billsacks]

Okay, great. Would you be able to try out these changes, and then we can get a PR in to CDEPS once things seem to be working correctly?

[olyson]

Yes. I might press ahead first with the user_nl_datm_streams file I have now to make sure I can run an I-case with it, and then figure out how to modify stream_definition_datm.xml to generate the new datm.streams.xml file correctly.

[billsacks]

That makes sense. I'm happy to try to help with the modification of the xml file if it isn't clear.

[mvertens]

I'm also happy to help answer questions and solve issues if they come up. @billsacks is correct in that the naming of the atmImp_Faxa_b* fields were there to make the prealpha coupler history test pass with the existing files that are already out there for the test. It would be good to create a new set of testing files once the names are changed.

[adamrher]

An update from CAM. We are scheduled to have a cam tag for use for the next phase of the coupled eval, by next Friday. So if you complete the validation of cplhist in nuopc next week, we could use the following week to do the land spinup needed for the coupled sims.

[olyson]

@billsacks @adamrher @wwieder , an update on coupler history verification:
See document here:

https://docs.google.com/document/d/1ylQPmk8Ol274WZN0ivUwWPiVs-K1u0Jx/edit?usp=share_link&ouid=104026900087612515425&rtpof=true&sd=true

3-year F-case and I-case simulations have been completed using cesm2_3_alpha12c.
Most comparisons of forcing variables look reasonable (you can get a plot of a comparison of each variable by clicking on the link in the Verification Fields table).
Except for downward solar radiation (FSDSVD, FSDSVI, FSDSND, FSDSNI) which show larger differences than we would expect. This appears to be due to a time shift in these fields at the diurnal scale (see the second to last diurnal plot in the document labeled "offset=2700" which shows hourly output for a 10-day time slice at a single point).
In investigating this, I see that the time stamps for the solar radiation stream are different between the new coupler history files and old ones, i.e.,

NEW: time = 0.0416667, 0.0833333, 0.125
OLD: time = 0.03125, 0.0625, 0.1041667

Note that the old and new offsets are the same (2700). I tried changing the offset by an hour (-900). This seemed to fix the shift (see the last diurnal plot in the document). The climatologies are much closer as well (plot linked at the end of the document).
The last thing is that the COSZEN appears to be slightly different between the two simulations. Difficult to see in the plots, but not a zero difference. I'm not sure why they would be different.

[billsacks]

I actually would expect the values of coszen to be half-hour offset in cam_dev vs. cam. That's really what cam_dev does, it sends radiative fluxes to the CPL that were computed in the prior nstep, and so the COSZEN reflects where the sun was in the sky in the previous nstep, to be consistent with radiative fluxes it's receiving.

So, thoughts on how we should handle this? Do we want to keep CPLHIST-forced I compsets consistent with F compsets in this respect? If so, I think we could introduce a new namelist flag in CDEPS/DATM to specify an offset for nextsw_cday and have its default value differ for CPLHIST cases. It might be too messy to distinguish between cam_dev-produced cplhist and non-cam_dev-produced cplhist, but I guess we could assume cam_dev-produced cplhist moving forward....???

Also, on an unrelated note, I have submitted a PR to fix the CMEPS issue @olyson ran into here: ESCOMP/CMEPS#345

[olyson]

I can confirm @adamrher expectations regarding coszen. That offset goes away in cam, the coszen is synched up in I and F cases. So it seems like we need to accommodate this somehow.

[adamrher]

In the F-case, we've overhauled the nextsw_cday calculation to be general regardless of where radiation is in the time-loop, by setting it at the time radiation is called. Whereas it was previously computed in the nuopc/mct caps assuming radiation was in a certain place in the time loop. You can see how I do this in the cap here, where nextsw_cday is brought in through a use statement since it's now a public variable set in radiation. But these are just details ...

I don't know how nextsw_cday being calculated when running a DATM, but it seems it make's an assumption that this is cam6 radiation ordering. I think we should have the flexibility to accommodate both cam_dev and cam6's coszen calc's. We could do a namelist ... but I wonder if we can add an attribute to the radiation streams in user_nl_datm_streams telling the model whether it's cam_dev or cam6?

[billsacks]

Here is the nextsw_cday calculation in datm:

https://github.com/ESCOMP/CDEPS/blob/b5d4929ae2e6bdb8fa3f459ee3adc1a357e1b19f/datm/atm_comp_nuopc.F90#L835-L885

As a first step: @adamrher and/or @olyson - would you be able to determine the changes needed in that subroutine to make it match what's happening with cam_dev and test this?

Then we can consider how to get this behavior set correctly out-of-the-box. @adamrher it's a good point that it makes sense to tie this to the radiation streams... the thing I'm struggling with is how to ensure that this is set correctly for a given set of cplhist forcing data. I can think of a somewhat convoluted mechanism of having CAM set a NUOPC attribute that describes the radiation ordering which is then used by CMEPS when writing the radiation streams to set a piece of global metadata on the CPLHIST files, which is then read by CDEPS. My intuition is that that's going to be messy, though. Do you have ideas for how this could work?

[adamrher]

Yes, I can get this subroutine to work for cam_dev. Parsing this code block:

    if (liradsw > 1) then
       delta_radsw = liradsw * dtime
       if (mod(tod+dtime,delta_radsw) == 0 .and. stepno > 0) then
          nextsw_cday = julday + 2*dtime/shr_const_cday
       else
          nextsw_cday = -1._r8
       end if
    else

I believe we can hack this to work with cam_dev by reverting the current time by one time-step. That is the tod and julday input arguments. The revised block would look like this:

    if (liradsw > 1) then
       delta_radsw = liradsw * dtime
       if (mod(tod,delta_radsw) == 0 .and. stepno > 0) then
          nextsw_cday = julday + 1*dtime/shr_const_cday
       else
          nextsw_cday = -1._r8
       end if
    else

I think that 2*dtime in the original expression for nextsw_cday assumes liradsw=2 when liradsw>0, which is the default setting for the 1deg model. If so, should it be liradsw*dtime for cam and (liradsw-1)*dtime for cam_dev, to work for any arbitrary physics time-step? I guess we should verify this with Mariana, as I may be missing something that forced her to set that to 2.

In terms of a sustainable, non-hacked implementation for use with both cam and cam_dev for use with CPLHIST ... What you lay-out @billsacks is what I was thinking when I proposed it. But there must be a more elegant solution.

The purpose of nextsw_cday is to inform CTSM that it needs to provide an albedo to CAM for use in computing radiation in it's next time-step, if that next time step is a radiation time-step. Why is this needed in DATM? Is CTSM's coszen and albedo completely dependent on nextsw_cday?

[edit - the answer to my question I believe is yes, for consistency. Back to the drawing board ...]

[mvertens]

@adamrher - I think your analysis is correct about the presence of 2*dtime. The logic here is exactly what was used in mct - so this goes back a long way.
The reason that this is in DATM is that CTSM needs this information for its radiation computation regardless of what the atmosphere component actually is- its not just to send albedos back to CAM. This would imply CLM computing how much radiation is actually absorbed/reflected by the surface (i.e. solar fluxes absorbed by veg and ground surface). And yes - CTSM's albedo computation is dependent on the nextsw_cday it receives. Hopefully I have this all correct. Maybe at @olyson could add some details here?

[olyson]

So..., now that the F-case that produces the coupler history files is running, I decided to repeat some of the coupler history verification work I did previously, by doing a short I-case forced by those coupler history files and comparing CLM history fields between the F- and I-cases. All of the differences look as expected based on my previous results, with the exception of PCO2. There seems to be larger differences for this field than expected.
After some research I found that CLM is not actually listening to the CO2 field provided by the coupler history fields. Instead, it is being forced by CCSM_CO2_PPMV (verified by changing this from the 1850 value of 284.7 to a value of 355).
Yes, you may ask how this appeared not to be a problem previously. The reason is that the previous F-case defined co2vmr = 284.7e-6 in atm_in which is the exact same value as prescribed by CCSM_CO2_PPMV. In the new F-case, CO2 appears to be prescribed from a boundary file:

flbc_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-20150116_CMIP6_0p5degLat_c180905.nc'

The values in that file end up being slightly different from 287.4.

At a minimum, it seems like "flds_co2" needs to be true in order for these fields to be "advertised" in components/cdeps/datm/datm_datamode_cplhist_mod.F90:

if (flds_co2) then
call dshr_fldList_add(fldsExport, 'Sa_co2prog')
call dshr_fldList_add(fldsExport, 'Sa_co2diag')
end if

But it looks like there should also be some code in lnd_import_export.F90 and esmFldsExchange_cesm_mod.F90, However, the code there keys off of flds_co2a, flds_co2b, flds_co2c, and I'm not sure how flds_co2 is actually supposed to be used, if at all. So, some review and guidance from someone who understands this would be appreciated.

[mvertens]

@olyson - I am happy to help with this. Can you please outline exactly how I can reproduce the problem and what you expect to see more clearly. I am also happy to set up a quick call. I actually think that might be the most efficient way to move forwards.

[klindsay28]

@olyson , does you case have CLM_CO2_TYPE=diagnostic in env_run.xml?

[olyson]

Thanks @klindsay28 and @mvertens . No, it does not have CLM_CO2_TYPE=diagnostic. It is set to constant out of the box for a CPLHIST compset. I had tried setting it to diagnostic, but got the following error:

(lnd_import_export:import_fields) ERROR: must have Sa_co2diag in import state if co2_type equal diagnostic

@mvertens , my I-case is here:

/glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_VALIDATE_1850AD

I'm using cesm2_3_alpha14a. The case is driven by coupler history files from an F-case. I expect CLM to be forced by atmImp_Sa_co2diag in the coupler history files through Sa_co2diag as assigned in datm.streams.xml. But it appears to be forced by CCSM_CO2_PPMV.

A meeting would be great, probably best for me after about 11am MDT tomorrow (I'd like to see some of the CESM workshop talks today). I'll keep looking at it today and maybe a solution will present itself.

[adamrher]

Pls incl me in this mtg. The switch to pull co2 from the flbc file is new to the low top model (the FLT/BLT compsets); this was previously done in WACCM though.

[olyson]

@mvertens , you should be able to re-create my case by running this script (you'll probably have to change some paths that are hardcoded to oleson):

/glade/u/home/oleson/run_hist_1850_files/SPINUP/setup_run_CPLHIST_SPINUP.csh

The case runs fine, it's just that the co2 provided by the coupler history files are not being used by CLM.

[olyson]

Ok, tentatively, it looks CLM is forced by coupler history co2 if I do the following:

./xmlchange CCSM_BGC=CO2A
./xmlchange CLM_CO2_TYPE=diagnostic
Add the following to user_nl_datm: flds_co2 = .true.

It looks like some logic needs to be added to get these settings out of box in a CPLHIST compset.
In particular, for example, the logic in namelist_definition_datm.xml sets flds_co2 to .true., but if DATM_CO2_TSERIES=none (as in a CPLHIST compset), it is set to .false.

[mvertens]

@olyson -I think you have effectively resolved the issue! That's great. Do you think it would still be good to meet?

[olyson]

No, thanks, appreciate the offer, but I think it is working now with those changes. Just need to implement them so that we get that behavior out of the box, I'll discuss with others.

[adamrher]

Thanks, Keith. My only remaining concern is that this is not also a problem in FLTHIST/BLTHIST. Do you have suggestions for how I might check this?

[olyson]

I think this is exclusively an I-case (DATM%CPLHIST) problem. I've compared the co2 field from the coupler history files generated by the F-case with the CLM co2 history field in the same F-case and they were within roundoff. The problem arose when I compared the CLM co2 history field in the F-case and the I-case, when the I-case was driven by coupler history files from the F-case. It is simply an issue of the I-case not being setup correctly, such that co2 from the coupler history files was not being used by CLM.
I don't have any reason to suspect there is anything wrong with the FLT1850 (or FLTHIST) cases.

[adamrher]

ah, I see. Thanks for putting my concerns to rest.

[olyson]

@wwieder , please see NCAR/amwg_dev#301 for F-case spinup results/discussion.

[olyson]

I've begun a cold-start AD spinup forced by years 25-44 of the coupler history files from f.cam6_3_112.FLT1850.ne30.landspinup.001 (NCAR/amwg_dev#301). I'm using cesm2_3_alpha14a (which has ctsm5.1.dev123).

/glade/work/oleson/cesm2_3_alpha14a/cime/scripts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850AD
/glade/scratch/oleson/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850AD/run

Unfortunately, it has crashed at year 12 with a methane conservation error:

1038: CH4 Conservation Error in CH4Mod during diffusion, nstep, c, errch4 (mol /m^2.t
1038: imestep) 206875 1 NaN
1038: Latdeg,Londeg= -3.13304571417066 26.5000000000000
1038:iam = 1002: local column index = 1
1038:iam = 1002: global column index = 7298
1038:iam = 1002: global landunit index = 2638
1038:iam = 1002: global gridcell index = 1003
1038:iam = 1002: gridcell longitude = 26.5000000
1038:iam = 1002: gridcell latitude = -3.1330457
1038:iam = 1002: column type = 1
1038:iam = 1002: landunit type = 1

I'll start tracking it down today....

[olyson]

Turns out the methane error was a red herring. I ran some segments to get more frequent restart files to troubleshoot and it ran past the point where the error had occurred.
I see some errors like this before the methane error:

MPT: rank 59 dropping unexpected RC packet from 1431 (10459:10460), presumed failover - 328 0 264

which has recently been noted in the Daily Bulletin as being as system problem.
I reran but it crashed at year 15 with a dropped packet error.
I will try setting #PBS -l place=group=rack as suggested in the bulletin.

[adamrher]

I'm all too familiar with these dropped packet errors. Cecile and I have noticed that in the past week, we are encountering this much less often. But CISL told me last Friday it's going to be a couple weeks to replace the problematic hardware.

[olyson]

The 1850 AD and post-AD simulations are complete (<3% disequilibrium in TOTECOSYSC). They took 240 and 560 years, respectively.
The restart/initial file resulting from this is:
/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc

I'll start a production run to get monthly history files so that the diagnostics package can be run. One possible simulation for comparison is the end of the CLM5.0 1850 spinup run I did for the CLM5.0/CTSM5.1 comparisons:

https://www2.cgd.ucar.edu/tss/clm/diagnostics/clm5.0_ctsm5.1/index.html

In particular, I have data from this simulation:

clm50_cesm23a02cPPEn08ctsm51d030_1deg_GSWP3V1_1850pSASU

which was used to initialize the CLM5.0 historical that was run at that time.

[wwieder]

@olyson should @adamrher, @cecilehannay and others use this restart for land initial conditions in upcoming tests?

/glade/p/cgd/tss/people/oleson/CLM5_restarts/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc

[olyson]

I think yes. NCAR/amwg_dev#349

[olyson]

Actually, that wouldn't be a good simulation to compare to since it uses GSWP3 forcing. Since the forcing for the spinup is unique, I've run the diagnostics for model-obs, not model-model:

https://webext.cgd.ucar.edu/I1850/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850/lnd/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850-obs/setsIndex.html

[wwieder]

Thanks, Keith. What about comparing to the CESM2 PI control or the first ~20 years of a CESM2 historical run? Is that easy enough to do?

[olyson]

That makes sense, I'll see what I can find.

[olyson]

There is a comparison against a segment of the CESM2 piControl here:

https://webext.cgd.ucar.edu/I1850/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850/lnd/ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850.11_30-b.e21.B1850.f09_g17.CMIP6-piControl.001.1080_1099/setsIndex.html

[wwieder]

@olyson can you take a look at the following:

• dead PFT's globally, I think we had an old NCL script that did this.
• persistent snow across the arctic, but especially in Siberia, which killed plants in CESM2 (this may be easier with polar views of minimum annual H2OSNOW and SNOWDP + maximum GPP and ELAI).

[dlawrenncar]

For the persistent snow problem in Siberia, I would note that I don't think we ever looked at whether or not snow never melted in the summer, or snow persisted too long into the season to allow vegetation to establish. So, a simple metric inquiring whether or not snow ever goes to zero may not be sufficient. The real question is whether or not the area with vegetation that does not establish in the high Arctic has decreased.

…

On Tue, Jul 25, 2023 at 10:46 AM will wieder ***@***.***> wrote: @olyson <https://github.com/olyson> should @adamrher <https://github.com/adamrher>, @cecilehannay <https://github.com/cecilehannay> and others use this restart for land initial conditions in upcoming tests? /glade/p/cgd/tss/people/oleson/CLM5_restarts/ ctsm51_cesm23a14a_ne30pg3ne30pg3mg17_CPLHIST_1850pAD.clm2.r.0561-01-01-00000.nc — Reply to this email directly, view it on GitHub <#1951 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFABYVGDARRCJNX42ZHEDEDXR7Z7DANCNFSM6AAAAAAU5LISK4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[olyson]

Briefly discussed this with @wwieder and @slevis-lmwg at our liaison meeting today. I'll start looking at this next week.