Describe underlying maths and numerical implementations within vignettes

[TimTaylor]

Highlighting this issue from finalsize as the principle will apply across multiple packages.

Currently the finalsize documentation is missing any real description of the methods employed, both in terms of the underlying mathematics and the numerical solver implementations. Whilst references to papers are given, as a user I would appreciate the package being self contained with all relevant information, including methodology assumptions, available in a vignette.

Do others agree with this requirement? If so - should it be made a "release blocker" (for want of a better term) for CRAN releases of Epiverse packages?

Due to the background required, I suspect these will often require drafting from the PI.

[jamesmbaazam]

I'm glad that you asked, Tim. I do see it as a great addition to our methods packages.

I discussed this with Seb sometime back and raised a similar issue on epichains (See epiverse-trace/epichains#35).

I assigned @sbfnk as he originally derived some of the analytical solutions.

[jamesmbaazam]

Related to the issue of self-containment, bpmodels also has a vignette with a bibliography of applications of branching processes in epidemiology.

I thought it would be a good idea for users to be able to see how the methods have been applied elsewhere. I have also outlined how users can contribute to the bibliography list. Here is an example of a PR contributing to the list.

[jamesmbaazam]

Do others agree with this requirement? If so - should it be made a "release blocker" (for want of a better term) for CRAN releases of Epiverse packages?

I think it's a nice standard to establish.

[pratikunterwegs]

Do others agree with this requirement? If so - should it be made a "release blocker" (for want of a better term) for CRAN releases of Epiverse packages?

I agree that the packages should have their maths and their algorithms documented, but for it to be made a release blocker we'd need to get the PIs or more mathematically minded folks to commit to contributing this documentation.

The methodology assumptions are IMO slightly different, in that those could be written by RSEs without necessarily a full understanding of the maths (see e.g. the "what we assume" callouts in {finalsize} vignettes). These could, by themselves, be very useful to many users. It really depends on who we think the target users are, and whether they will have or want mostly an empirical understanding of our packages ("run these commands in sequence") or a deeper theoretical one as well.

[TimTaylor]

I agree that the packages should have their maths and their algorithms documented, but for it to be made a release blocker we'd need to get the PIs or more mathematically minded folks to commit to contributing this documentation.

Yes - but that is a good thing!

It really depends on who we think the target users are, and whether they will have or want mostly an empirical understanding of our packages ("run these commands in sequence") or a deeper theoretical one as well.

I think it's best to assume users that want to gain the knowledge and making that as frictionless as possible should be the goal.

[pratikunterwegs]

Yes indeed it's a good thing - my point (poorly phrased) was that we should build consensus around this and get the maths folks to agree to doing it before adopting it as a standard, rather than adopting it first and then springing this requirement on them.

[avallecam]

I think it's best to assume users that want to gain the knowledge and making that as frictionless as possible should be the goal.

I agree with this statement.

I'm thinking about where to add this information, depending on the defined audience for one vignette.

We can have two types of vignettes:

• "how-to guides"-like vignettes, more focused on steps and code, with the minimally required amount of concept/method explanations, and
• "explanation"-heavy vignettes, closer to book/course level details when required.

An additional advantage of this could be in the review process. It should be easier to track a step/equation in a line of a vignette than in a paper. Alternatively, add numbered equations from papers when available in R files.

[adamkucharski]

I think there's a balance to be struck, as a lot of the methodology in our packages will be about numerical efficiency rather than epidemiological concepts. For example, to properly understand what an SIR simulation with {deSolve} is doing, the user would need to know the fundamentals of calculus and numerical integration (e.g. why Runke-Kutta preferable to Euler method).

To take {finalsize} as an example, the underlying concept is fairly straightforward (i.e. it gives the final size of the epidemic we'd expect from SIR model) but the steps required to get their efficiently require a few tricks (e.g. dividing the SIR ODEs to get dS/dI, integrating the resulting equation, then using a numerical solver like Newton's method). I'd say the {cfr} package is different, because the key delay calculation doesn't require much beyond thinking through how to add up vector elements, so equation more accessible.

When I've taught modelling over the years, I've found that including brief technical details can sometimes be worse than just pointing to further reading for the subset who are interested - it can make the approach more intimidating than it needs to be for occasional users, and can also be unsatisfying (mathematical tricks like the dS/dI step above feel a bit like pulling a rabbit out of a hat). Routinely providing book/course level details would also require a massive time commitment (more than is feasible for many packages).

A few options that may be more doable as routine practice (some of which noted above):

• Give equations briefly where relevant/accessible and point to external resources, e.g. like the vaccineff pages, rather than writing a full methodological tutorial. The {bpmodels} vignette also seems like reasonable compromise, giving the equations but not the full generating function theory or derivation. The underlying equations/methods are already coded in the packages, so this would be an easier starting point from a development point of view, then would just need some additional conceptual context and references.
• Include figures to highlight key concepts, like the EpiEstim pages have
• Feature examples of wider applications and case studies, as well as good reference lists, so users can explore these.
• Point to review papers or introductory concept guides already out there, e.g. this one from PRISM

[TimTaylor]

I think there's a balance to be struck, as a lot of the methodology in our packages will be about numerical efficiency rather than epidemiological concepts. For example, to properly understand what an SIR simulation with {deSolve} is doing, the user would need to know the fundamentals of calculus and numerical integration (e.g. why Runke-Kutta preferable to Euler method).

Agree and the suggestion is not to go in to the exact details of numerical methods ...

To take {finalsize} as an example, the underlying concept is fairly straightforward (i.e. it gives the final size of the epidemic we'd expect from SIR model) ...

Yes but even the underlying concept nor input assumptions are really described in the {finalsize} documentation

but the steps required to get their efficiently require a few tricks (e.g. dividing the SIR ODEs to get dS/dI, integrating the resulting equation, then using a numerical solver like Newton's method).

Yes and I really do think these details should be in a vignette.

When I've taught modelling over the years, I've found that including brief technical details can sometimes be worse than just pointing to further reading for the subset who are interested - it can make the approach more intimidating than it needs to be for occasional users, and can also be unsatisfying (mathematical tricks like the dS/dI step above feel a bit like pulling a rabbit out of a hat).

As @avallecam alluded to, R packages can contain multiple vignettes which have different foci (and even different audiences). I often do not read all package vignettes, only the ones I am interested in. My main point is this information should be in a vignette.

Routinely providing book/course level details would also require a massive time commitment (more than is feasible for many packages).

If there is remaining funding I think there would be a massive benefit to bring some dedicated modellers in to the group to enable this time commitment.

From my own personal / organisation perspective - EpiVerse packages have a massive opportunity to really stand out and set the standard here. Having a decent level of information contained within the package documentation rather than hidden behind a linked-list of papers (many paywalled) and books (likely expensive) would be a massive improvement for knowledge accessibility.

[sbfnk]

I broadly agree that it's really useful if a package contains the vignette laying out the methodology that's used in the code - also helps users/reviewers checking that the functions do what they're supposed to do.
I recently added a PR with a background vignette to bpmodels here in case anyone has comments/suggestions for improvements in this context:
epiforecasts/bpmodels#134

See also the "Model definitions" vignettes we recently added to EpiNow2:
https://epiforecasts.io/EpiNow2/dev/

[TimTaylor]

Yes - they're exactly the sort of thing that I was thinking for content (minor pr incoming for bpmodels theory)

[adamkucharski]

Yep, like the bpmodels example. Agree good to improve access to open access materials on methods where possible. As our training materials expand – which will have dedicated resources – alongside links with organisations focused on open training (like AppliedEpi), it should become clearer what sits best where. Expect there will also be quite a lot of overlap in places, so can decide whether replication or linking makes more sense.

[adamkucharski]

Started a draft with finalsize theory vignette (fortunately had the equations latexed up in some old draft documents – may not be so lucky with other packages!) Will add varying susceptibility shortly. Hard to avoid lots of integrals (especially as don't think we can use the dS/dR trick for multiple age groups) epiverse-trace/finalsize@88808cd

[pratikunterwegs]

Thanks, if you could ping me when complete I can link to this from the other vignettes too

[Bisaloo]

As an aside, Mathpix is quite good to convert equations from papers or handwritten equations to LaTeX / Mathjax.

[adamkucharski]

I've updated the vignette – might be good to get a fresh pair of eyes on it from @BlackEdder before doing a PR, just to make sure the version of heterogeneous susceptibility I've outlined is appropriated given the code implementation.