Larger max size for README and CHANGELOG

[rydmike]

Max size of README and CHANGELOG too small

The maximum char size (128kB) I assume, for README and CHANGELOG in pub.dev packages is too small.

Can you please increase the size?

Shame to have to delete history from CHANGELOG, which I now have to do for the change log in this case. OK so I like to be verbose, I don't do terse 😄

I could move the CHANGELOG history to an offsite link, and only offer link to content for every version in the log. Already had to move the docs to its own site.

Publishing is forever; packages cannot be unpublished.
Policy details are available at https://pub.dev/policy

Do you want to publish flex_color_scheme 6.0.0-dev.1 to https://pub.dartlang.org (y/N)? y
Uploading...
`CHANGELOG.md` exceeds the maximum content length (131072 bytes).
pub finished with exit code 1

[sigurdm]

I'm fine with upping the limit to 512k or whatever as long as we have a limit in place.

@jonasfj and @isoos do you see any problems here?

[jonasfj]

side note: @rydmike you have an awesome changelog 🚀

We absolute should encourage writing great changelogs.

I think datastore will limit us to 1MB (so long as we keep it unindexed), and we probably have to leave room for a few other attributes.

So, yeah, bumping the limit to 512k seems reasonably, but we should do a test case on staging to make sure it works.

---

I think we only did this because wanted to set some limit to prevent degenerate changelog files.
And ofcourse there is some point at which the solution is to put a link a the bottom saying that for versions prior to 1.2.3, see CHANGELOG-pre-1.2.3.md or something like that.

[isoos]

I am 50-50% on this, maybe slightly to concur: I think we should rather keep the current limit.

One part is Datastore: besides the single entity limit, we also have an overall transaction limit, and at upload time we write a lot of entries in a single transaction. (We could separate the content extraction from the main transaction though, but that introduces an additional backfill process).

The other part is the sheer size: the current changelog for flex_color_scheme is 40+ pages. It is impressively detailed, but I doubt that users of v5.1 are interested in changes that happened 1.5+ years ago in v1.3.

I think we should figure out how these files could grow and how they should be displayed. Maybe we need to customize the content extraction of the changelog, and display only the last-N versions (or up until 128 Kb). That way the input file could keep its size, and we also limit the size to the common query patterns.

[jonasfj]

The other part is the sheer size: the current changelog for flex_color_scheme is 40+ pages.

Yeah, it's not clear that users would suffer from having to follow a link to read the CHANGELOG for very old versions.
Or we could support something like git does with a file per release:
https://github.com/git/git/tree/master/Documentation/RelNotes

[sigurdm]

Just as an example: https://github.com/emacs-mirror/emacs has a split changelog.

Maybe something like that is the way to go.

I think for now we will not up the limit.

If this is an issue for you, please vote, and/or reply.

[rydmike]

@sigurdm, @jonasfj @isoos, thanks for the discussions and input.

Any changelog can of course easily be split into e.g. two parts. Having the one with latest changes named CHANGELOG.md, that then at end links to one with all older less relevant changes. This certainly works fine both technically and for all practical usage purposes for older logs too.

I did however not do so yet, since I was uncertain how pana would treat that?

Is there a score penalty if it cannot find a change log entry for any released version in the CHANGELOG.md file itself? At least pana (or it might be pub publish that checks it) nags if you try to publish a new version and there is no changelog entry for the version to be published.

If it checks for an entry for every published release, it becomes quite a bit more tedious to migrate to older logs. Still doable of course, just keep the version entry heading, and add linked files with the actual content, starting at oldest releases, newer still "inlined". But quite tedious to migrate to this. I would probably just keep dummy/version header entries in CHANGELOG.md then for everything, and move entire changelog content out of repo to the docs repo. Simpler to maintain.

[isoos]

I did however not do so yet, since I was uncertain how pana would treat that? [...]
or it might be pub publish that checks it [...]
If it checks for an entry for every published release [...]

The expectation of these tools is (and likely will remain) that the current release contains a changelog entry for the current version, and uses consistent styling so that we can separate one version's entry from another. In a distant future pub.dev may collect and organize these entries outside of the specific release-changelog pairs (e.g. display the changelog of 1.0.0 from the released version of 1.1.0, in case there was a retrospective update of typo fix in the later release). However, we won't expect that a changelog has entries for all versions release.

[rydmike]

Ok and thanks that's good to know.

I think I will move changelog content from each release to its own file and leave the version number in the file, starting from oldest ones. I will just move enough old entries per new release, to get required chars free for next changelog entry.

This way the chore is not so big, and the approach should be compatible with most analysis that might be done now and later on by pub and pana.

[jonasfj]

I think I will move changelog content from each release to its own file and leave the version number in the file, starting from oldest ones. I will just move enough old entries per new release, to get required chars free for next changelog entry.

This seems like a good solution.

If lots of people run into this issue, then we might need to reconsider our approach. But most packages don't have many versions, and most versions have small changelogs (not that we should discourage large changelogs).

[spydon]

We just ran into this issue with Flame too, it's very unfortunate that this limitation isn't posted anywhere (that I have seen at least) so that you only get to know about it during a release. Is there anywhere that this limitation could be made public, if it not already is that is?

This is our changelog:
https://github.com/flame-engine/flame/blob/main/packages/flame/CHANGELOG.md

If we'd go the route that @rydmike has gone in his package we would have 389 more files in our repository, but that is of course something that we could consider but currently Melos is not supporting generating one file per version.

[rydmike]

Hi Lukas @spydon, as far as I know the max 128kb limit is not documented anywhere. It applies to both readme.md and changelog.md, I have hit it on both. It should definitely be documented.

To get around the limit, the readme I made a minimal starter file and moved actual docs to its own web site and linked to it. For the change log, I started with oldest versions, left the version header, moved log entries for each version to its own file and added a link to that file under each header for moved older change logs content. Tedious, but worked well, and should be compatible even if tooling will start to check for an entry for every released version.

The 128kb limit on readme and changelog is imo silly small for complex and heavily continuously developed packages, like eg Flame. Imo 512kb would be more reasonable, even if it is also quite small.

Btw, since this issue is closed, I doubt anybody that is not tagged will notice your issue comment. So probably only me, unless we mention a few more 😃

I was thinking about your documentation question and comment about the limit. I think that is a good check and if not existing to document properly. It is also easily actionable, but probably better to open a new issue for it and reference this case.

[AlexV525]

Just added a check for packages to make sure we don't run into this during releases. cfug/dio#1977

[spydon]

@jonasfj @isoos @sigurdm is it possible to get you to reconsider bumping the limit of the chanelog? 😇
Like @rydmike says, a 128KB changeling really isn't that big, and I haven't found any other package repositories that impose similar restrictions.

[sigurdm]

I haven't found any other package repositories that impose similar restrictions.

I think having some sanity restriction is absolutely necessary given how we currently display the entire thing in a single page-load. Also we have internal restrictions in how much we can store in a single datastore entry.

I also think that looking at the flame changelog it really shines in showing useful information - there's not really anything I could point to and want to take away.

I think we have a limit on 1000 versions of a package on pub.dev, and if we want to give each version (on average) a 1kB entry in the changelog we need to allow at least 1MB, and that might even be on the small side.

I hope in the future we can merge our "versions" tab and the "changelog" tab, and have the changelog entry for each version be "foldable", and perhaps do some kind of paging when there are many versions, perhaps also by default hide versions not supported by the latest dart/flutter sdk.

Short term I think we should just bump the limit to something like 256 kB - it doesn't seem overly large.

[AlexV525]

If you can raise warnings progressively, like Your CHANGELOG.md is >128kb (the maximum is 256kb), consider split it into separate files., and decrease pub points, that makes more sense.

[spydon]

If you can raise warnings progressively, like Your CHANGELOG.md is >128kb (the maximum is 256kb), consider split it into separate files., and decrease pub points, that makes more sense.

I think decreasing pub points for having a good changelog really sends out the wrong signals.

[AlexV525]

I think decreasing pub points for having a good changelog really sends out the wrong signals

AFAIK it's already there, you'll get less points if your docs are not following specific conversions.

[spydon]

AFAIK it's already there, you'll get less points if your docs are not following specific conversions.

Following the conventions has nothing to do with the size of the changelog, a good changelog is following the conventions currently imposed so that one (and systems) can know which entry is bound to which version, this totally makes sense.
Just having a long changelog because you're having well documented versions is not breaking such convention and should not decrease points. But a warning when you are getting near to the limit is a good idea!