Peer sends message with sequence 16 as the first message

[boreq]

I just noticed the following sequence of messages exchanged with a peer (irrelevant payload redacted):

level=debug 
ts="2023-01-17 22:33:29.5185650 (UTC)" 
source=golang 
id="@CIlwTOK+m6v1hT2zUVOCJvvZq7KE/65ErN6yA2yrURY=.ed25519" 
header.flags="<stream=true endOrError=false bodyType={json}>" 
header.number=3 
header.bodyLength=14020 
body="{\"@3PGWnoMIY5ebs6sJjkwc7DKjxrgcrOeIUC2YegkMtIY=.ed25519\":0}" 
name=gossb.peer-2188669348362399538.raw 
message="sending a message"

level=debug 
ts="2023-01-17 22:33:29.8222370 (UTC)" 
source=golang 
id="@CIlwTOK+m6v1hT2zUVOCJvvZq7KE/65ErN6yA2yrURY=.ed25519" 
header.flags="<stream=true endOrError=false bodyType={json}>" 
header.number=-3 header.bodyLength=438 
body="{\"previous\":\"%MXYkWNlD73IZUpEgIZvTyShNR0Eb0JsfRbBXfB2v6Qw=.sha256\",\"author\":\"@3PGWnoMIY5ebs6sJjkwc7DKjxrgcrOeIUC2YegkMtIY=.ed25519\",\"sequence\":16,\"timestamp\":1494275440982,\"hash\":\"sha256\",\"content\":{\"type\":\"pub\",\"address\":{\"host\":\"ssb.nootch.net\",\"port\":8008,\"key\":\"@1ZCF+Z9cUjcXGHkOPd5jNnoiuu3VJn+o6L8XEliV6Cs=.ed25519\"}},\"signature\":\"EFtsPPoYb2kl3U62Qt8dNRQ/taix+bBttlN0yMCjlUYiweEAhUsMZGpYDGnxk9S4i7VBUi40CKX+odSQ3wfxBw==.sig.ed25519\"}" name=gossb.peer-2188669348362399538.raw 
message="received a message"

...

level=debug
ts="2023-01-17 22:33:30.5232780 (UTC)" source=golang
feed="@3PGWnoMIY5ebs6sJjkwc7DKjxrgcrOeIUC2YegkMtIY=.ed25519"
sequence_in_database=nil
sequences_in_buffer="16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27"
messages_that_can_be_persisted=0
name=gossb.message_buffer
message="persisting messages"

Log gobot-2023-01-17_22-33.log. The peer is Planetary System Pub 1.

Related issues:

• Message buffer health is often low during replication #168

[boreq]

I feel like this is a nonetheless a bug with the way the JS stack caches some things.

[boreq]

I am letting this go all the way back to "backlog" as I don't know what to do with this right now. It probably requires fixes to the JS stack and I haven't seen this in a while.

[boreq]

Findings:

• the fix for this is here ssbc/epidemic-broadcast-trees@8d1d2db
• v8.0.3 is the earliest version of ssbc/epidemic-broadcast-trees that has this fix
• we use https://github.com/ahdinosaur/ssb-pub
• https://github.com/ahdinosaur/ssb-pub uses RUN npm install -g ssb-server@15.3.0
• ssb-server@15.3.0 uses "ssb-ebt": "^5.6.1" which means <6.0.0
• ssb-ebt 5.6.7 is the latest release <6.0.0 and it uses "epidemic-broadcast-trees": "7"
• therefore the fix for this is not running on our pubs

docker logs 88c16dbabfac 2>&1 | head 

> ssb-server@15.3.0 start

[boreq]

Easy fix: run $ rm ssb-pub-data/ebt/* every couple of minutes. 😩

[boreq]

Other solutions:

• update the pubs (we failed to do this due to some kind of a bootloop)
• set up an extra node running scuttlego that is not even a pub just caches messes (works out of the box)