Migration to automerge

[dmaretskyi]

Topics for discussion

Database design

• Consistency model: do we gurantee update atomicity between objects in a single space.
• Storing entire space in one automerge document vs a separate document per object.
• Versioning.

Architecture

• How ECHO objects are modeled in Automerge 2.
• On-disk storage protocol.
• How is an index of documents created. Does it need to be consistent between peers.
• Communication between the worker and app threads.
  • Sending full snapshots
  • vs generate/receiveSyncMessage
• Sync protocol and it's integration into DXOS networking stack.
  • How do we know when we are in sync with the remote peer?
• Architecture diagram.
• Rich text editor integration.
  • Fix remote change handling automerge/automerge-codemirror#4

A very high level diagram of ECHO/HALO for reference:

References

• https://github.com/automerge/automerge-codemirror

[dmaretskyi]

We have begun a POC integration between Automerge & ECHO:

• The goal is to do the switch from YJS to Automerge without changing the public ECHO Api.
• Existing data in spaces will be migrated to Automerge during the Epoch creation process.
• We use automerge-repo to handle replication & storage with adapters to use existing storage backend & networking subsystem
• Entire space is stored inside a single automerge document. We plan to change this later so each ECHO object is stored in a separate document.
• Data is currently replicated to any peer that knows the document ID. Later we plan to integrate with HALO to replicated only with peers that have permission to view the data.

[wittjosiah]

@dmaretskyi it's not just yjs this would be replacing, but the hypercore feeds as well, correct? Not needing to plug in yjs to do text is then a side effect of that?

[dmaretskyi]

it's not just yjs this would be replacing, but the hypercore feeds as well, correct?

Yes, although we're gonna keep using hypercore for HALO where auditability is more important.

Not needing to plug in yjs to do text is then a side effect of that?

Yes automerge can handle text as well.

[wittjosiah]

Yes, although we're gonna keep using hypercore for HALO where auditability is more important.

Interesting, so going to keep both? Would someone be able to configure ECHO to use hypercore instead of automerge? What is automerge missing re: auditability that you wouldn't want to use it for HALO?

[jessmartin]

As of today, Automerge is the default storage mechanism for ECHO Spaces. We haven't yet removed Yjs/Hypercore, but that's coming next.

[wittjosiah]

I believe Hypercore is still being used in some places, but just not for the primary data store anymore. There are still control feeds for every space that are Hypercore feed afaik. Is that accurate @dmaretskyi?

[jessmartin]

Actually, @dmaretskyi would be good to have a more general update here on what decisions were made and where we ended up with this migration.