Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Distribution RFC #98

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Distribution RFC #98

wants to merge 16 commits into from

Conversation

sordina
Copy link
Contributor

@sordina sordina commented Feb 15, 2024

RFC for distribution aspects of connectors.

Related specifications:

While the precursor specifications outline the structure and mechanisms of packaging, this RFC outlines how the packages are owned, distributed, and indexed. A layered solution is outlined from storage up to user-applications and how they can be leveraged by CI in order to automatically publish updated versions and scrape topics for community contribution discovery.

See the Rendered Version of the RFC here.

@sordina sordina self-assigned this Feb 15, 2024

The intuition for this system is inspired by other package management systems such as NPM, Cabal, etc.

There was a previous implementation of these concepts as described (TODO: Get docs links from Shraddha)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might be working on the v2 of this legacy implementation. We could probably try to accommodate this spec in this v2 implementation.

cc: @imperfect-fourth

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


The CLI commands will include the following:

* hasura3 package create
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the new DX implementation, we always have hasura3 <verb> <noun>. So, please consult with DX team to see if this is okay.

* hasura3 package publish
* hasura3 package revoke
* hasura3 package search
* hasura3 package fetch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we aim to include the word connector somehow in the package-related commands? Because package might be something that we want the user to relate to Hasura connectors rather than it being like a top-level concept in Hasura.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! Package could be implicit by the operations on the connector.

@paf31
Copy link
Collaborator

paf31 commented Feb 15, 2024

Can you please add some notes about the motivation here, and specifically why we're considering building something new, as opposed to the Git-based approach that worked for the v2 GDC registry? My concern is primarily that a PR-based workflow is a lot simpler for connector authors than learning and using some custom API or CLI.

@sordina
Copy link
Contributor Author

sordina commented Feb 16, 2024

@paf31 I've added a Motivation and Why and How this Differs from Existing Solutions section.

The main reason to change the existing system is simply that if we're packaging definitions independently of the git registry we should have somewhere to index/reference them. The rest follows from there. That being said if people find the registry workflows convenient there's no reason why this couldn't be preserved as a tributary to the system.

The current system already has the DB/API components mentioned so it's not that radically different in practice.

Maybe I wasn't explicit enough in the RFC, but the ultimate goal is to have automation take care of publication rather than have authors manually using the CLI etc.

What do you think?


* Pull-based ingestion of registry - Changes should ideally propagate instantaneously
* No ability to treat connectors independently - Need to PR to central registry
* Artefacts are not distributable outside of Github references - Tied to Github
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a simple workaround for this - link to the release tarball on GitHub, or any other tarball hosted anywhere else.


Issues with the current system:

* Pull-based ingestion of registry - Changes should ideally propagate instantaneously
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we fix this by running the job more frequently, or in a commit hook?

@paf31
Copy link
Collaborator

paf31 commented Feb 16, 2024

@sordina Thanks. I've added a couple of notes on two of the three points you noted. On the third:

No ability to treat connectors independently - Need to PR to central registry

I think this is a feature, not a bug. If the registry is considered as a community-driven resource, then I think a PR is the appropriate way to suggest and review changes. Otherwise, we have to build an awful lot of custom versions of open tools.

@sordina
Copy link
Contributor Author

sordina commented Feb 18, 2024

@paf31 How about if the existing behaviour is preserved in the new system as one tributary? The new proposed system very closely matches the existing system anyway.


The initial change to the metadata format would look as follows for the [Postgres Connector](https://github.com/hasura/ndc-hub/blob/lyndon/distribution-rfc/registry/postgres/metadata.json):

```json
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would the CLI plugin binary URL also go in here?

// New stanza
"packages": [
{
"uri": "https://foobar.com/releases/postgres-postgresql-v0.2.0-9283dh9283u...hd092ujdf2ued.tar.gz",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this tarball contain exactly? A Dockerfile + any scaffolding files?

For connectors which don't have a separate build step, where does the prebuilt docker image come from?

sordina added a commit that referenced this pull request Mar 12, 2024
RFC for distribution aspects of connectors.

Related specifications:

* [NDC specification](http://hasura.github.io/ndc-spec/)
* [Deployment
Specification](https://github.com/hasura/ndc-hub/blob/main/rfcs/0000-deployment.md)
* [Packaging Specification
(WIP)](https://github.com/hasura/ndc-hub/pull/89/files)
* [Umbrella specification](#98)

While the precursor specifications outline the structure and mechanisms
of packaging, this RFC outlines the initial distribution of package
definitions.

See the [Rendered
Version](https://github.com/hasura/ndc-hub/blob/lyndon/distribution-rfc-milestone-1/rfcs/0002-distribution-gh.md)
of the RFC here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants