extension registry management

[dret]

i just discovered the extension registry and it's great to see this getting started (as per #1351, i guess). my suggestion would be to decouple this from the spec and turn it into a repo and site of its own. one of the main motivations for registries is to decouple spec and extension management, and managing these two things separately usually is a good idea.
another thing i would suggest is to track registry updates, and to have clear extension policies. maybe https://tools.ietf.org/html/draft-wilde-registries-01 can help with better understanding the issues that a registry should follow if it is expected to be reliable and robust over time.

[MikeRalphson]

Thanks @dret We did debate having a separate repo for this, but came down on the side of using the gh-pages branch as a pseudo-separate repository with its own history. We can revisit this decision easily if it becomes necessary. The registry has a site of its own at spec.openapis.org/registry.

The gh-pages branch can (and should) have its own CONTRIBUTING.md guidelines, and I'll be sure to work through the IETF draft you linked to. This is one reason we haven't removed the large "DRAFT" watermark from the site yet.

[dret]

On 2019-01-13 22:42, Mike Ralphson wrote: Thanks @dret <https://github.com/dret> We did debate having a separare repo for this, but came down on the side of using the |gh-pages| branch as a pseudo-separate repository with its own history. We can revisit this decision easily if it becomes necessary. The registry has a site of its own at spec.openapis.org/registry <https://spec.openapis.org/registry>.

ok. better than nothing, i guess. i still think that these things should be entirely decoupled. after all, form example there may be extensions for AsyncAPI (if that ever gets to be a proper OpenAPI-based thing) which then maybe also should be listed in that registry?

The |gh-pages| branch can (and /should/) have its own |CONTRIBUTING.md| guidelines, and I'll be sure to work through the IETF draft you linked to. This is one reason we haven't removed the large "DRAFT" watermark from the site yet.

one thing is the mechanical aspect. the more important one are the semantics: once you register, you cannot make breaking changes. things cannot be unregistered, they will simply be marked as deprecated. submissions must be in some specific format. there must be associated online resources for submissions. the list goes on, but i think it would be essential to define these things before the registry gets populated. if you feel like there is anything wrong/missing in the draft, please ping me or even better, raise an issue. i haven't worked on it in a little while, but it is on my list of things i'd like to pick up again.

[MikeRalphson]

once you register, you cannot make breaking changes. things cannot be unregistered, they will simply be marked as deprecated.

Yep, I think that's understood by all.

submissions must be in some specific format

Submissions will be made by GitHub PR (by the requester, or by an @OAI/tsc member on their behalf), a GitHub PR template will capture all mandatory metadata (such as a base_type for a format, a schema for an extension etc).

there must be associated online resources for submissions

Could you expand on the need for "online resources", if external, that sounds fragile.

@tedepstein @earth2marsh I think you've both mentioned the need for process around this, it would be really helpful if you could start to work up a draft CONTRIBUTING.md / Pull Request template, as this would help to stop this looking like a one-man crusade 😄

[MikeRalphson]

@dret

there may be extensions for AsyncAPI (if that ever gets to be a proper OpenAPI-based thing)

Just FYI, at the moment AsyncAPI is pursuing a course with their own foundation outside the OAI (which sidesteps issues with our charter scope).

[dret]

On 2019-01-14 02:54, Mike Ralphson wrote: @dret <https://github.com/dret> here may be extensions for AsyncAPI (if that ever gets to be a proper OpenAPI-based thing) Just FYI, at the moment AsyncAPI is pursuing a course with their own foundation outside the OAI (which sidesteps issues with our charter scope).

i know. i just wanted to use an example that at least semi-practically makes clear why separation of concerns is a good thing here.

[dret]

On 2019-01-14 01:16, Mike Ralphson wrote: once you register, you cannot make breaking changes. things cannot be unregistered, they will simply be marked as deprecated. Yep, I think that's understood by all.

great. i haven't looked at how the registry is defined and described in the spec, and how the registry page talks about submissions, but this should be very prominent in both places.

submissions must be in some specific format Submissions will be made by GitHub PR (by the requester, or by an @OAI/tsc <https://github.com/orgs/OAI/teams/tsc> member on their behalf), a GitHub PR template will capture all mandatory metadata (such as a |base_type| for a format, a |schema| for an extension etc).

very good. most importantly, there must be an "abstract schema" for what submissions must contain (fields, their types, mandatory or optional). how to represent and submit also matters, but is secondary to that.

there must be associated online resources for submissions Could you expand on the need for "online resources", if external, that sounds fragile.

yes it is. but that's how things are in decentralized systems. it is unlikely that your submissions will capture *all* data. so you will probably also ask for a URI to a persistent resource that fully documents the extension. this could disappear, that's true, but part of the submission requirements would be that it would supposed to be persistent and stable (i.e., it doesn't get changed in terms of the semantics).

@tedepstein <https://github.com/tedepstein> @earth2marsh <https://github.com/earth2marsh> I think you've both mentioned the need for process around this, it would be really helpful if you could start to work up a draft |CONTRIBUTING.md| / Pull Request template, as this would help to stop this looking like a one-man crusade 😄

sounds good. let me know if you guys need an extra set of eyes. i think this is the opposite of a one-man crusade. it is what happens when standards grow up: people get invested enough that they want to use but extend the standard. good standards are really careful about enabling and managing that part of the standards journey carefully. i've seen at least four registries already being considered, right? that's a good sign, i think. here's an idea: these four registries should all use the same general approach. anybody interested in using this as a great real-life example for coming up with something that would support the registry mechanics we're talking about here, but that would be reusable?

[MikeRalphson]

Thanks @dret, that's all very helpful and I really appreciate the offer of an extra set of eyes.

The four initial registries all use the same basic jekyll static site collections to achieve their persistent URLs from metadata + markdown.

The registry site itself could be easily re-used (say by JSON Schema for their own formats registry, or by AsyncAPI) and is all Apache-2.0 licensed like the spec. itself. It wouldn't take very much to make it a white-box solution.

[dret]

On 2019-01-14 10:04, Mike Ralphson wrote: Thanks @dret <https://github.com/dret>, that's all very helpful and I really appreciate the offer of an extra set of eyes. The four initial registries all use the same basic jekyll static site collections to achieve their persistent URLs from metadata + markdown. The registry site itself could be easily re-used (say by JSON Schema for their own formats registry, or by AsyncAPI) and is all Apache-2.0 licensed like the spec. itself. It wouldn't take very much to make it a white-box solution.

that looks great. any interest in actually doing this? i'd be on board. my claim is that many orgs actually could use registries but that they are a bit hard to get off the ground. having a repo i can fork and brand and then simply configure (i want these five registries, and here is my config data for them) would be very helpful.

[tedepstein]

@MikeRalphson wrote:

@tedepstein @earth2marsh I think you've both mentioned the need for process around this, it would be really helpful if you could start to work up a draft CONTRIBUTING.md / Pull Request template, as this would help to stop this looking like a one-man crusade 😄

Happy to help. I'll try to have a draft by end of this week. @earth2marsh , please ping me if you'd like to pair-program, tag-team, or whatever. :-)

[tedepstein]

@MikeRalphson, I'm starting on the Pull Request template and CONTRIBUTING.md guide for the registries.

I'm not clear yet on what a PR for a new registry entry should look like, and I assume we don't yet have an example of such a PR. I have the impression, based on what you've written here and in related issues, that the PR format would be fairly obvious if I understood how Jekyll works; I just don't have any experience with Jeckyll.

If Jeckyll is the right place to start, I'm happy to do some background reading, and hopefully the pieces will all fall in place. You pointed me to the Jeckyll docs in response to an earlier question. Should I start there?

[MikeRalphson]

I'm not clear yet on what a PR for a new registry entry should look like, and I assume we don't yet have an example of such a PR

Exactly so. The PR should add one new markdown file with YAML front-matter, where the template section is based on an existing entry for the same registry. (The template is a mixture of values brought in from the YAML front-matter and other Jekyll collection/site settings and free-form text like examples).

The PR template would have a number of check-boxes to ensure the relevant data has been collected. I'm not sure whether one generic PR template would do, or if one PR template per registry is better... I've just thought of one wrinkle, how does the PR author add the issue YAML property linking to a PR they haven't yet finished creating? This could be solved by always having an issue first then a linked PR, with the PR and the YAML referencing the original issue number.

You pointed me to the Jeckyll docs in response to an earlier question. Should I start there?

Yes, I think that's a good starting point. For googling other resources, drop the stray c from Jekyll. 😁

Once you've read the basics, have a look at Jekyll collections (https://jekyllrb.com/docs/collections/) which is the feature we're exploiting to build the registries out.

[dret]

On 2019-01-16 16:11, Ted Epstein wrote: I'm not clear yet on what a PR for a new registry entry should look like, and I assume we don't yet have an example of such a PR. I have the impression, based on what you've written here and in related issues, that the PR format would be fairly obvious if I understood how Jekyll works; I just don't have any experience with Jeckyll.

i wouldn't spend time learning jekyll right now. jekyll simply has a way of picking up structured data from pages, and this structured data has to be provided in the preamble (the initial part of the page) and has to use YAML format. jekyll then parses this data and makes it available to scripting. some of the data has built-in support in jekyll or certain packages, other is generic data that is parsed and simply made available to jekyll template code (which uses a language call liquid).

If Jeckyll is the right place to start, I'm happy to do some background reading, and hopefully the pieces will all fall in place. You pointed me to the Jeckyll docs in response to an earlier question. Should I start there?

again, i'd highly recommend to first have the data model of what's required, and i think we discussed that already somewhere. it basically should be everything that matters for representing the registry, so that i could extract and rebuild the registry somewhere else simply by using the data i can get through the API.

[MikeRalphson]

so that i could extract and rebuild the registry somewhere else simply by using the data i can get through the API.

I'm so not sure that's an actual use-case we would want to enable. 🙂 Though it may be a worthwhile intellectual exercise to ensure we have moved all the relevant data into the YAML front-matter.

[tedepstein]

@MikeRalphson , are registries considered part of the 3.x development scope?

@darrelmiller, should we add this to #1466, the Big List of Possibilities, so we can keep it on our radar?