Removal of shuttered APIs

[MikeRalphson]

Following on from @andypiper's comment on issue #1052 - should we remove the definitions of shuttered APIs?

I see a few pros and cons.

• It would reduce the risk of pointing developers at APIs they can no longer use and thus maybe wasting their time
• It would enable developers to find actively developed APIs quicker
• Accessing old API definitions would become more complex for researchers / those comparing API changes / those wanting to re-implement an API (i.e. old git commits would have to be searched)
• The update logs would be less full of 404 errors
• If we are not informed, how long between an API url 404ing and removal?
• What if the API definition URL has simply changed and we don't know. Searching for moved URLs takes time

Another option would be to move old/shuttered/defunct/404ing APIs to another directory within the repo, this would hide them from the UI and the API but keep the files around for research and potential moving back to "live".

Would love to hear some thoughts.

[andypiper]

I've also been away for a few months (!) and I'm no longer involved at Twitter. It seems as though "advertising" shuttered or closed APIs via this project is only going to lead to disappointment for anyone wanting to use them, so best to remove, in my opinion.

[MikeRalphson]

Hi @andypiper - random question, but I see from your RSVP you might be at Oxford Python this evening?

[andypiper]

That's the plan, about to drive that way now - see you later!

[pimterry]

old git commits would have to be searched

I think that's fine - well within what anybody digging into outdated API data is going to expect, and convenient enough for everybody involved. It's also very easy to bring things back from Git history, which I think negates the value of any 'recycle bin' approach.

What if the API definition URL has simply changed and we don't know. Searching for moved URLs takes time

This is the hardest part I think... It would be really useful to get a feel for the typical reasons why a spec disappears to know how much this matters and how it's normally handled. Is there a list of the current 404s easily available somewhere? It would be interesting to do a quick sample of a few at random, and see if any obvious patterns jump out immediately.

There are some cases where we could be more nuanced though, for example in the GitHub-hosted case at least, we can handle 'GitHub temporarily unavailable' vs 'repo deleted' vs 'spec file deleted' all separately. Also in official vs unofficial spec cases - if an unofficial spec is removed, to me that sounds much less likely that the API is completely gone, and so maybe we should keep hosting it ourselves, whereas if the official spec is removed then something really has changed on the provider side (either a real update or a removal entirely) and we should be doing something to handle it.

[MikeRalphson]

@pimterry I need to publish some proper development/contributing guidelines (see #1109) once I've taken a view on whether to open-source the registry package which now drives APIs.guru (it's currently a private repo/package on Github). There was a thought to allowing other interested parties to set up their own compatible repositories of internal and external API dependencies, but we never got any interest shown. There was also a consideration of what should be non-open in case someone wanted to acquire APIs.guru, but that seems a remote likelihood at the moment.

Opening things up would mean the registry metadata (currently a 3Mb blob of YAML, but I'm looking at Acebase) could move into the main repo too. The new API submission automation scripts will probably move into Github workflows when they're more battle-tested.

For the cases where the API still exists in a 1:1 form and the URL just needs updating, would you be able to raise some PRs against the info.x-origin.url properties in the main repo YAML files? You don't need to amend the format or version properties or rename the files, that'll get done by an update sweep.

I feel a blog post coming on where we could shame API providers who break OpenAPI URLs over time!

[MikeRalphson]

I have a solution for data:base64 and blob:// URL definitions (as used by ReDoc) so that one will be ok.

[pimterry]

There was a thought to allowing other interested parties to set up their own compatible repositories of internal and external API dependencies, but we never got any interest shown. There was also a consideration of what should be non-open in case someone wanted to acquire APIs.guru, but that seems a remote likelihood at the moment.

I have a sort-of similar situation for HTTP Toolkit - my conclusion so far is that licensing as AGPL with a tiny CLA for contributors (https://cla-assistant.io/httptoolkit/) works pretty well, and lets you go fully open-source without getting pinned into a corner forever. AGPL blocks any competition from closed-source direct forks of everything, and the CLA leaves you the option to change the license in future as required (because it explicitly gives you permission to use any license you like for the contributions you receive).

You can never retroactively stop people using the open source version once it's out, but this would allow you/somebody buying APIs.guru to say "from now on all future changes are open-source/closed source" or "yes, you can buy a non-GPL version for self-hosting, we will relicense it to you under a custom agreement for $$$" or similar.

I'm definitely not a lawyer, but it seems like a pretty tidy option AFAICT.

For the cases where the API still exists in a 1:1 form and the URL just needs updating, would you be able to raise some PRs against the info.x-origin.url properties in the main repo YAML files?

Sure, I'll take a look at that tomorrow 👍

I'm not totally clear what you mean about the data:base64 solution - what should go in the origin in that case? The entire base64 blob???

[MikeRalphson]

It's slightly like something out of 1984 (we have always been at war with Eurasia) but I've used git filter-branch to retroactively license https://github.com/Mermade/api-registry as AGPL, added a CLA workflow, and open-sourced it. Thanks for the prompt.

[pimterry]

Awesome, great to hear, thanks @MikeRalphson! It'll be great to be able to dig into that directly next time I run into issues, that's super helpful 👍