Writable API

[rixx]

This issue tracks the design and implementation of a new, writable REST API. The development of this feature is possible due to NLNet and their NGI Zero Entrust grant. Thank you! 💖

                   

Project plan / Progress tracker

• Overview and planning
  • Prior art: look at APIs by similar project (by technology and topic)
  • Current status: collect problems, design mistakes and frequently requested features in current API
  • Plan transition: communication strategies for breaking changes, support windows for old APIs etc
• Implementation
  • Code
    • Migration and deprecation of old API
    • handling of multilang strings
    • handling of nested resources (turning embeds on/off)
    • Migration of read-only endpoints to new, consistent output
      • Proposals (Schedule/submission API changes #1345)
      • Schedules (Finalise and document /schedules/ API #1367)
      • Tags
      • Speakers
      • Reviews
      • Rooms
      • Questions
      • Answers
    • Implementation of write actions
      • Proposals (including state changes)
      • Schedules (including releases)
      • Tags
      • Speakers
      • Reviews
      • Rooms
      • Questions (and options)
      • Answers
    • Time permitting: Addition of further writable endpoints
      • Teams and members (Have an API Endpoint for Teams #1417)
      • Speaker information
      • Event settings
      • CfP settings
      • Session types
      • Tracks
      • ActivityLog
      • QueuedMail
      • MailTemplate
      • AccessCode
      • Feedback
  • Documentation
    • Versioning
  • Tests
• Authentication and authorisation
  • Guarantee synced use of pretalx permissions in views and APIs
  • Authentication token system for scoped / limited API access
    • Permission concept (ro, writable, restriction by resource? handling of nested resources?)
    • Tests
    • Security review
• Release
  • Process accessibility audit findings
  • Published pretalx release
  • Announcement post

This checklist omits (by necessity) a lot of details, but tracks the project plan submitted to (and approved by) NLNet. Please note that I can't provide a timeline. I hope to get the whole thing done in 2024, but, y'know, this estimate is provided AS IS etc etc.

Further updates will follow in comments to this issue.

[rixx]

Current problems

The first step in rebuilding/expanding the API is an assessment of its shortcomings. These include:

• read-only, of course
• lack of endpoints (see list above), lack of data (e.g. event settings)
• no versioning
• inconsistent use of references by ID, vs by name (we always want to use references by ID/code as default). I'd say this is the biggest problem with the current pretalx API as it stands.
• no clear expectations/defaults for plugin APIs
• 1 token = 1 user, no possibility of permission scoping ("this token can only do x")
• [possible problem?] i18nstrings are not friendly for consumers
• [possible problem?] documentation not generated automatically. Upside: more incentive/opportunity to write proper text. Downside: not pretty, easy to drift away from actual API behaviour

[rixx]

Prior art

APIs I want to imitate are on the one hand the pretix API (same tech, general peeking at style/handling) and on the other hand the Stripe API.

• pretix API
  • Clear documentation of breaking changes and API reliability expectations
  • Conditional fetching with Last-Modified / If-Modified-Since / 304 Not Modified
  • File upload API
  • Uses tokens that convey permissions (though I don't think we want to tie tokens to teams? needs further thought, but I think API tokens should at least have clear read/write permissions, though probably not entity-based, as that woud work terribly with expansion …)
  • Use of JSON schema when permitting edits to JSON fields
  • Exception logging for better user support
• Stripe API
  • Versioning: attaches API version to tokens/auth. This version will then be consistently used for all future requests. Can be overridden temporarily via request header or permanently via web interface (permitting downgrades to previous version for 72h). docs
  • Expansion: Defaults to IDs everywhere, but permits expansion via request params
  • Separate changelog for API versions/updates
  • API docs. I don't think there will be the time to build anything approaching the Stripe docs in the scope of this issue, sadly

[rixx]

Transition plan

Technical migration

Given that we want to switch to Stripe-style versioning, without an API version in the URL, this is actually quite doable without interruption of existing APIs:

• New endpoints will be added, but this causes no interruption
• Users will be migrated to use tokens (of some scope and some kind), but their existing token can be migrated to have their current full permissions
• The migrated tokens will have version 2023.1.0. We will deprecate this version immediately, and set an end-of-life date, which also gives us basically free testing of the deprecation warning mechanism.
• We implement the new API alongside the old one, forcing us immediately to have solid mechanisms for version detection, temporary upgrades, and clean code for version selection (as I imagine this can clutter up rather fast).

Communications

We will have a big blog post explaining the new API. I don't think listing differences to the current one will be useful, but we should stress that old embedding behaviour can still be introduced, and that users will be safe on the old API for a set amount of time.

Additionally, pretalx needs to warn users using a token for a deprecated API about the deprecation timeline, once per token (via interface as dismissible alert? But definitely via email). This warning should include the timeline, the endpoints used, to be triggered only after use (as all users have tokens!), and in this specific case, link to our blog post in addition to the documentation.

Support timelines

As I'm running pretalx on my own, and I'm fairly confident that future API changes will be less disruptive than this one, I think version upgrades should go like this:

• release 0: endpoint exists at version 1
• release 1: endpoint exists at version 1, version 2 is introduced
• release 2: endpoint is deprecated at version 1 (warnings are sent on use, once per token)
• release 3: endpoint v1 is removed

This means I have to support two versions of the same endpoint for two releases, which feels doable and fairer than giving users only a single release to upgrade. It also allows for more relaxed testing of the new version – if we deprecate the old version immediately, while the new version is still unproven or may have problems, that leaves users with the choice of "new but sucks" and "deprecated but stable", which I'd prefer to avoid.

This plan assumes that pretalx will release 2-4 times a year (which seems like a safe assumption right now), giving users at least half a year to test the new API and migrate (or at least ~3 months after receiving the notification). If releases get more frequent, we should revisit this.

A downside of this choice is that users who use pretalx only once per year (for their event), will always have to update their tokens/versions. But as I don't think endpoint upgrades will be all that frequent or disruptive (if we turn "new field in output" into a versionable offence, that'll be an easy upgrade for almost everybody), I think that should be alright, and I'm not really willing to provide multi-year support for deprecated APIs as a solo developer.

[FlorianWilhelm]

Yes, this sounds awesome :-) As we are planing already PyConDE/PyData 2024 and of course use Pretalx again, will a first useable version be released before the end of this year? I'm currently weighing up whether it makes sense to adapt Pytanis to the new API before the review process in Januar or not.

EDIT: Okay, carefully rereading your initial post, it seems that it will take until February 2024. Then it will be rather subject for PyConDE / PyData 2025. CC @alanderex

[rixx]

Yeah, I'm afraid there's absolutely no chance of any actual writable part landing this year – parts of the rewrite might be merged this year, but I'm involved in running a large event at the end of the year, so most of the progress will take place next year. Even then, February is probably wildly optimistic, and I'd be more confident about something like April. Sorry! Update to avoid confusion: I still hope to get this done this year, but due to other work, illness and other problems, I can't provide an actual timeline.

[saerdnaer]

@rixx If you have time, could you add an update on the progress or update the estimates? If there's anything I could help with, feel free to let me know.

[rixx]

If you have time, could you add an update on the progress or update the estimates?

This issue represents the current progress. The initial issue statement contains a big checklist, which is updated whenever things get finished. I assume your impatience is due to the time estimates there – I'll remove them to avoid future confusion. It'll be done when it'll be done.