Add an HTTP API to configure contacts and contactgroups

[nilmerg]

Current State

At the moment contacts can only be configured by using the UI. Contactgroups cannot be configured at all. (see #174)

Problem

We can safely assume that consumers already have their users and usergroups defined somewhere. Maintaining them again for Icinga Notifications shouldn't be necessary.

Solution

We should provide a way to automate their creation. The easiest way is to provide a basic HTTP API to do so:

Authorization

The API must require the notifications/api/v1 permission.

Endpoints

Users: notifications/api/v1/contacts[/<identifier> | ?<filter>]
Groups: notifications/api/v1/contactgroups[/<identifier> | ?<filter>]

Parameters

identifier
This is a UUID. For resources created in the UI, this is a UUIDv4.

filter
A usual filter query string

Request Body Validity

• A request's body is, if accepted, always expected to be JSON. Since it always describes a single resource, a JSON object. No envelope structure is required. The object is the resource definition itself
• Every key not followed by a question mark (?) is mandatory
• The default channel of a user must be an existing channel's name (created manually in the UI)
• Referenced resources (contact -> groups, group -> contacts) must already exist

Resource Structure (Request and Response)

Contact

{
    id: identifier,
    full_name: string,
    username?: string,
    default_channel: string,
    groups?: identifier[],
    addresses?: {}
}

Contactgroup

{
    id: identifier,
    name: string,
    users?: identifier[]
}

Methods

GET

• If a filter is passed
  • Respond with a HTTP 200 code and a list of resources, even if empty
• If no filter and no identifier is passed
  • Respond with a HTTP 200 code and a full list of resources, even if empty
• If an identifier is passed
  • Respond with a HTTP 200 code and the resource, if it exists
  • Respond with a HTTP 404 code, if it does not exist

POST

• If a filter or an invalid request body is passed
  • Respond with a HTTP 400 code
• If an identifier is passed
  • Respond with a HTTP 422 code if the resource already exists and is not renamed
  • Respond with a HTTP 201 code if the resource is renamed and include the resulting resource endpoint (with new identifier) in the Location header
• If no identifier is passed
  • Respond with a HTTP 422 code if the resource already exists
  • else, respond with a HTTP 201 code, create the resource and include the resulting resource endpoint (with identifier) in the Location header

PUT

• If no identifier or an invalid request body is passed
  • Respond with a HTTP 400 code
• If a valid request body is passed
  • Respond with a HTTP 201 code, if the resource is created
  • Respond with a HTTP 204 code, if the resource is updated

DELETE

• If no identifier is passed
  • Respond with a HTTP 400 code
• If the resource is not found
  • Respond with a HTTP 404 code
• If the resource exists
  • Respond with a HTTP 204 code and remove it

Schema Changes

• Add column external_uuid to contact and contactgroup icinga-notifications#192

Implementation Requirements

• A single controller for each endpoint
  • Don't rely on other parts of this project (e.g. forms)
  • Straight forward code, no unnecessary optimizations or modularisation attempts
  • Quick, but not dirty
• Bypass the ORM, exclusively utilize ipl-sql
  • Implement GET ?filter responses like https://github.com/Icinga/icingadb-web/blob/v1.1.2/library/Icingadb/Data/JsonResultSetUtils.php#L70-L100 and disable PHP's output buffering

[julianbrost]

User

{
    full_name: string,
    username: identifier,
    default_channel: string,
    groups?: identifier[]
}

Group

{
    name: identifier,
    users?: identifier[]
}

So there will be two places where group memberships can be updated? Will the behavior be that on an update, memberships that aren't given, will be removed, i.e. to add a group to a user, you have to query the user first? Does omitting the groups/users attribute mean, that memberships remain unchanged?

• The username column of the contact table must not be nullable

That column is used to store an optional reference to an Icinga Web user. That would imply that each contact is linked to one then.

• The name column of the contactgroup table must be unique

That column currently stores a display name. Sure, that can be encoded as part of an URL, but is this desired here?

But in general, sounds like you want/need a user-chosen primary key for those tables. I wouldn't rule out just doing this instead.

[nilmerg]

So there will be two places where group memberships can be updated? Will the behavior be that on an update, memberships that aren't given, will be removed, i.e. to add a group to a user, you have to query the user first? Does omitting the groups/users attribute mean, that memberships remain unchanged?

Yes. Yes. Yes. My expectation is that whoever uses this API, just imports data from another source, so there's no need to fetch anything first, as all information is already available.

But in general, sounds like you want/need a user-chosen primary key for those tables. I wouldn't rule out just doing this instead.

I had a discussion with Eric about the use of UUIDs, which I'd chosen first. Though, they'd still need to reference an identifier (UUIDv5) that is known to both sides, i.e. mandatory in any case. Otherwise (UUIDv4) we'd had to prevent changes in the UI to resources created through the API and vice versa. The primary key isn't an option, hence the username. The group name is indeed more of a label right now.

My goal was, not to limit edits in any way. A resource created through the API should be changeable in the UI. This means, by creating one in the UI, the identifier must be provided, just the same as when creating it through the API.

[julianbrost]

I had a discussion with Eric about the use of UUIDs, which I'd chosen first. Though, they'd still need to reference an identifier (UUIDv5) that is known to both sides, i.e. mandatory in any case. Otherwise (UUIDv4) we'd had to prevent changes in the UI to resources created through the API and vice versa. The primary key isn't an option, hence the username. The group name is indeed more of a label right now.

My goal was, not to limit edits in any way. A resource created through the API should be changeable in the UI. This means, by creating one in the UI, the identifier must be provided, just the same as when creating it through the API.

Sounds like you think something is a problem where I'd say it's perfectly fine. UUIDs are an obvious choice, so let's stick to that. I'd say it's perfectly fine to make the primary key a UUID without any restrictions on the version. If a contact ist created using Web, it just gets a random UUID (v4) assigned. If a contact is created using the API, the client chooses the UUID however it desires. If it syncs from somewhere else that already uses an UUID to identify the source object, use that, otherwise, use a hash-based UUID (v5) or even use a random UUID and keep some state, doesn't matter for us, that would be a decision for the API client author.

Creating a contact in Web and then updating it using the API should be possible, yes, but if your sync source is the primary data source anyways, why wouldn't you create all contacts using the API? If your use-case is to just update individual attributes like an e-mail address for existing contacts, the API client could still query the contact by username using the filter mechanism and then update it by the returned ID. Or it could just query all contacts and then update those, where an update is necessary. So that would still be possible without a predictable ID.

[nilmerg]

Fine. Let's use UUIDs as identifier. contact.username and contactgroup.name are left untouched then. The UUID of a resource will be part of the structure as id key. But I wouldn't make the UUID the new primary key. We'd have to change every reference in the schema to contacts and groups then.

Oh, btw, I forgot to include addresses. -.-

[julianbrost]

But I wouldn't make the UUID the new primary key. We'd have to change every reference in the schema to contacts and groups then.

Yes, but it sounds like the way cleaner option, doesn't it? Would this result in an unreasonable amount of work in Notifications Web?

[nilmerg]

Yes, but it sounds like the way cleaner option, doesn't it?

It's not required. I'd rather add a new column for this, as a start. We still don't have versioning, so the schema isn't stable anyway..

[julianbrost]

However, if we just replace the primary key type of these two columns, it's a bit of an arbitrary mix.

Another idea for how two different columns could make sense in my opinion: keep the current numeric ID as-is and add a second column, something like external_id or external_key that is nullable and unique. For objects created using the web interface, this value is just not set, but if objects are created via the API, it's an optional field the client can use to store auxiliary information to later identify the same record. I.e. the default identifier stays the numeric ID, but if desired, the API client could access the objects also by this external ID (or username for that matter), all of these would be fast due to the existence of a corresponding index. The type could either be UUID/16 bytes or even just some string type so that the client could store whatever they want, like LDAP DN, ID reference to whatever other database, etc. without requiring any hashing, thus allowing lookups in the other direction as well.

[nilmerg]

the default identifier stays the numeric ID

Nope. That goes against everything I read. If we keep our numeric ID, it's won't be exposed in the API.

The type could either be...

A single type. I really don't want to support multiple ways if UUID is one of them. It should be the only one.

thus allowing lookups in the other direction as well.

This is something we already solved and agreed on:

Creating a contact in Web and then updating it using the API should be possible, yes, but if your sync source is the primary data source anyways, why wouldn't you create all contacts using the API? If your use-case is to just update individual attributes like an e-mail address for existing contacts, the API client could still query the contact by username using the filter mechanism and then update it by the returned ID.

However, if we just replace the primary key type of these two columns, it's a bit of an arbitrary mix.

Then let's introduce a new column of type UUID and make it required.

[julianbrost]

the default identifier stays the numeric ID

Nope. That goes against everything I read. If we keep our numeric ID, it's won't be exposed in the API.

Why not? Also, I'm not really sure what you read.

The type could either be...

A single type. I really don't want to support multiple ways if UUID is one of them. It should be the only one.

Of course we should pick one. Just wanted to say that the exact choice won't matter for the rest I wrote.

thus allowing lookups in the other direction as well.

This is something we already solved and agreed on:

That's not what I wanted to say with that. If you have a field large enough to store an unhashed reference, an API client could retrieve a list of all contacts, look them up by say a stored LDAP DN, and check if anything needs to be updated. Not sure how commonly someone would want to build something like this, I just wanted to say that's something that would additionally be possible if it was a "store whatever you want" type.

[nilmerg]

Why not? Also, I'm not really sure what you read.

To prevent enumeration attacks.

If you have a field large enough to store an unhashed reference, an API client could retrieve a list of all contacts, look them up by say a stored LDAP DN, and check if anything needs to be updated.

If the identifier is a UUID, chosen by the client, no checks are necessary. The client just PUTs the data it has and nothing else. And all with a "store whatever you want" type.

[julianbrost]

Why not? Also, I'm not really sure what you read.

To prevent enumeration attacks.

Yes, in general, unpredictable IDs add an additional layer of defense. But isn't the API supposed to allow listing all objects anyways?

If the identifier is a UUID, chosen by the client, no checks are necessary. The client just PUTs the data it has and nothing else. And all with a "store whatever you want" type.

Syncing object deletion would be annoying that way though as you don't really have a way of telling how that UUID was generated. If you have an contact that says external_id = 'uid=poorguy,ou=something,dc=example,dc=com', a sync client could simply check if that still exists in LDAP and if it doesn't, issue a DELETE request.

[nilmerg]

We concluded that two columns (pk + external_uuid), both required, are sufficient for now. Making the pk the UUID can be done at a later point, together with a custom fact for the client to identify its own resources. (To allow safe removals)