Proposal for Messages API

[milan-cvetkovic]

Problem

There are already several issues related to OSM Messaging and the visibility of messages to the user base:

• PR #4070 - cosmetic modification on how number of new messages is presented
• Issue #3129 - (closed) Block function to individual users
• Forum discussion

As a main form of communicating to users OSM uses email. In some scenarios, OSM website displays a message notification indicator. In other scenarios, i.e. changeset comments, notes and diary entries activities, OSM website only sends emails to subscribed users. In some cases, there are RSS feeds as well.

If user uses alternate applications to contribute to OSM, there is limited API support for querying information related to messages users would be interested in:

• Querying user details shows the summary counters for direct messages. Some applications use this info to display a notification and redirect users to OSM web site to access their messages.
• There is no support for reading, replying or modifying messages.
• Extracting the changeset comments user is subscribed to has limited support. There are active PRs improving the support in this area: PR #4359. As of now, there is indirect support through RSS feed. This is not in scope for this github issue.

Description

Proposal - Extend API v0.6 to include Messages API

This is the proposal for OSM Messages API addressing direct messages only.

Requirements

• applications need to ask the user's permission to access the messages on OSM on their behalf. The existing OAuth2 authorization framework would support the authorization, however the messaging is not covered by existing scopes. We propose to add 2 additional scopes, one for reading messages, and an additional scope for creating new messages and sending them.
• Messagesing API requires the client to be properly authorized by authenticated user.
  Only messages related to the authorizing user are accessible - the user must be either the recipient or the originator of the message.
• To prevent bulk messaging, sending a message must be throttled, similar to how changeset submission is throttled.

This proposal assumes the existing data model for messages in OSM. A message is shared by the originator and recipient. The messages can be marked as read/unread or deleted by the user viewing them. OSM website displays the messages separately in "inbox" and "outbox".

In addition, the API provisions for paginating the list of messages. Message id is used for ordering since for any other fields, for example ordering by message timestamp, would require an additional index.

Throttling Considerations

To prevent abuse, we would implement throttling in a similar manner to the existing implementation for API from PR #4319 or extending it to apply to the messaging API as well.

### Client Transitioning

We plan to work on transitioning OSM web site to use the Messaging API instead of directly accessing the content of the database. This will improve the user's experience by adding the pagination to inbox and outbox view. It would also be subject to the throttling limitations imposed by the API.

As the first client of this API, this will give us a chance to learn about API usage ergonomics.

API Summary

API would allow 3rd party applications to implement front-end for complete management of direct messages sent to or received by OSM users. The following functionality would be provided:

• List messages received by the logged in user, limit the count of messages, support pagination and filtering by read/unread messages only:

  GET /api/0.6/user/messages/inbox
  

• List messages sent by the logged in user, limit the count of messages, support pagination:

  GET /api/0.6/user/messages/outbox
  

• Retrieve the content of a specific message:

  GET /api/0.6/user/messages/#id
  

• Update the read status of a message

  POST /api/0.6/messages/#id
  

• Delete a message:

  DELETE /api/0.6/messages/#id
  

• Send a new message:

  POST /api/0.6/messages
  

Screenshots

No response

[tomhughes]

To prevent bulk messaging, sending a message must be throttled, similar to how changeset submission is throttled.

Mesage sending is already rate limited - has been since 81d47fe with further enhancements in 25510b6.

We plan to work on transitioning OSM web site to use the Messaging API instead of directly accessing the content of the database.

This will definitely be controversial - there have been differing opinions over the years and at one point I understood that the plan was to move in this direction but more recently things have swung back the other way and I know @gravitystorm is not a fan of moving things client side and have the web site work via API calls.

[milan-cvetkovic]

We posted a more detailed API draft on wiki: https://wiki.openstreetmap.org/wiki/Messaging_API_proposal

[milan-cvetkovic]

I believe we would need to apply rate limiting for sending messages from the API as well. I would also put a limit on some read operations, for example for retrieving the list of inbox messages.

This will definitely be controversial - there have been differing opinions over the years and at one point I understood that the plan was to move in this direction but more recently things have swung back the other way and I know @gravitystorm is not a fan of moving things client side and have the web site work via API calls.

Our intent was to have it there as a proof of concept, hoping it was a good thing to do. If this is against the OSM way, we can leave it out.

[gravitystorm]

We plan to work on transitioning OSM web site to use the Messaging API instead of directly accessing the content of the database.

As @tomhughes anticipates, that's a "hard no" from me on this approach. 😄 We already do this for a couple of parts of the website (e.g. Notes) and we're working on specifically undoing this. It's overcomplicated, it's unnecessary indirection and causes way more hassle for little to no tangible benefit.

So I'm in favour of a Messages API (nb not 'Messaging') but the web interface should continue to be a straightforward server-side-html / html-over-the-wire implementation like the rest of the site.

In other scenarios, i.e. changeset comments, notes and diary entries activities, OSM website only sends emails to subscribed users.

This is another thing that will need to be clarified up-front. The topics of Messages and Notifications are two different topics. We currently lack a proper Notifications system, and this often leads to confusion between notifications and messages. In particular, since there is a email notification sent for new Messages, and you can respond to that notification and it's treated as a new Message, that's a source of confusion. Additionally, the "Unread Messages" indicator on the site makes people think that it is a "notifications count", when it is not, and we don't have any way to view notifications on the site (e.g. view a list of changeset comment notifications that you have received).

I hope we can build a proper notifications system in the near future, and refactor all our existing event notifications to use it.

In the meantime, it's important that these two topics are understood as being two separate things, and this proposal needs to be updated to clarify that they are different topics to avoid (even more!) confusion when other people read it.

• one for reading messages, and an additional scope for creating new messages and sending them.

It's worth clarifying which scope you think "change read status" should be included in, I could be persuaded either way.

In addition, the API provisions for paginating the list of messages.

It might be worth spinning this off into a precursor project - to paginate the existing inbox and outbox implementation. That way, all the complications around cursor-based pagination (see e.g. @tomhughes recent work with diary entries and traces and @AntonKhorev recent work on diary_comments) can be sorted out separately. That will reduce the complexity of reviewing the API implementation.

[tomhughes]

I believe we would need to apply rate limiting for sending messages from the API as well.

Absolutely, the rate limit is is a limit on sending messages and how they are sent is not important so it should apply to the API in the same way it does to sending via the web site.

[AntonKhorev]

In addition, the API provisions for paginating the list of messages.

It might be worth spinning this off into a precursor project - to paginate the existing inbox and outbox implementation.

It requires adding a searching functionality to replace Ctrl+F.

[AntonKhorev]

We plan to work on transitioning OSM web site to use the Messaging API instead of directly accessing the content of the database.

As @tomhughes anticipates, that's a "hard no" from me on this approach. 😄 We already do this for a couple of parts of the website (e.g. Notes) and we're working on specifically undoing this.

Nobody is working on it. Changing things to use the api instead of the db directly or vice versa is going to affect how user blocks work. Currently they cut off the api access.

[milan-cvetkovic]

We plan to work on transitioning OSM web site to use the Messaging API instead of directly accessing the content of the database.

As @tomhughes anticipates, that's a "hard no" from me on this approach. 😄 We already do this for a couple of parts of the website (e.g. Notes) and we're working on specifically undoing this. It's overcomplicated, it's unnecessary indirection and causes way more hassle for little to no tangible benefit.

My suggestion was to make use Message API from web site as a proof of concept. I was not aware of (failed) attempts to do similar things in the past. As suggested, we will not use Message API from web site.

So I'm in favour of a Messages API (nb not 'Messaging') but the web interface should continue to be a straightforward server-side-html / html-over-the-wire implementation like the rest of the site.

Agreed.

Will also change the naming to refer to Messages instead of Messaging.

...
I hope we can build a proper notifications system in the near future, and refactor all our existing event notifications to use it.

To clarify, the efforts related to this github issue are only with regards to direct messages to and from the users. There is no attempt to (partially) fix the notification of any kind, including the notifications about received direct messages.

• one for reading messages, and an additional scope for creating new messages and sending them.

It's worth clarifying which scope you think "change read status" should be included in, I could be persuaded either way.

This is what I had in mind:

• message_consume to be able to read, messages, update their status to read/unread or deleted. Affects only messages owned by the user.
• message_send - to be able to send messages to other users. Affects other users as well.