Data limitation request

[FilipAben]

Problem
We would like to have the option to curtail a request for data if this would generate an excessive amount of data.

Describe the solution you'd like
A way of informing the requesting party that only part of the requested data was returned and that additional requests should be made to retrieve the remaining data.
Additionally we should be able to rate-limit requests for protection purposes. http status codes + quota http headers could achieve this.

[nickdickinson]

Paging is a norm in the REST API space with a custom link to an endpoint at the end of the response data to get the next page of data. According to the documentation the pages should be limited to 8 kB. Shall we add a paging structure to the API response format? "This documentation aims to account for the speed of the system as a whole by limiting the resource response of a data requests to about 8 kB. For more information see specific API calls." - wiki

[FilipAben]

Yes, that would be good.

[Stroomversnelling]

Nick recommends (not yet documented in YAML) is to have this added to each response:

{total_pages: 4, current_page: 2, previous_page: https://someurl/api/random_page_url, next_page: https://someurl/api/random_page_url}

The page urls are used to fetch the next page of data or previous page. It is also possible to add first and last urls for jumping to the start or end of timeseries for example. By default the first page is always returned.