Paging

[sckott]

various paging patterns:

• using query parameters
  • limit & offset - specify no. desired, and which record to start at
  • per_page & page - specify no. desired, and which page to return (page size variable)
• using link headers
  • headers return some combination of next, first, last, or prev links - if these provided, in all cases should probably use these - Github recommends that at least

examples

• github:
  • params: page, per_page
  • strategy: GH suggests using link headers to do paging b/c sometimes paging is based on SHA's instead of page numbers
• GBIF
  • params: limit, offset
  • strategy: page via query parameters
• Crossref
  • params: rows, offset
• idigbio
  • params: limit, offset (Elasticsearch backed, but not exposed directly)
• vertnet
  • params: limit, cursor
• Tropicos
  • params: pagesize, startrow
• NOAA NCDC v2
  • params: limit, offset
• CKAN API
  • params: limit, offset
• Berkeley Ecoengine
  • params: page_size, page
• iNaturalist
  • params: per_page, page
• DataCite
  • params: rows, start (solr backed)
• PLOS Search API
  • params: rows, start (solr backed)
• Europeana API
  • params: rows, start (solr backed)
• ORCID API
  • params: rows, start (solr backed)
• DPLA API
  • params: page_size, page (Elasticsearch backed)
• Twitter search API
  • params: count (that's it, AFAICT, not sure this is accurate)
• Enigma
  • params: limit, page

approach

Automagically figure out what params to pass and their values given user input. We'll need some user input:

• name of query parameters
• how many records they want
• what record to start at
• maximum records allowed (if known)

Things we can figure out automatically

• whether API uses link headers or not (just look for link headers)
• if a cursor is used (e.g., Vertnet API) look for a cursor

[sckott]

This probably will require using something that can maintain state, like an R6 class

[sckott]

paging notes

per_page/page interface --- assuming max is > 20
limit = 20 -> per_page = 5
offset = 15 -> page = 4
chunk_size = 5
stop -> 20 (so, get 4 pages)

limit/offset interface --- assuming max is > 20
limit = 20
offset = 0
chunk_size = 5
stop -> 20 (so, get 4 pages ({limit = 5, offset = 0}, {limit = 5, offset = 5}, etc.))

[sckott]

moving to next milestone, trying to get MVP out #10