Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pagination #56

Open
LadaRudnitckaia opened this issue Jul 27, 2021 · 1 comment · May be fixed by #57
Open

Pagination #56

LadaRudnitckaia opened this issue Jul 27, 2021 · 1 comment · May be fixed by #57
Assignees
@MKyhos
Labels
koboclass Kobo Class

Comments

@LadaRudnitckaia
Copy link
Contributor

LadaRudnitckaia commented Jul 27, 2021 •
edited by MKyhos

Design and implement pagination for GET

A reasonable requirement for kbtbr is to handle paginated server responses correctly, such that users to not silently receive only a partial response. This should actually be implemented before the first release, at best.

To that end, please

  1. Generate test data for GET methods (i.e. a number of assets such that pagination actually happens)
  2. Add the option for pagination to the get of the Kobo class

TBD

  • whether to use the Paginator from crul or write the code ourselves.
  • if we write the code ourselves, via adding a loop, we should also take care of the connection problems that might arise for different iterations of the loop, i.e. making the process robust against failure.
@MKyhos MKyhos self-assigned this Jul 27, 2021
@MKyhos
Copy link
Member

MKyhos commented Jul 28, 2021 •
edited

First finding: in the current version of {crul}, only offset-based pagination is implemented in the Paginator class:

#' Supported now:
#'
#' - `limit_offset`: the most common way (in my experience), so is the default.
#' This method involves setting how many records and what record to start at
#' for each request. We send these query parameters for you.
#' - `page_perpage`: set the page to fetch and (optionally) how many records
#' to get per page
#'
#' Supported later, hopefully:
#'
#' - `link_headers`: link headers are URLS for the next/previous/last
#' request given in the response header from the server. This is relatively
#' uncommon, though is recommended by JSONAPI and is implemented by a
#' well known API (GitHub).
#' - `cursor`: this works by a single string given back in each response, to
#' be passed in the subsequent response, and so on until no more records
#' remain. This is common in Solr

(see [here])https://github.com/ropensci/crul/blob/master/R/paginator.R#L27-L43)

As Kobotoolbox is using a link-header style pagination, we hence have to implement it by ourselves. However, the design of Paginator might be useful/inspiring :)

cc @dimitri-mar @LadaRudnitckaia

@MKyhos MKyhos added the koboclass Kobo Class label Jul 28, 2021
@MKyhos MKyhos linked a pull request Aug 6, 2021 that will close this issue
[#56] | Implement pagination #57
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MKyhos MKyhos

Labels
koboclass Kobo Class
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#56] | Implement pagination CorrelAid/kbtbr
2 participants
@MKyhos @LadaRudnitckaia