Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow user-defined offset #157

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Allow user-defined offset #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
46749b4
Allow user-defined offset
flynn-d Oct 11, 2018
ab4dc57
Adding test for offset
flynn-d Oct 12, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 13 additions & 7 deletions R/RSocrata.R
View file
Open in desktop
Expand Up @@ -366,13 +366,19 @@ read.socrata <- function(url, app_token = NULL, email = NULL, password = NULL,
# PAGE through data and combine
# if user limit is provided do not page
# if no limit $provided, loop until all data is paged
while (nrow(page) > 0 & !limitProvided) {
query <- paste(validUrl, if(is.null(parsedUrl$query)) {'?'} else {"&"},
'$limit=50000&$offset=', nrow(result), sep='')
response <- getResponse(query, email, password)
page <- getContentAsDataFrame(response)
result <- rbind.fill(result, page) # accumulate
}

# If an offset has been used, and limit has not been provided, skip the paging and return a message
if(!is.null(parsedUrl$query$`$offset`) & !limitProvided){
message(paste('Returning results from offset of', format(parsedUrl$query$`$offset`, big.mark=",")))
} else {
while (nrow(page) > 0 & !limitProvided) {
query <- paste(validUrl, if(is.null(parsedUrl$query)) {'?'} else {"&"},
'$limit=50000&$offset=', nrow(result), sep='')
response <- getResponse(query, email, password)
page <- getContentAsDataFrame(response)
result <- rbind.fill(result, page) # accumulate
}
}
if (is.null(dataTypes)) {
warning("Dates and currency fields will be converted to character")
} else {
Expand Down
11 changes: 11 additions & 0 deletions tests/testthat/test-all.R
View file
Open in desktop
Expand Up @@ -550,3 +550,14 @@ test_that("getContentAsDataFrame does not get caught in infinite loop", {
expect_equal("data.frame", class(df), label="class")
})


context("User-defined offset")

test_that("User can specify offset", {

rows_hist = 108000 # rows of data previously read
df <- read.socrata(url = paste0("https://data.cambridgema.gov/resource/gxzm-dpwp.csv?$offset=", rows_hist))

expect_equal("data.frame", class(df), label="class")
})