new "Direct Data Download and Ingestion" section

[wibeasley]

• consolidate all the scattered advice about reading a csv/json/xml from a url
• update all the advice involving read*() functions and TLS/SSL urls. A lot of still says that base R functions (eg, read.csv()) cannot handle an https url
• there are a lot of common packages that accept https that aren't listed. Of the top of my head, data.table, readr, arrow.
• File types for direct ingestion:
  • maybe refer html to the web scraping section in the task view.
  • maybe yaml

[pachadotdev]

@wibeasley hi! do you prefer yaml?

[wibeasley]

Yes, I typically prefer yaml if the (a) data has a nested or non-rectangular structure and (b) the file is a human entered/edited. I tend to use json for machine-generated datasets.

But there are some tabular/rectangular files that I have started expressing as yaml because they're easier to read & adjust. A small downside is that it requires a little more work (for the ingesting code) to verify the yaml politely transforms to a data.frame.

Here's an example of a tabular structure that I felt was a better fit for yaml than csv: https://github.com/OuhscBbmc/REDCapR/blob/main/inst/misc/validation-transformation.yml

I don't do it much, but the yaml package can load a file from a https url:

yaml::yaml.load_file(
  "https://raw.githubusercontent.com/OuhscBbmc/REDCapR/main/inst/misc/validation-transformation.yml"
)

Since we already have bullets for csv, xml, html, & json ...I thought yaml could be included for completeness. But as always, I'm happy following your lead. Tell me if you think tangents like this are more distracting than helpful.

Are there scenarios where you do/don't format a data file as yaml?