Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding export.socrata function #187

Open
wants to merge 6 commits into
base: dev
Choose a base branch
from
Open

Adding export.socrata function #187

wants to merge 6 commits into from

Commits on Dec 4, 2017

  1. Configuration menu
    Copy the full SHA
    46a488d View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2018

  1. Save data.json to file system; handle non-data files

    Save data.json to file system
    ------------------------------
    A copy of the data.json file at the beginning of the download process is
    saved alongside the actual downloaded data. Since `export.socrata()` uses
    data.json as the index to download data, this will allow users to
    cross-reference the downloaded data with other metadata associated with it
    available through [Project Open Data](https://project-open-data.cio.gov).
    
    Handle non-data file
    ---------------------
    Socrata lists non-data files, such as Socrata Stories--HTML websites that
    contain text but no machine-readable data--in the data.json file. This
    causes errors when trying to download those sites because they do not have
    a "distribution URL". While it's arguable that these "sites" should not be
    included in the first place, the script now simply skips those files.
    
    Since a copy of the data.json file is downloaded (see above), users will
    have transparency into which URLs were not downloaded.
    Tom Schenk Jr committed Oct 28, 2018
    Configuration menu
    Copy the full SHA
    aafcf15 View commit details
    Browse the repository at this point in the history
  2. Ignores HTML content

    Socrata supports external links which direct to web pages (e.g., HTML).
    These would cause an error when `export.socrata()` attempted to download
    them. This fix will simply skip those files and proceed to the next file.
    Tom Schenk Jr committed Oct 28, 2018
    Configuration menu
    Copy the full SHA
    8b601c6 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2020

  1. Handles non-CSV file types #126

      * Ignores HTML files (e.g., Socrata Pages)
      * Ignores on occassions there isn't any data
      * Will download (uncompressed) PDFs, Word, Excel, PowerPoint, plain text attachments.
    Tom Schenk Jr committed Jan 5, 2020
    Configuration menu
    Copy the full SHA
    ccc4c96 View commit details
    Browse the repository at this point in the history
  2. Rebased branch; generated documentation

    Rebased branch with most recent `dev` branch and generated documentation.
    
    Merge branch 'dev' into issue126
    
    # Conflicts:
    #	DESCRIPTION
    #	R/RSocrata.R
    Tom Schenk Jr committed Jan 5, 2020
    Configuration menu
    Copy the full SHA
    c565ef9 View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2020

  1. Several clean-up items for export.socrata()

    * Removed user-defined option for file output (not available yet)
    * Clarified documentation where `export.socrata()` files will be located.
    * Fixed incorrect date in `DESCRIPTION` file.
    * Iterating build number.
    Tom Schenk Jr committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    f9ec527 View commit details
    Browse the repository at this point in the history