Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#425] Extend harvester to support include/exclude filtering by tags #427

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open

[#425] Extend harvester to support include/exclude filtering by tags #427

wants to merge 84 commits into from

Commits on Nov 12, 2020

  1. [ckan#425] Extend harvester to support include/exclude filtering by tags

    @scottlimmer
    scottlimmer committed Nov 12, 2020
    Configuration menu
    Copy the full SHA
    c20cff7 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2020

  1. Fix version check in templates to work on 2.10+

    @amercader
    amercader committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    762ee14 View commit details
    Browse the repository at this point in the history

  2. Use pg 12 on 2.9

    @amercader
    amercader committed Nov 23, 2020
    Configuration menu
    Copy the full SHA
    faed627 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2020

  1. Test master allowing failures

    @amercader
    amercader committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    8365eb3 View commit details
    Browse the repository at this point in the history

  2. Fix syntax

    @amercader
    amercader committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    646a060 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'travis-pg-again'

    @amercader
    amercader committed Nov 24, 2020
    Configuration menu
    Copy the full SHA
    d3432e9 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2020

  1. Fix flaky redis test

    @amercader
    amercader committed Dec 11, 2020
    Configuration menu
    Copy the full SHA
    edccfc4 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2020

  1. Migrate tests from Travis CI to GitHub Actions 

    After Travis stopped supporting Open Source projects we decided to move
to GitHub Actions, which has turned out to be excellent.

The workflow uses two jobs, one for basic syntax checking with Flake8
and a matrix of test run using the supported CKAN versions (not master
yet, more on this later). The tests run inside a Docker Container
pre-configured with a CKAN dev install (currently the
openknowledge/ckan-dev images are used). Additionally all services are
also Dockerized, using ckan/ckan-solr-dev and ckan/postgres-dev and the
standard redis image.
    @amercader
    amercader committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    76c0118 View commit details
    Browse the repository at this point in the history

  2. Show badge just for master

    @amercader
    amercader committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    2cb3ab3 View commit details
    Browse the repository at this point in the history

  3. Remove test-ga.ini, use env instead

    @amercader
    amercader committed Dec 15, 2020
    Configuration menu
    Copy the full SHA
    95e82c2 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2020

  1. Replace default path to CKAN core config file with the one on the con… 

    …tainer
    @amercader
    amercader committed Dec 16, 2020
    Configuration menu
    Copy the full SHA
    d005538 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2021

  1. Fix last harvest time to avoid timeouts

    @avdata99
    avdata99 committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    eb24643 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2021

  1. Merge pull request ckan#431 from avdata99/fix_timeout 

    Fix last harvest time to avoid timeouts
    @amercader
    amercader committed Feb 4, 2021
    Configuration menu
    Copy the full SHA
    6222e38 View commit details
    Browse the repository at this point in the history

  2. Add check that redis key is available before running .strptime to avo… 

    …id error
    @thejuliekramer
    thejuliekramer committed Feb 4, 2021
    Configuration menu
    Copy the full SHA
    15849ba View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2021

  1. Add flake8 step to workflow

    @Zharktas
    Zharktas committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    3c83f41 View commit details
    Browse the repository at this point in the history

  2. flake8

    @Zharktas
    Zharktas committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    078077c View commit details
    Browse the repository at this point in the history

  3. flake8

    @Zharktas
    Zharktas committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    2f1a2d3 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2021

  1. Merge pull request ckan#436 from ckan/add_flake8_step_to_workflow 

    Add flake8 step to Github Actions workflow
    @tino097
    tino097 committed Mar 21, 2021
    Configuration menu
    Copy the full SHA
    44bf788 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2021

  1. Optimize last_error_free_job to be more efficient

    @Zharktas
    Zharktas committed Mar 22, 2021
    Configuration menu
    Copy the full SHA
    52b2139 View commit details
    Browse the repository at this point in the history

  2. Oops. wrong syntax

    @Zharktas
    Zharktas committed Mar 22, 2021
    Configuration menu
    Copy the full SHA
    1c64d0f View commit details
    Browse the repository at this point in the history

  3. Use .outerjoin instead to support old sqlalchemy

    @Zharktas
    Zharktas committed Mar 22, 2021
    Configuration menu
    Copy the full SHA
    4f27de9 View commit details
    Browse the repository at this point in the history

  4. flake8

    @Zharktas
    Zharktas committed Mar 22, 2021
    Configuration menu
    Copy the full SHA
    106e205 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2021

  1. Include webassets.yml in MANIFEST.in

    @amercader
    amercader committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    94b17d4 View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#437 from ckan/optimize_last_error_free_job 

    Optimize last error free job
    @amercader
    amercader committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    80e4e60 View commit details
    Browse the repository at this point in the history

  3. Merge pull request ckan#432 from thejuliekramer/fix_strptime_error_re… 

    …dis_key

check that redis key is available before running .strptime
    @amercader
    amercader committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    ae554f0 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of github.com:ckan/ckanext-harvest

    @amercader
    amercader committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    dff1fce View commit details
    Browse the repository at this point in the history

  5. Bump version

    @amercader
    amercader committed Mar 26, 2021
    Configuration menu
    Copy the full SHA
    e2e8acb View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2021

  1. Update README -fix run-test command name 

    `run_test` command is actually spelled as `run-test`
    @alantygel
    alantygel committed Apr 18, 2021
    Configuration menu
    Copy the full SHA
    d4064eb View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2021

  1. Update README - replace _ by - in all commands

    @alantygel
    alantygel committed Apr 20, 2021
    Configuration menu
    Copy the full SHA
    5458ca2 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2021

  1. fix for py3 also works for py2; https://stackoverflow.com/questions/4… 

    …8374667/object-of-type-dict-items-is-not-json-serializable
    @nickumia-reisys
    nickumia-reisys committed Jul 19, 2021
    Configuration menu
    Copy the full SHA
    fd5a6b7 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2021

  1. Removes call to no longer used `render_jinja2' function 

    Removes calls to `render_jinja2`, which was removed in recent CKAN
versions.
    @benjwadams
    benjwadams committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    8ee5eb3 View commit details
    Browse the repository at this point in the history

  2. Add summary of changes to CHANGELOG.rst

    @benjwadams
    benjwadams committed Jul 30, 2021
    Configuration menu
    Copy the full SHA
    a5dac48 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2021

  1. Allow GET requests to harvester.refresh

    @bzar
    bzar committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    35a9ba2 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2021

  1. Merge pull request ckan#439 from alantygel/patch-1 

    Update README -fix run-test command name
    @metaodi
    metaodi committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    ceaabd4 View commit details
    Browse the repository at this point in the history

  2. Add note about changed command names to README

    @metaodi
    metaodi committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    415b7d8 View commit details
    Browse the repository at this point in the history

  3. Merge pull request ckan#454 from ckan/readme-changed-command-names 

    Add note about changed command names to README
    @amercader
    amercader committed Sep 1, 2021
    Configuration menu
    Copy the full SHA
    e3c70bf View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2021

  1. Allow clearing a harvest source with a GET request

    @bzar
    bzar committed Sep 13, 2021
    Configuration menu
    Copy the full SHA
    eb7f6fc View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#452 from bzar/patch-1 

    Allow GET requests to harvester.refresh
    @amercader
    amercader committed Sep 13, 2021
    Configuration menu
    Copy the full SHA
    d91ea66 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2021

  1. Add harvest info to solr 

    This allows the harvest information on the package to be easily searched
and removes complicated logic for adding/removing extras
    @jbrown-xentity
    jbrown-xentity committed Sep 20, 2021
    Configuration menu
    Copy the full SHA
    d06444e View commit details
    Browse the repository at this point in the history

  2. Use py2 compatible formatting

    @jbrown-xentity
    jbrown-xentity committed Sep 20, 2021
    Configuration menu
    Copy the full SHA
    cb38910 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2021

  1. replace render_jinja2 with render

    @TomeCirun
    TomeCirun committed Sep 22, 2021
    Configuration menu
    Copy the full SHA
    fc8e723 View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#459 from TomeCirun/456-replace-render_jinja2-… 

    …with-render

replace render_jinja2 with render
    @Zharktas
    Zharktas committed Sep 22, 2021
    Configuration menu
    Copy the full SHA
    51f85ee View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2021

  1. set the default MQ_TYPE to be redis

    @TomeCirun
    TomeCirun committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    3d2b398 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2021

  1. Merge pull request ckan#463 from TomeCirun/change-default-MQ_TYPE 

    set the default MQ_TYPE to be redis
    @tino097
    tino097 committed Sep 29, 2021
    Configuration menu
    Copy the full SHA
    7e14f92 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2021

  1. Merge pull request ckan#21 from GSA/json-serialization-fix 

    fix for py3 also works for py2;
    @nickumia-reisys
    nickumia-reisys committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    99538a4 View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#20 from GSA/fix-pkg-extras 

    Fix pkg extras
    @nickumia-reisys
    nickumia-reisys committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    b653113 View commit details
    Browse the repository at this point in the history

  3. Revert "Fix pkg extras"

    @jbrown-xentity
    jbrown-xentity committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    b3b519a View commit details
    Browse the repository at this point in the history

  4. Merge pull request ckan#23 from GSA/revert-20-fix-pkg-extras 

    Revert "Fix pkg extras"
    @jbrown-xentity
    jbrown-xentity committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    eca0ae0 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2021

  1. Merge pull request ckan#451 from benjwadams/fix_render_function 

    Fix render function
    @amercader
    amercader committed Oct 1, 2021
    Configuration menu
    Copy the full SHA
    7f565cc View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#450 from GSA/json-serialization-fix 

    Harvest Source Extra Fields are not encoded into JSON appropriately
    @amercader
    amercader committed Oct 1, 2021
    Configuration menu
    Copy the full SHA
    63701ad View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2021

  1. Adjust solr index from live tests

    @jbrown-xentity
    jbrown-xentity committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    6995fbc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'datagov-py3' into fix-pkg-extras

    @jbrown-xentity
    jbrown-xentity committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    9d2faea View commit details
    Browse the repository at this point in the history

  3. Fix lint

    @jbrown-xentity
    jbrown-xentity committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    c0b3823 View commit details
    Browse the repository at this point in the history

  4. Remove unnecessary code, add tests

    @jbrown-xentity
    jbrown-xentity committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    a418ed4 View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2021

  1. Merge pull request ckan#24 from GSA/fix-pkg-extras 

    Adjust solr index from live tests
    @nickumia-reisys
    nickumia-reisys committed Oct 5, 2021
    Configuration menu
    Copy the full SHA
    496a0df View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2021

  1. Explicitly set harvest extra info

    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    bfa0da7 View commit details
    Browse the repository at this point in the history

  2. Contain fix for both data_dict and validated

    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    2cd3d8d View commit details
    Browse the repository at this point in the history

  3. Only update if extras exist 

    Should never be necessary in data.gov, but for upstream.
    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    14ba290 View commit details
    Browse the repository at this point in the history

  4. Create extras if not there

    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    ff40069 View commit details
    Browse the repository at this point in the history

  5. Fix lint

    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    925b0ee View commit details
    Browse the repository at this point in the history

  6. Merge pull request ckan#26 from GSA/bug/clean-harvest-info 

    Bug/clean harvest info
    @jbrown-xentity
    jbrown-xentity committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    04586fe View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2021

  1. Merge pull request ckan#458 from GSA/fix-pkg-extras 

    Fix harvest extras for packages
    @amercader
    amercader committed Oct 8, 2021
    Configuration menu
    Copy the full SHA
    2910693 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2021

  1. remove force_import_val

    @TomeCirun
    TomeCirun committed Oct 15, 2021
    Configuration menu
    Copy the full SHA
    254ffb0 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2021

  1. use render_jinja2 for when it is available for ckan core <= 2.9.4

    @FuhuXia
    FuhuXia committed Oct 20, 2021
    Configuration menu
    Copy the full SHA
    878b81f View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2021

  1. flake8

    @FuhuXia
    FuhuXia committed Oct 21, 2021
    Configuration menu
    Copy the full SHA
    7783919 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2021

  1. Merge pull request ckan#470 from FuhuXia/469-use-render_jinja2-when-a… 

    …vailable

use render_jinja2 for when it is available for ckan core <= 2.9.4
    @amercader
    amercader committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    419befe View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into 466-cli-run-test-throws-an-error

    @TomeCirun
    TomeCirun committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    8f8cde8 View commit details
    Browse the repository at this point in the history

  3. Merge pull request ckan#467 from TomeCirun/466-cli-run-test-throws-an… 

    …-error

remove force_import_val
    @tino097
    tino097 committed Oct 22, 2021
    Configuration menu
    Copy the full SHA
    9d5679f View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2021

  1. Adds test for queue.resubmit_objects()

    @seitenbau-govdata
    seitenbau-govdata committed Nov 20, 2021
    Configuration menu
    Copy the full SHA
    0a06cad View commit details
    Browse the repository at this point in the history

  2. Fixes timeout date calculation on different time zones by using utcnow

    @seitenbau-govdata
    seitenbau-govdata committed Nov 20, 2021
    Configuration menu
    Copy the full SHA
    2891b46 View commit details
    Browse the repository at this point in the history

  3. Adds debug logging for re-sending objects to queue

    @seitenbau-govdata
    seitenbau-govdata committed Nov 20, 2021
    Configuration menu
    Copy the full SHA
    a934f27 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2021

  1. Clears redis db before queue test 

    Makes the test independent of the test execution order
Prevents the test from failing on a server that is not freshly installed
    @seitenbau-govdata
    seitenbau-govdata committed Nov 22, 2021
    Configuration menu
    Copy the full SHA
    d2cf517 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2021

  1. Merge pull request ckan#482 from GovDataOfficial/improve-test-coverag… 

    …e-and-fixes-timeout-calculation

Improve test coverage and fixes timeout calculation
    @metaodi
    metaodi committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    d84d847 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2021

  1. Update changelog and setup.py for version 1.3.4

    @seitenbau-govdata
    seitenbau-govdata committed Dec 1, 2021
    Configuration menu
    Copy the full SHA
    57a551d View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2021

  1. Add option keep-actual to clearsource_history command

    @seitenbau-govdata
    seitenbau-govdata committed Dec 18, 2021
    Configuration menu
    Copy the full SHA
    6b238f8 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2021

  1. Rename keep-actual to keep-current and updated documentation

    @seitenbau-govdata
    seitenbau-govdata committed Dec 22, 2021
    Configuration menu
    Copy the full SHA
    d2b7340 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2022

  1. Futureproof version check to work also on 3.0+

    @Zharktas
    Zharktas committed Jan 18, 2022
    Configuration menu
    Copy the full SHA
    e650be8 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2022

  1. Merge pull request ckan#484 from GovDataOfficial/fix-clearsourcehisto… 

    …ry-command-keep-latest-objects

Add option keep-current to clearsource_history command
    @amercader
    amercader committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    b702bf9 View commit details
    Browse the repository at this point in the history

  2. Merge pull request ckan#487 from ckan/futureproof_version_check 

    Futureproof version check to work also on 3.0+
    @amercader
    amercader committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    ddc891f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'release-version-1-3-4' of https://github.com/GovDataOff… 

    …icial/ckanext-harvest into GovDataOfficial-release-version-1-3-4
    @amercader
    amercader committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    9d8d346 View commit details
    Browse the repository at this point in the history

  4. Add more changelog entries

    @amercader
    amercader committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    0ae24e1 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'GovDataOfficial-release-version-1-3-4'

    @amercader
    amercader committed Jan 24, 2022
    Configuration menu
    Copy the full SHA
    694a8cc View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2022

  1. [ckan#425] Extend harvester to support include/exclude filtering by tags

    @scottlimmer
    scottlimmer committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    be6822a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feature/425-tags-filter-include-exclude' of https://git… 

    …hub.com/scottlimmer/ckanext-harvest into feature/425-tags-filter-include-exclude

� Conflicts:
�	CHANGELOG.rst
    @scottlimmer
    scottlimmer committed Apr 4, 2022
    Configuration menu
    Copy the full SHA
    edf6c7d View commit details
    Browse the repository at this point in the history