Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Website crawler #376

Merged
merged 9 commits into from
May 29, 2024
Merged

Website crawler #376

merged 9 commits into from
May 29, 2024

Commits on Apr 29, 2024

  1. added environment / requirements for easy setup

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    29129c1 View commit details
    Browse the repository at this point in the history

  2. fixed type error & wrong error message

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    d9dd230 View commit details
    Browse the repository at this point in the history

  3. updated settings to not crawl images

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    fa813e2 View commit details
    Browse the repository at this point in the history

  4. updated install instructions

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e275a6b View commit details
    Browse the repository at this point in the history

  5. simplified write pipeline (only write raw html and json file)

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    06ba393 View commit details
    Browse the repository at this point in the history

  6. added new website (whole domain) crawler

    @bigabig
    bigabig committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    b12fd57 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. fixed bug in filename generation

    @bigabig
    bigabig committed May 3, 2024
    Configuration menu
    Copy the full SHA
    80ae13f View commit details
    Browse the repository at this point in the history

  2. impoved website crawler

    @bigabig
    bigabig committed May 3, 2024
    Configuration menu
    Copy the full SHA
    679f015 View commit details
    Browse the repository at this point in the history

  3. disabled selenium in settings

    @bigabig
    bigabig committed May 3, 2024
    Configuration menu
    Copy the full SHA
    3f20d35 View commit details
    Browse the repository at this point in the history