Skip to content

spacesailor24/github-dependents-scraper

Repository files navigation

Github Dependents Scraper

This is designed to scrape the data from the Github dependency graph page into a JSON file

Getting Started

Prerequisites

  • Install yarn
  1. Clone the repo
  2. Run yarn - Installs dependencies
  3. Run npx tsc - Compiles index.ts
  4. Run the scrapper node index.js repoOwner/repo dependents.json
    • The command line arguments for the scrapper are as follows:
      1. (githubOwnerAndRepo) repoOwner/repo - This is what's displayed in the Github URL when on the repo page e.g. for this repo it would be spacesailor24/github-dependents-scraper
      2. (dependentsFile) anything.json - This file can be named anything, but it needs to be a valid JSON file ending with the .json file extension
      3. (resumeCrawl) true or false - Eventually this crawler will get rate limited by Github, this flag allows you to run the crawler from where it left off before receiving the rate limit page
        • So if the crawler dies because of rate limiting, you'd start it up again with:
        node index.js repoOwner/repo dependents.json true
        NOTE Starting it with false will override the dependentsFile and start scrapping from the first dependents page

Sorting the Scrapped Data

Maybe I'll extend crawler to be able to sort the data, but for now, there's a nifty online sorter that'll do the trick!

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published