Skip to content

N0taN3rd/memgatorBulkDownload

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Memgator Bulk TimeMap Downloader

Have you ever had a need to download 100 or 1 million TimeMaps using oduwsdl/memgator?

With the caveat that it must be done in a timely manner?

If so then you are in luck because this project has you covered.

Requirements

Requires python 3

Be sure to install the dependencies first

  • [sudo] pip install -r requirements.txt

You also need a running instance of oduwsdl/MemGator

If you do not have one. You can get one at oduwsdl/MemGator/releases

Usage

Basic usage

$ python download.py -m {MGURL} {FORMAT2} -d {DUMDIR} -u {LIST}
# MGURL   => http://localhost:1208
# FORMAT  => link|json|cdxj
# FORMAT2 => (-l, --link)|(-j, --json)|(-c, --cdxj)
# DUMDIR  => Path to directory where timemaps will be dumped
# LIST    => Path to URL list

Full Usage

$ python download.py --help
usage: download [-h] [-m MEMURL] [-w WORKERS] [-r REQUESTS] [-d DUMP] -u URLS
                [-k KEY] [-j | -l | -c]

Bulk download TimeMaps using a local memgator instance

optional arguments:
  -h, --help            show this help message and exit
  -m MEMURL, --memurl MEMURL
                        URL for running memgator instance. Defaults to
                        http://localhost:1208/timemap/json
  -w WORKERS, --workers WORKERS
                        Max number of worker processes spawned. Defaults to 5
  -r REQUESTS, --requests REQUESTS
                        How many requests should be queued per chunk. Defaults
                        to 10
  -d DUMP, --dump DUMP  Directory to dump the TimeMaps in. Defaults to
                        <cwd>/timemaps
  -u URLS, --urls URLS  Path to file (.txt, .csv, .json) containing list of
                        URLs. File type detected by considering extension. If
                        .csv must supply -k <key> so we know where to get the
                        url
  -k KEY, --key KEY     The csv key for the urls
  -j, --json            Download TimeMaps in json format. Default format
  -l, --link            Download TimeMaps in link format
  -c, --cdxj            Download TimeMaps in cdxj format

URL List Format

  • .txt: 1 URL per line
  • .csv: Requires -k or --key {KEY} argument. KEY is the csv column containing the URL
  • .json: List of URLs

License

MIT