plnx-grabber

Transfer trade history from Poloniex into a local MongoDB database

• Every pair and time period
• Chunk wise without using much RAM
• One time, multiple times or continuous
• Smart outputs on history available on Poloniex as well as locally

Installation

pip install https://github.com/polakowo/plnx-grabber/archive/master.zip

Setup

import plnxgrabber
from pymongo import MongoClient
import logging

logging.basicConfig(format='%(asctime)s - %(name)s - %(funcName)s() - %(levelname)s - %(message)s',
                    datefmt='%d/%m/%Y %H:%M:%S',
                    level=logging.INFO)
# or your preferred logger

client = MongoClient('localhost:27017')
db = client['TradeHistory']
mongo_ts = plnxgrabber.MongoTS(db)
grabber = plnxgrabber.Grabber(mongo_ts)

Usage

General

Get information on history available on Poloniex:

grabber.remote_info(['USDT_BTC', 'USDT_ETH', 'USDT_LTC', 'USDT_BCH])

Get information on pairs stored locally:

grabber.db_info()

Get progress of currently stored history relative to overall available on Poloniex:

grabber.progress()

Grab one pair

To collect trade history for a single pair, use Grabber.one()

Collect the entire history:

grabber.one('USDT_BCH')

# or grabber.one('USDT_BCH', from_dt=plnxgrabber.begin(), to_dt=plnxgrabber.now())

Collect the history between 1/9/2017 12:00:00 to 1/9/2017 18:00:00:

from datetime import datetime
import pytz

from_dt = datetime(2017, 9, 1, 12, 0, 0, tzinfo=pytz.utc)
to_dt = datetime(2017, 9, 1, 18, 0, 0, tzinfo=pytz.utc)
grabber.one('USDT_BTC', from_dt=from_dt, to_dt=to_dt)

Collect the last hour:

grabber.one('USDT_BTC', from_dt=plnxgrabber.ago(hours=1))

# or grabber.one('USDT_BTC', from_dt=plnxgrabber.ago(hours=1), to_dt=plnxgrabber.now())

• If collection not empty, it gets extended
• Use oldest to auto-fill the timestamp of the oldest record in the collection
• Use newest to auto-fill the timestamp of the youngest record

Collect everything below the oldest record in the collection (backward):

grabber.one('USDT_BTC', to_dt='oldest')

# or grabber.one('USDT_BTC', from_dt=plnxgrabber.begin(), to_dt='oldest')

Collect everything above the newest record in the collection (forward):

grabber.one('USDT_BTC', from_dt='newest')

# or grabber.one('USDT_BTC', from_dt='newest', to_dt=plnxgrabber.now())

Drop currently stored pair and recollect:

grabber.one('USDT_BCH', from_dt='oldest', to_dt='newest', drop=True)

---

Grab row of pairs

To collect trade history for a row of pairs, use Grabber.row()

For the following 4 pairs, collect the history from 1/9/2017 12:00:00 to 1/9/2017 18:00:00:

from_dt = datetime(2017, 9, 1, 12, 0, 0, tzinfo=pytz.utc)
to_dt = datetime(2017, 9, 1, 18, 0, 0, tzinfo=pytz.utc)
grabber.row(['USDT_BTC', 'USDT_ETH', 'USDT_LTC', 'USDT_BCH'], from_dt=from_dt, to_dt=to_dt)

• Pass 'ticker' instead of pair to perform an action on all pairs traded on Poloniex
• Pass 'db' to perform an action on all pairs stored locally
• Or use Regex

For each pair in the current ticker, collect the last 5 minutes:

grabber.row('ticker', from_dt=plnxgrabber.ago(minutes=5))

Recollect each collection:

grabber.row('db', from_dt='oldest', to_dt='newest', drop=True)

For each ETH pair, collect the last minute:

grabber.row(r'(ETH_+)', from_dt=plnxgrabber.ago(minutes=1))

---

Update row of pairs

To constantly collect the most recent records for a row of pairs, use Grabber.ring()

Keep updating a row of pairs every 60 sec:

grabber.ring(["USDT_BTC", "USDT_ETH"], every=60)

Update continuously:

grabber.ring(["USDT_BTC", "USDT_ETH"])

Continue updating all collections:

grabber.ring('db')