Skip to content

Predicting Arbitrage in the Crypto Market with Machine Learning

License

Notifications You must be signed in to change notification settings

peddlem25/cryptolyticapp

 
 

Repository files navigation

Cryptolytic

Project Overview

Cryptolytic is a platform for beginners tinkering with cryptocurrency to the seasoned trader. It provides you with recommendations on when to buy and sell based on technical indicators and assesses the markets to predict arbitrage opportunities before they even happen.

Application Website

See demo

Contributers

Team Lead

Stanley Kusmier

Data Science Team

Alfredo Quintana Elizabeth Ter Sahakyan Marvin A Davila Nathan Van Wyck Taylor Bickell

Application Architechture

How it Works

We gather the historical data from each exchange’s API into a database for all of our supported exchanges and trading pairs and implement Lambda functions in Cloud9 to collect live data from the cryptowat.ch API in the respective tables. This allows us to have the most up to date data in our databases for predictions.

Random forest classifier models were then trained on that dataset, stored in S3 buckets, and more Lambda functions were used to load those models and make predictions on the live data every 3 minutes. Those predictions are inserted into the database in a new table that stores all of the predictions.

The Flask app retrieves the most recent predictions from the database and can return them to a user or be available for a backend team to use via API. The API was deployed on Elastic Beanstalk.

Overall we created a data pipeline and a backend that makes use of 15 different models to generate and store predictions on a recurring basis, which can then be accessed via API endpoints.

Product Canvas and Set Up

Cryptolytic Product Canvas Notion

Cryptolytic Set-Up Guide

Tech Stack

Python, SQL, Flask, AWS (Elastic Beanstalk, RDS, Lambda Functions, Cloud9, KMS, Sagemaker), PostgreSQL

Models

The modeling/models/ folder contains two zip files, with a total of 15 models:

tr_pickles.zip contains 9 pickled trade recommender models.

arb_models.zip contains 6 pickled arbitrage models.

All 15 models use a Random Forest Classifier algorithm.

Each trade recommender model recommends trades for a particular trading pair on a particular exchange by predicting whether the closing price will increase by enough to cover the costs of executing a trade.

The arbitrage models predict arbitrage opportunities between two exchanges for a particular trading pair. Predictions are made ten minutes in advance. To count as an arbitrage opportunity, a price disparity between two exchanges must last for at least 30 minutes, and the disparity must be great enough to cover the costs of buying on one exchange and selling on the other.

Features

Each of the nine trade recommender models is trained on 67 features. Of those 67 features, five are taken directly from the OHLCV data (open, high, low, close, base_volume), one indicates where gaps were present in the data (nan_ohlcv), three indicate the time (year, month, day), and the remainder are technical analysis features.

Each of the 6 arbitrage models is trained on 70 features. Of those 70 features, three features indicate the time (year, month, day), and four indicate the degree and length of price disparities between two exchanges (higher_closing_price, pct_higher, arbitrage_opportunity, window_length). The remaining 63 features are the original OHLCV features and technical analysis features specific to the two exchanges in a given arbitrage dataset and are labeled with the suffixes "exchange_1" and "exchange_2".

Technical analysis features were engineered with the Technical Analysis Library. They fall into five categories:

(1) Momentum indicators

(2) Volume indicators

(3) Volatility indicators

(4) Trend indicators

(5) Others indicators

Documentation for the technical analysis features features is available here:

Technical Analysis Library Documentation

Data Sources

We obtained all of our data from the Cryptowatch, Bitfinex, Coinbase Pro, and HitBTC APIs. Documentation for obtaining that data is listed below:

Cryptowatch API OHLCV Data Documentation

Bitfinex API OHLCV Data Documentation

Coinbase Pro API OHLCV Data Documentation

HitBTC OHLCV Data Documentation

Kraken OHLCV Data Documentation

Gemini OHLCV Data Documentation

Python Notebooks

Arbitrage Prediction Models

Data Processing Notebook

Modeling Notebook

Model Evaluation Notebook

Trade Recommender Models

Data Processing and Modeling Notebook

Visualization Notebook

How to connect to the Cryptolytic API

http://www.cryptolyticapp.com/ (running on AWS but models outdated!)

Trade API /trade

Method: ["GET"]

Returns: {"results": "{('exchange', 'trading_pair'): [{ 'p_time': 'time', ‘period’: ‘minutes’, 'prediction': 'result'}], }"}

Arbitrage API /arbitrage

Note: Unavailable (AWS Cost High)

Method: ["GET"]

Returns: {"results":"{ ('exchange_1', 'exchange_2', 'trading_pair'): [ {'p_time': 'time', 'prediction': 'result'} ]}

Medium Articles

We wrote article on how to do some of the technical aspects of this project:

Contributing

When contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change.

Please note we have a code of conduct. Please follow it in all your interactions with the project.

Issue/Bug Request

If you are having an issue with the existing project code, please submit a bug report under the following guidelines:

  • Check first to see if your issue has already been reported.
  • Check to see if the issue has recently been fixed by attempting to reproduce the issue using the latest master branch in the repository.
  • Create a live example of the problem.
  • Submit a detailed bug report including your environment & browser, steps to reproduce the issue, actual and expected outcomes, where you believe the issue is originating from, and any potential solutions you have considered.

Feature Requests

We would love to hear from you about new features which would improve this app and further the aims of our project. Please provide as much detail and information as possible to show us why you think your new feature should be implemented.

Pull Requests

If you have developed a patch, bug fix, or new feature that would improve this app, please submit a pull request. It is best to communicate your ideas with the developers first before investing a great deal of time into a pull request to ensure that it will mesh smoothly with the project.

Remember that this project is licensed under the MIT license, and by submitting a pull request, you agree that your work will be, too.

Pull Request Guidelines

  • Ensure any install or build dependencies are removed before the end of the layer when doing a build.
  • Update the README.md with details of changes to the interface, including new plist variables, exposed ports, useful file locations and container parameters.
  • Ensure that your code conforms to our existing code conventions and test coverage.
  • Include the relevant issue number, if applicable.
  • You may merge the Pull Request in once you have the sign-off of two other developers, or if you do not have permission to do that, you may request the second reviewer to merge it for you.

Attribution

These contribution guidelines have been adapted from this good-Contributing.md-template.

About

Predicting Arbitrage in the Crypto Market with Machine Learning

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Jupyter Notebook 96.2%
  • Python 3.4%
  • HTML 0.4%