remove out-dated jupyternotebook, rename nntrader to pgportfolio

README.md

@@ -17,132 +17,3 @@ Note that this library is a part of our main project, which is several versions
 ## Platform Support
 The python 3.5 in windows and python 2.7 in linux are supported.
 
-
-# User Guide
-## Configuration File
-Under the `nntrader/nntrader` directory, there is a json file called `net_config.json`,
- holding all the configuration of the agent and could be modified outside the program code.
-### Network Topology
-* `"layers"`
-    * layers list of the CNN, including the output layer
-    * `"type"`
-        * domain is {"ConvLayer", "FullyLayer", "DropOut", "MaxPooling",
-        "AveragePooling", "LocalResponseNormalization", "SingleMachineOutput",
-        "LSTMSingleMachine", "RNNSingleMachine"}
-    * `"filter shape"`
-        * shape of the filter (kernal) of the Convolution Layer
-* `"input"`
-    * `"window_size"`
-        * number of columns of the input matrix
-    * `"coin_number"`
-        * number of rows of the input matrix
-    * `"feature_number"`
-        * number of features (just like RGB in computer vision)
-        * domain is {1, 2, 3}
-        * 1 means the feature is ["close"], last price of each period
-        * 2 means the feature is ["close", "volume"]
-        * 3 means the features are ["close", "high", "low"]
-
-### Market Data
-* `"input "`
-    * `"start_date"`
-        * start date of the global data matrix
-        * format is yyyy/MM/dd
-    * `"end_date"`
-        * start date of the global data matrix
-        * format is yyyy/MM/dd
-    * `"is_permed"`
-        * if the samples inside the mini-batch of will be permed
-        * the start point of each mini-batch will be permed by default
-    * `"volume_average_days"`
-        * number of days of volume used to select the coins
-    * `"online"`
-        * if it is not online, the program will select coins and generate inputs
-        from the local database.
-        * if it is online, new data that dose not exist in the database would be saved
-
-## Training and Tuning the hyper-parameters
-1. First, change the content in the `nntrader/nntrader/net_config.json` file.
-2. make sure current directory is under `nntrader` and type `python main.py --mode=generating --repeat=10`
-    * this will make 10 subfolders under the `train_package`
-    * in each subfloder, there is a copy of the `net_config.json`
-    * `--repeat=` could follow any positive integers
-3. type `python main.py --mode=generating --processes=1`
-    * this will start training one by one of the 10 floders created just now
-    * do not start more than 1 processes if you want to download data online
-4. after that, check the summary of the training in `nntrader/train_package/train_summary`
-5. tune the hyper-parameters based on the summary, and goto 1 again.
-
-## Logging
-There are three types of logging of each training.
-* In each subfloder
-    * There is a text file called `programlog`, which is the log generated by the running programming.
-        * If the number of process is 1, the logging level of the file is `DEBUG`, else it is `INFO`
-        * If the number of process is 1, the console will output the logging with level `INFO`
-    * There is a `tensorboard` folder saves the data about the training process which could be viewed by tensorboard.
-        * type `tensorboard --logdir=train_package/1` to use tensorboard
-* The summary infomation of this training, including network configuration, portfolio value on validation set and test set etc., will be saved in the `train_summary.csv` under `train_pakage` folder
-
-## Save and Restore of the Model
-* The trained weights of the network are saved at `train_package/1` named as `netfile` (including 3 files). 
-
-## Download Data
-* Type `python main.py --mode=download_data` you can download data without starting training
-* The program will use the configurations in `nntrader/nntrader/net_config` to select coins and
-  download necessary data to train the network.
-* The downloading speed is very slow and sometimes even have timeout error on computers in China
-    * It seems that vpn did not help a lot.
-    * The workstation in xjtlu sometimes have acceptable downloading speed.
-    * Amazon and Google cloud servers download very fast.
-
-## Back-test
-* Type `python main.py --mode=backtest --algo=1` to conduct
-back test with rolling train(i.e. online learning in supervised learning)
-on the target model.
-paper trading
-* `--algo` could be either the name of traditional method or the index of training folder
-
-## Tradition Agent
-see progress on [wiki](https://github.com/ZhengyaoJiang/nntrader/wiki/Current-Progress-of-Traditional-Agents)
-
-OLPS summary:
-
-![](https://github.com/DexHunter/nntrader/blob/dev/images/olps_algo.png)
-
-## Ploting
-* type `python main.py --mode=plot --algos=crp,olmar,40 --labels=crp,olmar,nnagent
-`,for example, to plot
-* `--algos` could be the name of the tdagent algorithms or
-the index of nnagent
-* `--labels` is the name that will be shown in the legend
-* result is
-![](http://oan6f7zbh.bkt.clouddn.com/17-4-23/91567996-file_1492914862957_4f40.png)
-* a sample csv file https://drive.google.com/open?id=0Bz5He3MSOBUtNnZFRGFaRUhDZW8
-* In order to run this sample, a larger database file, covering longer timespan, may be needed https://drive.google.com/open?id=0Bz5He3MSOBUtQUJVVkpoMTBDcG8
-
-## present backtest results in a table
-* type `python main.py --mode=table --algos=40,olmar,ons --labels=nntrader,olmar,ons`
-* `--algos` and `--lables` are the same as in plotting case
-* result is like:
-```
-          max_drawdown  portfolio_value  sharpe_ratio
-nntrader      0.360347       156.357158      0.099530
-olmar         0.452494         7.968085      0.045061
-ons           0.231059         1.799372      0.033449
-```
-* use `--format` arguments to change the format of the table,
- could be `raw` `html` or `latex`
-    * sample of latex format:
-
-```
-\begin{tabular}{lrrr}
-\toprule
-{} &  max\_drawdown &  portfolio\_value &  sharpe\_ratio \\
-\midrule
-nntrader &      0.360347 &       156.357158 &      0.099530 \\
-olmar    &      0.452494 &         7.968085 &      0.045061 \\
-ons      &      0.231059 &         1.799372 &      0.033449 \\
-\bottomrule
-\end{tabular}
-
-```

main.py

@@ -6,11 +6,11 @@
 from argparse import ArgumentParser
 from datetime import datetime
 
-from nntrader.tools.configprocess import preprocess_config
-from nntrader.tools.configprocess import load_config
-from nntrader.tools.trade import save_test_data
-from nntrader.tools.shortcut import execute_backtest
-from nntrader.resultprocess import plot
+from pgportfolio.tools.configprocess import preprocess_config
+from pgportfolio.tools.configprocess import load_config
+from pgportfolio.tools.trade import save_test_data
+from pgportfolio.tools.shortcut import execute_backtest
+from pgportfolio.resultprocess import plot
 
 
 def build_parser():
@@ -38,7 +38,7 @@ def build_parser():
     parser.add_argument("--device", dest="device", default="cpu",
                         help="device to be used to train")
     parser.add_argument("--folder", dest="folder", type=int,
-                        help="folder(int) to load the config, neglect this option if loading from ./nntrader/net_config")
+                        help="folder(int) to load the config, neglect this option if loading from ./pgportfolio/net_config")
     return parser
 
 
@@ -51,19 +51,19 @@ def main():
         os.makedirs("./" + "database")
 
     if options.mode == "train":
-        import nntrader.autotrain.training
+        import pgportfolio.autotrain.training
         if not options.algo:
-            nntrader.autotrain.training.train_all(int(options.processes), options.device)
+            pgportfolio.autotrain.training.train_all(int(options.processes), options.device)
         else:
             for folder in options.train_floder:
                 raise NotImplementedError()
     elif options.mode == "generate":
-        import nntrader.autotrain.generate as generate
+        import pgportfolio.autotrain.generate as generate
         logging.basicConfig(level=logging.INFO)
         generate.add_packages(load_config(), int(options.repeat))
     elif options.mode == "download_data":
-        from nntrader.marketdata.datamatrices import DataMatrices
-        with open("./nntrader/net_config.json") as file:
+        from pgportfolio.marketdata.datamatrices import DataMatrices
+        with open("./pgportfolio/net_config.json") as file:
             config = json.load(file)
         config = preprocess_config(config)
         start = time.mktime(datetime.strptime(config["input"]["start_date"], "%Y/%m/%d").timetuple())

nntrader/autotrain/generate.py → pgportfolio/autotrain/generate.py

@@ -7,10 +7,10 @@
 
 def add_packages(config, repeat=1):
     train_dir = "train_package"
-    package_dir = path.realpath(__file__).replace('nntrader/autotrain/generate.pyc',train_dir)\
-        .replace("nntrader\\autotrain\\generate.pyc", train_dir)\
-                  .replace('nntrader/autotrain/generate.py',train_dir)\
-        .replace("nntrader\\autotrain\\generate.py", train_dir)
+    package_dir = path.realpath(__file__).replace('pgportfolio/autotrain/generate.pyc',train_dir)\
+        .replace("pgportfolio\\autotrain\\generate.pyc", train_dir)\
+                  .replace('pgportfolio/autotrain/generate.py',train_dir)\
+        .replace("pgportfolio\\autotrain\\generate.py", train_dir)
     all_subdir = [int(s) for s in os.listdir(package_dir) if os.path.isdir(package_dir+"/"+s)]
     if all_subdir:
         max_dir_num = max(all_subdir)

nntrader/autotrain/training.py → pgportfolio/autotrain/training.py

@@ -6,8 +6,8 @@
 import os
 import time
 from multiprocessing import Process
-from nntrader.learn.tradertrainer import TraderTrainer
-from nntrader.tools.configprocess import load_config
+from pgportfolio.learn.tradertrainer import TraderTrainer
+from pgportfolio.tools.configprocess import load_config
 
 
 def train_one(save_path, config, log_file_dir, index, logfile_level, console_level, device):

nntrader/constants.py → pgportfolio/constants.py

@@ -3,10 +3,10 @@
 from os import path
 
 DATABASE_DIR = path.realpath(__file__).\
-    replace('nntrader/constants.pyc','/database/Data.db').\
-    replace("nntrader\\constants.pyc","database\\Data.db").\
-    replace('nntrader/constants.py','/database/Data.db').\
-    replace("nntrader\\constants.py","database\\Data.db")
+    replace('pgportfolio/constants.pyc','/database/Data.db').\
+    replace("pgportfolio\\constants.pyc","database\\Data.db").\
+    replace('pgportfolio/constants.py','/database/Data.db').\
+    replace("pgportfolio\\constants.py","database\\Data.db")
 CONFIG_FILE_DIR = 'net_config.json'
 LAMBDA = 1e-4  # lambda in loss function 5 in training
    # About time

nntrader/learn/nnagent.py → pgportfolio/learn/nnagent.py

@@ -2,8 +2,8 @@
 import tflearn
 import tensorflow as tf
 import numpy as np
-from nntrader.constants import *
-import nntrader.learn.network as network
+from pgportfolio.constants import *
+import pgportfolio.learn.network as network
 
 class NNAgent:
     def __init__(self, config, restore_dir=None, device="cpu"):

nntrader/learn/rollingtrainer.py → pgportfolio/learn/rollingtrainer.py

@@ -1,7 +1,7 @@
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
-from nntrader.learn.tradertrainer import TraderTrainer
+from pgportfolio.learn.tradertrainer import TraderTrainer
 import logging
 import tflearn
 

nntrader/learn/tradertrainer.py → pgportfolio/learn/tradertrainer.py

@@ -11,8 +11,8 @@
 import numpy as np
 import pandas as pd
 import tensorflow as tf
-from nntrader.learn.nnagent import NNAgent
-from nntrader.marketdata.datamatrices import DataMatrices
+from pgportfolio.learn.nnagent import NNAgent
+from pgportfolio.marketdata.datamatrices import DataMatrices
 import logging
 Result = collections.namedtuple("Result",
                                 [
@@ -200,7 +200,7 @@ def train_net(self, log_file_dir="./tensorboard", index="0"):
         return self.__log_result_csv(index, time.time() - starttime)
 
     def __log_result_csv(self, index, time):
-        from nntrader.trade import backtest
+        from pgportfolio.trade import backtest
         dataframe = None
         csv_dir = './train_package/train_summary.csv'
         tflearn.is_training(False, self._agent.session)

nntrader/marketdata/coinlist.py → pgportfolio/marketdata/coinlist.py

@@ -1,12 +1,12 @@
 from __future__ import absolute_import
 from __future__ import print_function
 from __future__ import division
-from nntrader.marketdata.poloniex import Poloniex
-from nntrader.tools.data import get_chart_until_success
+from pgportfolio.marketdata.poloniex import Poloniex
+from pgportfolio.tools.data import get_chart_until_success
 import pandas as pd
 from datetime import datetime
 import logging
-from nntrader.constants import *
+from pgportfolio.constants import *
 
 
 class CoinList(object):

nntrader/marketdata/datamatrices.py → pgportfolio/marketdata/datamatrices.py

@@ -1,13 +1,13 @@
 from __future__ import print_function
 from __future__ import absolute_import
 from __future__ import division
-import nntrader.marketdata.globaldatamatrix as gdm
+import pgportfolio.marketdata.globaldatamatrix as gdm
 import numpy as np
 import pandas as pd
 import logging
-from nntrader.tools.configprocess import parse_time
-from nntrader.tools.data import get_volume_forward, get_type_list
-import nntrader.marketdata.replaybuffer as rb
+from pgportfolio.tools.configprocess import parse_time
+from pgportfolio.tools.data import get_volume_forward, get_type_list
+import pgportfolio.marketdata.replaybuffer as rb
 
 MIN_NUM_PERIOD = 3
 

nntrader/marketdata/globaldatamatrix.py → pgportfolio/marketdata/globaldatamatrix.py

@@ -2,11 +2,11 @@
 from __future__ import absolute_import
 from __future__ import print_function
 
-from nntrader.marketdata.coinlist import CoinList
+from pgportfolio.marketdata.coinlist import CoinList
 import numpy as np
 import pandas as pd
-from nntrader.tools.data import panel_fillna
-from nntrader.constants import *
+from pgportfolio.tools.data import panel_fillna
+from pgportfolio.constants import *
 import sqlite3
 from datetime import datetime
 import logging

nntrader/resultprocess/plot.py → pgportfolio/resultprocess/plot.py

@@ -7,9 +7,9 @@
 import json
 import numpy as np
 import datetime
-from nntrader.tools.indicator import max_drawdown, sharpe, positive_count, negative_count, moving_accumulate
-from nntrader.tools.configprocess import parse_time, check_input_same
-from nntrader.tools.shortcut import execute_backtest
+from pgportfolio.tools.indicator import max_drawdown, sharpe, positive_count, negative_count, moving_accumulate
+from pgportfolio.tools.configprocess import parse_time, check_input_same
+from pgportfolio.tools.shortcut import execute_backtest
 
 # the dictionary of name of indicators mapping to the function of related indicators
 # input is portfolio changes
@@ -44,7 +44,7 @@
 def plot_backtest(config, algos, labels=None):
     """
     @:param config: config dictionary
-    @:param algos: list of strings representing the name of algorithms or index of nntrader result
+    @:param algos: list of strings representing the name of algorithms or index of pgportfolio result
     """
     results = []
     for i, algo in enumerate(algos):
@@ -105,7 +105,7 @@ def table_backtest(config, algos, labels=None, format="raw",
     """
     @:param config: config dictionary
     @:param algos: list of strings representing the name of algorithms
-    or index of nntrader result
+    or index of pgportfolio result
     @:param format: "raw", "html", "latex" or "csv". If it is "csv",
     the result will be save in a csv file. otherwise only print it out
     @:return: a string of html or latex code

nntrader/tdagent/algorithms/__init__.py → pgportfolio/tdagent/algorithms/__init__.py

@@ -1,3 +1,3 @@
 from __future__ import absolute_import
-from nntrader.tdagent import *
+from pgportfolio.tdagent import *
 # -*- coding: utf-8 -*-

nntrader/tdagent/algorithms/bcrp.py → pgportfolio/tdagent/algorithms/bcrp.py

@@ -1,5 +1,5 @@
 from ..tdagent import TDAgent
-from nntrader.tdagent.algorithms.crp import CRP
+from pgportfolio.tdagent.algorithms.crp import CRP
 import numpy as np
 from scipy.optimize import minimize
 
@@ -22,10 +22,10 @@ def get_weight(self, data):
 
     def decide_by_history(self, x, last_b):
         if self.last_b is None:
-            from nntrader.tools.trade import get_test_data
-            from nntrader.tools.configprocess import preprocess_config
+            from pgportfolio.tools.trade import get_test_data
+            from pgportfolio.tools.configprocess import preprocess_config
             import json
-            with open("nntrader/net_config.json") as file:
+            with open("pgportfolio/net_config.json") as file:
                 config = json.load(file)
             config = preprocess_config(config)
             data = get_test_data(config)
@@ -44,10 +44,10 @@ def opt_weights(X, max_leverage=1):
 
 
 if __name__ == '__main__':
-    from nntrader.tools.backtest import get_test_data
-    from nntrader.tools.configprocess import preprocess_config
+    from pgportfolio.tools.backtest import get_test_data
+    from pgportfolio.tools.configprocess import preprocess_config
     import json
-    with open("nntrader/net_config.json") as file:
+    with open("pgportfolio/net_config.json") as file:
         config = json.load(file)
     config = preprocess_config(config)
     data = get_test_data(config)

nntrader/tdagent/algorithms/best.py → pgportfolio/tdagent/algorithms/best.py

@@ -11,10 +11,10 @@ def __init__(self, last_b=None):
 
     def decide_by_history(self, data, last_b=None):
         if self.last_b is None:
-            from nntrader.tools.trade import get_test_data
-            from nntrader.tools.configprocess import preprocess_config
+            from pgportfolio.tools.trade import get_test_data
+            from pgportfolio.tools.configprocess import preprocess_config
             import json
-            with open("nntrader/net_config.json") as file:
+            with open("pgportfolio/net_config.json") as file:
                 config = json.load(file)
             config = preprocess_config(config)
             data = get_test_data(config)