IMPORTANT: Adding parquet functionality in

[robcarver17]

As discussed here, this is to ensure that dependencies on arctic db can be removed, and so that the required packages and python version can be updated to the current versions.

IF YOU WANT TO USE PARQUET (recommended):

• add a directory to your private config: parquet_store: '/home/rob/data/parquet/' (you do not need to create it, or subdirectories this is done automatically)
• install pyarrow
• backup all your data just in case. Stop any processes using adjusted price data.
• pull and install the latest commit
• run python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py
• you should now be able to get data from parquet rather than arctic
• your arctic data will still be there, but will no longer be updated

IF YOU DO NOT WANT TO USE PARQUET BUT WANT TO GET THE LATEST COMMIT

• Modify the classes assigned in sysproduction.data.production_data_objects to point back to arctic
• Modify the classes assigned in sysdata.sim.db_futures_sim_data to point back to arctic

WHAT WILL HAPPEN IF YOU DON'T DO EITHIER AND PULL THE LATEST COMMIT

• You will find you can't certain data and many things will break. To fix just follow one of the two choices above.

[yk2k]

Thank you for the great tool!
I needed to add this to backup_capital() to make it run when parquet data directory is empty.

try:                                                                   
    parquet_data = data.parquet_capital.get_capital_pd_df_for_strategy(
        strategy_name                                                  
    )                                                                  
except missingData:                                                    
    parquet_data = [] 

if len(parquet_data) >= len(strategy_capital_data):
   ...

[rorymac]

yk2k

A question if you would.

There are two backup_capital( ) functions in PST - I assume you are talking about the one in ~/pysystemtrade/sysinit/transfer/backup_ arctic_to_parquet.py ?

Thank you

[oldlore]

yes that is the file. line 488 or so.

[t3chap]

Rob, et al:
If I'm installing from scratch, do I need to go through the steps to install arctic and mongodb and then run the scripts to transition to parquet?
I have build my own methods of importing the data that work with the previous (pre-parquet, mongodb using arctic) regime but I was trying a fresh install so I could use the latest versions of ubuntu, pandas, etc.

I guess another way of phrasing the question is can I go straight from a clean install ov 1.80 straight into parquet without ever touching mongodb or arctic?

Thanks.

[tgibson11]

can I go straight from a clean install ov 1.80 straight into parquet without ever touching mongodb or arctic?

Without installing arctic, yes. MongoDB is still used. Only the time series data has been moved to Parquet so far.

[tgibson11]

Regarding MongoDB, I've been running version 7 (the latest) on my development machine, along with pymongo 3.11.3, which is documented as compatible, only that it doesn't support newer MongoDB features that pysystemtrade wouldn't be using anyway. This has been working fine for me.

TL;DR - you shouldn't need to worry about installing an older version of MongoDB if that was a concern.

[oldlore]

This is a sort of related question. I'm running Linux Mint 20.3. Adding parquet seems straightforward enough, but is it worth the trouble of upgrading to Mint 21.x and upgrading python from 3.8.10? To maintain compatibility with future libraries perhaps?

[rorymac]

Todd
Question if you would be so kind.
I have all the data backed up in the standard csv files, if I attempt to do a clean install How do I get that into parquet?
I am not having any luck trying the transition, my environment is handing out module not found errors.
Would like to attempt a clean install

[rorymac]

A question Rob, when you say pull and install the latest commit, I assume you mean the droparctic branch?

[robcarver17]

No, I mean develop.

[rorymac]

I have done a full new install of the latest version. [05184ea] The Master branch.
As well as Pyenv
All looked good the test worked - copied over the DB
BUT on running python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py
Got the following error - which is puzzling
rorym@rorym-VirtualBox:~/pysystemtrade$ python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py Configuring sim logging Traceback (most recent call last): File "/home/rorym/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py", line 40, in <module> from sysdata.arctic.arctic_futures_per_contract_prices import ( File "/home/rorym/pysystemtrade/sysdata/arctic/arctic_futures_per_contract_prices.py", line 8, in <module> from sysdata.arctic.arctic_connection import arcticData File "/home/rorym/pysystemtrade/sysdata/arctic/arctic_connection.py", line 2, in <module> from arctic import Arctic ModuleNotFoundError: No module named 'arctic'

As you can see the test bed is a Virtual Machine (which I believe should not make a difference.)
Any ideas out there?

Regards

[craigmediaservices]

It appears your machine thinks the Python artic extension is not installed? Were you able to fix this?

[rorymac]

Yes , I did. You are correct , on the new install , somehow I messed up the requirements.txt file and had not installed arctic.
I hacked it though by doing a mongodump of the production DB and restored it on the new machine.
I also had to fiddle with the csv backup files for the pnl took the back up csv and wrote it to parquet then the system worked.
I took the long way around, created a VM for a test and once I had that working, then installed everything on my server.
I still have a small glitch in that my pnl report won't run. But that is not that much of an issue for me.

[craigmediaservices]

VM is nice for isolation. Right now I've built out everything in Docker so I could theoretically clone the project rapidly on any machine.

[oldlore]

So I've tried to do this as well in a VM box.

1. Ran into the same problem with 'ModuleNotFoundError: No module named 'syscore' error. This occurs when I try to run something that goes through the run.py like interactive_diagnostics. I suspect it has something to do with the virtual environment because I can import modules fine in the python command line or in VS Code. After getting past that, I'm stuck in the parquet config setup.

PermissionError: [Errno 13] Permission denied: '/home/me'

I have the parquet_store variable set up in the private_config with the correct directory and even changed the default config but for some reason it won't read it and keeps looking in /home/me.

[rorymac]

oldlore
I finally figured it out (The module not found error) - you need to check that your system path in python has the correct paths.
procedure is as follows:
open python on the command line
`
rorym@rorym-VirtualBox:~/pysystemtrade$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
import pprint
pprint.pprint(sys.path)
['',
'/usr/lib/python38.zip',
'/usr/lib/python3.8',
'/usr/lib/python3.8/lib-dynload',
'/home/rorym/.local/lib/python3.8/site-packages',
'/usr/local/lib/python3.8/dist-packages',
'/home/rorym/pysystemtrade',
'/usr/local/lib/python3.8/dist-packages/statsmodels-0.12.2-py3.8-linux-x86_64.egg',
'/usr/lib/python3/dist-packages',
'/usr/lib/python3.8/dist-packages']

`
So if the sys.path does not have your pysystemtrade path correct you will get the error. Also another complication if your python version changes then some of the pip packages will not be found.
I have no idea how I messed up the pysys path as that is done when you run the setup.py - but there it is.

I did mange to get the DB transferred to parquet - all except Capital, but that is enough to get by.

The procedure.
Is exactly as described by Rob above BUT do not install - (by that think he means run the setup.py ) if you do that then you get the pip package errors, module not found.

As to your last comment I had something similar on one of the iterations - I do think that has something to do with the change of the python version.