Manually setting capital for has errors with parquet

[t3chap]

I'm trying to get a production system going that, at least at first, I intend to use to generate orders and manually inter them into IB. I also plan to enter the capital manually until I get things further setup/integrated with IB.

when I run the interactive_update_capital_manual() function with no previous capital set (and not pulling it from IB), I get an error when it tries to delete the __global_capital.parquet file that doesn't yet exist.

File ~/pysystemtrade/sysdata/parquet/parquet_access.py:36, in ParquetAccess.delete_data_given_data_type_and_identifier(self, data_type, identifier)
     30 def delete_data_given_data_type_and_identifier(
     31     self, data_type: str, identifier: str
     32 ):
     33     filename = self._get_filename_given_data_type_and_identifier(
     34         data_type=data_type, identifier=identifier
     35     )
---> 36     os.remove(filename)

FileNotFoundError: [Errno 2] No such file or directory: '/root/pysystemtrade/data/parquet/capital/__global_capital.parquet'

I worked around this by editing capital.py to add a try: except surrounding the delete command on or about 190, figuring that this workaround would allow me to at least proceed enough to get orders.
However, I found the RETURN for EXIT doesn't work at the menu, it just loops back to the menu.
After exiting this, I attempted to run the update_system_backtest() function, and get the following errors:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/pysystemtrade/sysdata/parquet/parquet_capital.py:36, in parquetCapitalData.get_capital_pd_df_for_strategy(self, strategy_name)
     35 try:
---> 36     pd_df = self.parquet.read_data_given_data_type_and_identifier(
     37         data_type=CAPITAL_COLLECTION, identifier=strategy_name
     38     )
     39 except:

File ~/pysystemtrade/sysdata/parquet/parquet_access.py:54, in ParquetAccess.read_data_given_data_type_and_identifier(self, data_type, identifier)
     51 filename = self._get_filename_given_data_type_and_identifier(
     52     data_type=data_type, identifier=identifier
     53 )
---> 54 return pd.read_parquet(filename)

File ~/miniconda3/lib/python3.11/site-packages/pandas/io/parquet.py:670, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, filesystem, filters, **kwargs)
    668 check_dtype_backend(dtype_backend)
--> 670 return impl.read(
    671     path,
    672     columns=columns,
    673     filters=filters,
    674     storage_options=storage_options,
    675     use_nullable_dtypes=use_nullable_dtypes,
    676     dtype_backend=dtype_backend,
    677     filesystem=filesystem,
    678     **kwargs,
    679 )

File ~/miniconda3/lib/python3.11/site-packages/pandas/io/parquet.py:265, in PyArrowImpl.read(self, path, columns, filters, use_nullable_dtypes, dtype_backend, storage_options, filesystem, **kwargs)
    263     to_pandas_kwargs["split_blocks"] = True  # type: ignore[assignment]
--> 265 path_or_handle, handles, filesystem = _get_path_or_handle(
    266     path,
    267     filesystem,
    268     storage_options=storage_options,
    269     mode="rb",
    270 )
    271 try:

File ~/miniconda3/lib/python3.11/site-packages/pandas/io/parquet.py:139, in _get_path_or_handle(path, fs, storage_options, mode, is_dir)
    130 if (
    131     not fs
    132     and not is_dir
   (...)
    137     # fsspec resources can also point to directories
    138     # this branch is used for example when reading from non-fsspec URLs
--> 139     handles = get_handle(
    140         path_or_handle, mode, is_text=False, storage_options=storage_options
    141     )
    142     fs = None

File ~/miniconda3/lib/python3.11/site-packages/pandas/io/common.py:872, in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
    870 else:
    871     # Binary mode
--> 872     handle = open(handle, ioargs.mode)
    873 handles.append(handle)

FileNotFoundError: [Errno 2] No such file or directory: '/root/pysystemtrade/data/parquet/capital/rob_sys.parquet'

During handling of the above exception, another exception occurred:

missingData                               Traceback (most recent call last)
File ~/pysystemtrade/sysproduction/strategy_code/run_system_classic.py:82, in runSystemClassic._get_currency_and_capital(self)
     81 try:
---> 82     notional_trading_capital = capital_data.get_current_capital_for_strategy(
     83         strategy_name
     84     )
     85 except missingData:
     86     # critical log will send email

File ~/pysystemtrade/sysproduction/data/capital.py:164, in dataCapital.get_current_capital_for_strategy(self, strategy_name)
    163 try:
--> 164     capital_value = self.db_capital_data.get_current_capital_for_strategy(
    165         strategy_name
    166     )
    167 except missingData:

File ~/pysystemtrade/sysdata/production/capital.py:153, in capitalData.get_current_capital_for_strategy(self, strategy_name)
    152 def get_current_capital_for_strategy(self, strategy_name: str) -> float:
--> 153     capital_series = self.get_capital_pd_df_for_strategy(strategy_name)
    154     return float(capital_series.iloc[-1, 0])

File ~/pysystemtrade/sysdata/parquet/parquet_capital.py:40, in parquetCapitalData.get_capital_pd_df_for_strategy(self, strategy_name)
     39 except:
---> 40     raise missingData(
     41         "Unable to get capital data from parquet for strategy %s"
     42         % strategy_name
     43     )
     45 return pd_df

missingData: Unable to get capital data from parquet for strategy rob_sys

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 update_system_backtests()

Cell In[1], line 29, in update_system_backtests()
     25 for strategy_name in list_of_strategies:
     26     system_backtest_runner = strategyRunner(
     27         data, strategy_name, process_name, backtest_function
     28     )
---> 29     system_backtest_runner.run_strategy_method()

File ~/pysystemtrade/syscontrol/strategy_tools.py:41, in strategyRunner.run_strategy_method(self)
     39 method = self.strategy_method
     40 # no arguments. no return. no explanations
---> 41 method()

File ~/pysystemtrade/sysproduction/strategy_code/run_system_classic.py:56, in runSystemClassic.run_backtest(self)
     53 strategy_name = self.strategy_name
     54 data = self.data
---> 56 base_currency, notional_trading_capital = self._get_currency_and_capital()
     58 system = self.system_method(
     59     notional_trading_capital=notional_trading_capital,
     60     base_currency=base_currency,
     61 )
     63 function_to_call_on_update = self.function_to_call_on_update

File ~/pysystemtrade/sysproduction/strategy_code/run_system_classic.py:91, in runSystemClassic._get_currency_and_capital(self)
     87     error_msg = (
     88         "Capital data is missing for %s: can't run backtest" % strategy_name
     89     )
     90     data.log.critical(error_msg)
---> 91     raise Exception(error_msg)
     93 currency_data = dataCurrency(data)
     94 base_currency = currency_data.get_base_currency()

Exception: Capital data is missing for rob_sys: can't run backtest

Any ideas how to get around this one? It is very possible that I've missed a step somewhere that would setup the capital values in parquet correctly, but I haven't found anything yet.

[jurriaantressel]

I think I overcame this error by creating a blank __global_capital.parquet file and next running interactive_update_capital_manual()

[t3chap]

That works, thank you.

[t3chap]

I also copied the blank file and renamed it for the system name that I was trying to use, as that error occurred next.

[rorymac]

I have the same issue BUT having created the file :
import pyarrow.parquet as pq import pandas as pd df = pd.DataFrame() df.to_parquet('/home/rorym/data/parquet/capital/__global_capital.parquet')

Still get the following error

Traceback (most recent call last):
  File "/home/rorym/pysystemtrade/sysproduction/linux/scripts/run.py", line 66, in <module>
    func(*args, **kwargs)
  File "/home/rorym/pysystemtrade/sysproduction/interactive_update_capital_manual.py", line 44, in interactive_update_capital_manual
    function_to_run(data)
  File "/home/rorym/pysystemtrade/sysproduction/interactive_update_capital_manual.py", line 101, in setup_initial_capital
    data_capital.create_initial_capital(
  File "/home/rorym/pysystemtrade/sysproduction/data/capital.py", line 88, in create_initial_capital
    self.total_capital_calculator.create_initial_capital(
  File "/home/rorym/pysystemtrade/sysdata/production/capital.py", line 476, in create_initial_capital
    self.delete_all_global_capital(are_you_really_sure=are_you_really_sure)
  File "/home/rorym/pysystemtrade/sysdata/production/capital.py", line 513, in delete_all_global_capital
    self.capital_data.delete_all_global_capital(
  File "/home/rorym/pysystemtrade/sysdata/production/capital.py", line 135, in delete_all_global_capital
    self.delete_all_capital_for_strategy(
  File "/home/rorym/pysystemtrade/sysdata/production/capital.py", line 192, in delete_all_capital_for_strategy
    self._delete_all_capital_for_strategy_no_checking(strategy_name)
  File "/home/rorym/pysystemtrade/sysdata/parquet/parquet_capital.py", line 48, in _delete_all_capital_for_strategy_no_checking
    self.parquet.delete_data_given_data_type_and_identifier(
  File "/home/rorym/pysystemtrade/sysdata/parquet/parquet_access.py", line 36, in delete_data_given_data_type_and_identifier
    os.remove(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/rorym/data/parquet/capital/__global_capital.parquet'

perhaps is someone can please show the basic df from _global_capital.parquet - then I can create it and try that instead.
Just the headers should do it.

[emretezel]

Hi Rory,

Were you able to resolve this issue? I am running into the same problem in master.

[emretezel]

I have created an empty file under data parquet, and ran the interactive manual capital update and then tried the transfer again, now getting the following error.

It seems the transfer to parquet script is not working in master branch.

(pysystemtrade-user) pysystemtrade@emre-OptiPlex-3080:~/opt/pysystemtrade/sysinit/transfer$ python backup_arctic_to_parquet.py 
Configuring sim logging
2024-03-29 16:17:10 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-03-29 16:17:10 DEBUG backup_arctic_to_parquet Dumping from arctic, mongo to parquet files
Do futures contract prices?n
FX?n
Multiple prices?n
Adjusted prices?n
Strategy positions?n
Contract positions?n
Capital?y
Traceback (most recent call last):
  File "/home/pysystemtrade/opt/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py", line 548, in <module>
    backup_arctic_to_parquet()
  File "/home/pysystemtrade/opt/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py", line 95, in backup_arctic_to_parquet
    backup_capital(backup_data)
  File "/home/pysystemtrade/opt/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py", line 454, in backup_capital
    if len(parquet_data) > strategy_capital_data:
  File "/home/pysystemtrade/anaconda3/envs/pysystemtrade-user/lib/python3.10/site-packages/pandas/core/generic.py", line 1519, in __nonzero__
    raise ValueError(
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[rorymac]

Yeah I did manage to, the long way around, from the csv backup files for capital, I created a parquet files put those into the directory and it worked.