IMPORTANT version 1.47 add hourly and daily data #756
Replies: 3 comments 3 replies
-
|
BTW is anyone has the pandas issue: the quick fix is: # in pdutils.py change this func to:
def merge_data_with_different_freq(list_of_data: list):
from sysobjects.futures_per_contract_prices import futuresContractPrices
list_of_data = [pd.DataFrame(x) for x in list_of_data]
list_as_concat_pd = pd.concat(list_of_data, axis=0)
sorted_pd = list_as_concat_pd.sort_index()
unique_pd = uniquets(sorted_pd)
return futuresContractPrices(unique_pd)I will work on a deeper understanding |
Beta Was this translation helpful? Give feedback.
-
|
That's weird - and I'm not seeing that error. |
Beta Was this translation helpful? Give feedback.
-
|
@robcarver17 I have a few queries about this, would you mind clarifying a few things? My data is different, it didn't come from IB, so I don't have the 11pm magic marker. For me, end of day timestamps are either 5 or 6am the next day (presumably midnight in Chicago converted to UTC, depending on the time of year). So
It would be helpful if you could reformat the post above so that the entire text isn't wrapped in a code block. It looks like maybe the markdown headers (###) should be outside the code block? And paragraph 4 could do with a resolution |
Beta Was this translation helpful? Give feedback.
-
|
"it looks like your converter script extracts end of day prices (marked with your special magic number 11pm) and puts them in one arctic collection, and puts everything else in another - is that right? So I would need to figure out a way to extract the daily prices for my data?" That's correct. You would need to have a play with closing_date_rows_in_pd_object(pd_object) and intraday_date_rows_in_pd_object(pd_object) in syscore.pdutils. If you were using a single time rather than 5am or 6am, then you could just modify NOTIONAL_CLOSING_TIME. A brand new user would eithier need to
I will modify the documentation and I'll edit the code block. |
Beta Was this translation helpful? Give feedback.
-
|
Brilliant - thank you! |
Beta Was this translation helpful? Give feedback.
-
|
How would the strategies change when dealing with intra-day hourly data instead of inter-day daily data? |
Beta Was this translation helpful? Give feedback.
-
This version adds support for seperate hourly (cleanest fix for #729)
Your must run
~/pysystemtrade/sysinit/futures$ python3 create_hourly_and_daily.pybefore you next update your prices. This will create seperate hourly and daily data series. If you don't do this, then your existing data ('merged') will be wiped out.As usual you should backup everything up before running this script.
These two series will then be updated seperately by the usual daily prices updating code, with spike checks for both, and then a merged dataframe written which is what will be read downstream whenever these prices are required (principally to create multiple and adjusted prices). So to an extent the hourly and daily prices are invisible to the user (if there is demand they could be added to interactive_diagnostics).
NOTE: The parameter that stops daily data from being written when intraday data is not available is no longer used, since these processes operate indepedently.
Let's start by looking at the existing data (which is merged)
These are in fact the original merged prices - we can't inspect the hourly and daily (will implement if demand for it)
These won't be modified when we create hourly and daily data, until we run the update multiple and adjusted prices (which will continue to be on a mixed time sequence).
Now let's do a standard manual price update
We seperately write the prices for daily and hourly. Next step is to merge these together to form the merged prices that will be used downstream.
On this line we're doing the write of the merged prices - this is a write, not an update so the number is the total number of rows
Now we want to update the multiple and adjusted prices - this will read the merged prices
Let's see what's happened
The new prices are there. Note we finish on an end of day price
Now let's look at the updated multiple prices
Beta Was this translation helpful? Give feedback.
All reactions