You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently fetch data from Binance one token at at time.
So if there are 10 coins (BTC, ETH, DOT, ..) then it takes 10x longer.
TODOs / DoD
Parallelize fetching data, in lake/ohclv_data_factory.py. It's likely a straightforward conversion of a for loop
Related github issues
#804 "[Sim, make $] Make benchmarking parallel, via threading"
Note: we could also parallelize grabbing data within a feed. However this isn't as easy, because the algorithm needs to check what's there already. So consider this for later. But also maybe not needed because at some point we'll have historical data repo / bundle.
The text was updated successfully, but these errors were encountered:
I don't recommend doing this now, since the structure is changing for lake either way. It will only result in either difficult conflicts or entirely lost while fixing said conflicts. I agree it is a good thing, but we should wait until the lake/ETL part is done.
I don't recommend doing this now, since the structure is changing for lake either way. It will only result in either difficult conflicts or entirely lost while fixing said conflicts. I agree it is a good thing, but we should wait until the lake/ETL part is done.
OK. Makes sense.
trentmc
changed the title
[Lake] Parallelize fetching data across >>1 tokens
[Lake] Parallelize fetching data across >>1 coins
Apr 30, 2024
Background / motivation
We currently fetch data from Binance one token at at time.
So if there are 10 coins (BTC, ETH, DOT, ..) then it takes 10x longer.
TODOs / DoD
lake/ohclv_data_factory.py
. It's likely a straightforward conversion of a for loopRelated github issues
Note: we could also parallelize grabbing data within a feed. However this isn't as easy, because the algorithm needs to check what's there already. So consider this for later. But also maybe not needed because at some point we'll have historical data repo / bundle.
The text was updated successfully, but these errors were encountered: