Add get forex data #258

Boreas813 · 2020-10-06T05:50:23Z

add get_forex_data, this method added some parameters not in stock model, details are in the comments of the method and readme

…add_get_forex_data

enzoampil

Thanks for this PR! 😁

Starting with some initial comments around whether there's a way to do the data pull with a date filter right away so that the processing time can be shortened. At the moment, it seems to take very long. Also, added a comment about including get_forex_data in the init.py of the data module.

enzoampil · 2020-10-06T08:37:46Z

python/fastquant/data/forex/tests.py

@@ -0,0 +1,127 @@
+import unittest


Can we move this file to fastquant/python/tests/? This is where we store the tests and run them via pytest. I'm not sure if this will automatically run as a class via pytest, but if now, we can address that in a later PR 😄

oh, it's a unittest file, I‘ve never used pytest. I can rewrite it use pytest later

Currently, unittest works fine but we have to convert this to use pytest for consistency. And we make sure that this test is also run in Travis-CI. @Boreas514 , pytest structure is simpler in that you don't have to define classes but only functions and assert statements (see tests).

enzoampil · 2020-10-13T01:57:15Z

python/fastquant/data/forex/forextester.py

+
+    logger.info(f'start downloading forex {symbol} data zip file...')
+    file_size = 0
+    res = requests.get(f'http://www.forextester.com/templates/data/files/{symbol}.zip', stream=True)


@Boreas514 So it seems that we download the full historical data from each data pull? Is there a way to query only the data from a specific time period? This make the get data process very slow for users 😄

For the first time to use, it will download data from online and make local cache, after that set get_forex_data's parameter read_from_local=False so that it will read data from local.

I agree with enzoampil that we should avoid the caching all forex data. We should cache only what is requested. Is there an easy implementation you can try?

Makeing one symbol and one period cache in single process needs 15 minutes and 2GB RAM, is the cost acceptable?
A minimum of 2GB of memory is required to load 1 min chart of forex data from 2000 to 2018.

Can you explain why such large memory is needed?

enzoampil · 2020-10-13T01:58:32Z

python/fastquant/data/forex/forex.py

+from fastquant.data.forex.forextester import get_forextester_data, get_local_data
+
+
+def get_forex_data(symbol, start_date=None, end_date=None, source="forextester", time_frame='D1', read_from_local=False):


Let's also import get_forex_data in the __init__.py of the data module (here).

jpdeleon · 2020-10-17T09:38:03Z

README.md

@@ -110,6 +112,39 @@ get_crypto_data("BTCUSDT", "2019-01-01", "2019-03-01")

 *R does NOT have support for backtesting yet*

+## Get forex data
+
+All symbols from [forextester.com](https://forextester.com/) are accessible via `get_forex_data`.This mehtod fetch one minute data from source and shape it to other time frame, M1, M15, H1, D1, W1 are supported time frame now.


Suggested change

All symbols from [forextester.com](https://forextester.com/) are accessible via `get_forex_data`.This mehtod fetch one minute data from source and shape it to other time frame, M1, M15, H1, D1, W1 are supported time frame now.

All symbols from [forextester.com](https://forextester.com/) are accessible via `get_forex_data`.This method fetch one minute data from source and shape it to other time frame, M1, M15, H1, D1, W1 are supported time frame now.

jpdeleon · 2020-10-17T09:41:20Z

README.md

+    2020-08-30 22:00:00  1.1906  1.1918  1.1901  1.1916
+[6127 rows x 5 columns]
+```
+this process download online data, transfer it, save cache files in disk, first call will need 25 minutes or more and 7GB free RAM to generate cache files.


Many of our users may not have 7GB free RAM so we cannot implement this. Let's try to cache only data the user actually querries.

Makeing one symbol and one period cache in single process needs 15 minutes and 2GB RAM, is the cost acceptable?
A minimum of 2GB of memory is required to load 1 min chart of forex data from 2000 to 2018.

jpdeleon · 2020-10-17T09:42:29Z

README.md

+```
+this process download online data, transfer it, save cache files in disk, first call will need 25 minutes or more and 7GB free RAM to generate cache files.
+
+After then, set `read_from_local` to True and you get a second response if don't need update local data to newest.


We can read cache automatically when available else download so probably no need to add the read_from_local flag.

jpdeleon

Only partial caching needs to change and we can merge.

…tquant into add_get_forex_data

Boreas813 · 2020-10-22T13:51:38Z

I modified the function to reduce the hardware requirements to 2.3GB and a 20-minute conversion process, have a try.

jpdeleon · 2020-10-26T14:43:22Z

README.md


-After then, set `read_from_local` to True and you get a second response if don't need update local data to newest.
+After then, set `read_from_local=Ture` and you will get a second response if don't need update local data to newest.


read_from_local=True

jpdeleon

No errors when I ran the branch in colab here: https://colab.research.google.com/drive/128gBYBjc4DcoS_QN3Kf4wdV1524vAjOA?hl=en#scrollTo=7sulhEFAWPl0

I like the logging functionality with ETA info.
But it is still very time consuming so we may need to implement faster solutions.

enzoampil · 2020-10-27T03:47:37Z

Thanks again for this @Boreas514 ! I agree with @jpdeleon , it's still kind of long. Is there a way that we can have a get function that follows the pattern of get_stock_data taking around < 10 seconds to get a specific date range?

Keen to get your thoughts on this 😄

Boreas813 added 5 commits September 23, 2020 19:04

add most method

fd8bbb2

get forex data method done

491e86b

Merge branch 'master' of https://github.com/enzoampil/fastquant into …

ee54c17

…add_get_forex_data

fix bugs, use mulitprocess, write readme

e496a81

Merge branch 'master' of https://github.com/enzoampil/fastquant into …

2d11fd0

…add_get_forex_data

enzoampil self-requested a review October 6, 2020 08:26

enzoampil linked an issue Oct 6, 2020 that may be closed by this pull request

[FEATURE] Pull data for forex pairs #160

Open

enzoampil requested changes Oct 13, 2020

View reviewed changes

repair work

4073e30

jpdeleon reviewed Oct 17, 2020

View reviewed changes

Rename tset_forex.py to test_forex.py

3cf0a9a

jpdeleon reviewed Oct 17, 2020

View reviewed changes

Boreas813 added 2 commits October 22, 2020 21:41

split function, cancel multi - process

61731c5

Merge branch 'add_get_forex_data' of https://github.com/Boreas514/fas…

b45783f

…tquant into add_get_forex_data

jpdeleon reviewed Oct 26, 2020

View reviewed changes

jpdeleon requested changes Oct 26, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add get forex data #258

Add get forex data #258

Boreas813 commented Oct 6, 2020

enzoampil left a comment

enzoampil Oct 6, 2020

Boreas813 Oct 13, 2020

jpdeleon Oct 26, 2020

enzoampil Oct 13, 2020

Boreas813 Oct 13, 2020

jpdeleon Oct 17, 2020

Boreas813 Oct 17, 2020

jpdeleon Oct 26, 2020

enzoampil Oct 13, 2020

jpdeleon Oct 17, 2020

jpdeleon Oct 17, 2020

Boreas813 Oct 17, 2020

jpdeleon Oct 17, 2020

jpdeleon left a comment

Boreas813 commented Oct 22, 2020

jpdeleon Oct 26, 2020

jpdeleon left a comment •

edited

enzoampil commented Oct 27, 2020

		from fastquant.data.forex.forextester import get_forextester_data, get_local_data


		def get_forex_data(symbol, start_date=None, end_date=None, source="forextester", time_frame='D1', read_from_local=False):

	All symbols from [forextester.com](https://forextester.com/) are accessible via `get_forex_data`.This mehtod fetch one minute data from source and shape it to other time frame, M1, M15, H1, D1, W1 are supported time frame now.
	All symbols from [forextester.com](https://forextester.com/) are accessible via `get_forex_data`.This method fetch one minute data from source and shape it to other time frame, M1, M15, H1, D1, W1 are supported time frame now.


		After then, set `read_from_local` to True and you get a second response if don't need update local data to newest.
		After then, set `read_from_local=Ture` and you will get a second response if don't need update local data to newest.

Add get forex data #258

Are you sure you want to change the base?

Add get forex data #258

Conversation

Boreas813 commented Oct 6, 2020

enzoampil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpdeleon left a comment

Choose a reason for hiding this comment

Boreas813 commented Oct 22, 2020

Choose a reason for hiding this comment

jpdeleon left a comment • edited

Choose a reason for hiding this comment

enzoampil commented Oct 27, 2020

jpdeleon left a comment •

edited