Implement forex python #225

macanudo527 · 2024-03-04T09:09:30Z

What?

Fixes #224
Dependent on #223
This allows users to make use of CSV files for Forex pricing instead of rely on exchangerate.host

Testing

I have not tested this with real world data at all. This is curently WIP.

macanudo527 · 2024-03-21T14:00:25Z

@eprbell this reads in missing forex data from CSV files. But, I did finally find this API that can basically do the same thing, but via a REST API. Should I keep both and use the CSV files as a fallback?

eprbell · 2024-03-22T03:48:44Z

@eprbell this reads in missing forex data from CSV files. But, I did finally find this API that can basically do the same thing, but via a REST API. Should I keep both and use the CSV files as a fallback?

Sounds good, but should it be a separate plugin, so that the user can decide which to use and with what priority?

macanudo527 · 2024-03-25T01:33:01Z

Sounds good, but should it be a separate plugin, so that the user can decide which to use and with what priority?

I'm wondering about how to implement that. Maybe have two different CCXT pair converters one with the REST API, and one with the importing of CSV?

eprbell · 2024-03-25T02:02:47Z

Sounds good, but should it be a separate plugin, so that the user can decide which to use and with what priority?

I'm wondering about how to implement that. Maybe have two different CCXT pair converters one with the REST API, and one with the importing of CSV?

Yes, that's what I was thinking: that should give the user the most flexibility.

macanudo527 · 2024-03-26T02:28:15Z

@eprbell btw, I'm going to try to cherry pick and splice the code from your PR. Hopefully, I can get it all put together here soon.

Should I implement this with inheritance, or by parameter that will load a particular class to handle forex processing? I'm thinking making ccxt pair converter abstract and the forex processing has to inherit from it. OR, make the REST API default, and then have another CSV class inherit and override the defaults.

eprbell · 2024-03-26T03:29:55Z

@eprbell btw, I'm going to try to cherry pick and splice the code from your PR. Hopefully, I can get it all put together here soon.

Sounds good, no worries.

Should I implement this with inheritance, or by parameter that will load a particular class to handle forex processing? I'm thinking making ccxt pair converter abstract and the forex processing has to inherit from it. OR, make the REST API default, and then have another CSV class inherit and override the defaults.

I'm leaning toward making the ccxt-based converter abstract and have the two forex processing plugins inherit from it. In theory we could do something similar to the input plugins and have csv and rest subdirectories, but that would not be backward compatible, would require changes in users config files, and would make this change more complex: let's leave it as is. If needed, we'll tackle that issue as a separate change later.

macanudo527 · 2024-05-13T05:33:18Z

@eprbell This should fix most Forex Issues. It is lacking docs, but I thought I would let you get a peek at it.

eprbell · 2024-05-13T06:09:04Z

@eprbell This should fix most Forex Issues. It is lacking docs, but I thought I would let you get a peek at it.

Sounds good: I'll take a look (the diff in Github is large so it may take some time for me to go through it).

macanudo527 · 2024-05-13T14:26:31Z

Sounds good: I'll take a look (the diff in Github is large so it may take some time for me to go through it).

I figured it would take a while. I tried to figure out a way to break down into simpler parts, but kind of just had to be done all at once. Sorry.

eprbell

No worries, here's a first review.

eprbell · 2024-05-30T03:40:26Z

docs/configuration_file.md

@@ -485,7 +484,7 @@ aliases = <em>&lt;untradeable_assets&gt;</em>
 Where:
 * `<historical_price_type>` is one of `open`, `high`, `low`, `close`, `nearest`. When DaLi downloads historical market data, it captures a `bar` of data surrounding the timestamp of the transaction. Each bar has a starting timestamp, an ending timestamp, and OHLC prices. You can choose which price to select for price lookups. The open, high, low, and close prices are self-explanatory. The `nearest` price is either the open price or the close price of the bar depending on whether the transaction time is nearer the bar starting time or the bar ending time.
 * `default_exchange` is an optional string for the name of an exchange to use if the exchange listed in a transaction is not currently supported by the CCXT plugin. If no default is set, Kraken(US) is used. If you would like an exchange added please open an issue. The current available exchanges are "Binance.com", "Gate", "Huobi" and "Kraken". Please note that the Binance.com API is not accessible from the US, Canada, Netherlands, and other [prohibited countries](https://www.binance.com/en/legal/list-of-prohibited-countries).
-* `fiat_access_key` is an optional access key that can be obtained from [Exchangerate.host](https://exchangerate.host/). It is required for any fiat conversions, which are typically required if the base fiat is other than USD.
+* The CCXT plugin makes use of the default forex exchange API, Frankfurter, which provides daily rates from the European Central Bank. Rates for bank holidays and weekends are taken from the previous trading day, so if a rate is requested for Saturday, the Friday rate will be used. Two other plugins are also available for forex - one that makes use of Exchangerate.host free plan and requires a fiat_access_key and another plugin that makes use of a CSV file for forex rates.


This seems to be a spurious bullet in this bullet list (which is about ccxt pair converter plugin parameters): should it be moved elsewhere?

eprbell · 2024-05-30T03:42:30Z

docs/configuration_file.md

+-------------------|------|------|------|------|------
+2008-02-22 00:00:00|210.45|215.45|211.56|213.45|8900
+
+Historical CSV Files are available from several places on the web for free. [Forexsb.com](https://forexsb.com/historical-forex-data) is a good source. Currently only 1 day candles are supported. One row per day a rate is needed is required. For example, if you want to use the [official yearly rates from the IRS](https://www.irs.gov/individuals/international-taxpayers/yearly-average-currency-exchange-rates), you will need to copy and paste the same rate for all days of the calendar year. CSV files can contain multiple years of rates.


Typo: "is needed is required"

eprbell · 2024-05-30T03:46:03Z

docs/configuration_file.md

+
+#### Note on Forex CSV file
+
+If you do not want to use Exchangerate.host or the Frankfurter API for Forex prices, you can use CSV files by saving them to the folder `.dali_cache/forex`. The files must be in the format of quote_asset_base_asset.csv (e.g. USD_JPY.csv). The format should be Time, high, low, open, close, volume. Like the below example:


Suggested change

If you do not want to use Exchangerate.host or the Frankfurter API for Forex prices, you can use CSV files by saving them to the folder `.dali_cache/forex`. The files must be in the format of quote_asset_base_asset.csv (e.g. USD_JPY.csv). The format should be Time, high, low, open, close, volume. Like the below example:

If you do not want to use Exchangerate.host or the Frankfurter API for Forex prices, you can use CSV files by saving them to the folder `.dali_cache/forex`. The files must be in the format of <quote_asset>_<base_asset>.csv (e.g. USD_JPY.csv). The format should be Time, high, low, open, close, volume. Like the below example:

eprbell · 2024-05-30T03:51:56Z

src/dali/abstract_ccxt_pair_converter_plugin.py

+# It appears Kraken public API is limited to around 12 calls per minute.
+# There also appears to be a limit of how many calls per 2 hour time period.
+# Being authenticated lowers this limit.
+_REQUEST_DELAYDICT: Dict[str, float] = {_KRAKEN: 5.1, _BITFINEX: 5.0}


suggestion
_REQUEST_DELAY_DICT

eprbell · 2024-05-30T03:55:01Z

src/dali/abstract_ccxt_pair_converter_plugin.py

+}
+
+# If Exchangerates.host is not available or the user does not have an access key, we can use this list
+DEFAULT_FIAT_LIST: List[str] = ["AUD", "CAD", "CHF", "EUR", "GBP", "JPY", "NZD", "USD"]


Should this list have the same number of elements as FIAT_PRIORITY?

eprbell · 2024-05-30T04:18:43Z

src/dali/plugin/pair_converter/ccxt.py

@@ -1,4 +1,4 @@
-# Copyright 2022 Neal Chambers
+# Copyright 2024 Neal Chambers


The file is not new, so the date should not change.

eprbell · 2024-05-30T04:26:06Z

src/dali/plugin/pair_converter/ccxt.py

+
+                    for day in range((end_of_year - beginning_of_year).days + 1):
+                        current_day: datetime = beginning_of_year + timedelta(days=day)
+                        print(f"current_day: {current_day}, timestamp: {key.timestamp}")


Use the logger module instead of print.

eprbell · 2024-05-30T04:30:50Z

src/dali/plugin/pair_converter/ccxt_fiat_from_csv.py

@@ -0,0 +1,120 @@
+# Copyright 2024 Neal Chambers


Shouldn't this file live in the csv subdirectory?

eprbell · 2024-05-30T04:34:48Z

src/dali/plugin/pair_converter/csv/kraken.py

@@ -62,7 +62,7 @@
 _GOOGLE_DRIVE_DOWNLOAD_URL: str = "https://drive.usercontent.google.com/download"

 # File ID for the unified CSV file ID. This will need to be replaced every quarter.
-_UNIFIED_CSV_FILE_ID: str = "16YKyFkYlvawCHv3W7WuTFzM8RYgMRWMt"
+_UNIFIED_CSV_FILE_ID: str = "11WtjXA9kvVYV9KDoebGV5U75dmcA3bJa"


Can you add the full URL to the file into a comment, so that people can download it manually and inspect it, if they wish to do so?

eprbell · 2024-05-30T04:40:22Z

tests/test_plugin_ccxt.py

@@ -1,4 +1,4 @@
-# Copyright 2022 Neal Chambers
+# Copyright 2024 Neal Chambers


No need to change the date.

macanudo527 self-assigned this Mar 4, 2024

macanudo527 mentioned this pull request Mar 18, 2024

Implement Binance.US REST API Data Loader Plugin #4

Open

macanudo527 and others added 5 commits March 21, 2024 22:57

Add CSV Read for Forex Prices

e973ac1

Add Documentation on How to Save CSV Files

f20656b

Fix Formatting

6ad1f1f

Add Tests for Forex CSV

120fe35

Fix Spacing (modified by Black)

176a314

macanudo527 force-pushed the implement_forex_python branch from 6811add to bf60891 Compare March 21, 2024 13:57

macanudo527 force-pushed the implement_forex_python branch 6 times, most recently from 8b754be to 9a57efb Compare April 17, 2024 00:43

macanudo527 force-pushed the implement_forex_python branch 2 times, most recently from 0b2ea73 to 8aebb3c Compare April 19, 2024 00:48

macanudo527 mentioned this pull request Apr 19, 2024

fiat access key required when not needed #238

Open

macanudo527 force-pushed the implement_forex_python branch 6 times, most recently from 6286ef4 to 6414ff7 Compare April 27, 2024 00:21

macanudo527 force-pushed the implement_forex_python branch from 6414ff7 to 19047f5 Compare May 9, 2024 07:46

macanudo527 force-pushed the implement_forex_python branch from ba6864a to 5b3aa5c Compare May 13, 2024 05:27

macanudo527 mentioned this pull request May 13, 2024

Forex monkey patch #236

Closed

macanudo527 marked this pull request as ready for review May 13, 2024 05:29

macanudo527 mentioned this pull request May 13, 2024

Reorganize pair converter code #218

Open

Make Compatible with Other PR

f2522be

macanudo527 force-pushed the implement_forex_python branch from 2c33b00 to f2522be Compare May 13, 2024 14:02

Neal Chambers added 2 commits May 13, 2024 23:17

Delete Gzip encoding due to Bug

de95ed2

Fix String Concatenation Bug

c165d3f

macanudo527 force-pushed the implement_forex_python branch from 785412c to c165d3f Compare May 13, 2024 14:18

Merge branch 'main' into implement_forex_python

8546089

macanudo527 changed the title ~~[WIP] Implement forex python~~ Implement forex python May 14, 2024

macanudo527 and others added 5 commits May 16, 2024 09:58

Merge branch 'main' into implement_forex_python

c329140

Add Docs for Forex Plugins

1484add

Update Google File Id

5995e02

Make Logger Protected

498f35d

Fix cache_modifier in exchangerate_host

fe722cf

macanudo527 force-pushed the implement_forex_python branch from e257d7e to fe722cf Compare May 24, 2024 02:19

Remove CCXT specific methods from abstract_pair_converter

66c0a38

macanudo527 force-pushed the implement_forex_python branch from 2943ca8 to 40fcb09 Compare May 28, 2024 07:42

macanudo527 added 2 commits May 28, 2024 16:49

Squash Zeroing Out Bug

bde959b

Fix Pylint Error

a2199d4

macanudo527 force-pushed the implement_forex_python branch from 40fcb09 to a2199d4 Compare May 28, 2024 07:49

Fix Forex Bug

22d5af8

eprbell requested changes May 30, 2024

View reviewed changes

macanudo527 force-pushed the implement_forex_python branch from f390aea to 398fb05 Compare May 31, 2024 07:48

[WIP] Pruning

226eb8c

macanudo527 force-pushed the implement_forex_python branch from 398fb05 to 226eb8c Compare June 4, 2024 00:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement forex python #225

Implement forex python #225

macanudo527 commented Mar 4, 2024 •

edited

macanudo527 commented Mar 21, 2024

eprbell commented Mar 22, 2024

macanudo527 commented Mar 25, 2024

eprbell commented Mar 25, 2024

macanudo527 commented Mar 26, 2024

eprbell commented Mar 26, 2024

macanudo527 commented May 13, 2024

eprbell commented May 13, 2024

macanudo527 commented May 13, 2024 •

edited

eprbell left a comment

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024

eprbell May 30, 2024


		#### Note on Forex CSV file

		If you do not want to use Exchangerate.host or the Frankfurter API for Forex prices, you can use CSV files by saving them to the folder `.dali_cache/forex`. The files must be in the format of quote_asset_base_asset.csv (e.g. USD_JPY.csv). The format should be Time, high, low, open, close, volume. Like the below example:

		@@ -1,4 +1,4 @@
		# Copyright 2022 Neal Chambers
		# Copyright 2024 Neal Chambers

Implement forex python #225

Are you sure you want to change the base?

Implement forex python #225

Conversation

macanudo527 commented Mar 4, 2024 • edited

What?

Testing

macanudo527 commented Mar 21, 2024

eprbell commented Mar 22, 2024

macanudo527 commented Mar 25, 2024

eprbell commented Mar 25, 2024

macanudo527 commented Mar 26, 2024

eprbell commented Mar 26, 2024

macanudo527 commented May 13, 2024

eprbell commented May 13, 2024

macanudo527 commented May 13, 2024 • edited

eprbell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macanudo527 commented Mar 4, 2024 •

edited

macanudo527 commented May 13, 2024 •

edited