-
Notifications
You must be signed in to change notification settings - Fork 4
'Prices' module configuration notes
The Prices module implements the concept of untrusted oracle that allows to retrieve price or rate values for currencies and stocks from publicly available sources and store them in blockchain. The Prices module validates the received values using historical data, rejecting possibly cheating prices values.
The Prices module might simultaneously poll several different data sources, called feeds, and store their returned values along with corresponding prices or rates symbols.
The stored price and rate values history could be retrieved with 'prices' rpc call as a json array.
This note explains how to add prices feeds to the komodod daemon on its startup.
The Prices module supports http/https data sources that return result data as a json object. The Prices module has an internal parser that supports extracting values from the resulting json using RFC 6901 'Json Pointer' addressing. The internal parser can extract the value specified by json pointer itself or calculate average value for specified value paths.
The Prices module also allows to add a custom shared object library that could retrieve values if the internal parser features are not sufficient. The library should implement a specific parsing function that will be provided with the returned json and configuration parameters for data retrieving and should return the value.
The Prices module contains a preconfigured feed that always retrieves values for BTC_USD, BTC_EUR and BTC_GBP quotes.
The feed configuration is a string that is passed as -ac_feeds parameter to the komodod daemon on each startup. To make the Prices module work there should be also -ac_cbopret, -ac_cclib=prices and -ac_cc=n (where n >=2) komodod parameters.
Example of the -ac_feeds parameter:
ac_feeds='[{"name":"stocks", "url":"https://api.iextrading.com/1.0/tops/last?symbols=AAPL,ADBE", "results":[{"symbol":"AAPL","valuepath":"/0/price"}, {"symbol":"ADBE","valuepath":"/1/price"}], "multiplier":1000000, "interval":120 }, {other feed config...}]'
The ac_feeds is a quoted string that contains json array of feed configuration objects. The example configuration has a single feed config that allows to receive ADBE and ADBE stocks symbols from api.iextrading.com web service server. The result data is processed by the internal parser.
The configuration contains several parameters, like "name":"stocks" under which this configuration item will be stored in the configuration object, "url" is the address which will be polled, "results" array that describes how to extract prices values from the resulting json. Each element of "results" has "symbol" property which is the name of the symbols and json pointer in "valuepath" property that points to the value in the resulting json.
There are also "multiplier" parameters, a number that the value will be multiplied by to convert it from value with decimal point to integer (like forex prices that returned like "345.5678" should be converted to "3455678") and "interval" which is a time in seconds between polls (120 sec is minimum).
Example of the -ac_feeds parameter with substitutes:
-ac_feeds='[{"name":"metals", "url":"https://forex-data-feed.swissquote.com/public-quotes/bboquotes/instrument/%s/USD", "substitutes":["XAU","XPT"], "quote":"USD", "results":{"averagevaluepaths":["/*/spreadProfilePrices/*/ask","/*/spreadProfilePrices/*/bid"] }, "multiplier":10000, "interval":120 }]'
The example configuration has a single feed config that allows to receive XAU_USD and XPT_USD symbols from forex-data-feed.swissquote.com web service server. The result data is processed by the internal parser.
The configuration contains "name" under which this configuration item will be stored in the configuration object and "url" which is the address that will be polled. Note that the url has '%s' symbols that mean that on their place will be placed symbols from the "substitutes" array, so the poll will be executed two times. Processing of the returned json is described in the "results" object, in this case it denotes that the price value is the average value retrieved with json pointers "/*/spreadProfilePrices/*/ask" and "/*/spreadProfilePrices/*/bid". Note '*' in the pointers that means that this is an array of values in the resulting json and all the array's elements will be used for the average value calculation.
There is also optional "quote" parameter which will be added to the price symbol to form value pair like "XAU_USD" and "XPT_USD". The "quote" could be empty if substitutes strings are complete symbols. If "substitutes" parameters are used in configuration, the "result" parameter is an object and it is applied to resulting json object from each poll which is executed for each substitute.
Example of the -ac_feeds configuration with custom library:
-ac_feeds='[{"name":"metals", "url":"https://forex-data-feed.swissquote.com/public-quotes/bboquotes/instrument/%s/USD", "substitutes":["XAU","XPT"], "quote":"USD", "customlib":"libpricessampleparser.so", "results":{"customdata":"/0/spreadProfilePrices/0/ask"}, "multiplier":10000, "interval":120 }]'
The parameters that differentiate this custom parser feed configuration are "customlib" parameter which contains the name of a custom parser shared lib and "customdata" which are arbitraty data that will be passed to the custom library function.
The parameter -ac_feed is a quoted string with a json array inside it. Each array element is an object with several parameters. The complete parameter list of a feed configuration object:
| parameter | type | description | example |
|---|---|---|---|
| "name" | mandatory, string | name of configuration item | "name":"stocks" |
| "url" | mandatory, string | url of feed web service. The url might contain '%s', in such case there should be a "substitutes" property | with no '%s': "url": "https://api.trade.com/1.0/", with '%s': "url": "https://api.fin.com/api/?symbol=%sBTC"
|
| "substitutes" | optional, array of strings | list of strings to substitute '%s' symbols in "url" parameter to create requests. It is supposed that each request returns a single value. If no "substitutes" array in the config, then a poll could return many values | "substitutes":["XAU","XPT"] |
| "quote" | optional, string | quote symbol to form currency pair like "USD_BTC" | "quote":"BTC" |
| "customlib" | optional, string | name of custom parser library | "customlib": "libmyparser.so" |
| "results" | mandatory, object or array | contains parameters to parse returned json. The "results" is either object if "substitutes" is present (so it is applied to each poll result to get one value) or array (so it allows to get several values from the json). See the members in the next tables | |
| "multiplier" | optional, number | integer multiplier to convert the value to integer, default = 1 | for forex prices "multiplier":10000, for crypto "multiplier":100000000
|
| "interval" | optional, number | poll interval in sec, should be more or equal 120 (default) | "interval":120 |
"results" array item members:
| name | type | description | example |
|---|---|---|---|
| "symbol" | optional, string | symbol name which value is retrieved | "symbol":"USD_BTC" |
| "valuepath" | optional, string | json pointer to value | "valuepath":"/BTC/price" |
| "averagepaths" | optional, array of strings | list of json pointers to values. If present, the average value is calculated. In position of array index in the json pointer it could be '*' which means to use all the array elements | "averagepaths":["/prices/*/bid", /prices/*/ask"] |
| "customdata" | optional, string | arbitrary data passed to custom lib function | "customdata":"/price" |
Notes:
- "results" must be array of objects if "substitutes" parameter is NOT used (otherwise it should be only the object, not an array)
- If no customlib is used then "symbol" property should be set
- If no customlib is used then either "valuepath" or "averagepaths" property should be set
"results" object members:
| name | type | description | example |
|---|---|---|---|
| "valuepath" | optional, string | json pointer to value | "valuepath":"/BTC/price" |
| "averagepaths" | optional, array of strings | list of json pointers to values. If present, the average value is calculated. In position of array index in the json pointer it could be '*' which means to use all the array elements | "averagepaths":["/prices/*/bid", /prices/*/ask"] |
| "customdata" | optional, string | arbitrary data passed to custom lib function | "customlib":"/price" |
Notes:
- "results" must be the object if "substitutes" parameter is used
- no "symbol" parameter is used in the "results" object as the symbol name is constructed from the "substitutes" item and optional "quote" property
- If no customlib is used then either "valuepath" or "averagepaths" property should be set in each array item
Custom result json parser is a shared library that should be placed with its source code with building instructions in src/cc/priceslibs directory. The custom library should implement the single C-language function with declaration specified in pricesfeeds.h:
extern "C" {
int pricesJsonParser(const char *sjson /*in*/, const char *symbol /*in*/, const char *customdata, uint32_t multiplier /*in*/, uint32_t *value /*out*/);
}On each call the function should retrieve a single value and place it in the '*value' variable.
The function receives the following parameters: string with the json returned by web service, the symbol to retrieve, custom data from the configuration (the custom data might contain a hint how to find the value in the json) and multiplier used to convert the price value to integer.
The function should return 1 if success and 0 if it could not extract the value.
See the custom parser lib example PricesResultParserSample.cpp in src/cc/priceslibs directory in komodod sources.
-ac_cbopret parameter should be set into 1 in normal cases when a new chain started with the Prices module. To provide compatibilty with the old chain where the previous Prices module versions are working, which did not support -ac_feed parameter, -ac_cbopret should be set either to 7 or 15 depending on whether -ac_prices only or both -ac_prices and -ac_stock are set on this chain. Also -ac_feed should have exactly the same symbols in the feed configuration as the old chain currently has.
It is also possible to create feeds configuration in the source code. The configuration object is feedconfig in src/cc/pricesfeeds.cpp source. Use an existing config item with name="basic" as an example.