[WIP][Do Not Merge] Fix #685 - DuckDB - E2E Integration #758
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* stop loop if subgraph starts returning same data again * write test for fixed use case flow and order payouts response * order subgraph data
* issue-975 - bronze pdr slots query * updated sql query for more consistent data * extend bronze slots test to verify table lenght * use different table names for join tables * added back query filtering --------- Co-authored-by: Norbert <katunanorbert@gmail.com>
* Fix #889: [Pdr bot] Add approach 3: like approach 2, but 1ss (larger-smaller) (#891) Fix #889 * Fix #892: [Bug] UnixTimeS & UnixTimeMS.now() ignore time zone (PR #893) * Fix #892 * Bump pandas from 2.2.1 to 2.2.2 (#888) Bumps [pandas](https://github.com/pandas-dev/pandas) from 2.2.1 to 2.2.2. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Commits](pandas-dev/pandas@v2.2.1...v2.2.2) --- updated-dependencies: - dependency-name: pandas dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump web3 from 6.16.0 to 6.17.1 (#897) Bumps [web3](https://github.com/ethereum/web3.py) from 6.16.0 to 6.17.1. - [Changelog](https://github.com/ethereum/web3.py/blob/v6.17.1/docs/releases.rst) - [Commits](ethereum/web3.py@v6.16.0...v6.17.1) --- updated-dependencies: - dependency-name: web3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ccxt from 4.2.87 to 4.2.98 (#898) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.2.87 to 4.2.98. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.2.87...4.2.98) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump scikit-learn from 1.4.1.post1 to 1.4.2 (#885) Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.4.1.post1 to 1.4.2. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](scikit-learn/scikit-learn@1.4.1.post1...1.4.2) --- updated-dependencies: - dependency-name: scikit-learn dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix #899 [Bug] ImportError: cannot import name 'ContractName' from 'eth_typing' * undo last commit, it should go through a PR * Fix #899 [Bug] ImportError: cannot import name 'ContractName' from 'eth_typing' (PR #900) Fix #899 * Towards #879: Refactors in 'exchange/', and timezone bug fix (PR #896) #879 is [Bug] dydx api ignores 'fromISO' time-date spec Towards that, this PR makes it easier to work with dydx separately from the rest. - rename `safe_fetch_ohlcv*()` -> `fetch_ohlcv*()`. Why: less verbose, same effect - move `fetch_ohlcv_ccxt()` into its own module of the same name - move `fetch_ohlcv_dydx()` "" - timezone-related bug fix in `fetch_ohlcv_dydx()` - when it got the timestamp from the API call, it didn't set it as UTC. - so, I first created a new routine `time_types.py::from_iso_timestr(iso_timestr: str)`, plus unit test - where the new routine tacks on timezone info - then used that new routine in `fetch_ohlcv_dydx()` * Fix #901: [Warning] datetime.utcfromtimestamp() is deprecated, scheduled for removal (PR #902) Fix #901 * #701 - Predictoor Agent with PredSubmitterManager (#877) * #701 - Update PredictoorSS to include prediction feeds and the data sources for training these feeds (#828) * Trimming apart old implementation * Make predict_feed an array * Rename to predict_feeds * MultiFeedMixin * Update key to feeds * Update structure * Update verify_feed_dependencies for multiple feeds * Create PredictFeedMixin * PredictFeedMixin test * parse_feed_obj func and fixes * test_parse_feed_obj and multi predict feeds * pred_submitter_mgr property * Convert PredictoorSS to PredictFeedMixin * Formatting * Update predictoor_ss_test_dict * Add prepare_stakes function * Break on cutoff * Update check balances to check contracts * Load pred_submitter_mgr_addr * Update take_step and remove unused funcs * Add PredictFeed and PredictFeeds classes * add feeds property * Update predictoor_ss_test_dict * Update predictoor ss tests * convert to str * Add feeds_str and fixes * Update keys * Black * Fixes * Black * Enforce typing and fixes * Type fixes * Type fixes * Fix & update ppss tests * Formatting * Feeds structure * Remove removed functions tests * enable assert attributes * Use list * Set submitter addr * Update sim engine * Update sim engine test * Update feed list * Update AI model factory tests * Use PredictFeeds * Return PredictFeeds * Use argfeed * BLack * PredictFeed update to have 1 -> N * Fix test * feeds_list func for comparison * Single predict feed * Enforce types * Usel ist * Set self.feeds and remove extra properties * Call to_list * Fix tests for single predict feed * Black * Update tests for single pred feed and structure * Add feeds property back * Update tests to support new structure and single pred feed * Fix check network tests * Fix mock args * Black * Replace all, arg fix * Fix args * Fix tests * Update dict * Update dict * Black * create_xy on given feeds * Add filter_feeds_from_candidates to mixin * Fallback to ss.feeds * Formatting * Linter * Linter * Revert "#701 - Update PredictoorSS to include prediction feeds and the data s…" (#878) This reverts commit ac6899b. * #701 - Multi Feed Predictions and PredSubmitter in Predictoor Agent (#855) * Trimming apart old implementation * Make predict_feed an array * Rename to predict_feeds * MultiFeedMixin * Update key to feeds * Update structure * Update verify_feed_dependencies for multiple feeds * Create PredictFeedMixin * PredictFeedMixin test * parse_feed_obj func and fixes * test_parse_feed_obj and multi predict feeds * pred_submitter_mgr property * Convert PredictoorSS to PredictFeedMixin * Formatting * Update predictoor_ss_test_dict * Add prepare_stakes function * Break on cutoff * Update check balances to check contracts * Load pred_submitter_mgr_addr * Update take_step and remove unused funcs * Add PredictFeed and PredictFeeds classes * add feeds property * Update predictoor_ss_test_dict * Update predictoor ss tests * convert to str * Add feeds_str and fixes * Update keys * Black * Fixes * Black * Enforce typing and fixes * Type fixes * Type fixes * Fix & update ppss tests * Formatting * Feeds structure * Remove removed functions tests * enable assert attributes * Use list * Set submitter addr * Update sim engine * Update sim engine test * Update feed list * Update AI model factory tests * Use PredictFeeds * Return PredictFeeds * Use argfeed * BLack * PredictFeed update to have 1 -> N * Fix test * feeds_list func for comparison * Single predict feed * Enforce types * Usel ist * Set self.feeds and remove extra properties * Call to_list * Fix tests for single predict feed * Black * Update tests for single pred feed and structure * Add feeds property back * Update tests to support new structure and single pred feed * Fix check network tests * Fix mock args * Black * Replace all, arg fix * Fix args * Fix tests * Update dict * Update dict * Black * create_xy on given feeds * Add filter_feeds_from_candidates to mixin * Fallback to ss.feeds * Formatting * Linter * Linter * Add missing import * Add get_predict_feed func * Add PredictionSlotsData class * Multi slot support * Fixes, ready to predict * Argpair attributes * Payout function and todos * Todo sim engine * Add minimum_timeframe_seconds func * Add get_min_epoch_s_left to predictoor agent * Fix balance check test * Fix empty init test * Add mock functions * Fix test_predictoor_agent_calc_stakes2_1feed test * Move pred_submitter_mgr to ganache conftest * Pass in pred_submitter_mgr * Pass in pred_submitter_mgr between mocks * pred_submitter_mgr test dict * Fix test_predictoor_agent_calc_stakes2_2feeds test * Make optional * Fix tests * Chain id attr * Comment out sanity checks * Update wait_for_transaction_receipt mock * Black formatting * Add pair_str * Update sim engine for single predict feed * Update sim engine tests * Update sim engine to include feed in MultisimEngine initialization * Formatting * Update sim engine to include feed in MultisimEngine initialization * Formatting * Resolve linter issues * Implementing unique epoch * Add min_epoch_seconds func * Remove removed vars * Black Formatting * Linter * Final touchups * Rename to get * Todo * Fix system test predictoor * Formatting * fix mypy errors * Transfer automatically * Compile * Approve tokens * Fix token transfer issue in PredSubmitterMgr.sol * Compiled contract * Update test * Black * Add info * Fix .exchange * Fix order * make it more explicit * Update predictoor docs * Update cli help * Update * Update docs * Remove unused variable * Use logger.info * Test PredictFeed and parse_feed_obj * Add tests for PredictFeeds * Formatting * Remove cutoff seconds * Remove cut_off * Update mock func name * Formatting * Resolve linter issues and bugs * Update todo comment * Linter * Remove types * Formatting * Improve docs * Quotes * Remove 1s * Add log message * Fix typo * Fix assert * Fix typo * Refactor aimodel_data_factory.py to handle feeds and x_dim_len dynamically * Fix #907: Opportunity: solve #901 depr'cn warning, while working in older py (PR #903) Fix #907 * Fix #912: [PPSS, Aimodel] predictoor_ss.aimodel_ss.input_feeds is … (#913) Fix #912: [PPSS, Aimodel] predictoor_ss.aimodel_ss.input_feeds is redundant with new predictoor_ss.feeds.train_on Includes refactoring to what is now PredictTrainFeedset etc. * Fix #917: multisim sweeping in ppss.yaml (PR #920) Co-authored with @calina-c * Bump web3 from 6.17.1 to 6.17.2 (#925) Bumps [web3](https://github.com/ethereum/web3.py) from 6.17.1 to 6.17.2. - [Changelog](https://github.com/ethereum/web3.py/blob/v6.17.2/docs/releases.rst) - [Commits](ethereum/web3.py@v6.17.1...v6.17.2) --- updated-dependencies: - dependency-name: web3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ccxt from 4.2.98 to 4.3.4 (#924) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.2.98 to 4.3.4. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.2.98...4.3.4) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump polars from 0.20.17 to 0.20.22 (#923) Bumps [polars](https://github.com/pola-rs/polars) from 0.20.17 to 0.20.22. - [Release notes](https://github.com/pola-rs/polars/releases) - [Commits](pola-rs/polars@py-0.20.17...py-0.20.22) --- updated-dependencies: - dependency-name: polars dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump streamlit from 1.32.2 to 1.33.0 (#922) Bumps [streamlit](https://github.com/streamlit/streamlit) from 1.32.2 to 1.33.0. - [Release notes](https://github.com/streamlit/streamlit/releases) - [Commits](streamlit/streamlit@1.32.2...1.33.0) --- updated-dependencies: - dependency-name: streamlit dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump plotly from 5.20.0 to 5.21.0 (#921) Bumps [plotly](https://github.com/plotly/plotly.py) from 5.20.0 to 5.21.0. - [Release notes](https://github.com/plotly/plotly.py/releases) - [Changelog](https://github.com/plotly/plotly.py/blob/master/CHANGELOG.md) - [Commits](plotly/plotly.py@v5.20.0...v5.21.0) --- updated-dependencies: - dependency-name: plotly dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * For #927, in predictoor.md: streamline how t plot * For #927, in trader.md, Streamline how to plot in simulation * Fix #929 [Sim] predictoor.md & trader.md say "Warning: simulation is temporarily disabled", but sim works * Undo change to ppss.yaml, that was accidentally put into previous commit. * Update vps.md Add nohup * Update vps.md: update nohup instructions * #918 - Remove ganache dependency from predictoor tests (#934) * Use a mock for pred submitter manager * Remove ganache dependency from predictoor tests * Formatting * #930 - Remove CLI approach arg from predictoor (#931) * Set predictoor parser to _ArgParser_PPSS_NETWORK * Remove approach * Set approach in system test * Remove approach from command * Remove approach text check * Fix #933 [Bug][Pdr Submitter Mgr] Predictions are submitted before <= 60s (#935) * Fix epoch_s_thr * Revert * Smart prediction submisison * Fix epoch_s_thr and add epoch cache in predictoor_agent.py * type annotation for "epoch_cache" * Logging and clean code * Fix system tests * Add a test for test_calc_stakes_across_feeds * Mock calc_stakes_2ss_model * Improve logging * Divide into lines * Approve only cand feeds * Improve logs * Divide lines * Fix type * Formatting * Split lines * Fix typo * Convert info to debug This should be at debug level and used to see what value are being submitted to the smart contract * Update predictoor.md (#926) Co-authored-by: Idiom <69865342+idiom-bytes@users.noreply.github.com> * Bump ccxt from 4.3.4 to 4.3.11 (#947) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.3.4 to 4.3.11. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.3.4...4.3.11) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump statsmodels from 0.14.1 to 0.14.2 (#946) Bumps [statsmodels](https://github.com/statsmodels/statsmodels) from 0.14.1 to 0.14.2. - [Release notes](https://github.com/statsmodels/statsmodels/releases) - [Changelog](https://github.com/statsmodels/statsmodels/blob/main/CHANGES.md) - [Commits](statsmodels/statsmodels@v0.14.1...v0.14.2) --- updated-dependencies: - dependency-name: statsmodels dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump eth-keys from 0.5.0 to 0.5.1 (#945) Bumps [eth-keys](https://github.com/ethereum/eth-keys) from 0.5.0 to 0.5.1. - [Changelog](https://github.com/ethereum/eth-keys/blob/main/CHANGELOG.rst) - [Commits](ethereum/eth-keys@v0.5.0...v0.5.1) --- updated-dependencies: - dependency-name: eth-keys dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump freezegun from 1.4.0 to 1.5.0 (#944) Bumps [freezegun](https://github.com/spulec/freezegun) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/spulec/freezegun/releases) - [Changelog](https://github.com/spulec/freezegun/blob/master/CHANGELOG) - [Commits](spulec/freezegun@1.4.0...1.5.0) --- updated-dependencies: - dependency-name: freezegun dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump polars from 0.20.22 to 0.20.23 (#943) Bumps [polars](https://github.com/pola-rs/polars) from 0.20.22 to 0.20.23. - [Release notes](https://github.com/pola-rs/polars/releases) - [Commits](pola-rs/polars@py-0.20.22...py-0.20.23) --- updated-dependencies: - dependency-name: polars dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix #879 - Include toISO in dydx api requests `fetch_ohlcv_dydx` (#951) * Add TIMEFRAME_TO_DYDX_RESOLUTION_SECONDS into constants * Update fetch_ohlcv_dydx.py to include _time_delta_from_timeframe function and toISO arg * Update test_fetch_ohlcv_dydx.py to fix fromISO issue and update tests * Add test cases for _time_delta_from_timeframe function * Fix #942 - Move pred_submitter_mgr under bot_only (#950) * Move pred_submitter_mgr under bot_only * Formatting * Convert streamlit dashboard to Dash (#938) * Convert streamlit dashboard to Dash. Main cases. * Scaffold for lineplot response with var imps selection. * Allow running with custom multi_id and ports. * Adds slider for state selection. * Reorganise files to separate layout, callbacks and utils. * Adds argparse and readme updates. * Linter fixes. * Draft add 1-var interactivity. * Adds deselecting. * style update to make plots take up the entire screen width * update justify content attribute * add width to graph components * removed todo comment --------- Co-authored-by: trentmc <gtrent@gmail.com> Co-authored-by: Norbert <katunanorbert@gmail.com> * Fix #963: [Sim, log] Main summary log line takes up >1 line, even in ultrawide stretched terminal (#964) Fix #963 * Fix #890: [Aimodel] Add Jaime's techniques: balance class weights in logistic regression, calibrate with isotonic (#959) Fix #890 * Fixes to sim plotting (#962) * Adds waiting around unpickling error * Adjust colours in 3-line plots. * Adjust labels for slider. * Fix #965: add log loss metric to logs (#966) * Fix #965 * Fix #970 [CLI, sim plots] READMEs should say to open the url in the browser * Fix #977: [CLI] 'pdr view accounts' is too verbose (#978) Fix #977 * #971 - Organize payout slots and contract addresses based on shared slots (#972) Bug fix, no breaking changes. Updated payout function to organize payout slots and contract addresses based on shared slots before passing them to the smart contract. * Change sim plots entrypoint, add tests (#979) * Adds new entrypoint for dash. * Add tests for dashboard. * Update md files. * fix multiple plots on the same row are not shrinking --------- Co-authored-by: Norbert <katunanorbert@gmail.com> * Issue 804 - Thread multisim (#919) * Make multisim threaded. * Feature #981 - Integrate dydx exchange and sim (#987) Adds DydxExchange class. Implements DydxExchange into exchange_mgr.py and other tiny compatibility fixes in fetch_ohlcv_dydx.py * Adds sim line logger. (#988) * Adds sim line logger. * Remove extra command line in fund_accounts parser. (#986) * Arrange logs and inits. (#985) * Plot log loss. (#984) * Fixing conflicts * Bump mypy from 1.9.0 to 1.10.0 (#994) Bumps [mypy](https://github.com/python/mypy) from 1.9.0 to 1.10.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump eth-typing from 4.1.0 to 4.2.2 (#995) Bumps [eth-typing](https://github.com/ethereum/eth-typing) from 4.1.0 to 4.2.2. - [Changelog](https://github.com/ethereum/eth-typing/blob/main/docs/release_notes.rst) - [Commits](ethereum/eth-typing@v4.1.0...v4.2.2) --- updated-dependencies: - dependency-name: eth-typing dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump dash from 2.16.1 to 2.17.0 (#996) Bumps [dash](https://github.com/plotly/dash) from 2.16.1 to 2.17.0. - [Release notes](https://github.com/plotly/dash/releases) - [Changelog](https://github.com/plotly/dash/blob/dev/CHANGELOG.md) - [Commits](plotly/dash@v2.16.1...v2.17.0) --- updated-dependencies: - dependency-name: dash dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump plotly from 5.21.0 to 5.22.0 (#998) Bumps [plotly](https://github.com/plotly/plotly.py) from 5.21.0 to 5.22.0. - [Release notes](https://github.com/plotly/plotly.py/releases) - [Changelog](https://github.com/plotly/plotly.py/blob/master/CHANGELOG.md) - [Commits](plotly/plotly.py@v5.21.0...v5.22.0) --- updated-dependencies: - dependency-name: plotly dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix black and pylint. * Fix tests in gql data factory. * Fix the rest of ppss config tests. * Removed stale function tests from test_table, add test for append_to_db. * Fix #990: Add predictoor log line. (#999) * Process dates with natural language processing. (#1004) * Add another assertion that passes locally. * added prints * added more prints * undo check change * Removing priint statements * Attempting to fix chunking issue --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Trent McConaghy <gtrent@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> Co-authored-by: trizin <25263018+trizin@users.noreply.github.com> Co-authored-by: Robin Lehmann <w1kke@users.noreply.github.com> Co-authored-by: Norbert <37236152+KatunaNorbert@users.noreply.github.com> Co-authored-by: Norbert <katunanorbert@gmail.com>
* solving conflicts * Reverted code added, attempting to make build pass w/ latest code * Reverting conflict import
* Fix #889: [Pdr bot] Add approach 3: like approach 2, but 1ss (larger-smaller) (#891) Fix #889 * Fix #892: [Bug] UnixTimeS & UnixTimeMS.now() ignore time zone (PR #893) * Fix #892 * Bump pandas from 2.2.1 to 2.2.2 (#888) Bumps [pandas](https://github.com/pandas-dev/pandas) from 2.2.1 to 2.2.2. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Commits](pandas-dev/pandas@v2.2.1...v2.2.2) --- updated-dependencies: - dependency-name: pandas dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump web3 from 6.16.0 to 6.17.1 (#897) Bumps [web3](https://github.com/ethereum/web3.py) from 6.16.0 to 6.17.1. - [Changelog](https://github.com/ethereum/web3.py/blob/v6.17.1/docs/releases.rst) - [Commits](ethereum/web3.py@v6.16.0...v6.17.1) --- updated-dependencies: - dependency-name: web3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ccxt from 4.2.87 to 4.2.98 (#898) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.2.87 to 4.2.98. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.2.87...4.2.98) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump scikit-learn from 1.4.1.post1 to 1.4.2 (#885) Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.4.1.post1 to 1.4.2. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](scikit-learn/scikit-learn@1.4.1.post1...1.4.2) --- updated-dependencies: - dependency-name: scikit-learn dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix #899 [Bug] ImportError: cannot import name 'ContractName' from 'eth_typing' * undo last commit, it should go through a PR * Fix #899 [Bug] ImportError: cannot import name 'ContractName' from 'eth_typing' (PR #900) Fix #899 * Towards #879: Refactors in 'exchange/', and timezone bug fix (PR #896) #879 is [Bug] dydx api ignores 'fromISO' time-date spec Towards that, this PR makes it easier to work with dydx separately from the rest. - rename `safe_fetch_ohlcv*()` -> `fetch_ohlcv*()`. Why: less verbose, same effect - move `fetch_ohlcv_ccxt()` into its own module of the same name - move `fetch_ohlcv_dydx()` "" - timezone-related bug fix in `fetch_ohlcv_dydx()` - when it got the timestamp from the API call, it didn't set it as UTC. - so, I first created a new routine `time_types.py::from_iso_timestr(iso_timestr: str)`, plus unit test - where the new routine tacks on timezone info - then used that new routine in `fetch_ohlcv_dydx()` * Fix #901: [Warning] datetime.utcfromtimestamp() is deprecated, scheduled for removal (PR #902) Fix #901 * #701 - Predictoor Agent with PredSubmitterManager (#877) * #701 - Update PredictoorSS to include prediction feeds and the data sources for training these feeds (#828) * Trimming apart old implementation * Make predict_feed an array * Rename to predict_feeds * MultiFeedMixin * Update key to feeds * Update structure * Update verify_feed_dependencies for multiple feeds * Create PredictFeedMixin * PredictFeedMixin test * parse_feed_obj func and fixes * test_parse_feed_obj and multi predict feeds * pred_submitter_mgr property * Convert PredictoorSS to PredictFeedMixin * Formatting * Update predictoor_ss_test_dict * Add prepare_stakes function * Break on cutoff * Update check balances to check contracts * Load pred_submitter_mgr_addr * Update take_step and remove unused funcs * Add PredictFeed and PredictFeeds classes * add feeds property * Update predictoor_ss_test_dict * Update predictoor ss tests * convert to str * Add feeds_str and fixes * Update keys * Black * Fixes * Black * Enforce typing and fixes * Type fixes * Type fixes * Fix & update ppss tests * Formatting * Feeds structure * Remove removed functions tests * enable assert attributes * Use list * Set submitter addr * Update sim engine * Update sim engine test * Update feed list * Update AI model factory tests * Use PredictFeeds * Return PredictFeeds * Use argfeed * BLack * PredictFeed update to have 1 -> N * Fix test * feeds_list func for comparison * Single predict feed * Enforce types * Usel ist * Set self.feeds and remove extra properties * Call to_list * Fix tests for single predict feed * Black * Update tests for single pred feed and structure * Add feeds property back * Update tests to support new structure and single pred feed * Fix check network tests * Fix mock args * Black * Replace all, arg fix * Fix args * Fix tests * Update dict * Update dict * Black * create_xy on given feeds * Add filter_feeds_from_candidates to mixin * Fallback to ss.feeds * Formatting * Linter * Linter * Revert "#701 - Update PredictoorSS to include prediction feeds and the data s…" (#878) This reverts commit ac6899b. * #701 - Multi Feed Predictions and PredSubmitter in Predictoor Agent (#855) * Trimming apart old implementation * Make predict_feed an array * Rename to predict_feeds * MultiFeedMixin * Update key to feeds * Update structure * Update verify_feed_dependencies for multiple feeds * Create PredictFeedMixin * PredictFeedMixin test * parse_feed_obj func and fixes * test_parse_feed_obj and multi predict feeds * pred_submitter_mgr property * Convert PredictoorSS to PredictFeedMixin * Formatting * Update predictoor_ss_test_dict * Add prepare_stakes function * Break on cutoff * Update check balances to check contracts * Load pred_submitter_mgr_addr * Update take_step and remove unused funcs * Add PredictFeed and PredictFeeds classes * add feeds property * Update predictoor_ss_test_dict * Update predictoor ss tests * convert to str * Add feeds_str and fixes * Update keys * Black * Fixes * Black * Enforce typing and fixes * Type fixes * Type fixes * Fix & update ppss tests * Formatting * Feeds structure * Remove removed functions tests * enable assert attributes * Use list * Set submitter addr * Update sim engine * Update sim engine test * Update feed list * Update AI model factory tests * Use PredictFeeds * Return PredictFeeds * Use argfeed * BLack * PredictFeed update to have 1 -> N * Fix test * feeds_list func for comparison * Single predict feed * Enforce types * Usel ist * Set self.feeds and remove extra properties * Call to_list * Fix tests for single predict feed * Black * Update tests for single pred feed and structure * Add feeds property back * Update tests to support new structure and single pred feed * Fix check network tests * Fix mock args * Black * Replace all, arg fix * Fix args * Fix tests * Update dict * Update dict * Black * create_xy on given feeds * Add filter_feeds_from_candidates to mixin * Fallback to ss.feeds * Formatting * Linter * Linter * Add missing import * Add get_predict_feed func * Add PredictionSlotsData class * Multi slot support * Fixes, ready to predict * Argpair attributes * Payout function and todos * Todo sim engine * Add minimum_timeframe_seconds func * Add get_min_epoch_s_left to predictoor agent * Fix balance check test * Fix empty init test * Add mock functions * Fix test_predictoor_agent_calc_stakes2_1feed test * Move pred_submitter_mgr to ganache conftest * Pass in pred_submitter_mgr * Pass in pred_submitter_mgr between mocks * pred_submitter_mgr test dict * Fix test_predictoor_agent_calc_stakes2_2feeds test * Make optional * Fix tests * Chain id attr * Comment out sanity checks * Update wait_for_transaction_receipt mock * Black formatting * Add pair_str * Update sim engine for single predict feed * Update sim engine tests * Update sim engine to include feed in MultisimEngine initialization * Formatting * Update sim engine to include feed in MultisimEngine initialization * Formatting * Resolve linter issues * Implementing unique epoch * Add min_epoch_seconds func * Remove removed vars * Black Formatting * Linter * Final touchups * Rename to get * Todo * Fix system test predictoor * Formatting * fix mypy errors * Transfer automatically * Compile * Approve tokens * Fix token transfer issue in PredSubmitterMgr.sol * Compiled contract * Update test * Black * Add info * Fix .exchange * Fix order * make it more explicit * Update predictoor docs * Update cli help * Update * Update docs * Remove unused variable * Use logger.info * Test PredictFeed and parse_feed_obj * Add tests for PredictFeeds * Formatting * Remove cutoff seconds * Remove cut_off * Update mock func name * Formatting * Resolve linter issues and bugs * Update todo comment * Linter * Remove types * Formatting * Improve docs * Quotes * Remove 1s * Add log message * Fix typo * Fix assert * Fix typo * Refactor aimodel_data_factory.py to handle feeds and x_dim_len dynamically * Fix #907: Opportunity: solve #901 depr'cn warning, while working in older py (PR #903) Fix #907 * Fix #912: [PPSS, Aimodel] predictoor_ss.aimodel_ss.input_feeds is … (#913) Fix #912: [PPSS, Aimodel] predictoor_ss.aimodel_ss.input_feeds is redundant with new predictoor_ss.feeds.train_on Includes refactoring to what is now PredictTrainFeedset etc. * Fix #917: multisim sweeping in ppss.yaml (PR #920) Co-authored with @calina-c * Bump web3 from 6.17.1 to 6.17.2 (#925) Bumps [web3](https://github.com/ethereum/web3.py) from 6.17.1 to 6.17.2. - [Changelog](https://github.com/ethereum/web3.py/blob/v6.17.2/docs/releases.rst) - [Commits](ethereum/web3.py@v6.17.1...v6.17.2) --- updated-dependencies: - dependency-name: web3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump ccxt from 4.2.98 to 4.3.4 (#924) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.2.98 to 4.3.4. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.2.98...4.3.4) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump polars from 0.20.17 to 0.20.22 (#923) Bumps [polars](https://github.com/pola-rs/polars) from 0.20.17 to 0.20.22. - [Release notes](https://github.com/pola-rs/polars/releases) - [Commits](pola-rs/polars@py-0.20.17...py-0.20.22) --- updated-dependencies: - dependency-name: polars dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump streamlit from 1.32.2 to 1.33.0 (#922) Bumps [streamlit](https://github.com/streamlit/streamlit) from 1.32.2 to 1.33.0. - [Release notes](https://github.com/streamlit/streamlit/releases) - [Commits](streamlit/streamlit@1.32.2...1.33.0) --- updated-dependencies: - dependency-name: streamlit dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump plotly from 5.20.0 to 5.21.0 (#921) Bumps [plotly](https://github.com/plotly/plotly.py) from 5.20.0 to 5.21.0. - [Release notes](https://github.com/plotly/plotly.py/releases) - [Changelog](https://github.com/plotly/plotly.py/blob/master/CHANGELOG.md) - [Commits](plotly/plotly.py@v5.20.0...v5.21.0) --- updated-dependencies: - dependency-name: plotly dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * For #927, in predictoor.md: streamline how t plot * For #927, in trader.md, Streamline how to plot in simulation * Fix #929 [Sim] predictoor.md & trader.md say "Warning: simulation is temporarily disabled", but sim works * Undo change to ppss.yaml, that was accidentally put into previous commit. * Update vps.md Add nohup * Update vps.md: update nohup instructions * #918 - Remove ganache dependency from predictoor tests (#934) * Use a mock for pred submitter manager * Remove ganache dependency from predictoor tests * Formatting * #930 - Remove CLI approach arg from predictoor (#931) * Set predictoor parser to _ArgParser_PPSS_NETWORK * Remove approach * Set approach in system test * Remove approach from command * Remove approach text check * Fix #933 [Bug][Pdr Submitter Mgr] Predictions are submitted before <= 60s (#935) * Fix epoch_s_thr * Revert * Smart prediction submisison * Fix epoch_s_thr and add epoch cache in predictoor_agent.py * type annotation for "epoch_cache" * Logging and clean code * Fix system tests * Add a test for test_calc_stakes_across_feeds * Mock calc_stakes_2ss_model * Improve logging * Divide into lines * Approve only cand feeds * Improve logs * Divide lines * Fix type * Formatting * Split lines * Fix typo * Convert info to debug This should be at debug level and used to see what value are being submitted to the smart contract * Update predictoor.md (#926) Co-authored-by: Idiom <69865342+idiom-bytes@users.noreply.github.com> * Bump ccxt from 4.3.4 to 4.3.11 (#947) Bumps [ccxt](https://github.com/ccxt/ccxt) from 4.3.4 to 4.3.11. - [Release notes](https://github.com/ccxt/ccxt/releases) - [Changelog](https://github.com/ccxt/ccxt/blob/master/CHANGELOG.md) - [Commits](ccxt/ccxt@4.3.4...4.3.11) --- updated-dependencies: - dependency-name: ccxt dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump statsmodels from 0.14.1 to 0.14.2 (#946) Bumps [statsmodels](https://github.com/statsmodels/statsmodels) from 0.14.1 to 0.14.2. - [Release notes](https://github.com/statsmodels/statsmodels/releases) - [Changelog](https://github.com/statsmodels/statsmodels/blob/main/CHANGES.md) - [Commits](statsmodels/statsmodels@v0.14.1...v0.14.2) --- updated-dependencies: - dependency-name: statsmodels dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump eth-keys from 0.5.0 to 0.5.1 (#945) Bumps [eth-keys](https://github.com/ethereum/eth-keys) from 0.5.0 to 0.5.1. - [Changelog](https://github.com/ethereum/eth-keys/blob/main/CHANGELOG.rst) - [Commits](ethereum/eth-keys@v0.5.0...v0.5.1) --- updated-dependencies: - dependency-name: eth-keys dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump freezegun from 1.4.0 to 1.5.0 (#944) Bumps [freezegun](https://github.com/spulec/freezegun) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/spulec/freezegun/releases) - [Changelog](https://github.com/spulec/freezegun/blob/master/CHANGELOG) - [Commits](spulec/freezegun@1.4.0...1.5.0) --- updated-dependencies: - dependency-name: freezegun dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump polars from 0.20.22 to 0.20.23 (#943) Bumps [polars](https://github.com/pola-rs/polars) from 0.20.22 to 0.20.23. - [Release notes](https://github.com/pola-rs/polars/releases) - [Commits](pola-rs/polars@py-0.20.22...py-0.20.23) --- updated-dependencies: - dependency-name: polars dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix #879 - Include toISO in dydx api requests `fetch_ohlcv_dydx` (#951) * Add TIMEFRAME_TO_DYDX_RESOLUTION_SECONDS into constants * Update fetch_ohlcv_dydx.py to include _time_delta_from_timeframe function and toISO arg * Update test_fetch_ohlcv_dydx.py to fix fromISO issue and update tests * Add test cases for _time_delta_from_timeframe function * Fix #942 - Move pred_submitter_mgr under bot_only (#950) * Move pred_submitter_mgr under bot_only * Formatting * Convert streamlit dashboard to Dash (#938) * Convert streamlit dashboard to Dash. Main cases. * Scaffold for lineplot response with var imps selection. * Allow running with custom multi_id and ports. * Adds slider for state selection. * Reorganise files to separate layout, callbacks and utils. * Adds argparse and readme updates. * Linter fixes. * Draft add 1-var interactivity. * Adds deselecting. * style update to make plots take up the entire screen width * update justify content attribute * add width to graph components * removed todo comment --------- Co-authored-by: trentmc <gtrent@gmail.com> Co-authored-by: Norbert <katunanorbert@gmail.com> * Fix #963: [Sim, log] Main summary log line takes up >1 line, even in ultrawide stretched terminal (#964) Fix #963 * Fix #890: [Aimodel] Add Jaime's techniques: balance class weights in logistic regression, calibrate with isotonic (#959) Fix #890 * Fixes to sim plotting (#962) * Adds waiting around unpickling error * Adjust colours in 3-line plots. * Adjust labels for slider. * Fix #965: add log loss metric to logs (#966) * Fix #965 * Fix #970 [CLI, sim plots] READMEs should say to open the url in the browser * Fix #977: [CLI] 'pdr view accounts' is too verbose (#978) Fix #977 * #971 - Organize payout slots and contract addresses based on shared slots (#972) Bug fix, no breaking changes. Updated payout function to organize payout slots and contract addresses based on shared slots before passing them to the smart contract. * Change sim plots entrypoint, add tests (#979) * Adds new entrypoint for dash. * Add tests for dashboard. * Update md files. * fix multiple plots on the same row are not shrinking --------- Co-authored-by: Norbert <katunanorbert@gmail.com> * Issue 804 - Thread multisim (#919) * Make multisim threaded. * Feature #981 - Integrate dydx exchange and sim (#987) Adds DydxExchange class. Implements DydxExchange into exchange_mgr.py and other tiny compatibility fixes in fetch_ohlcv_dydx.py * Adds sim line logger. (#988) * Adds sim line logger. * Remove extra command line in fund_accounts parser. (#986) * Arrange logs and inits. (#985) * Plot log loss. (#984) * Bump mypy from 1.9.0 to 1.10.0 (#994) Bumps [mypy](https://github.com/python/mypy) from 1.9.0 to 1.10.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump eth-typing from 4.1.0 to 4.2.2 (#995) Bumps [eth-typing](https://github.com/ethereum/eth-typing) from 4.1.0 to 4.2.2. - [Changelog](https://github.com/ethereum/eth-typing/blob/main/docs/release_notes.rst) - [Commits](ethereum/eth-typing@v4.1.0...v4.2.2) --- updated-dependencies: - dependency-name: eth-typing dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump dash from 2.16.1 to 2.17.0 (#996) Bumps [dash](https://github.com/plotly/dash) from 2.16.1 to 2.17.0. - [Release notes](https://github.com/plotly/dash/releases) - [Changelog](https://github.com/plotly/dash/blob/dev/CHANGELOG.md) - [Commits](plotly/dash@v2.16.1...v2.17.0) --- updated-dependencies: - dependency-name: dash dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Bump plotly from 5.21.0 to 5.22.0 (#998) Bumps [plotly](https://github.com/plotly/plotly.py) from 5.21.0 to 5.22.0. - [Release notes](https://github.com/plotly/plotly.py/releases) - [Changelog](https://github.com/plotly/plotly.py/blob/master/CHANGELOG.md) - [Commits](plotly/plotly.py@v5.21.0...v5.22.0) --- updated-dependencies: - dependency-name: plotly dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> * Fix #990: Add predictoor log line. (#999) * Process dates with natural language processing. (#1004) * Towards #1009: Seasonal decomposition: Part 1/2: backend, unit test plots (#1016) Towards #1009 Seasonal decomposition - Add real crypto data: 10000 five-min epochs for Binance BTC/USDT & ETH/USDT. In `./pdr_backend/lake/test/merged_ohlcv_df_BTC-ETH_2024-02-01_to_2024-03-08.csv` - add `aimodel/seasonal.py` which has: - `pdr_seasonal_decompose()` - Wraps statsmodels' seasonal_decompose() with predictoor-specific inputs - `SeasonalPlotdata` - Simple class to manage many inputs going to plot_seasonal." - `plot_seasonal(d: SeasonalPlotdata)` - Plot seasonal decomposition of the feed, via 4 figures - add `aimodel/test/test_seasonal.py` which has: - via the methods added: pull in real BTC data, decompose it, and plot it * Fixing merge --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Trent McConaghy <gtrent@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Călina Cenan <calina@cenan.net> Co-authored-by: trizin <25263018+trizin@users.noreply.github.com> Co-authored-by: Robin Lehmann <w1kke@users.noreply.github.com> Co-authored-by: Norbert <37236152+KatunaNorbert@users.noreply.github.com> Co-authored-by: Norbert <katunanorbert@gmail.com>
… a validation report (#956) * Working towards implementing lake validation, which may extend from LakeInfo * adding basic validations around lake and expected tables * Added a couple of basic validations and some loop logic * Cleaning up cli commands, and hooking them up via secondary commands-only * Getting validations to work, added query to summarize prediction data, and run gappiness check. * Implemented gap validation, such that we can get a feel for gaps in bronze_predictions. * Finalizing logic for validating, so i can merge and continue iterating from there. * Iterating on validation test * Added lake validation test that shows expected SQL format, and verifies the transform is working as expected. * fomatting, cleanup, type enforcing... * Moving commands around, such that we can still expose ohlcv, while taking over the monicker lake * Making lake raw & etl update work. Fixing some lake command extensions such as ohlcv * preparing code for PR/merge * cleaning up formatting, linter, type enforcing * Continuing to fix tests and making them pass * Fixin lint and black * cleaning up logging * Updating logging where applicable * Adjusting set validation * Fixing linter * Making tests pass or skip, GQL queries need to be fixed * Making tests pass * Making tests pass * making tests pass * fixing linter * Move forward with test fixes. * Cleaning up validations * passing pylint * Testing commands, and cleaning update command. * fixing black formatting * Adjusting key names to be pythonic * continue * fixing black.... somewhere * Cleaning lint * getting tests to pass * Fixing tests * fixing tests and cicd * Fixing linter, mypy, black * fix black, formatting, etc... * Removed null record so validations can pass nearly 100% * Fixing tests --------- Co-authored-by: Calina Cenan <calina@cenan.net>
* Add schema cleanup. * Linter fixes. * Move subgraph primitives.
* Cleanup table names and map from table, dataclass or name.
* fix different class used on subgraph and gql validation for slots * Covert PredictSlot to Slot inside subgraph test
* fix prepare temp tables fails if production table has no values * Fixed pdr drop so that it works w/ ppss, like the rest of the commands * helping test pass --------- Co-authored-by: idiom-bytes <idiom.bytes@gmail.com>
* Remove PredictSlot. * Fix linters.
* use table_name, and max() to help crystalize etl.py * Fixing query st_ts checks * passing tests
* Adds CSV datastore simplifications. * Linter fixes. * Remove CSVDS datastore entirely to rely on identifier. * Renaming CSVSDIdentifier to CSVDataStore --------- Co-authored-by: idiom-bytes <idiom.bytes@gmail.com>
* Add parent type for LakeMapper schemas. * Enforce schema check. * Removing TODO --------- Co-authored-by: idiom-bytes <idiom.bytes@gmail.com>
* Adds html dashboard. * Fix linters. * Add min timestamp to mirror changes by Roberto. * Truncate preview to 100 lines. * Test fixes. * Adds validations as part of lake describe html. * Separate rendering in lake info. * Adds validations in LakeInfo. * Integrate Roberto's changes for duplication detection. * Small tweak to logger name. * Updated README, updated help, simplified command, improved logging * Fixing mypy * Fixing test to use HTML --------- Co-authored-by: idiom-bytes <idiom.bytes@gmail.com>
…are generating local time (#1071) * Fixing end_ts to be affixed across the whole gqldf._update loop. * Implementing test and basic logic to show issue w/ timestamp * Fix localisation of dates. * Add new tests. --------- Co-authored-by: Calina Cenan <calina@cenan.net>
* Enforcing UnixTimeMs so we always have a valid timestamp for dropping rows * Removed UnixTimeMs casts
…1083) * Renamed from persistent-data-store to duckdb-data-store * Cleaning up parameter names * renaming from duckdb to simply db, as its one of the most commonly used names for referencing the db connection * Continuing to clean up tests and renaming things to db * Fixing black * fixing cli dcumentation * Fixing comments/documentation of cli too
…e2e ETL (#1077) * scoped down queries, ran e2e, and reviewed output * Fixing drop command * Continuing to trim extra tables away from code, for now * cleaning linter * Removing print statements * Fixing table registry test * Removing comments * Removed casting to UnixTimeMs, leaving it TBD in-the-call * Fixed temp table implementations in ETL. Simplified GQLDF + ETL definition to improve DRY and legibility. * fix etl does not have raw tables property * remove raw table names property access from etl test * Fixing test documentation and comments * Removing half-implemented subscriptions from fixture * Fixing test * fixing black --------- Co-authored-by: Norbert <katunanorbert@gmail.com>
* First draft to remove table registry. * Simplify fetcher functions. * Fix linters and typings.
* Remove gql get tables to keep only constant list. * Remove table object. * Fix linters.
Open
3 tasks
…ss => to fixed values, as expected by lake/etl systems (#1106) * Fixing issues with _natural language time_ parameters in ppss causing problems w/ time drifting in the ETL process. Addressed this at the root interface, such that ST_TS and END_TS are fixed during the process. * Fixing issues with natural language in lake commands
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #685