Feature/add predicate hints and json predicate hints parameters #295
Conversation
Signed-off-by: Shingo OKAWA <shingo.okawa.g.h.c@gmail.com>
Signed-off-by: Shingo OKAWA <shingo.okawa.g.h.c@gmail.com>
|
I will fix these errors. |
Signed-off-by: Shingo OKAWA <shingo.okawa.g.h.c@gmail.com>
Signed-off-by: Shingo OKAWA <shingo.okawa.g.h.c@gmail.com>
| pd.testing.assert_frame_equal(pdf, expected) | ||
|
|
||
|
|
||
| @pytest.mark.skipif(not ENABLE_INTEGRATION, reason=SKIP_MESSAGE) | ||
| @pytest.mark.parametrize( | ||
| "fragments,version,timestamp,error", | ||
| "fragments,jsonPredicateHints,predicateHints,version,timestamp,error", |
There was a problem hiding this comment.
Is there a unit test that specifies these predicate hints and verifies that they are used correctly?
There was a problem hiding this comment.
@chakankardb
Thank you for the comment, and yes, I have to add an unit test to check if these new parameters are used correctly. I will ping you again when I add the unit test.
|
Hi @chakankardb, Would you mind if I requested you to share the table schemas (e.g., the available columns in the test data) for implementing the unit test? From what I understand, the configurations for the test server can be found at: https://github.com/delta-io/delta-sharing/tree/main/server/src/test/scala/io/delta/sharing/server However, I couldn't find any explicit definition for the relating Delta tables. |
|
Hello everyone - is there anything else blocking this feature? What needs to be done to move forward? |
At the moment
load_as_pandasdoes not support SQL expressions for Filtering and JSON Predicates for Filtering.This PR expands
load_as_pandasto support those parameters.predicateHintsandjsonPredicateHintsparameters toload_as_pandas.Regarding the
load_as_spark, I didn't mention it in the previous discussion as I thought it could be treated as a separate feature request. However, I can work on it too if needed.closes #292