Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add spark table metric repository #502

Open
charlieyou opened this issue Aug 21, 2023 · 4 comments
Open

[FEATURE] Add spark table metric repository #502

charlieyou opened this issue Aug 21, 2023 · 4 comments
Labels
enhancement New feature or request

Comments

@charlieyou
Copy link

Is your feature request related to a problem? Please describe.
Given that the rest of Deequ relies on Spark, it seems incongruous that there is no support for loading metrics from a Spark table. Saving to a JSON works fine for now, but as we scale up, we would like to take advantage of the data catalog/governance that comes along with using Spark tables (specifically with Databricks in our case, but can imagine it being generally useful outside of that).

Describe the solution you'd like
An implementation of MetricsRepository using Spark tables as the data source.

Describe alternatives you've considered
This can be hacked together by dumping a spark table to a JSON file and then reading that with the FS MR, but it's quite inelegant.

Additional context
Happy to take a crack at the implementation myself when I have more capacity in a few days.

@charlieyou charlieyou added the enhancement New feature or request label Aug 21, 2023
@VenkataKarthikP
Copy link
Contributor

VenkataKarthikP commented Oct 22, 2023

I can take a stab at this, just want to check if PR's are accepted for this feature? @rdsharma26

@mentekid
Copy link
Contributor

@VenkataKarthikP we do take Pull Requests, feel free to work on this and thank you in advance

@VenkataKarthikP
Copy link
Contributor

@mentekid @charlieyou i worked on #518 to implement this request, please take a look. thanks in advance.

@VenkataKarthikP
Copy link
Contributor

@mentekid @rdsharma26 can we close this as we have PR merged. Also, can we get a release tag with latest changes. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants