Load and Compare Datasets

[lettergram]

Howdy!

I'm a maintainer on the DataProfiler library, we built the DataProfiler as an improved replacement for pandas-profiling. Some additional features:

• Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL data = Data("your_filepath_or_url.csv")
• Profile data in a multi-threaded way with robust PII detection profile = Profiler(data)
• Merge profiles: profile3 = profile1 + profile2
• Take differences between profiles profile_diff = profile1.diff(profile2)
• Generate reports: readable_report = profile.report(report_options={"output_format": "compact"})

Would love to connect and / or see how we could be useful to this project!

import json
from dataprofiler import Data, Profiler

data = Data("your_filepath_or_url.csv") # Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL

print(data.data.head(5)) # Access data directly via a compatible Pandas DataFrame

profile = Profiler(data) # Calculate Statistics, Entity Recognition, etc

readable_report = profile.report(report_options={"output_format": "compact"})

print(json.dumps(readable_report, indent=4))

[shir22]

Hi!

Happy to get to know your package, and indeed lets connect.
About potential collaboration - actually one of the Checks we thought about adding is a DatasetInfo Check #235.
A brief overview of our package's structure - all of our logic is implemented in "Checks" that can be run independently or as part of a Suite that contains several checks. They receive a Dataset Object which contains a DataFrame.
Perhaps it can be implemented using your package, would you like to check it out and consider contributing?
Also, we're still putting some things in place (guidelines, documentation etc.), and of course we'd love to be in touch and to take an active part in helping to implement it.

[lettergram]

Hello @shir22 the task itself seems pretty straight forward, but I'm not familiar with the current libraries the architecture so that could be a bit of a challenge.

Perhaps it can be implemented using your package, would you like to check it out and consider contributing?

I'll definitely check it out and get back to you! I understand you're still putting some contribution guidelines in place, so perhaps we will blaze a bit of a trail here haha. Happy to help.

[shir22]

Hi again @lettergram,

That sounds great. Yes indeed putting those things in place, and we'd be happy to focus on things in that area that will help you. By the way, we just uploaded the first (still partial) version of our documentation, I believe that the How-to-Guides and the API Reference can be of use. Anyway we'll wait to hear from you, and also @matanper will be happy to assist in any architecture, package logic and structure, conventions-related issues, or in anything else.
Also, you're invited to join our Slack Community, and we can continue our get-to-know there 🙂.