Interactively explore unstructured datasets from your dataframe.
Spotlight helps you to understand unstructured datasets fast. You can quickly create interactive visualizations and leverage data enrichments (e.g. embeddings, prediction, uncertainties) to identify critical clusters in your data.
Spotlight supports most unstructured data types including images, audio, text, videos, time-series and geometric data. You can start from your existing dataframe:
And start Spotlight with just a few lines of code:
from renumics import spotlight
spotlight.show(df, dtype={"image": spotlight.Image, "embedding": spotlight.Embedding})Machine learning and engineering teams use Spotlight to understand and communicate on complex unstructured data problems. Here are some examples on publicly available datasets along with code snippets (👨💻), interactive demos (🕹️) and blog articles (📝):
| Modality | Task | Description | Link |
|---|---|---|---|
| 🖼️ Image | [Classification] | Find Issues in Any Image Classification Dataset | 👨💻 📝 🕹️ |
| Find data issues in the CIFAR-100 image dataset | 🕹️ | ||
| Fine-tuning image classification models from Bing image search | 👨💻📝 | ||
| 🔊 Audio | [Classification] | Find Issues in Any Audio Classification Dataset | 👨💻 📝🕹️ |
| Debug pre-trained gender detection models on the emodb dataset | 📝 🕹️ | ||
| Compare gender detection models on the emodb dataset | 📝 🕹️ | ||
| 📝 Text | [Classification] | Find Issues in Any Text Classification Dataset | 👨💻 📝 |
| 📈🖼️ Mixed | [EDA] | Explore results from the Formula1 Montreal 2023 GP | 🕹️ |
| Explore a crash simulation dataset | 🕹️ |
Get started by installing Spotlight and loading your first dataset.
- Python version 3.8-3.11
Install Spotlight via pip
pip install renumics-spotlightWe recommend installing Spotlight and everything you need to work on your data in a separate virtual environment.
import pandas as pd
from renumics import spotlight
df = pd.read_csv("https://renumics.com/data/mnist/mnist-tiny.csv")
spotlight.show(df, dtype={"image": spotlight.Image})pd.read_csv loads a sample csv file as a pandas DataFrame.
spotlight.show opens up spotlight in the browser with the pandas dataframe ready for you to explore. The dtype argument specifies custom column types for the browser viewer.
Load a Hugging Face audio dataset with embeddings and a pre-defined layout
import datasets
from renumics import spotlight
ds = datasets.load_dataset('renumics/emodb-enriched', split='all')
layout= spotlight.layouts.debug_classification(label='gender', prediction='m1_gender_prediction', embedding='m1_embedding', features=['age', 'emotion'])
spotlight.show(ds, layout=layout)Here, the data types are discovered automatically from the dataset and we use a pre-defined layout for model debugging. Custom layouts can be built programmatically or via the UI.
The
datasets[audio]package can be installed via pip.
We have added crash report and performance collection. We do NOT collect user data other than an anonymized Machine Id obtained by py-machineid, and only log our own actions. We do NOT collect folder names, dataset names, or row data of any kind only aggregate performance statistics like total time of a table_load, crash data, etc. Collecting Spotlight crashes will help us improve stability. To opt out of the crash report collection define an environment variable called SPOTLIGHT_OPT_OUT and set it to true. e.G.export SPOTLIGHT_OPT_OUT=true
- Open an issue on Github
- Have a coffee talk with us
- Join our channel on Discord
- 🤗 Huggingface example spaces and datasets
- 🏀 Playbook for data-centric AI workflows
- 🍰 Sliceguard library for automatic slice detection
We are currently participating in the running Hacktoberfest 2023.
If you would like to contribute to Spotlight, the easiest way is to have a look at our Contribution Docs and the CONTRIBUTING.md.
We are also equally happy about non-code contributions -- whether it's reporting bugs, suggesting features, contributing design ideas, or offering feedback, every non-code contribution is highly valued and helps make our project better for everyone.