Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simplify adding metadata to vectordb #8953

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

dusvyat
Copy link
Contributor

@dusvyat dusvyat commented Mar 19, 2024

Description

simplify insert with vectordb, extra columns which aren't in schema will be assumed to be part of metadata

e.g.

insert into my_vectordb(
select 
  id, 
  body as content,  
  /* all other columns go to metadata */ 
    mail_id, to_field, from_field, subject, subject, datetime
from email_datasource.emails)

Fixes #issue_number

Type of change

(Please delete options that are not relevant)

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ⚡ New feature (non-breaking change which adds functionality)
  • 📢 Breaking change (fix or feature that would cause existing functionality not to work as expected)
  • 📄 This change requires a documentation update

Verification Process

To ensure the changes are working as expected:

  • Test Location: Specify the URL or path for testing.
  • Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

  • I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

  • My code follows the style guidelines(PEP 8) of MindsDB.
  • I have appropriately commented on my code, especially in complex areas.
  • Necessary documentation updates are either made or tracked in issues.
  • Relevant unit and integration tests are updated or added.

The tests for the chromadb_handler have been updated with additional SQL queries to validate insertion of specific records from a third dataframe. In the vectordatabase_handler, if provided columns do not match the predefined schema, extra columns are now handled and stored in a metadata field. This ensures all relevant information is captured.
@dusvyat dusvyat requested a review from ea-rus March 19, 2024 13:52
@dusvyat dusvyat added the documentation Documentation needs to be added or changed label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Documentation needs to be added or changed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant