Add Metadata filtering on Pgvector #410

humcqc · 2024-03-26T14:15:40Z

Implement Metadata filtering on Pgvector with 3 implementations to store Metadata: JSON, JSONB, COLUMNS
Add indexes on metadata
use Langchain4j metadata filter test
Change tests to @QuarkusTest. QuarkusUnitTest extension does not support ParametrizedTest with custom classes as argument. to check with quarkus team.

Not sure if this should be in quarkus-langchain4j or in langchain4j ?
PR open for discussion.

geoand · 2024-03-26T14:17:05Z

Thanks!

I'll let @jmartisk review this

geoand · 2024-03-26T14:17:24Z

We will definitely need the PR rebased onto main so we can skip the merge commits

humcqc · 2024-03-26T14:19:40Z

yes, will cleanup after review and comments. I know i should not sync with the github site, but too late

…store Metadata: JSON, JSONB, COLUMNS - Add indexes on metadata - use Langchain4j metadata filter test - Change tests to @QuarkusTest. QuarkusUnitTest extension does not support ParametrizedTest with custom classes as argument.

jmartisk · 2024-03-27T07:42:16Z

I'll try to look into it asap, I'm not sure I'll have enough time before leaving until Tuesday.
Maybe @sebastienblanc would like to have a look too, as he implemented the Pgvector store.

jmartisk · 2024-03-27T11:29:13Z

I really do like it, very nice job! But yeah a lot of this looks like it could be moved to upstream langchain4j, which currently doesn't support metadata filtering on pgvector.
@langchain4j what do you think?
It could also benefit from the strategies (json, jsonb, columns) that you introduce for storing metadata

langchain4j · 2024-03-27T11:39:47Z

Yeah, would be awesome to have Filter mapping logic in https://github.com/langchain4j/langchain4j ! And the rest probably as well :)

humcqc · 2024-03-27T11:52:49Z

ok will move it to langchain4j, and adapt this one.

sebastienblanc · 2024-03-28T09:07:34Z

Will be happy to review this PR and also the parts moving to @langchain4j

humcqc · 2024-04-02T10:03:21Z

last commit integrates : langchain4j/langchain4j#851 , this will no build because last langchain pgvector is not yet available. just for @jmartisk review.

- Fix asciidoctor doc generation: interfaces that extends interfaces produce duplicated parameter in doc that make the build failing. - add integration test - Add buildStep to include dependencies in index - Add default no args constructor for cdi - !!! Need to do some tests in native mode.

jmartisk · 2024-04-03T11:42:00Z

Thanks, I'm going to look into it ASAP

jmartisk · 2024-04-03T12:36:43Z

I can't get this to work somehow...
Does this add some new requirements on the application to specify some properties?
On startup I get:

Caused by: org.postgresql.util.PSQLException: Unknown type vector.
	at org.postgresql.jdbc.PgPreparedStatement.setPGobject(PgPreparedStatement.java:537)
	at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1049)
	at io.agroal.pool.wrapper.PreparedStatementWrapper.setObject(PreparedStatementWrapper.java:291)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.addAllInternal(PgVectorEmbeddingStore.java:324)
	... 26 more

So I tried adding

quarkus.datasource.jdbc.additional-jdbc-properties.datatype.vector=com.pgvector.PGvector

in the application's properties, but then I get

2024-04-03 14:26:27,559 WARN  [io.agr.pool] (agroal-11) Datasource '<default>': Unable to load the class com.pgvector.PGvector responsible for the datatype vector
(...)
java.lang.RuntimeException: Sql statement issue: org.postgresql.util.PSQLException: Unable to load the class com.pgvector.PGvector responsible for the datatype vector 
Last query: init
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.initTable(PgVectorEmbeddingStore.java:172)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.<init>(PgVectorEmbeddingStore.java:81)
	at io.quarkiverse.langchain4j.pgvector.PgVectorEmbeddingStore.<init>(PgVectorEmbeddingStore.java:43)
	at io.quarkiverse.langchain4j.pgvector.runtime.PgVectorEmbeddingStoreRecorder.lambda$embeddingStoreFunction$1(PgVectorEmbeddingStoreRecorder.java:26)
	at io.quarkiverse.langchain4j.pgvector.PgVectorEmbeddingStore_NcsXENBovnGlSJZ1aaFjoohrtKM_Synthetic_Bean.createSynthetic(Unknown Source)
	... 36 more

Then I tried adding

quarkus.class-loading.parent-first-artifacts=com.pgvector:pgvector

but I am back at the first error

jmartisk · 2024-04-03T12:42:05Z

To reproduce this problem, for example, take the samples/chatbot application, change the quarkus-langchain4j-redis dependency into quarkus-langchain4j-pgvector and add quarkus.langchain4j.pgvector.dimension=1536 into application.properties and then try mvn quarkus:dev

(I built upstream langchain4j from your commit fcce3f1f26da1f1bba78b51267de6066a260ff1b and quarkus-langchain4j from 2dc8653 before trying this)

humcqc · 2024-04-03T12:44:43Z

interesting, does the rag-with-pgvector integration test work for you ?
I think you face the same issue when i try to use the extension in my projects, there is some issue with pgvector and classloading.
I've put some question in the review where there are possible issues.

sebastienblanc · 2024-04-03T12:45:24Z

@jmartisk Are you using a pg container that has pgvector ? It should mention it in the log if not https://github.com/quarkiverse/quarkus-langchain4j/blob/main/pgvector/runtime/src/main/java/io/quarkiverse/langchain4j/pgvector/PgVectorEmbeddingStore.java#L114

humcqc · 2024-04-03T12:50:10Z

@sebastienblanc should be automatic now with #399

jmartisk · 2024-04-03T12:53:32Z

Right, the integration test is failing too.
The image isn't the problem, it's using pgvector/pgvector:pg16

humcqc · 2024-04-03T12:57:56Z

Curious they are working for me. Could you show me the error ?
And do you have some glue about extension class loading when an extension (quarkus langchain ) have a dependency ( langchain) that have a dependency (pgvector).
I think there is an issue when we want to configure the agroal datasource with pgvector vector.

jmartisk · 2024-04-03T13:14:20Z

When running the integration tests I get (this is the relevant part)

2024-04-03 15:12:42,956 INFO  [tc.pgvector/pgvector:pg16] (build-41) Container pgvector/pgvector:pg16 is starting: f1c5cdc29c85b6a1e002dad4a4a1f1b457439d927cc514868338ce6920bfb0bf
2024-04-03 15:12:46,368 INFO  [tc.pgvector/pgvector:pg16] (build-41) Container pgvector/pgvector:pg16 started in PT3.496847295S
2024-04-03 15:12:46,369 INFO  [tc.pgvector/pgvector:pg16] (build-41) Container is started (JDBC URL: jdbc:postgresql://localhost:46373/quarkus?loggerLevel=OFF)
2024-04-03 15:12:46,369 INFO  [io.qua.dev.pos.dep.PostgresqlDevServicesProcessor] (build-41) Dev Services for PostgreSQL started.
2024-04-03 15:12:46,370 INFO  [io.qua.dat.dep.dev.DevServicesDatasourceProcessor] (build-41) Dev Services for default datasource (postgresql) started - container ID is f1c5cdc29c85
2024-04-03 15:12:51,787 INFO  [ai.djl.uti.Platform] (main) Found matching platform from: jar:file:///home/jmartisk/.m2/repository/ai/djl/huggingface/tokenizers/0.26.0/tokenizers-0.26.0.jar!/native/lib/tokenizers.properties
2024-04-03 15:12:52,054 WARN  [io.agr.pool] (main) Datasource '<default>': JDBC resources leaked: 0 ResultSet(s) and 1 Statement(s)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 14.37 s <<< FAILURE! -- in org.acme.example.openai.RAGWithMetadataFilterTest
[ERROR] org.acme.example.openai.RAGWithMetadataFilterTest -- Time elapsed: 14.37 s <<< ERROR!
java.lang.RuntimeException: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:638)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptBeforeAllMethod(QuarkusTestExtension.java:706)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.quarkus.runner.bootstrap.StartupActionImpl.run(StartupActionImpl.java:285)
	at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:252)
	at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:605)
	at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:655)
	... 1 more
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: Unknown type vector.
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.addAllInternal(PgVectorEmbeddingStore.java:345)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.addInternal(PgVectorEmbeddingStore.java:294)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.add(PgVectorEmbeddingStore.java:210)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.add(PgVectorEmbeddingStore.java:36)
	at io.quarkiverse.langchain4j.pgvector.PgVectorEmbeddingStore_NcsXENBovnGlSJZ1aaFjoohrtKM_Synthetic_ClientProxy.add(Unknown Source)
	at org.acme.example.EmbeddingStoreIngestor.embed(EmbeddingStoreIngestor.java:29)
	at org.acme.example.EmbeddingStoreIngestor.ingest(EmbeddingStoreIngestor.java:22)
	at org.acme.example.EmbeddingStoreIngestor_Observer_ingest_UItmqUFF_K3-3VnNeybAPBBp55c.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	... 8 more
Caused by: org.postgresql.util.PSQLException: Unknown type vector.
	at org.postgresql.jdbc.PgPreparedStatement.setPGobject(PgPreparedStatement.java:537)
	at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1049)
	at io.agroal.pool.wrapper.PreparedStatementWrapper.setObject(PreparedStatementWrapper.java:291)
	at dev.langchain4j.store.embedding.pgvector.PgVectorEmbeddingStore.addAllInternal(PgVectorEmbeddingStore.java:324)
	... 22 more

humcqc · 2024-04-03T13:27:48Z

weird:

2024-04-03 14:53:24,758 INFO  [tc.pgvector/pgvector:pg16] (build-86) Reusing container with ID: b263c49bad38c2cde3f87e32ece37bd12cb50e787a1e49de89433ce85a44a3be and hash: 40c16b4a38334133b9e756b168c97be7428fecb2
2024-04-03 14:53:24,759 INFO  [tc.pgvector/pgvector:pg16] (build-86) Reusing existing container (b263c49bad38c2cde3f87e32ece37bd12cb50e787a1e49de89433ce85a44a3be) and not creating a new one
2024-04-03 14:53:25,080 INFO  [tc.pgvector/pgvector:pg16] (build-86) Container pgvector/pgvector:pg16 started in PT1.305414S
2024-04-03 14:53:25,080 INFO  [tc.pgvector/pgvector:pg16] (build-86) Container is started (JDBC URL: jdbc:postgresql://localhost:57599/quarkus?loggerLevel=OFF)
2024-04-03 14:53:25,080 INFO  [io.qua.dev.pos.dep.PostgresqlDevServicesProcessor] (build-86) Dev Services for PostgreSQL started.
2024-04-03 14:53:25,081 INFO  [io.qua.dat.dep.dev.DevServicesDatasourceProcessor] (build-86) Dev Services for default datasource (postgresql) started - container ID is b263c49bad38
2024-04-03 14:53:33,574 INFO  [ai.djl.uti.Platform] (main) Found matching platform from: jar:file:///Users/humberto/.m2/repository/ai/djl/huggingface/tokenizers/0.26.0/tokenizers-0.26.0.jar!/native/lib/tokenizers.properties
2024-04-03 14:53:34,417 INFO  [io.quarkus] (main) quarkus-langchain4j-integration-test-rag-pgvector 999-SNAPSHOT on JVM (powered by Quarkus 3.8.2) started in 16.997s. Listening on: http://localhost:8081
2024-04-03 14:53:34,425 INFO  [io.quarkus] (main) Profile test activated. 
2024-04-03 14:53:34,425 INFO  [io.quarkus] (main) Installed features: [agroal, awt, cdi, jdbc-postgresql, langchain4j, langchain4j-openai, langchain4j-pgvector, narayana-jta, poi, qute, rest-client-reactive, rest-client-reactive-jackson, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, vertx]
WARNING: A Java agent has been loaded dynamically (/Users/humberto/.m2/repository/net/bytebuddy/byte-buddy-agent/1.14.10/byte-buddy-agent-1.14.10.jar)
WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
WARNING: Dynamic loading of agents will be disallowed by default in a future release
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.51 s -- in org.acme.example.openai.RAGWithMetadataFilterTest
2024-04-03 14:53:36,369 INFO  [io.quarkus] (main) quarkus-langchain4j-integration-test-rag-pgvector stopped in 0.091s
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Perhaps link to the fact it re-use a previous pg16 container where connections have been set correctly in my side.

Let me remove the container.

By the way could you check my review question , I think it's link to the issue.

…roduce bad behavior during first getConnection - add vector management in LangChain4jPgVectorBaseTest#clearstore in case it's the first call to the datasource. - add pgvector artifact classloader setting in runtime pom instead of application - rename PgVectorEmbeddingStore as QuarkusPgVectorEmbeddingStore

humcqc · 2024-04-04T07:32:07Z

I've change the way we handle connection settings. some weird behavior at the first usage of the store in certain condition.
Should be ok.
I still need to find a better way when the db settings and the table is created by external script.

…groalpoollistener - remove other settings. - getConnection return just the connection - Add an integretion test when embeddings table created by flyway.

humcqc · 2024-04-04T19:20:10Z

@jmartisk , @sebastienblanc, should be ok. If you can give a try to validate your external project. I've added 2 integrations tests for rag integration. one with automatic table creation and another with flyway. For the connection settings I use a pool listener to configure pgvector settings. Then getConnection just give datasource connection. I've tested with internal test and external project. And with fresh pg containers. First time I have this behavior.

jmartisk · 2024-04-05T07:32:42Z

Thanks, I will look into it today!

.../src/main/java/io/quarkiverse/langchain4j/pgvector/runtime/PgVectorEmbeddingStoreConfig.java

jmartisk · 2024-04-05T08:51:45Z

Truly outstanding work, and it works for me now!

I have just a few comments.
On this PR, I added a few Javadoc clarification suggestions.

On the upstream PR, I commented that we need to avoid using GSON directly (langchain4j/langchain4j#851 (comment)) - after fixing that, could you please add an exclusion for gson, just like we have it in the OpenAI module (https://github.com/quarkiverse/quarkus-langchain4j/blob/0.10.3/openai/openai-common/runtime/pom.xml#L42) - we have a dependency on jackson which should be picked up instead of GSON.

jmartisk · 2024-04-05T11:31:47Z

Works for me!
I won't give a +1 yet because we need to wait for this to be in a released version of langchain4j, but given the current status of this PR plus the current status of langchain4j/langchain4j#851, I think it's good.
Thanks so much, this is great.

geoand · 2024-04-22T09:09:27Z

What's the status of this?

langchain4j · 2024-04-22T12:19:58Z

@geoand I have to re-review it. How urgent is it?

geoand · 2024-04-22T12:22:50Z

Not terrible urgent from point of view, but it does seem like a very contribution

humcqc · 2024-04-25T11:26:20Z

For this one i will create a new PR when langchain4j will be released with : langchain4j/langchain4j#851

… JSONB and Columns (#851) Implement metadata filtering for PgVectorEmbeddingStore with 3 implementations: COLUMN_PER_KEY, COMBINED_JSON and COMBINED_JSONB Related: quarkiverse/quarkus-langchain4j#410

humcqc · 2024-05-15T08:27:37Z

Waiting for langchain4J 0.31.0

langchain4j · 2024-05-15T09:03:53Z

@humcqc will be released next week

humcqc · 2024-05-23T20:29:59Z

Too many conflicts.
Replaced by #619
This PR will be closed soon

humcqc requested a review from a team as a code owner March 26, 2024 14:15

humcqc force-pushed the main branch from df29626 to d2743e4 Compare March 26, 2024 14:27

humcqc mentioned this pull request Mar 26, 2024

Implement embedding filtering where appropriate #368

Open

6 tasks

Clean up

24e04e8

humcqc mentioned this pull request Mar 28, 2024

Implement Metadata filtering on Pgvector with 3 implementations JSON, JSONB and Columns langchain4j/langchain4j#851

Merged

- Integrate last pgvector

7240e5e

humcqc and others added 4 commits April 2, 2024 14:26

- minor cleanup

0b35c01

Merge branch 'quarkiverse:main' into main

34115d2

- revert bad doc commit

2dc8653

humcqc added 3 commits April 4, 2024 13:06

- Doc Fix + default constructor for CDI

f9dec33

- Configure properly the connections with pgvector settings through a…

5010ed9

…groalpoollistener - remove other settings. - getConnection return just the connection - Add an integretion test when embeddings table created by flyway.

- Remove Logger, seems prohibited in recorder

00b606a

jmartisk reviewed Apr 5, 2024

View reviewed changes

.../src/main/java/io/quarkiverse/langchain4j/pgvector/runtime/PgVectorEmbeddingStoreConfig.java Outdated Show resolved Hide resolved

jmartisk reviewed Apr 5, 2024

View reviewed changes

.../src/main/java/io/quarkiverse/langchain4j/pgvector/runtime/PgVectorEmbeddingStoreConfig.java Outdated Show resolved Hide resolved

jmartisk reviewed Apr 5, 2024

View reviewed changes

.../src/main/java/io/quarkiverse/langchain4j/pgvector/runtime/PgVectorEmbeddingStoreConfig.java Outdated Show resolved Hide resolved

jmartisk reviewed Apr 5, 2024

View reviewed changes

.../src/main/java/io/quarkiverse/langchain4j/pgvector/runtime/PgVectorEmbeddingStoreConfig.java Outdated Show resolved Hide resolved

- @jmartisk review 1

b3c778b

Merge branch 'main' into main

f6beb19

- langchain4j review impact

718ce82

humcqc added 2 commits April 22, 2024 18:10

Merge branch 'main' into main

061ae28

Merge branch 'quarkiverse:main' into main

c18216b

humcqc added 2 commits April 30, 2024 17:19

Merge branch 'main' into main

6ffc41c

Merge branch 'main' into main

71f5d71

humcqc mentioned this pull request May 23, 2024

Replace quarkus PGVectorEmbeddingStore with new Langchain4J PGVectorEmbeddingStore #619

Open

humcqc closed this May 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Metadata filtering on Pgvector #410

Add Metadata filtering on Pgvector #410

humcqc commented Mar 26, 2024 •

edited

geoand commented Mar 26, 2024

geoand commented Mar 26, 2024

humcqc commented Mar 26, 2024

jmartisk commented Mar 27, 2024

jmartisk commented Mar 27, 2024

langchain4j commented Mar 27, 2024

humcqc commented Mar 27, 2024

sebastienblanc commented Mar 28, 2024

humcqc commented Apr 2, 2024 •

edited

jmartisk commented Apr 3, 2024

jmartisk commented Apr 3, 2024

jmartisk commented Apr 3, 2024 •

edited

humcqc commented Apr 3, 2024 •

edited

sebastienblanc commented Apr 3, 2024

humcqc commented Apr 3, 2024

jmartisk commented Apr 3, 2024

humcqc commented Apr 3, 2024

jmartisk commented Apr 3, 2024

humcqc commented Apr 3, 2024 •

edited

humcqc commented Apr 4, 2024

humcqc commented Apr 4, 2024

jmartisk commented Apr 5, 2024

jmartisk commented Apr 5, 2024 •

edited

jmartisk commented Apr 5, 2024

geoand commented Apr 22, 2024

langchain4j commented Apr 22, 2024

geoand commented Apr 22, 2024

humcqc commented Apr 25, 2024 •

edited

humcqc commented May 15, 2024

langchain4j commented May 15, 2024

humcqc commented May 23, 2024

Add Metadata filtering on Pgvector #410

Add Metadata filtering on Pgvector #410

Conversation

humcqc commented Mar 26, 2024 • edited

geoand commented Mar 26, 2024

geoand commented Mar 26, 2024

humcqc commented Mar 26, 2024

jmartisk commented Mar 27, 2024

jmartisk commented Mar 27, 2024

langchain4j commented Mar 27, 2024

humcqc commented Mar 27, 2024

sebastienblanc commented Mar 28, 2024

humcqc commented Apr 2, 2024 • edited

jmartisk commented Apr 3, 2024

jmartisk commented Apr 3, 2024

jmartisk commented Apr 3, 2024 • edited

humcqc commented Apr 3, 2024 • edited

sebastienblanc commented Apr 3, 2024

humcqc commented Apr 3, 2024

jmartisk commented Apr 3, 2024

humcqc commented Apr 3, 2024

jmartisk commented Apr 3, 2024

humcqc commented Apr 3, 2024 • edited

humcqc commented Apr 4, 2024

humcqc commented Apr 4, 2024

jmartisk commented Apr 5, 2024

jmartisk commented Apr 5, 2024 • edited

jmartisk commented Apr 5, 2024

geoand commented Apr 22, 2024

langchain4j commented Apr 22, 2024

geoand commented Apr 22, 2024

humcqc commented Apr 25, 2024 • edited

humcqc commented May 15, 2024

langchain4j commented May 15, 2024

humcqc commented May 23, 2024

humcqc commented Mar 26, 2024 •

edited

humcqc commented Apr 2, 2024 •

edited

jmartisk commented Apr 3, 2024 •

edited

humcqc commented Apr 3, 2024 •

edited

humcqc commented Apr 3, 2024 •

edited

jmartisk commented Apr 5, 2024 •

edited

humcqc commented Apr 25, 2024 •

edited