When I attempt to insert new data into a table indexed with pg_search, I encounter failure. #1047
Comments
rebasedming
added
bug
|
There are two separate errors here that suggest that something is going wrong on the file system.
I haven't seen this before. Could you try again on our latest release? ( |
|
Hi @neilyio , I'll update the version and give it a try. I'll let you know if there's any progress. Thank you. |
|
Going to close this in the meantime. Please feel free to re-open the issue if you still face problems on |
|
Hi @neilyio @philippemnoel ,
book-test=> CALL paradedb.create_bm25(
index_name => 'index_book_local_embedding_text',
schema_name => 'public',
table_name => 'book_local_embedding',
...
);
book=> CALL paradedb.create_bm25(
index_name => 'index_book_local_embedding_text',
schema_name => 'public',
table_name => 'book_local_embedding',
...
);
# Success
book-test=> INSERT INTO "book_local_embedding" (...) VALUES ...;
# Error: failed to create index reader while retrieving index.
# The error log is in the subject.
book=> INSERT INTO "book_local_embedding" (...) VALUES ...;
|
|
After observing for a while, I've noticed that the error persists. The successful insertion after renaming was just a coincidence. The issue is indeed related to the inserted data. This is the insert SQL. |
Thank you for reporting, sorry we missed this. We'll take a look and get back to you |
|
I'm experiencing the identical problem even after updating to version v0.6.1. |
Can you provide a reproduction? Are you able to share the data you're inserting? |
philippemnoel
changed the title
|
Thanks for response. Description: I'm encountering an error while performing bulk insertions into a ParadeDB table with HNSW and BM25 indexes.The error [215] ERROR: XX000: failed to create index reader while retrieving indexNotably, the error consistently occurs after approximately 5-6 insertions. Steps to Reproduce:
CREATE TABLE IF NOT EXISTS chunks
(
id BIGSERIAL PRIMARY KEY,
content TEXT,
is_deleted BOOLEAN,
index_bert Vector(768),
index_text TEXT
);
CREATE INDEX IF NOT EXISTS semantic_index_bert ON chunks USING hnsw (index_bert vector_cosine_ops);
CALL paradedb.create_bm25(
index_name => 'literal_index_text',
schema_name => 'public',
table_name => 'chunks',
key_field => 'id',
text_fields => '{index_text: {tokenizer: {type: "default"}, fast: true}}'
);
import random
import string
import psycopg
from tqdm import tqdm
def generate_random_word(length=5):
return ''.join(random.choice(string.ascii_lowercase) for _ in range(length))
if __name__ == '__main__':
connection_string = "postgresql://test:test@localhost:5432/test_db"
sql_contexts = []
for i in tqdm(range(100)):
sql_context = "insert into chunks (content, is_deleted) values"
for j in range(1000):
sql_context += f"""('{generate_random_word(100)}', False)"""
if j != 999:
sql_context += ",\n"
sql_context += ";"
sql_contexts.append(sql_context)
with psycopg.connect(connection_string, autocommit=True) as conn:
for sql_context in tqdm(sql_contexts):
cursor = conn.cursor()
with conn.transaction():
cursor.execute(sql_context)Expected Behavior: The Python code should successfully insert 100,000 rows (100 insertions with 1000 rows each) into the chunks table Paradedb version: v0.7.3 |
Bug Description
When I attempt to insert new data into a table indexed with pg_bm25, I encounter failure.
How To Reproduce
logs
The text was updated successfully, but these errors were encountered: