[Bug]: Crash on insert or deletion into compressed chunk #6753
Labels
Comments
|
I also believe this is linked to this issue: #6031 |
|
Hi there, Could you send us the schema definitions of the hypertable and also your compression settings? It does look like the DELETE is the problem here. Thanks, |
To compress data we use Kubernetes cronjob, and we run queries below once a week: |
|
This seems like it could be related to #6717, which would make A workaround for this would be to use The fix above is not released yet, but it is very likely to be in the next release. Could you show me the part of the code that involves the |
|
There is too much code to share the whole code. So I extracted only relevant SQL. For each data upload, we run these queries: |
|
@hostops I tried to reproduce it in different ways but have not yet succeeded. Can you get access to the stack trace for the crash or show more in the log about what caused the crash? The log extract you showed above just shows that the delete statement cannot be executed because the server is in recovery mode, not what statement caused the crash. Here is an example of something I did: |
|
Hello! I did not know I missed the important part of the logs: |
|
I also tried the same test in our development environment as you @mkindahl, but I can only get it to fail in production. Here is the relevant code: async fn do_save(db: &PostgresService, signal: &Signal) -> Result<(), DatabaseError> {
let mut client = log_slow(
db.get_conn(&crate::prometheus::CONN_SIGNAL_SAVE),
"signal_repo, pool.get()",
5,
)
.await?;
let transaction = log_slow(
client
.build_transaction()
.isolation_level(Serializable)
.start(),
"signal_repo, transaction.start()",
5,
)
.await?;
log_slow(
clear_period(&transaction, signal),
"signal_repo, clear_period",
5,
)
.await?;
insert(&transaction, signal.id.into(), &signal.points)
.timed(&crate::prometheus::TIME_SIG_INSERT)
.await?;
log_slow(transaction.commit(), "signal_repo, transaction.commit()", 5).await?;
Ok(())
}
async fn insert(
transaction: &Transaction<'_>,
id: i64,
points: &[DataPoint],
) -> Result<(), DatabaseError> {
// open COPY FROM STDIN sink
let sink = transaction
.copy_in("COPY signal (id, time, value) FROM STDIN BINARY")
.await?;
let writer = BinaryCopyInWriter::new(sink, &[Type::INT8, Type::TIMESTAMP, Type::FLOAT8]);
pin_mut!(writer);
// write all points except last
for p in &points[0..points.len() - 1] {
if let Some(value) = p.value {
writer
.as_mut()
.write(&[&id, &wrap_time(p.timestamp), &value])
.await?;
} else {
writer
.as_mut()
.write(&[&id, &wrap_time(p.timestamp), &None::<&f64>])
.await?;
}
}
writer.finish().await?;
// insert last point with INSERT to utilise ON CONFLICT
// because we clear before inserting, conflict may arise only for last data point
// since last data point is always null, on conflict we want to preserve current value
let last_timestamp = (points.last().unwrap().timestamp as f64) / 1000.0;
let last_insert = format!(
"INSERT INTO signal VALUES ({:}, to_timestamp({:0.3}), NULL) ON CONFLICT DO NOTHING;",
id, last_timestamp
);
transaction.simple_query(&last_insert[..]).await?;
Ok(())
} |
|
I just tested simple delete query in compressed chunk and bug was reproduced. |
Do you have a stack trace for this? |
|
If you mean stack trace on client. I just executed this command using If |
I meant a stack trace using |
|
@mkindahl I believe this is the stack trace you are looking for. |
|
To save you some time: timescaledb/tsl/src/compression/compression.c Line 2553 in 402c1c3 |
|
@hostops Yes, that is what I was looking for. Thanks! |
mkindahl
added
segfault
|
Could you check what opexpr->args is in the coredump and maybe all the other fields of opexpr too to see what might be wrong. |
|
Hello! But I may have used version mismatched to docker image used. I may have forgotten to run update command from 2.14.0 to 2.14.2. But I am not sure and had no time to test and confirm if the issue still persists. So please wait until I have the time to test if this issue is still there on latest version. |
|
Hello, thx for the update i'll label the issue as waiting for your input. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What type of bug is this?
Crash
What subsystems and features are affected?
Compression, Data ingestion
What happened?
When we try to insert data into compressed chunk, database crashes.
TimescaleDB version affected
2.14.2
PostgreSQL version used
15.6
What operating system did you use?
Ubuntu 20.04.2 x64
What installation method did you use?
Docker, Other
What platform did you run on?
Amazon Web Services (AWS)
Relevant log output and stack trace
How can we reproduce the bug?
The text was updated successfully, but these errors were encountered: