Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000) #1694

Open
montag opened this issue Oct 13, 2022 · 5 comments
Labels
Bug Something isn't working

Comments

@montag
Copy link

montag commented Oct 13, 2022

Describe the bug
Promscale stops writing trace data with an 'Error sending samples to remote storage' error for each sample ingested.

To Reproduce
Attempt to store otel trace data with large SQL or stacktrace tag values.

Expected behavior
Expect traces to be stored and not dropped.

Screenshots
If applicable, add screenshots to help explain your problem.

Configuration (as applicable)

  • Promscale Connector: 0.7.0
  • TimescaleDB: Timescaledb-ha:pg14.5-ts2.8.1
  • Prometheus: 2.39.1
  • Jaeger: 1.38.1
  • OpenTelemetry: collector 0,61.9
    Version
  • Distribution/OS: linux (docker)
  • Promscale: 0.15.0
  • TimescaleDB: Timescaledb-ha:pg14.5-ts2.8.1

Additional context
Traces are collected without error for a short period of time then the error starts and traces are no longer stored.

Example log entries:

version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:31.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:36.182Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:36.611Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:41.218Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:41.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:46.253Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:46.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:51.288Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:51.610Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:55:56.303Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:55:56.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=info ts=2022-10-13T06:55:57.589Z caller=throughput.go:90 msg=“ingestor throughput” metric-metadata/sec=496 level=warn ts=2022-10-13T06:56:01.337Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:01.587Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:06.372Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:06.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.053Z level=warn ts=2022-10-13T06:56:11.403Z caller=write.go:244 msg=“Error sending samples to remote storage” err=“copier: writing series: error setting series ids: error filling labels: ERROR: index row size 3616 exceeds btree version 4 maximum 2704 for index \“label_pkey\” (SQLSTATE 54000)” num_samples=442 level=info ts=2022-10-13T06:56:11.588Z caller=throughput.go:90 msg=“ingestor throughput” samples/sec=442 metrics-max-sent-ts=2022-10-13T05:09:57.05

@ramonguiu
Copy link
Contributor

@montag could you explain how you are generating and sending the trace data to Promscale? I assume you are using the OpenTelemetry collector? What's the configuration you are using? Are you also sending metrics?

From this log message the issue seems to be with metric ingestion and not traces. Maybe you have something that is automatically generating metrics from trace data?

@sumerman
Copy link
Contributor

FTR: this is the same as #1091, which we fixed for tracing tables. The fix should be similar should we decide to go for it.

@montag
Copy link
Author

montag commented Oct 13, 2022

Here's my otel collector config

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp:
    endpoint: promscale:9202
    tls:
      insecure: true

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

And my app's ENV vars

OTEL_EXPORTER_OTLP_ENDPOINT: otel_collector:4317
OTEL_EXPORTER_OTLP_INSECURE: true
OTEL_EXPORTER_OTLP_TRACES_INSECURE: true
OTEL_SERVICE_NAME: ******
OTEL_PYTHON_LOG_CORRELATION: true      

@montag
Copy link
Author

montag commented Oct 13, 2022

I'm not using otel for app metrics but some of the instrumentation libraries might be sending some. I'm using these libs:

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

@montag
Copy link
Author

montag commented Oct 20, 2022

I worked around this issue by removing metrics from my otel collector config but at some point, I will need to be able to use them again.

@ramonguiu ramonguiu added the Bug Something isn't working label Dec 13, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants