No performance metrics are recorded. Upstream responds with 502 #2931
Comments
|
Do other features (like error reporting) work? Can you access the sentry UI via the web frontend? |
|
@azaslavsky Yes. Everything else seems to work correctly and I can access the web frontend without issues. We have two backend projects where I've implemented performance metrics. Only of them reports any performance metrics and they're implemented exactly the same. |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
|
Also, looking at issues on the internal-project on Sentry reports no issues. |
|
@azaslavsky You know what. I just noticed that we stopped receiving anything three days ago. Probably when I upgraded Sentry to 24.3.0. |
|
Here is a larger log output: |
|
I'm also seeing what I believe to be some Kafka errors intermittently when I run |
|
It seems like things are stuck in a loop? I'm seeing a lot of recurring errors in the log. I do not know if it's just logspam and expected or if it's part of a larger issue. Attaching a larger logfile: |
|
For "no performance metrics are recorded", this maybe relates to #2876. What I'd suggest is to replace all occurrences of But let's also take a look at your logs:
You should try to reset your kafka consumer group. See #1894 (comment)
Seeing this, you should check if your machine have a good Docker networking configured, probably relates to system's IOps or networking segment. |
|
We got the exact same error on 24.3.0, will try to upgrade to latest version this week to see if that solves our problems. Kafka logs[2024-04-19 08:11:52,835] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2024-04-19 08:12:01,970] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent customer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
[2024-04-19 08:12:02,031] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
[2024-04-19 08:12:14,909] ERROR [Log partition=ingest-replay-recordings-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/ingest-replay-recordings-0/00000000000000026795.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:15,326] ERROR [Log partition=processed-profiles-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/processed-profiles-0/00000000000000004708.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:16,339] ERROR [Log partition=outcomes-billing-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/outcomes-billing-0/00000000000000013656.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:17,548] ERROR [Log partition=profiles-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/profiles-0/00000000000000005287.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:17,903] ERROR [Log partition=ingest-events-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/ingest-events-0/00000000000000642592.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:18,052] ERROR [Log partition=ingest-replay-events-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/ingest-replay-events-0/00000000000000026801.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:18,224] ERROR [Log partition=outcomes-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/outcomes-0/00000000000015162257.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:18,532] ERROR [Log partition=ingest-sessions-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/ingest-sessions-0/00000000000000771737.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:20,499] ERROR [Log partition=snuba-commit-log-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/snuba-commit-log-0/00000000000000349422.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:20,660] ERROR [Log partition=transactions-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/transactions-0/00000000000014122668.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:20,856] ERROR [Log partition=ingest-transactions-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/ingest-transactions-0/00000000000014339943.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:21,097] ERROR [Log partition=profiles-call-tree-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/profiles-call-tree-0/00000000000000004708.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:21,261] ERROR [Log partition=events-0, dir=/var/lib/kafka/data] Could not find offset index file corresponding to log file /var/lib/kafka/data/events-0/00000000000000636874.log, recovering segment and rebuilding index files... (kafka.log.Log)
[2024-04-19 08:12:23,096] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2024-04-19 08:12:24,053] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
[2024-04-19 08:12:30,052] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2024-04-19 08:12:31,451] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
[2024-04-19 08:12:31,489] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
[2024-04-19 08:12:32,284] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
[2024-04-19 08:12:32,344] INFO Stat of the created znode at /brokers/ids/1001 is: 5357,5357,1713514352335,1713514352335,1,0,0,72057599696240641,180,0,5357
(kafka.zk.KafkaZkClient)
[2024-04-19 08:12:32,345] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 5357 (kafka.zk.KafkaZkClient)
[2024-04-19 08:12:33,129] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2024-04-19 08:12:33,284] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
[2024-04-19 08:17:48,733] WARN Client session timed out, have not heard from server in 17174ms for sessionid 0x100000151430001 (org.apache.zookeeper.ClientCnxn)
[2024-04-19 08:18:09,288] WARN Unable to reconnect to ZooKeeper service, session 0x100000151430001 has expired (org.apache.zookeeper.ClientCnxn)
[2024-04-19 08:18:17,652] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
[2024-04-19 08:18:17,824] INFO Stat of the created znode at /brokers/ids/1001 is: 5363,5363,1713514697751,1713514697751,1,0,0,72057599696240642,180,0,5363
(kafka.zk.KafkaZkClient)
|
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
|
I would try @aldy505 's suggestions in the comments above. In particular, we've been having issues with |
|
I totally forgot to follow up on this! Sorry! @aldy505's solution above solved our problems and Sentry runs like a charm :) |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
|
@azaslavsky Tried the solution yesterday and it was working for a couple of hours before the instance stopped capturing issues & performance data. I will try to add more juice to the machine and see if that helps. Worth mentioning is that we've been running Sentry with similar load on this machine for a couple of years and in the more recent versions (like, last 6 months) we've noticed that both CPU and memory usage is much higher. EDIT: Just checked in on our second instance to compare. It is running |
@karatekaneen But on the newer version instance, have you replace all occurrance of |
|
@aldy505 Yes I have, I also reset the kafka stuff from this comment. Think I have some time next week to do further experiments. I'll try with adding more memory and updating to latest as well as rolling back so the versions are identical and let you know what I figure out |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
|
@aldy505 's solution works like a charm, CPU was maxing out on the snuba rust-consumers, swapping it to non-rust reduced it drastically: |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
|
It worked by adding more juice to the VM and switching to the non-Rust consumers for us as well. However, we also needed to add |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Self-Hosted Version
24.3.0
CPU Architecture
x86_64
Docker Version
24.0.6
Docker Compose Version
2.21.0
Steps to Reproduce
Expected Result
Performance metrics should show up for the configured project.
Actual Result
No performance metrics are showing up for the affected projects. Some other projects we have are successfully reporting performance metrics.
For one of the affected projects I'm seeing this in the logs:
Event ID
No response
The text was updated successfully, but these errors were encountered: