Skip to content
This repository has been archived by the owner on Feb 18, 2021. It is now read-only.

Error from send: Message too long #71

Open
keyboardfann opened this issue Jun 20, 2017 · 2 comments
Open

Error from send: Message too long #71

keyboardfann opened this issue Jun 20, 2017 · 2 comments

Comments

@keyboardfann
Copy link

Hello All,
I start statsrelay instances and use statsd-tg to do performance testing. After push metrics minutes, statsrelay show error "Error from send: Message too long". Is it because of the queue full and drop message?

Test script

while true;do statsd-tg -d 10.62.4.240 -D 12000 -T 1 -s 0 -c 1000000 -t 0 -g 0 & sleep 0.5s;pkill statsd-tg;sleep 3s;done

statsrelay1.conf

cat statsrelay1.conf
statsd:
  bind: 10.62.4.158:12001
  tcp_cork: false
  validate: true
  shard_map:
    0: 10.62.4.159:12100:udp

ERROR

ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
@JeremyGrosser
Copy link
Contributor

This happens when you try to send metrics to a UDP backend that is larger than your network interface's MTU (probably somewhere around 1500 bytes). You can either use shorter metric keys or switch to TCP for the backend to resolve this problem.

@keyboardfann
Copy link
Author

Dear @JeremyGrosser ,
Because telegraf statsd plugin only supports UDP protocol at this moment, so I need to use UDP protocol.
setting max-send-queue: 512MB parameter seems work, please correct me if I am wrong.

theatrus referenced this issue in lyft/statsrelay Oct 29, 2017
Previously, the termination hooks would tear down resources (including
destroying server collections) while still running from within the
ev_loop. ev_break is only guaranteed to exit the active loop after all
pending signals in that invocation have been processed. This led to a
race and a (annoying but mostly harmless) crash.

The loop exit code after ev_run() already does all the teardown, so we
are re-using that teardown logic and only stop accepting new connections
from the signal handler (not that they would be processed most likely).
theatrus referenced this issue in lyft/statsrelay May 1, 2020
Previously, the termination hooks would tear down resources (including
destroying server collections) while still running from within the
ev_loop. ev_break is only guaranteed to exit the active loop after all
pending signals in that invocation have been processed. This led to a
race and a (annoying but mostly harmless) crash.

The loop exit code after ev_run() already does all the teardown, so we
are re-using that teardown logic and only stop accepting new connections
from the signal handler (not that they would be processed most likely).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants