Too high log rate in docker agent when connection error occurs

[johanvdw]

Component

agent

Describe the bug

We are running a woodpecker-ci agent in docker , which connects to the server on a different host. We suddenly saw a huge spike in error logs (several gigabytes). There was an actual error it seems, because dns was not available for some time, but the issue is the rate at which the errors are logged, is far too high:

{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514550725Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514576053Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514600056Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514609191Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.51465298Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514660152Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514716556Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514788238Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514800782Z"}
{"log":"{\"level\":\"error\",\"error\":\"rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp: lookup woodpecker-server on 10.126.0.1:53: read udp 172.16.0.4:41552-\u003e10.126.0.1:53: read: connection refused\\\"\",\"time\":\"2024-04-24T18:50:11Z\",\"message\":\"grpc error: done(): code: Unavailable\"}\n","stream":"stderr","time":"2024-04-24T18:50:11.514806631Z"}

ie 10 messages in 0.2 ms which would be about 40000 per second, so filling up a disk quickly.

System Info

woodpecker 2.4.1 in docker on linux

Additional context

No response

Validations

• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Checked that the bug isn't fixed in the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]