Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tempo doesn't boot. Something about eth0/en0 which don't exist #3590

Open
pcarrier opened this issue Apr 18, 2024 · 8 comments
Open

tempo doesn't boot. Something about eth0/en0 which don't exist #3590

pcarrier opened this issue Apr 18, 2024 · 8 comments
Labels
memberlist operations type/bug Something isn't working

Comments

@pcarrier
Copy link

pcarrier commented Apr 18, 2024 •
edited

Describe the bug
tempo doesn't start, logs:

Apr 18 22:54:56 x0.xmit.dev systemd[1]: Started Tempo service.
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=info ts=2024-04-18T22:54:56.437992824Z caller=main.go:225 msg="initialising OpenTracing tracer"
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=info ts=2024-04-18T22:54:56.439046423Z caller=main.go:118 msg="Starting Tempo" version="(version=2.4.1, branch=HEAD, revision=b1c4d3fbf)"
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=info ts=2024-04-18T22:54:56.551118075Z caller=cache.go:54 msg="caches available to storage backend" footer=false bloom=false offset_idx=false column_idx=false trace_id_idx=false page=false
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=info ts=2024-04-18T22:54:56.553367992Z caller=server.go:238 msg="server listening on addresses" http=[::]:3200 grpc=[::]:9095
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=warn ts=2024-04-18T22:54:56.647866147Z caller=netutil.go:90 msg="error getting addresses for interface" inf=eth0 err="route ip+net: no such network interface"
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=warn ts=2024-04-18T22:54:56.647949067Z caller=netutil.go:90 msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"
Apr 18 22:54:56 x0.xmit.dev tempo[3406473]: level=error ts=2024-04-18T22:54:56.647991706Z caller=main.go:121 msg="error running Tempo" err="failed to init module services: error initialising module: compactor: failed to create compactor: no useable address found for interfaces [eth0 en0]"
Apr 18 22:54:56 x0.xmit.dev systemd[1]: tempo.service: Main process exited, code=exited, status=1/FAILURE
Apr 18 22:54:56 x0.xmit.dev systemd[1]: tempo.service: Failed with result 'exit-code'.

To Reproduce
Steps to reproduce the behavior:

  1. Start Tempo 2.4.1 with the config below

Expected behavior
tempo starts successfully.

Environment:

  • Infrastructure: bare-metal
  • Deployment tool: manual

Additional Context

# cat /etc/systemd/system/tempo.service
[Unit]
Description=Tempo service
After=network.target

[Service]
Type=simple
User=tempo
ExecStart=/usr/bin/tempo -config.file /etc/tempo/config.yml
# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec = 120
Restart = on-failure
RestartSec = 2

[Install]
WantedBy=multi-user.target
# cat /etc/tempo/config.yml
server:
  http_listen_port: 3200
  grpc_listen_port: 9095

querier:
  frontend_worker:
    frontend_address: xi.xmit.dev:9095

query_frontend:
  search:
    duration_slo: 5s
  trace_by_id:
    duration_slo: 5s

distributor:
  ring: &1
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - 127.0.0.1:2379
        tls_enabled: true
        tls_cert_path: /etc/ssl/etcd/cert.pem
        tls_key_path: /etc/ssl/etcd/key.pem
        tls_ca_path: /etc/ssl/etcd/ca.pem
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317

ingester:
  lifecycler:
    ring: *1
  max_block_duration: 5m

compactor:
  ring: *1
  compaction:
    block_retention: 1h

metrics_generator:
  ring: *1
  storage:
    path: /srv/tempo/generator/wal
    remote_write:
      - url: http://127.0.0.1:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo
      endpoint: 127.0.0.1:9000
      insecure: true
      access_key: CENSORED
      secret_key: CENSORED
    wal:
      path: /srv/tempo/wal
      v2_encoding: snappy
    block:
      v2_encoding: zstd
overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]
# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:00:02:b2:41:98 brd ff:ff:ff:ff:ff:ff
    inet 37.27.16.132/32 metric 1024 scope global dynamic enp1s0
       valid_lft 77793sec preferred_lft 77793sec
    inet6 2a01:4f9:c012:4248::/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:2ff:feb2:4198/64 scope link
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    link/ether 86:00:00:66:8d:a4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.4/32 metric 1024 scope global dynamic enp7s0
       valid_lft 77793sec preferred_lft 77793sec
    inet6 fe80::8400:ff:fe66:8da4/64 scope link
       valid_lft forever preferred_lft forever
@pcarrier pcarrier changed the title eth0/en0!? tempo doesn't boot. Something about eth0/en0!? Apr 18, 2024
@pcarrier pcarrier changed the title tempo doesn't boot. Something about eth0/en0!? tempo doesn't boot. Something about eth0/en0 which don't exist Apr 18, 2024
@mapno
Copy link
Member

mapno commented Apr 19, 2024

Likely related grafana/loki#4483

@mapno mapno added type/bug Something isn't working operations memberlist labels Apr 19, 2024
@mapno
Copy link
Member

mapno commented Apr 19, 2024

The compactor, as many other Tempo components, uses memberlist to coordinate across instances. As described in the issue I linked, the problem happens when Tempo can't find a default network interface for memberlist. A quick solution is to define it manually in the config

compactor:
  ring:
    instance_addr: 127.0.0.1

Long term, we'll a solution like grafana/loki#4570

@pcarrier
Copy link
Author

pcarrier commented Apr 19, 2024 •
edited

Could I set the network interface name instead? If so, how? Also, I'm confused as to why this fails / tries to use non-existent network interfaces?

@joe-elliott
Copy link
Member

yes, you can configure custom interface names to check for the ip address.

image

there are few ring configs: distributor, ingester, compactor, and metrics generator that will need this change.

you can search the manifest for instance_interface_names

https://grafana.com/docs/tempo/latest/configuration/manifest/

@pcarrier
Copy link
Author

Switched to:

server:
  http_listen_port: 3200
  grpc_listen_port: 9095

querier:
  frontend_worker:
    frontend_address: xi.xmit.dev:9095

query_frontend:
  search:
    duration_slo: 5s
  trace_by_id:
    duration_slo: 5s

distributor:
  ring: &1
    instance_interface_names:
    - enp7s0
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - 127.0.0.1:2379
        tls_enabled: true
        tls_cert_path: /etc/ssl/etcd/cert.pem
        tls_key_path: /etc/ssl/etcd/key.pem
        tls_ca_path: /etc/ssl/etcd/ca.pem
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317

ingester:
  lifecycler:
    ring: *1
  max_block_duration: 5m

compactor:
  ring: *1
  compaction:
    block_retention: 1h

metrics_generator:
  ring: *1
  storage:
    path: /srv/tempo/generator/wal
    remote_write:
      - url: http://127.0.0.1:9090/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo
      endpoint: 127.0.0.1:9000
      insecure: true
      access_key: CENSORED
      secret_key: CENSORED
    wal:
      path: /srv/tempo/wal
      v2_encoding: snappy
    block:
      v2_encoding: zstd
overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

Fails with:

Apr 19 22:15:48 x0.xmit.dev systemd[1]: Started Tempo service.
Apr 19 22:15:48 x0.xmit.dev tempo[3477821]: failed parsing config: failed to parse configFile /etc/tempo/config.yml: yaml: unmarshal errors:
Apr 19 22:15:48 x0.xmit.dev tempo[3477821]:   line 17: field instance_interface_names not found in type ring.Config
Apr 19 22:15:48 x0.xmit.dev systemd[1]: tempo.service: Main process exited, code=exited, status=1/FAILURE

@mapno
Copy link
Member

mapno commented Apr 22, 2024

The ring config in the ingester is slightly different than in the distributor or compactor. Try with this

ingester:
  lifecycler:
    interface_names:
      - enp7s0
    ring:
      kvstore:
        store: etcd
        etcd:
          ...
  max_block_duration: 5m

@pcarrier
Copy link
Author

pcarrier commented Apr 22, 2024 •
edited

Ah, now looks like grafana-server tries to connect to :3200 over IPv6 but tempo only listens on IPv4… Does that sound about right?

@zalegrala
Copy link
Contributor

Tempo can operate using IPv4 or IPv6, but the ring can use only one at a time. See our docs on IPv6 if that is your intent. https://grafana.com/docs/tempo/latest/configuration/network/ipv6/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
memberlist operations type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pcarrier @joe-elliott @mapno @zalegrala