cannot derive external address port without listening on a secure port. #81870

RGBCube · 2024-02-05T10:00:01Z

What happened?

logger=ticker t=2024-02-05T10:36:28.496462788+01:00 level=info msg=starting first_tick=2024-02-05T10:36:30+01:00
logger=sqlstore t=2024-02-05T10:36:28.339411276+01:00 level=info msg="Connecting to DB" dbtype=postgres
logger=settings t=2024-02-05T10:36:28.338576283+01:00 level=info msg="App mode production"
logger=settings t=2024-02-05T10:36:28.338564591+01:00 level=info msg="Path Provisioning" path=/nix/store/7v9i6mkimy42wf1lhc22aw54vidc7fq4-grafana-provisioning
logger=settings t=2024-02-05T10:36:28.338553821+01:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2024-02-05T10:36:28.338543252+01:00 level=info msg="Path Logs" path=/var/lib/grafana/data/log
logger=settings t=2024-02-05T10:36:28.338532921+01:00 level=info msg="Path Data" path=/var/lib/grafana/data
logger=settings t=2024-02-05T10:36:28.338516989+01:00 level=info msg="Path Home" path=/var/lib/grafana
logger=settings t=2024-02-05T10:36:28.338500519+01:00 level=info msg=Target target=[all]
logger=settings t=2024-02-05T10:36:28.338487535+01:00 level=info msg="Config loaded from" file=/nix/store/8yrfznz7xd9kch6jpgs3pni9gl8kn1iw-config.ini
logger=settings t=2024-02-05T10:36:28.338450025+01:00 level=info msg="Config loaded from" file=/var/lib/grafana/conf/defaults.ini
logger=settings t=2024-02-05T10:36:28.335772108+01:00 level=info msg="Starting Grafana" version=10.3.1 commit=NA branch=main compiled=2024-02-05T10:36:28+01:00
logger=secrets t=2024-02-05T10:36:28.360905533+01:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2024-02-05T10:36:28.442449428+01:00 level=info msg="Query Service initialization"
logger=provisioning.datasources t=2024-02-05T10:36:28.466482236+01:00 level=info msg="inserting datasource from configuration" name=Prometheus uid=PBFA97CFB590B2093
logger=provisioning.datasources t=2024-02-05T10:36:28.465923427+01:00 level=info msg="deleted datasource based on configuration" name=Prometheus
logger=provisioning.alerting t=2024-02-05T10:36:28.489262978+01:00 level=info msg="finished to provision alerting"
logger=provisioning.alerting t=2024-02-05T10:36:28.489226316+01:00 level=info msg="starting to provision alerting"
logger=plugin.store t=2024-02-05T10:36:28.439355825+01:00 level=info msg="Plugins loaded" count=55 duration=49.157436ms
logger=plugin.store t=2024-02-05T10:36:28.390199487+01:00 level=info msg="Loading plugins..."
logger=ngalert.state.manager t=2024-02-05T10:36:28.496259754+01:00 level=info msg="State cache has been initialized" states=0 duration=4.832784ms
logger=ngalert.state.manager t=2024-02-05T10:36:28.491420467+01:00 level=info msg="Warming state cache for startup"
logger=ngalert.scheduler t=2024-02-05T10:36:28.496330913+01:00 level=info msg="Starting scheduler" tickInterval=10s
logger=ngalert.multiorg.alertmanager t=2024-02-05T10:36:28.494325352+01:00 level=info msg="Starting MultiOrg Alertmanager"
logger=ngalert.migration t=2024-02-05T10:36:28.449082443+01:00 level=info msg=Starting
logger=ngalert.migration CurrentType=UnifiedAlerting DesiredType=UnifiedAlerting CleanOnDowngrade=false CleanOnUpgrade=false t=2024-02-05T10:36:28.449856001+01:00 level=info msg="Migration already complete"
logger=migrator t=2024-02-05T10:36:28.354841941+01:00 level=info msg="migrations completed" performed=0 skipped=526 duration=956.449µs
logger=migrator t=2024-02-05T10:36:28.341513348+01:00 level=info msg="Starting DB migrations"
logger=local.finder t=2024-02-05T10:36:28.491757721+01:00 level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins
logger=local.finder t=2024-02-05T10:36:28.439264787+01:00 level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins
logger=local.finder t=2024-02-05T10:36:28.439055321+01:00 level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins-bundled
logger=local.finder t=2024-02-05T10:36:28.388120441+01:00 level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins
logger=live.push_http t=2024-02-05T10:36:28.446834+01:00 level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2024-02-05T10:36:28.461017013+01:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=http.server t=2024-02-05T10:36:28.494887695+01:00 level=info msg="HTTP Server Listen" address=[::1]:8000 protocol=http subUrl= socket=
logger=grafanaStorageLogger t=2024-02-05T10:36:28.494964381+01:00 level=info msg="Storage starting"
logger=grafana-apiserver t=2024-02-05T10:36:28.502640384+01:00 level=info msg="cannot derive external address port without listening on a secure port."

What did you expect to happen?

For it to run normally, the config worked on an older version, my config is like so:

    settings = {
      analytics.reporting_enabled = false;

      database.host = "/run/postgresql";
      database.type = "postgres";
      database.user = "grafana";

      server.domain    = fqdn;
      server.http_addr = "::";
      server.http_port = 8000;

      users.default_theme = "system";
    };

    settings.security = {
      admin_email    = "metrics@${domain}";
      admin_password = "$__file{${config.age.secrets."cube/password.grafana".path}}";
      admin_user     = "admin";

      cookie_secure    = true;
      disable_gravatar = true;

      disable_initial_admin_creation = true; # Just in case.
    };

    settings.smtp = {
      enabled = true;

      password        = "$__file{${config.age.secrets."cube/password.mail.grafana".path}}";
      startTLS_policy = "MandatoryStartTLS";

      ehlo_identity = "contact@${domain}";
      from_address  = "metrics@${domain}";
      from_name     = "Metrics";
      host          = "${config.mailserver.fqdn}:${toString config.services.postfix.relayPort}";
    };

Did this work before?

Yes, 10.2.3

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.3.1
OS: NixOS
Browser: -

Grafana platform?

A package manager (APT, YUM, BREW, etc.)

Datasource(s)?

The text was updated successfully, but these errors were encountered:

leahoswald · 2024-02-05T22:36:15Z

You now have to put the IPv6 address in square brackets, then it works. I have no idea where the change comes from. Can't find anything in the release notes.

RGBCube · 2024-02-06T06:16:40Z

You now have to put the IPv6 address in square brackets, then it works. I have no idea where the change comes from. Can't find anything in the release notes.

Maybe now the listening is done by something like $"{host}:{port}" now? Because v4 works fine...

leahoswald · 2024-02-06T10:23:51Z

I think this is still a bug and the issue should be kept open.

grafana/pkg/api/http_server.go

Line 392 in 4b90936

    
           // Remove any square brackets enclosing IPv6 addresses, a format we support for backwards compatibility

This and the code in the following lines sound a lot like using IPv6 without brackets should be the default and also the used code referring to the http_addr value should use IPv6 without the brackets if I read the docs of the parseIP function right: https://pkg.go.dev/net#ParseIP

The problem is that http_addr is parsed into HTTPAddr here

grafana/pkg/setting/setting.go

Line 1916 in eef491d

cfg.HTTPAddr = valueAsString(server, "http_addr", DefaultHTTPAddr)

but I can't find any change of usage of that variable between v10.2.3 and v10.3.1

Maybe its a problem with a changed behavior of an upstream library.

I would appreciate if you could reopen the issue and maybe change the title accordingly.

leahoswald · 2024-02-06T10:58:49Z

Ok now after a little debugging I see the net.JoinHostPort function of Go returns a string like [host]:port if host is a IPv6 address. We use it here:
https://github.com/grafana/grafana/blob/main/pkg/api/http_server.go#L395

And go seems to expect a version with square brackets, see: "If the host is a literal IPv6 address it must be enclosed in square brackets, as in "[2001:db8::1]:80" or "[fe80::1%zone]:80"" https://pkg.go.dev/net#Dial

So maybe some fallback code was removed in go. I would recommend trying to remove this line: https://github.com/grafana/grafana/blob/main/pkg/api/http_server.go#L393

RGBCube · 2024-02-07T07:27:52Z

Alright, I've reopened the issue. I'll try that fix soon

CJ-Jackson · 2024-02-13T06:57:23Z

In grafana.ini I had to wrap the IPv6 in square brackets for http_addr and grafana started working again. It worked fine as is in previous version of grafana.

DmitriK · 2024-02-21T16:18:31Z

I had this issue as well, and the square brackets fixed it. However, I would also like to note that the error message in the title is an info-level message, and my config originally limited logging to 'warn', so it took a while to figure out what the real problem was (I kept thinking the errors about provisioning directories being missing (which I don't use) was the issue).
Perhaps the error message itself can be upgraded to an error-level as part of the fix?

nkukard · 2024-03-06T20:56:20Z

Still seems to be a problem in 10.3.4...

logger=http.server t=2024-03-06T19:46:20.131578326Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=grafana-apiserver t=2024-03-06T19:46:20.132500013Z level=info msg="cannot derive external address port without listening on a secure port.

fernvenue · 2024-03-15T11:10:38Z

I had this issue as well, and the square brackets fixed it.

@DmitriK It works for me, thx :)

yincongcyincong · 2024-04-30T07:07:02Z

The reason is that the apiserver printed a fatal log and exited. i will fix it , hope next version this problem can be solved.
https://github.com/kubernetes/apiserver/blob/master/pkg/server/config.go#L686

yincongcyincong · 2024-04-30T08:13:14Z

finally, locate this line of code：https://github.com/grafana/grafana/blob/main/pkg/services/apiserver/config.go#L30
and i bring a pr #87108

* fix ipv6 startup fail #81870 * ipv6 startup fail * ipv6 startup fail

* fix ipv6 startup fail #81870 * ipv6 startup fail * ipv6 startup fail (cherry picked from commit ba8b4bd)

RGBCube changed the title ~~Product Area: Short description of bug~~ cannot derive external address port without listening on a secure port. Feb 5, 2024

RGBCube closed this as completed Feb 6, 2024

RGBCube reopened this Feb 7, 2024

usmangt added area/backend/config area/backend triage/needs-confirmation used for OSS triage rotation - reported issue needs to be reproduced labels Feb 16, 2024

papagian mentioned this issue Feb 19, 2024

grafana-server does not start after update #81991

Open

sometimes-i-send-pull-requests mentioned this issue Apr 26, 2024

IPv6 address format incompatibility in wait_for action grafana/grafana-ansible-collection#189

Open

yincongcyincong added a commit to yincongcyincong/grafana that referenced this issue Apr 30, 2024

fix ipv6 startup fail grafana#81870

6fb4e1d

yincongcyincong mentioned this issue Apr 30, 2024

fix ipv6 startup fail #81870 #87108

Merged

DanCech closed this as completed in #87108 May 6, 2024

DanCech pushed a commit that referenced this issue May 6, 2024

fix ipv6 startup fail #81870 (#87108)

ba8b4bd

* fix ipv6 startup fail #81870 * ipv6 startup fail * ipv6 startup fail

grafana-delivery-bot bot pushed a commit that referenced this issue May 6, 2024

fix ipv6 startup fail #81870 (#87108)

9d6bbd3

* fix ipv6 startup fail #81870 * ipv6 startup fail * ipv6 startup fail (cherry picked from commit ba8b4bd)

grafana-delivery-bot bot mentioned this issue May 6, 2024

[v11.0.x] fix ipv6 startup fail #81870 #87402

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot derive external address port without listening on a secure port. #81870

cannot derive external address port without listening on a secure port. #81870

RGBCube commented Feb 5, 2024 •

edited

leahoswald commented Feb 5, 2024

RGBCube commented Feb 6, 2024

leahoswald commented Feb 6, 2024

leahoswald commented Feb 6, 2024

RGBCube commented Feb 7, 2024 •

edited

CJ-Jackson commented Feb 13, 2024

DmitriK commented Feb 21, 2024

nkukard commented Mar 6, 2024

fernvenue commented Mar 15, 2024 •

edited

yincongcyincong commented Apr 30, 2024

yincongcyincong commented Apr 30, 2024

cannot derive external address port without listening on a secure port. #81870

cannot derive external address port without listening on a secure port. #81870

Comments

RGBCube commented Feb 5, 2024 • edited

What happened?

What did you expect to happen?

Did this work before?

Is the bug inside a dashboard panel?

Environment (with versions)?

Grafana platform?

Datasource(s)?

leahoswald commented Feb 5, 2024

RGBCube commented Feb 6, 2024

leahoswald commented Feb 6, 2024

leahoswald commented Feb 6, 2024

RGBCube commented Feb 7, 2024 • edited

CJ-Jackson commented Feb 13, 2024

DmitriK commented Feb 21, 2024

nkukard commented Mar 6, 2024

fernvenue commented Mar 15, 2024 • edited

yincongcyincong commented Apr 30, 2024

yincongcyincong commented Apr 30, 2024

RGBCube commented Feb 5, 2024 •

edited

RGBCube commented Feb 7, 2024 •

edited

fernvenue commented Mar 15, 2024 •

edited