DNS resolution issue in Kubernetes

[NiklasRosenstein]

I'm trying to deploy Outline in Kubernetes, but I'm facing a surprising DNS resolution error that I can't explain. Nothing is special to the Pod's configuration regarding DNS, in fact the resolution works fine with curl in the very same container.

Yet, Outline fails to resolve the full-qualified DNS of it's in-cluster Postgres service as well as the public record of my Keycloak instance. Interestingly, the database DNS resolution works without the .cluster.local suffix.

{"error":"getaddrinfo ENOTFOUND outline-db-rw.outline.svc.cluster.local","level":"error","message":"Failed to connect to database","stack":"SequelizeHostNotFoundError: getaddrinfo ENOTFOUND outline-db-rw.outline.svc.cluster.local\n    at Client._connectionCallback (/opt/outline/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:134:24)\n    at Client._handleErrorWhileConnecting (/opt/outline/node_modules/pg/lib/client.js:327:19)\n    at Client._handleErrorEvent (/opt/outline/node_modules/pg/lib/client.js:337:19)\n    at Connection.emit (node:events:518:28)\n    at Connection.emit (node:domain:488:12)\n    at Socket.reportStreamError (/opt/outline/node_modules/pg/lib/connection.js:58:12)\n    at Socket.emit (node:events:518:28)\n    at Socket.emit (node:domain:488:12)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"} 

Curl can resolve the entry:

<<K9s-Shell>> Pod: outline/outline-68c7995c5b-zlrc8 | Container: outline 
/opt/outline $ curl -v outline-db-rw.outline.svc.cluster.local
* Host outline-db-rw.outline.svc.cluster.local:80 was resolved.
* IPv6: (none)
* IPv4: 10.43.58.250
*   Trying 10.43.58.250:80...
*

I'm a bit lost as to what could be going on. Any hints are appreciated!

Image: outlinewiki/outline:0.75.2

Edit: The cluster is running on a Hetzner Dedicated Server

[kiguigui]

Is your router a FritzBox? Hab the same Problem some time ago as some dude bought fritz.box before AVm (Vendor of Fritz Box) could do that. This led to some super-strange behavior to kubedns and stuff. I can post the solution later, if needed.

[NiklasRosenstein]

Thanks for the quick reply! Actually I'm running Kubernetes on a Hetzner Dedicated Server, I don't think they're using a FritzBox! I'd still be curious what the solution was for you.

[kiguigui]

I've had to configure coredns in k8s to drop anything related to the tld fritz.box. extremely weird as something like "login.my.domain" was resolved to "login.my.domaon.fritz.box". It always appeded "fritz.box".

It was completely resolved by using a internal domain with ".internal" as tld since ICANN wants to define that as a domain for private networks.

[NiklasRosenstein]

I think my issue is actually very similar after all. I don't understand exactly what's going on, but updating my CoreDNS configuration

- forward . /etc/resolv.conf
+ forward . 100.100.100.100

fixed the issue. I use this IP because the server is connected to a headscale VPN. The resolv.conf in the CoreDNS Pod contained

search user.vpn.example.com
nameserver 100.100.100.100 nameserver 185.12.64.2
nameserver 185.12.64.1

Which seems odd, as I think it should contain either 100.100.100.100 or the other two nameservers, as they might resolve records differently.

Anyway, no issue with Outline at all.