Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[External-DNS] Sync mode does not delete/update DNS entries when changes are made on kubernetes side #4430

Closed
baptistebalmon opened this issue May 2, 2024 · 5 comments
Closed

[External-DNS] Sync mode does not delete/update DNS entries when changes are made on kubernetes side #4430

baptistebalmon opened this issue May 2, 2024 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@baptistebalmon
Copy link

baptistebalmon commented May 2, 2024 •
edited

What happened:
The externalDNS has been configured this way :

image: registry.k8s.io/external-dns/external-dns:v0.14.0
args:
- --policy=sync
- --source=ingress
- --source=service
- --annotation-filter=managed.by=external-dns
- --txt-owner-id=<rke cluster name>-ext-dns
- --txt-prefix=dbaas.

I'm using RFC2136 mode with PowerDNS.

this way, only services and ingresses that have the annotation "managed.by=external-dns" will be managed.
also txt-owner-id allows to identify the RKE Cluster on which the DNS entry should lead to
the txt-prefix allows to create CNAME entries (for ingresses mainly)

The creation works well :

  • 2 TXT records
  • A/CNAME record

Here is an example of a kubernetes service that needs to be created :

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: toto.example.org
    external-dns.alpha.kubernetes.io/ttl: "3600"
    loadbalancer.openstack.org/load-balancer-address: X.X.X.X
    managed.by: external-dns
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  sessionAffinity: None
  type: LoadBalancer

When I delete the kubernetes service or ingress, externalDNS does not delete them on the DNS.
When I update the kubernetes service or ingress, externalDNS does not update them on the DNS but creates a new entry.

What you expected to happen:
On the kubernetes resources (svc/ingress) deletion, cleaning the corresponding A/CNAME and TXT records from the DNS
On the kubernetes resources (svc/ingress) update, updating the corresponding A/CNAME and TXT records from the DNS.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy externalDNS version 0.14.0
  2. Create kubernetes services
  3. Check that the records are created on the DNS
  4. Delete the kubernetes service
  5. Check that the records are not deleted on the DNS

Anything else we need to know?:

Environment:

  • External-DNS version (use external-dns --version): 0.14.0
  • DNS provider: PowerDNS
  • Others: RFC2136
@baptistebalmon baptistebalmon added the kind/bug Categorizes issue or PR as related to a bug. label May 2, 2024
@leonardocaylent
Copy link
Contributor

@baptistebalmon Did you check if this is working correctly with v0.13.6? Can you attach the external-dns pod logs in Debug level?

@baptistebalmon
Copy link
Author

baptistebalmon commented May 7, 2024 •
edited

Hi @leonardocaylent
I've tried to downgrade the externalDNS image version to v0.13.6.
The issue is still the same.
Kubernetes resources are deleted but DNS records are not.
Please find the log in debug mode.

time="2024-05-07T13:44:03Z" level=debug msg="axfr is disabled"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from ingress: toto/haproxy-stats: [toto.org 3600 IN CNAME vip.org []]"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:44:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:44:03Z" level=debug msg="ApplyChanges (Create: 9, UpdateOld: 0, UpdateNew: 0, Delete: 0)"
time="2024-05-07T13:44:03Z" level=debug msg="Processing batch 0 of create changes"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto.org 3600 IN CNAME vip.org []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto.org 3600 CNAME vip.org"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X"
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.cname-toto.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.cname-toto.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:44:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:44:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:44:03Z" level=debug msg=SendMessage
time="2024-05-07T13:44:03Z" level=debug msg=SendMessage.success

time="2024-05-07T13:49:03Z" level=debug msg="axfr is disabled"
time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: titi/titi-pgbouncer: [titi.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:49:03Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto-replicas.org 3600 IN A X.X.X.X []]"
time="2024-05-07T13:49:03Z" level=debug msg="ApplyChanges (Create: 6, UpdateOld: 0, UpdateNew: 0, Delete: 0)"
time="2024-05-07T13:49:03Z" level=debug msg="Processing batch 0 of create changes"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=toto-replicas.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: toto-replicas.org 3600 A X.X.X.X"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=titi.org 3600 IN A X.X.X.X []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: titi.org 3600 A X.X.X.X"
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-toto-replicas.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:49:03Z" level=debug msg="AddRecord.ep=dbaas.a-titi.org 0 IN TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer" []"
time="2024-05-07T13:49:03Z" level=info msg="Adding RR: dbaas.a-titi.org 0 TXT "heritage=external-dns,external-dns/owner=kubernetes-cluster-ext-dns,external-dns/resource=service/titi/titi-pgbouncer""
time="2024-05-07T13:49:03Z" level=debug msg=SendMessage
time="2024-05-07T13:49:03Z" level=debug msg=SendMessage.success

Thank you for your help.

@leonardocaylent
Copy link
Contributor

@baptistebalmon I can see that your plan is not detecting anything to delete. I need to ask you 3 more questions:
1)Has this ever worked for you before?
2)Can you test again with the versions v0.12.0 and v0.10.0?
3)If you manually delete the records external-dns creates them again on the next round?

If you can add the creation logs that could also help.

Thank you for reporting this issue

@baptistebalmon
Copy link
Author

hi @leonardocaylent , thank you for helping me out on this.
It seems that externalDNS only creates (as we can see in the logs), I could never see UpdateOld, UpdateNew or Delete.
It explains why it recreates every entry but does not delete any.
Is it a normal behavior ?

  1. I guess that the delete worked once first time that I installed externalDNS.
    The only difference is that I didn't configure the annotation filter on the records to be managed by externalDNS.
    I've tried to remove this configuration but it still doesn't work.

  2. I've tried with both v0.12.0 and v0.10.0 and I've got same result.

  3. If I manually delete the records, they are created again on the next round.

Here are the logs of creation :
time="2024-05-29T08:38:01Z" level=debug msg="axfr is disabled"
time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from ingress: toto/haproxy-stats: [toto-haproxy-stats.io 3600 IN CNAME priv-toto.io []]"
time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy: [toto.io 3600 IN A X.X.X.X []]"
time="2024-05-29T08:38:01Z" level=debug msg="Endpoints generated from service: toto/toto-haproxy-replicas: [toto-replicas.io 3600 IN A X.X.X.X []]"
time="2024-05-29T08:38:01Z" level=debug msg="ApplyChanges (Create: 9, UpdateOld: 0, UpdateNew: 0, Delete: 0)"
time="2024-05-29T08:38:01Z" level=debug msg="Processing batch 0 of create changes"
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto-haproxy-stats.io 3600 IN CNAME priv-mtdbaas01stg.io []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto-haproxy-stats.io 3600 CNAME priv-mtdbaas01stg.io"
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto-replicas.io 3600 IN A X.X.X.X []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto-replicas.io 3600 A X.X.X.X"
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=toto.io 3600 IN A X.X.X.X []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: toto.io 3600 A X.X.X.X"
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto-haproxy-stats.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto-haproxy-stats.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.cname-toto-haproxy-stats.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.cname-toto-haproxy-stats.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=ingress/toto/haproxy-stats""
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto-replicas.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto-replicas.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas""
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.a-toto-replicas.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.a-toto-replicas.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy-replicas""
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.toto.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.toto.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-29T08:38:01Z" level=debug msg="AddRecord.ep=dbaas.a-toto.io 0 IN TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy" []"
time="2024-05-29T08:38:01Z" level=info msg="Adding RR: dbaas.a-toto.io 0 TXT "heritage=external-dns,external-dns/owner=cluster-stg-ext-dns,external-dns/resource=service/toto/toto-haproxy""
time="2024-05-29T08:38:01Z" level=debug msg=SendMessage
time="2024-05-29T08:38:01Z" level=debug msg=SendMessage.success

@baptistebalmon
Copy link
Author

Hi again @leonardocaylent , problem solved

I've found that AXFR is mandatory to update or delete DNS records.

The TSIG key that I was using was only configured with DYNUPDATE.
I've added :

  • AXFR on the TSIG key
  • rfc2136-tsig-axfr flag to true on the externalDNS container

Updates and Deletes work now fine.

Thank you for the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@baptistebalmon @leonardocaylent