External DNS servers provided with use.external.dns disabled

[kriegsmanj]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR / Isolated network

CLOUDSTACK VERSION

4.18.1.1

CONFIGURATION

Global setting:
use.external.dns = false

SUMMARY

When setting 'use.external.dns' is set to false, I expect to only receive the internal (VR) IP as DNS server.
However, it returns both internal + external DNS servers.

Providing external DNS servers is addition to the internal gives problems resolving instances hostnames in the same isolated network as these mappings are not available in external dns servers.

STEPS TO REPRODUCE

Set 'use.external.dns' to true
Restart isolated network with cleanup

Set 'use.external.dns' to false
Restart isolated network with cleanup

EXPECTED RESULTS

With setting on true: 
External DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<external_dns_ip_1>,<external_dns_ip_2>

With setting on false: 
Internal DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<virtual_router_ip>

ACTUAL RESULTS

With setting on true: 
External DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<external_dns_ip_1>,<external_dns_ip_2>

With setting on false: 
Internal + External DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<virtual_router_ip>,<external_dns_ip_1>,<external_dns_ip_2>

[weizhouapache]

@kriegsmanj
the description of the global setting is: Bypass internal dns, use external dns1 and dns2
it looks like the global setting is used to determine if internal dns is bypassed.
external dns1/dns2 are always used.

[kriegsmanj]

@kriegsmanj the description of the global setting is: Bypass internal dns, use external dns1 and dns2 it looks like the global setting is used to determine if internal dns is bypassed. external dns1/dns2 are always used.

To me this means: dns is not going "instance -> vr -> external dns", but bypasses internal, "instance -> external dns"

Using both internal + external at the same time, where the internal also has a host-file with all entries of instances in that network makes no sense. Some app use the resolvers in random and not just the first configured. This causes lookup errors for these hostnames when it randomly uses the external resolver to resolve an internal hostname

[weizhouapache]

@kriegsmanj the description of the global setting is: Bypass internal dns, use external dns1 and dns2 it looks like the global setting is used to determine if internal dns is bypassed. external dns1/dns2 are always used.

To me this means: dns is not going "instance -> vr -> external dns", but bypasses internal, "instance -> external dns"

Using both internal + external at the same time, where the internal also has a host-file with all entries of instances in that network makes no sense. Some app use the resolvers in random and not just the first configured. This causes lookup errors for these hostnames when it randomly uses the external resolver to resolve an internal hostname

if I understand correctly, internal dns means the internal dns1/dns2 in zone setting.
From what @kriegsmanj described, it seems internal dns also include the cloudstack VR

[kriegsmanj]

In case of an isolated network, the resolvers configured are the Virtual Router IP and external dns1/dns2 in zone setting.
In our environment we have no internal dns1/dns2 configured, so cannot say if those are added if those are set.

The DHCP should give only the Virtual Router IP as DNS servers in case of isolated network / vpc. Else the hostname entires in the VR make no sense if it cannot used by the virtual machines.

[weizhouapache]

In case of an isolated network, the resolvers configured are the Virtual Router IP and external dns1/dns2 in zone setting. In our environment we have no internal dns1/dns2 configured, so cannot say if those are added if those are set.

The DHCP should give only the Virtual Router IP as DNS servers in case of isolated network / vpc. Else the hostname entires in the VR make no sense if it cannot used by the virtual machines.

I got same result as @kriegsmanj described, even if internal dns1/dns2 are set.

With setting on true: 
External DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<external_dns_ip_1>,<external_dns_ip_2>
With setting on false: 
Internal + External DNS provided by DHCP (file /etc/dnsmasq.conf on VR)
dhcp-option=6,<virtual_router_ip>,<external_dns_ip_1>,<external_dns_ip_2>

[hrak]

I think the problem lies in the logic here. Based on the description in the comment, that should be either !dnsProvided && dhcpProvided or dnsProvided != dhcpProvided (former probably better match).

In the current state its causing the external DNS to be appended even when dnsProvided and dhcpProvided are both true.

[DaanHoogland]

To me it looks like you either want

• an extra setting use.internal.dns to be able to switch off the <virtual_router_ip> addition.
• an extra setting bypass.external.dns to be able to switch off the <external_dns_ip_1>,<external_dns_ip_2> additions.
  The current behaviour is actually as intended but documentation can always improve. ;)

[hrak]

Even if this is considered intended behavior, it still seems wrong. Adding external DNS's that don't know anything about the instances in the isolated network to the list of resolvers returned by DHCP results in a broken DNS config for the instances in the isolated network.

Any attempt to resolve another instance in the isolated network (say, a webserver looking for a mysql server) would randomly fail if systemd-resolved decides to pick another resolver than the primary (which it seems to randomly do quite frequently)

And the existence of this logic and the comment above it seem to suggest that this is not working as intended, as the code is not doing what the comment describes.

[weizhouapache]

Even if this is considered intended behavior, it still seems wrong. Adding external DNS's that don't know anything about the instances in the isolated network to the list of resolvers returned by DHCP results in a broken DNS config for the instances in the isolated network.

Any attempt to resolve another instance in the isolated network (say, a webserver looking for a mysql server) would randomly fail if systemd-resolved decides to pick another resolver than the primary (which it seems to randomly do quite frequently)

I have no idea how systemd-resolved works. Is it possible to enforce the order of DNS servers in systemd-resolved ?
Have you seen the issue in the VMs without systemd-resolved ?

And the existence of this logic and the comment above it seem to suggest that this is not working as intended, as the code is not doing what the comment describes.

the comment means, the VR will not be used as DNS resolver, if

• VR does not provide DNS service, OR
• the setting use.external.dns is set to true

I agree with Daan that this probably needs a new setting.

[DaanHoogland]

@kriegsmanj , @hrak , very sorry that it doesn't behave as you would expect, and we can certainly change it, but we'll have to do that in a backwards compatible way as it is working for lots of other installations.

As a workaround you can configure your internal DNS server as external DNS server as well, or not configure an external DNS for this network.

As for a changed functionality, I would suggest a threesome of settings:
dns.enable.external
dns.enable.internal
dns.enable.vr (which is basically the function of the current setting)
and mark use.external.dns as obsolete, or rename it as the description suggests; dns.bypass.internal .

[phsm]

As a workaround you can configure your internal DNS server as external DNS server as well, or not configure an external DNS for this network.

This particular workaround won't work as the only DNS resolver that keeps VM name -> VM address records is the DNS server running on the Virtual router.

There could be 3 scenario, lets consider the use case for each of them:
Instances use only VR IP as a resolver.
It is needed in case of Isolated networks and VPCs. The instances there are expected to be able to reach another instance within the same network by its name.

Instances use only External DNS IPs as resolvers
This is mostly suitable for Shared networks. Shared network instances typically have public IPs on them, they don't rely on the Virtual Router to reach the Internet. Thus, there is no need to bind them to a VR that can go offline for some reason.

Instances use both VR IP and External DNS IPs as resolvers (current behavior)
No specific use case for it which makes it a good default value.

I think the best approach to give flexibility to the CS users while also keeping the backwards compatibility would be to implement it as a Network Offering setting, e.g.:

Setting name: DHCP DNS servers policy
Possible values (select one): Virtual Router address, External DNS servers, Both (default)

This way the Cloudstack admins will decide what is best for their users on the Network Offering level, without having to set it per individual network.

[weizhouapache]

@phsm
the solution depends on your requirements
if you want to have setting globally or per zone, a configkey is better solution
if you want to have setting in network offering level, it is also good.