allow to configure dns_record_type with a system property #776

cpesch · 2024-03-07T12:02:29Z

My setup is a set of Keycloak 18.0.2 clusters in an environment with nomad and consul. We're using DNS_PING with SRV records for a long time now and the JGroups setup is pretty simple:

JGROUPS_DISCOVERY_PROTOCOL=dns.DNS_PING
JGROUPS_DISCOVERY_PROPERTIES="dns_query="${dns_query}",dns_record_type="SRV""
JGROUPS_TRANSPORT_STACK="tcp"

When migrating this to Keycloak 23.0.7, I found that the dns_record_type separated with a comma is not supported anymore. With the documentation I found on the internet and discussions and issue in the JGroups, Infinispan, Keycloak github projects, I failed to get SRV records running with something straight-forward like this:

<infinispan ...>

    <jgroups>
        <stack name="dns-ping">
            <dns.DNS_PING dns_record_type="SRV"
                          dns_query="${dns_query}"/>
            <TCP diag.enabled="true"
                 bind_addr="GLOBAL"
                 external_addr="{{ env "NOMAD_IP_jgroupscluster" }}"
                 external_port="{{ env "NOMAD_HOST_PORT_jgroupscluster" }}"/>
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport stack="dns-ping"...

When looking at org.jgroups.protocols.dns.DNS_PING I found that only a tiny tweak is necessary to reduce all this down to

KC_CACHE=ispn
KC_CACHE_STACK=kubernetes
JAVA_OPTS_APPEND="-Djgroups.dns.dns_query=${dns_query} -Djgroups.dns.dns_record_type=SRV

This works fine for me since some days.

@belaban I'd be really happy if you include this in an upcoming 5.2.x release

rhusar

With a brief look this looks good to me. Allowed workflow to run now.

n.b. maybe a bit unfortunate that we didn't name these jgroups.dns.query instead of duplicating the prefix like we do now jgroups.dns.dns_query, but let's keep this consistent as 'jgroups.dns.' + field name.

belaban · 2024-03-07T14:05:23Z

Hmm... adding the system property should not be required, although it certainly works, so no objections.
This could be done in the XML config directly: JGroups parses comma-deparated attributes and tries to substitute both system properties and environment variables.

belaban · 2024-03-07T14:06:04Z

In other words, no change and no release should be required...

rhusar · 2024-03-07T14:18:48Z

Hmm... adding the system property should not be required, although it certainly works, so no objections. This could be done in the XML config directly: JGroups parses comma-deparated attributes and tries to substitute both system properties and environment variables.

IIUC well, no - the whole point is so that you do not have to edit the XML as that's way more convoluted in a container rather than adding a property (this PR) and then simply adding -Djgroups.dns.dns_record_type=SRV to the start options.

belaban · 2024-03-07T14:22:47Z

Christian is already using a system prop (JGROUPS_DISCOVERY_PROPERTIES) to change the config, so it could be done there (IMO)...
BTW: a JIRA would be nice, so we can assign this change to a version

cpesch · 2024-03-07T14:57:37Z

IIUC well, no - the whole point is so that you do not have to edit the XML as that's way more convoluted in a container rather than adding a property (this PR) and then simply adding -Djgroups.dns.dns_record_type=SRV to the start option

@rhusar well said. I fully agree.

Christian is already using a system prop (JGROUPS_DISCOVERY_PROPERTIES) to change the config, so it could be done there (IMO)...

I'm confused: that was the Keycloak 18 legacy configuration.

Wait. Is there a way to set org.jgroups.protocols.dns.DNS_PING#dns_record_type without touching the XML? Just with a system property or environment variable?

belaban · 2024-03-07T15:07:44Z

Yes, in the config (XML or programmatic):

<dns.DNS dns_record_type="${dns.record_type,RECORD_TYPE:A}".../>

Default is A. System property dns.record_type or env var RECORD_TYPE overrird this.

cpesch · 2024-03-07T16:12:00Z

Ok, that's exactly the config I was struggeling with. I spent two days reading and experimenting – and couldn't even see this in the logs:

2024-03-07 11:36:59,929 TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] (jgroups-13,040f8502066a-48384) resolving DNS query: labs-keycloak-playground..... of a type: SRV

It might be caused by JGroups being integrated to Infinispan, and Infinispan being integrated in Keycloak. I've tried to find people who were successful with this. Could you spot, what's wrong here in my config?

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
    xmlns="urn:infinispan:config:14.0">

<jgroups>
    <stack name="dns-ping">
        <dns.DNS_PING dns_record_type="SRV"
                      dns_query="${dns_query}"
                      stack.combine="REPLACE"
                      stack.position="MPING"/>
        <TCP diag.enabled="true"
             bind_addr="GLOBAL"
             external_addr="{{ env "NOMAD_IP_jgroupscluster" }}"
             external_port="{{ env "NOMAD_HOST_PORT_jgroupscluster" }}"/>
    </stack>
</jgroups>

<cache-container name="keycloak">
    <transport lock-timeout="60000" stack="dns-ping"/>
    <local-cache name="realms" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <distributed-cache name="sessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="2">
        <expiration lifespan="-1"/>
    </distributed-cache>
    <local-cache name="authorization" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <memory max-count="10000"/>
    </local-cache>
    <replicated-cache name="work">
        <expiration lifespan="-1"/>
    </replicated-cache>
    <local-cache name="keys" simple-cache="true">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="3600000"/>
        <memory max-count="1000"/>
    </local-cache>
    <distributed-cache name="actionTokens" owners="2">
        <encoding>
            <key media-type="application/x-java-object"/>
            <value media-type="application/x-java-object"/>
        </encoding>
        <expiration max-idle="-1" lifespan="-1" interval="300000"/>
        <memory max-count="-1"/>
    </distributed-cache>
</cache-container>
</infinispan>

It looks simple but with this config, I don't see any output from the org.jgroups.protocols.dns.DefaultDNSResolver and I've tried a lot of variants. What am I missing?

pruivo · 2024-03-07T19:32:36Z

You need to extend the kubernetes stack:

      <stack name="dns-ping" extends="kubernetes">
         <dns.DNS_PING dns_record_type="SRV"/>
      </stack>

cpesch · 2024-03-11T14:45:45Z

Finally, I've found the road blocker for my efforts: the distributed cache configuration in Keycloak is part of the build time configuration and not the run time configuration. After putting this cache configuration file

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
        xmlns="urn:infinispan:config:14.0">

    <jgroups>
        <stack name="dns-ping" extends="kubernetes">
            <dns.DNS_PING dns_query="${jgroups.dns.query}"
                          dns_record_type="${jgroups.dns.record:A}"
            />
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="dns-ping"/>
...same as above...

into the docker image, running kc.sh build, I could configure it with

JAVA_OPTS_APPEND="-Xmx${kc_xmx}m -Djgroups.bind_addr=GLOBAL -Djgroups.external_addr={{ env "NOMAD_IP_jgroupscluster" }} -Djgroups.external_port={{ env "NOMAD_HOST_PORT_jgroupscluster" }} -Djgroups.dns.query=${dns_query} -Djgroups.dns.record=SRV"

at run time like for Keycloak 18 – and it works. This was the discussion that finally helped me.

allow to configure dns_record_type with a system property

6533376

rhusar approved these changes Mar 7, 2024

View reviewed changes

cpesch mentioned this pull request Mar 11, 2024

ISPN-15857 Allow to configure DNS record type for DNS_PING with a Java property infinispan/infinispan#12121

Merged

github-actions bot mentioned this pull request Apr 4, 2024

[14.0.x] ISPN-15857 Allow to configure DNS record type for DNS_PING with a Java property infinispan/infinispan#12243

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow to configure dns_record_type with a system property #776

allow to configure dns_record_type with a system property #776

cpesch commented Mar 7, 2024 •

edited

rhusar left a comment

belaban commented Mar 7, 2024

belaban commented Mar 7, 2024

rhusar commented Mar 7, 2024

belaban commented Mar 7, 2024

cpesch commented Mar 7, 2024

belaban commented Mar 7, 2024

cpesch commented Mar 7, 2024

pruivo commented Mar 7, 2024

cpesch commented Mar 11, 2024

allow to configure dns_record_type with a system property #776

Are you sure you want to change the base?

allow to configure dns_record_type with a system property #776

Conversation

cpesch commented Mar 7, 2024 • edited

rhusar left a comment

Choose a reason for hiding this comment

belaban commented Mar 7, 2024

belaban commented Mar 7, 2024

rhusar commented Mar 7, 2024

belaban commented Mar 7, 2024

cpesch commented Mar 7, 2024

belaban commented Mar 7, 2024

cpesch commented Mar 7, 2024

pruivo commented Mar 7, 2024

cpesch commented Mar 11, 2024

cpesch commented Mar 7, 2024 •

edited