Docker variant of Launch agents via SSH with SSH key injection always fails

[vdeyak]

Jenkins and plugins versions report

Environment

Jenkins: 2.401.1
OS: Linux - 5.14.0-319.el9.x86_64
Java: 11.0.18 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
ant:487.vd79d090d4ea_e
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
apache-httpcomponents-client-5-api:5.2.1-1.0
authentication-tokens:1.53.v1c90fd9191a_b_
bootstrap5-api:5.3.0-1
bouncycastle-api:2.28
branch-api:2.1105.v472604208c55
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloud-stats:267.v577e3742c282
cloudbees-folder:6.815.v0dd5a_cb_40e0e
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
credentials:1254.vb_96f366e7b_a_d
credentials-binding:604.vb_64480b_c56ca_
display-url-api:2.3.7
docker-commons:419.v8e3cd84ef49c
docker-java-api:3.3.1-79.v20b_53427e041
docker-plugin:1.4
durable-task:507.v050055d0cb_dd
echarts-api:5.4.0-5
email-ext:2.97
font-awesome-api:6.4.0-1
git:5.0.2
git-client:4.3.0
github:1.37.1
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1725.vd391eef681a_e
gitlab-api:5.2.0-86.v1ed41a_9cf486
gitlab-plugin:1.7.14
gradle:2.8
instance-identity:142.v04572ca_5b_265
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jersey2-api:2.39.1-2
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.7.0-1
junit:1207.va_09d5100410f
ldap:682.v7b_544c9d1512
mailer:457.v3f72cb_e015e5
matrix-auth:3.1.8
matrix-project:789.v57a_725b_63c79
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-145.vcb_8de402ef81
pam-auth:1.10
pipeline-build-step:491.v1fec530da_858
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2133.ve46a_6113dfc3
pipeline-model-definition:2.2133.ve46a_6113dfc3
pipeline-model-extensions:2.2133.ve46a_6113dfc3
pipeline-rest-api:2.32
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2133.ve46a_6113dfc3
pipeline-stage-view:2.32
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
resource-disposer:0.22
scm-api:672.v64378a_b_20c60
script-security:1251.vfe552ed55f8d
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-agent:333.v878b_53c89511
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.303.vefc7119b_ec23
structs:324.va_f5d6774f3a_d
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:596.v8c21c963d92d
workflow-api:1213.v646def1087f9
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3673.v5b_dd74276262
workflow-durable-task-step:1247.v7f9dfea_b_4fd0
workflow-job:1308.v58d48a_763b_31
workflow-multibranch:746.v05814d19c001
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

root@jenkins ~# cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

*on both (controller and agent)

Reproduction steps

• Go to Dashboard > Manage Jenkins > Nodes;
• Hit "+ New Node";
• Set Node name: {any}, Type: "Permanent Agent" and hit "Create" button;
• Set Remote root directory: /home/jenkins;
• Set Launch method: Docker variant of Launch agents via SSH with SSH key injection (provided by Docker plugin);
• Set Host: {valid agent's host available via SSH},
• Set Credentials: {valid SSH credentials like SSH username with a private key};
• Set Host Key Verification Strategy: Non verifying Verification Strategy
• Hit "Save" button.

Expected Results

New agent node is created.

Actual Results

Oops!
A problem occurred while processing the request.
Logging ID=0e25126c-9919-4dbe-bf35-b2cc0e9a0f06

Log output:
Jun 07, 2023 9:45:17 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException Caught unhandled exception with ID 0e25126c-9919-4dbe-bf35-b2cc0e9a0f06 org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:276) at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:862) at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:766) Caused: java.lang.IllegalArgumentException: Failed to instantiate class io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher from {"stapler-class":"io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher","$class":"io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher","host":"10.2.0.22","includeUser":"false","credentialsId":"4b5baa64-d620-4c03-88af-30ba63a99ada","":"3","sshHostKeyVerificationStrategy":{"stapler-class":"hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy","$class":"hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy"},"port":"22","javaPath":"","jvmOptions":"","prefixStartSlaveCmd":"","suffixStartSlaveCmd":"","launchTimeoutSeconds":"","maxNumRetries":"","retryWaitTime":"","tcpNoDelay":true,"workDir":""} at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:769) at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:549) at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:544) at hudson.model.Descriptor.bindJSON(Descriptor.java:622) at hudson.model.Descriptor.newInstance(Descriptor.java:592) Caused: java.lang.LinkageError: Failed to instantiate class io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher from {"stapler-class":"io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher","$class":"io.jenkins.docker.connector.DockerComputerSSHConnector$InjectKeySSHLauncher","host":"10.2.0.22","includeUser":"false","credentialsId":"4b5baa64-d620-4c03-88af-30ba63a99ada","":"3","sshHostKeyVerificationStrategy":{"stapler-class":"hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy","$class":"hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy"},"port":"22","javaPath":"","jvmOptions":"","prefixStartSlaveCmd":"","suffixStartSlaveCmd":"","launchTimeoutSeconds":"","maxNumRetries":"","retryWaitTime":"","tcpNoDelay":true,"workDir":""} at hudson.model.Descriptor.newInstance(Descriptor.java:595) at hudson.model.Descriptor$NewInstanceBindInterceptor.instantiate(Descriptor.java:669) at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:853) at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:766) at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:549) at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:875) at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:766) at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:549) at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:544) at hudson.model.Descriptor.bindJSON(Descriptor.java:622) at hudson.model.Descriptor.newInstance(Descriptor.java:592) at hudson.model.ComputerSet.doDoCreateItem(ComputerSet.java:304) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207) at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36) at org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48) at org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) Caused: javax.servlet.ServletException at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:818) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:224) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698) at org.kohsuke.stapler.Stapler.service(Stapler.java:248) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

The agent is reachable from the Jenkins controller, and account can log in on the target machine.

[Aniket-IN]

@vdeyak I'm facing the same issue as well, did you find any workaround? 🤔

[MarkEWaite]

I believe that https://github.com/ash-sxn/GSoC-2023-docker-based-quickstart is injecting an ssh key into an ssh agent, though it is not using the Docker plugin to launch that agent. Are you able to inject the ssh key when not using the Docker plugin launch method?