Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shell Commands fail to ever start #940

Open
ChtrMshipper opened this issue Feb 23, 2023 · 1 comment
Open

Shell Commands fail to ever start #940

ChtrMshipper opened this issue Feb 23, 2023 · 1 comment
Labels
bug An issue reporting a bug or a PR fixing one.

Comments

@ChtrMshipper
Copy link

Jenkins and plugins versions report

Environment 
OS: Linux - 3.10.0-1160.49.1.el7.x86_64
Java: 11.0.17 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
PrioritySorter:4.1.0
active-directory:2.30
ant:481.v7b_09e538fcca
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
artifactdeployer:1.3
authentication-tokens:1.4
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-timeout:1.28
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
cloudbees-folder:6.800.v71307ca_b_986b
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
data-tables-api:1.12.1-4
display-url-api:2.3.7
docker-commons:1.21
docker-compose-build-step:1.0
docker-java-api:3.2.13-68.va_875df25a_b_45
docker-plugin:1.3.0
docker-slaves:1.0.7
docker-workflow:563.vd5d2e5c4007f
durable-task:504.vb10d1ae5ba2f
echarts-api:5.4.0-1
email-ext:2.94
external-monitor-job:203.v683c09d993b_9
font-awesome-api:6.2.1-1
git:5.0.0
git-client:4.1.0
git-server:99.va_0826a_b_cdfa_d
github:1.37.0
github-api:1.303-400.v35c2d8258028
github-branch-source:1701.v00cc8184df93
gradle:2.3
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.2-319.v37853346a_229
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.16-2
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1177.v90374a_ef4d09
ldap:659.v8ca_b_a_fe79fa_d
lockable-resources:1123.v4002ee23c671
mailer:448.v5b_97805e3767
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
okhttp-api:4.10.0-125.v3593b_a_f8c97b_
pam-auth:1.10
pipeline-build-step:2.18.1
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.31
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.31
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
resource-disposer:0.21
scm-api:631.v9143df5b_e4a_a
script-security:1229.v4880b_b_e905a_6
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
subversion:2.17.0
throttle-concurrents:2.11
timestamper:1.22
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:1010.vf7a_b_98e847c1
workflow-cps:3618.v13db_a_21f0fcf
workflow-durable-task-step:1234.v019404b_3832a
workflow-job:1268.v6eb_e2ee1a_85a
workflow-multibranch:733.v109046189126
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44

What Operating System are you using (both controller, and any agents involved in the problem)?

Centos7

Reproduction steps

Jenkins pipeline jobs fail to spawn shell commands if they were previously in a suspended state when using a docker multibranch pipeline.
Please note, this only happens when the job enters a "Suspended" state and keeps building any ways.

Not sure how to provide exact details here as I am still learning a lot about jenkins, but here is the problem.

Jobs that enter a suspended state fail to execute shell commands inside the container.

From what I can tell there is some sort of IO failure with the spawning of the container.

Jenkins logs here Logs here:


Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@57bfaad6 to DockerClientParameters{dockerUri='tcp://localhost:7504', credentialsId='null', readTimeoutInMsOrNull=100000, connectTimeoutInMsOrNull=100000}
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedAgent
Provisioning 'teldev/centos7-builder:latest' number 1 (of 2) on 'docker-uat1'; Total containers: 0 (of 2)
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Will provision 'teldev/centos7-builder:latest', for label: 'null', in cloud: 'docker-uat1'
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for image "teldev/centos7-builder:latest"
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Started container ID 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): Feb 23, 2023 12:57:54 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Using /var/lib/jenkins/agent.log as an agent error log destination; output log will not be generated
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): channel started
Feb 23, 2023 12:57:56 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Feb 23, 2023 12:58:02 AM INFO hudson.slaves.NodeProvisioner update
Image of teldev/centos7-builder:latest provisioning successfully completed. We have now 6 computer(s)
Feb 23, 2023 12:58:44 AM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect LTWC Dev Server 1
Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'centos7-builder-jekins-slave1-002uuz9748r51'.
Feb 23, 2023 12:59:22 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel centos7-builder-jekins-slave1-002uuz9748r51
java.net.SocketException: Socket closed
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:51)
	at io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:33)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'centos7-builder-jekins-slave1-002uuz9748r51'.

Build log here:

[2023-02-23T00:57:51.862Z] Started by user Shipper, Michael
[2023-02-23T00:57:51.887Z] [Pipeline] Start of Pipeline
[2023-02-23T00:57:51.924Z] [Pipeline] library
[2023-02-23T00:57:51.933Z] Loading library teldev_build_tools@master
[2023-02-23T00:57:51.934Z] No credentials specified
[2023-02-23T00:57:51.934Z]  > /bin/git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@libs/b874e508a693557e2c5b7ccce06797e7c000ab4bef3a2f21cb534bf208455a8e/.git # timeout=10
[2023-02-23T00:57:51.941Z] Fetching changes from the remote Git repository
[2023-02-23T00:57:51.942Z]  > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git # timeout=10
[2023-02-23T00:57:51.948Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git
[2023-02-23T00:57:51.948Z]  > /bin/git --version # timeout=10
[2023-02-23T00:57:51.955Z]  > git --version # 'git version 1.8.3.1'
[2023-02-23T00:57:51.955Z]  > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:57:52.183Z]  > /bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10
[2023-02-23T00:57:52.189Z] Checking out Revision 09d77cf0b427b671ead030a77dab2cb6b03a7b70 (refs/remotes/origin/master)
[2023-02-23T00:57:52.189Z]  > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:57:52.205Z]  > /bin/git checkout -f 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:52.212Z] Commit message: "fixed bad var name"
[2023-02-23T00:57:52.213Z]  > /bin/git rev-list --no-walk 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:53.084Z] [Pipeline] node
[2023-02-23T00:57:57.391Z] Running on centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 in /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools
[2023-02-23T00:57:57.394Z] [Pipeline] {
[2023-02-23T00:57:57.477Z] [Pipeline] isUnix
[2023-02-23T00:57:57.483Z] [Pipeline] withEnv
[2023-02-23T00:57:57.485Z] [Pipeline] {
[2023-02-23T00:57:57.502Z] [Pipeline] sh
[2023-02-23T00:58:01.164Z] + docker inspect -f . teldev/debian-min-build
[2023-02-23T00:58:01.164Z] .
[2023-02-23T00:58:01.681Z] [Pipeline] }
[2023-02-23T00:58:01.702Z] [Pipeline] // withEnv
[2023-02-23T00:58:01.711Z] [Pipeline] withDockerContainer
[2023-02-23T00:58:02.608Z] centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 seems to be running inside container 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:03.287Z] $ docker run -t -d -u 798:798 --tmpfs=/var/lib/jenkins/.gnupg:rw,exec,mode=0700,size=10m,uid=798,gid=798 -w /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:rw,z -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** teldev/debian-min-build cat
[2023-02-23T00:58:03.781Z] $ docker top 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719 -eo pid,comm
[2023-02-23T00:58:03.983Z] [Pipeline] {
[2023-02-23T00:58:04.036Z] [Pipeline] withEnv
[2023-02-23T00:58:04.042Z] [Pipeline] {
[2023-02-23T00:58:04.071Z] [Pipeline] stage
[2023-02-23T00:58:04.076Z] [Pipeline] { (clone git branch)
[2023-02-23T00:58:04.115Z] [Pipeline] checkout
[2023-02-23T00:58:14.600Z] No credentials specified
[2023-02-23T00:58:14.600Z] Warning: JENKINS-30600: special launcher org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@20fbff43; decorates RemoteLauncher[hudson.remoting.Channel@193b6b15:centos7-builder-jekins-slave1-002uuz9748r51] will be ignored (a typical symptom is the Git executable not being run inside a designated container)
[2023-02-23T00:58:14.643Z] Cloning the remote Git repository
[2023-02-23T00:58:14.734Z] Cloning repository ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.845Z]  > /bin/git init /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools # timeout=10
[2023-02-23T00:58:14.981Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.981Z]  > /bin/git --version # timeout=10
[2023-02-23T00:58:14.986Z]  > git --version # 'git version 1.8.3.1'
[2023-02-23T00:58:14.986Z]  > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:27.602Z]  > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git # timeout=10
[2023-02-23T00:58:27.608Z]  > /bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:30.912Z] Avoid second fetch
[2023-02-23T00:58:30.962Z] Checking out Revision 4b93166c6d7431c35d6a2d217016b979f9b9fc10 (refs/remotes/origin/release/dev)
[2023-02-23T00:58:30.953Z]  > /bin/git rev-parse refs/remotes/origin/release/dev^{commit} # timeout=10
[2023-02-23T00:58:31.047Z]  > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:58:31.059Z]  > /bin/git checkout -f 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.694Z] Commit message: "rolling in priority fix"
[2023-02-23T00:58:32.738Z]  > /bin/git rev-list --no-walk 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.889Z] [Pipeline] }
[2023-02-23T00:58:32.912Z] [Pipeline] // stage
[2023-02-23T00:58:32.921Z] [Pipeline] stage
[2023-02-23T00:58:32.922Z] [Pipeline] { (Build)
[2023-02-23T00:58:32.960Z] [Pipeline] sh
[2023-02-23T00:59:00.301Z] Aborted by Shipper, Michael
[2023-02-23T00:59:00.344Z] Sending interrupt signal to process
[2023-02-23T00:59:15.303Z] Click here to forcibly terminate running steps
[2023-02-23T00:59:20.345Z] After 20s process did not stop
[2023-02-23T00:59:20.352Z] [Pipeline] }
[2023-02-23T00:59:20.378Z] [Pipeline] // stage
[2023-02-23T00:59:20.388Z] [Pipeline] stage
[2023-02-23T00:59:20.389Z] [Pipeline] { (install local copy of charter-rpm-tools)
[2023-02-23T00:59:20.403Z] Stage "install local copy of charter-rpm-tools" skipped due to earlier failure(s)
[2023-02-23T00:59:20.406Z] [Pipeline] }
[2023-02-23T00:59:20.437Z] [Pipeline] // stage
[2023-02-23T00:59:20.447Z] [Pipeline] stage
[2023-02-23T00:59:20.449Z] [Pipeline] { (push package)
[2023-02-23T00:59:20.463Z] Stage "push package" skipped due to earlier failure(s)
[2023-02-23T00:59:20.466Z] [Pipeline] }
[2023-02-23T00:59:20.496Z] [Pipeline] // stage
[2023-02-23T00:59:20.506Z] [Pipeline] stage
[2023-02-23T00:59:20.508Z] [Pipeline] { (Declarative: Post Actions)
[2023-02-23T00:59:20.528Z] [Pipeline] cleanWs
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deleting project workspace...
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deferred wipeout is used...
[2023-02-23T00:59:20.677Z] [WS-CLEANUP] done
[2023-02-23T00:59:20.682Z] [Pipeline] }
[2023-02-23T00:59:20.706Z] [Pipeline] // stage
[2023-02-23T00:59:20.711Z] [Pipeline] }
[2023-02-23T00:59:20.734Z] [Pipeline] // withEnv
[2023-02-23T00:59:20.740Z] [Pipeline] }
[2023-02-23T00:59:20.746Z] $ docker stop --time=1 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.301Z] $ docker rm -f --volumes 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.446Z] [Pipeline] // withDockerContainer
[2023-02-23T00:59:22.452Z] [Pipeline] }
[2023-02-23T00:59:22.485Z] [Pipeline] // node
[2023-02-23T00:59:22.495Z] [Pipeline] End of Pipeline
[2023-02-23T00:59:22.504Z] Finished: ABORTED

Expected Results

I expect the shell commands to run

Actual Results

No shell command is ever run.

Anything else?

If I hop into the container jenkins spawned, the docker is not running any commands.

just "cat" and "/bin/bash"
Waiting for input.
@ChtrMshipper ChtrMshipper added the bug An issue reporting a bug or a PR fixing one. label Feb 23, 2023
@ChtrMshipper
Copy link
Author

As a note, the command failed to spawn so I terminated the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug An issue reporting a bug or a PR fixing one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ChtrMshipper