Double URL encoding issue with Jenkins jobs

[dprangnell]

Issue Summary:

When Spinnaker goes to retrieve an artifact from a jenkins job, orca url encodes the job name and requests the artifact from igor. This works if the job does not have a space in it. If the job name has a space in it, then jenkins already url encodes it, then spinnaker/orca url encodes it again and queries igor, igor then returns a 404.

Environment:

1.32.4

Feature Area:

Executing external jenkins jobs that both:

• Have a space in their name
• Produce non-zero byte artifacts

Description:

I have two example jobs, both produce an artifact build.properties that is greater than 0 bytes.

When I call the jenkins job artifact-test-temporary it works
When I call the jenkins job promote docker image it gets a 404:

Exception ( Get Build Artifacts )
404 Not Found

Steps to Reproduce:

1. Create two jenkins jobs one called promote docker image the other called artifactory-test-temporary. Put this in the job definition/Jenkinsfile:

pipeline {
    agent any

    stages {
        stage('Hello') {
            steps {
               sh "echo -n 'dockerImageTag=exampleTag' > build.properties"
               archiveArtifacts 'build.properties'
            }
        }
    }
}

2. Call the job from Spinnaker, the one with the spaces will fail with a 404.

Additional Details:

I believe this is the problem, orca is blindly encoding the job:

  public Map<String, Object> getBuild(Integer buildNumber, String master, String job) {
    return this.igorFeatureFlagProperties.isJobNameAsQueryParameter()
        ? igorService.getBuildWithJobAsQueryParam(buildNumber, master, encode(job))
        : igorService.getBuild(buildNumber, master, encode(job));
  }

Either remove that or have igor try url decoding the job name more intelligently (or stop trying to get the artifact from jenkins entirely).

The orca logs are telling (I've added some newlines and tabs for readability). See context.buildInfo.url=https://build.example.org/job/Promote%20Docker%20Image/30993/ whereas the request from orca to igor is context.exception.details.url=http://spin-igor.spinnaker:8088/builds/artifacts/30993/build.example.org?job=Promote%2520Docker%2520Image :

2024-04-25 17:48:01.215  INFO 1 --- [       cancel-4] c.n.s.orca.igor.pipeline.CIStage         : [david.prangnell@example.org] Cancelling stage
(
    stageId: 01HWB5R4JEGQY14M4A3TSP5XQX,
    executionId: 01HWB5R4F1PPV7X9MSAEANR9CG context: {
        exception={
            exceptionType=RetrofitError,
            shouldRetry=false,
            details={
                responseBody={
                    "timestamp":"2024-04-25T17:48:01.077+00:00",
                    "status":404,
                    "error":"Not Found",
                    "exception":"com.netflix.spinnaker.kork.web.exceptions.GenericExceptionHandlers$RetrofitErrorWrapper",
                    "message":"404 Not Found",
                    "url":"https://build.example.org/job/Promote%2520Docker%2520Image/30993/api/xml?exclude=/*/action[not(totalCount)]&tree=actions[failCount,skipCount,totalCount,urlName],duration,number,timestamp,result,building,url,fullDisplayName,artifacts[displayPath,fileName,relativePath]"
                },
                kind=HTTP,
                error=Not Found,
                errors=[404 Not Found],
                url=http://spin-igor.spinnaker:8088/builds/artifacts/30993/build.example.org?job=Promote%2520Docker%2520Image,
                rootException=com.netflix.spinnaker.kork.web.exceptions.GenericExceptionHandlers$RetrofitErrorWrapper,
                status=404
            },
            operation=getBuildArtifacts,
            timestamp=1714067281079
        },
        consecutiveErrors=0,
        markUnstableAsSuccessful=true,
        queuedBuild=525764,
        buildInfo={
            duration=170293,
            result=SUCCESS,
            number=30993,
            name=Promote Docker Image,
            fullDisplayName=Promote Docker Image #30993,
            id=Promote Docker Image-30993,
            scm=[{
                sha1=5ca6cf46bc37c7d585fab7fe554f212faf44757c,
                name=master,
                remoteUrl=https://github.com/example/jenkins-global-library.git,
                branch=master
            }],
            building=false,
            url=https://build.example.org/job/Promote%20Docker%20Image/30993/,
            timestamp=1714067102237,
            artifacts=[{
                reference=build.properties,
                fileName=build.properties,
                relativePath=build.properties,
                name=Promote Docker Image,
                displayPath=build.properties,
                type=jenkins/file,
                version=30993,
                decorated=false
            }]
        },
        completeOtherBranchesThenFail=false,
        failPipeline=false,
        job=Promote Docker Image,
        parameters={
            docker_tag=58-a8c867,
            promote_to=production,
            promote_from=dev,
            docker_image=web-extra/pe-spinnakertest
        },
        buildNumber=30993,
        continuePipeline=false,
        master=build.example.org
    }
)

My manual curls from inside clouddriver shows this:

root@spin-clouddriver-85c897b586-8vd8p:/# curl http://spin-igor.spinnaker:8088/builds/artifacts/30993/build.example.org?job=Promote%2520Docker%2520Image
{"timestamp":"2024-04-25T18:03:42.187+00:00","status":404,"error":"Not Found","exception":"com.netflix.spinnaker.kork.web.exceptions.GenericExceptionHandlers$RetrofitErrorWrapper","message":"404 Not Found","url":"https://build.example.org/job/Promote%2520Docker%2520Image/30993/api/xml?exclude=/*/action[not(totalCount)]&tree=actions[failCount,skipCount,totalCount,urlName],duration,number,timestamp,result,building,url,fullDisplayName,artifacts[displayPath,fileName,relativePath]"}root@spin-clouddriver-85c897b586-8vd8p:/# 

root@spin-clouddriver-85c897b586-8vd8p:/# curl http://spin-igor.spinnaker:8088/builds/artifacts/30993/build.example.org?job=Promote%20Docker%20Image
[]

root@spin-clouddriver-85c897b586-8vd8p:/# curl http://spin-igor.spinnaker:8088/builds/artifacts/30993/build.icann.org?job=Promote+Docker+Image
[]

For the really observant, you'll notice another bug, the successful curl returns an empty json string [], but I don't really care about getting the artifact from Jenkins to spinnaker, I just don't want the stage to fail.

This is reflected in the job getting 404's trying to retrieve the artifact:

2024-04-25 04:24:32.682  INFO 1 --- [    scheduler-3] com.netflix.spinnaker.q.QueueProcessor   : [] Received message RunTask(executionType=pipeline, executionId=01HW9QT70AH1PPYD9SHA0BBDWD, application=okdtest, stageId=01HW9QT79SV5J84H6QDWG01VHZ, taskId=5, taskType=class com.netflix.spinnaker.orca.igor.tasks.GetBuildArtifactsTask)
2024-04-25 04:24:32.685  INFO 1 --- [    handlers-19] c.n.spinnaker.orca.igor.IgorService      : [david.prangnell@example.org] ---> HTTP GET http://spin-igor.spinnaker:8088/builds/artifacts/30958/build.example.org?job=Promote%2520Docker%2520Image
2024-04-25 04:24:32.795  INFO 1 --- [    handlers-19] brave.Tracer                             : [david.prangnell@example.org] {"traceId":"78e8aaeec114fe7d","id":"78e8aaeec114fe7d","kind":"CLIENT","name":"GET","timestamp":1714019072685718,"duration":109897,"localEndpoint":{"serviceName":"unknown"},"tags":{"http.method":"GET","http.path":"/builds/artifacts/30958/build.example.org","http.status_code":"404","error":"404"}}
2024-04-25 04:24:32.796  INFO 1 --- [    handlers-19] c.n.spinnaker.orca.igor.IgorService      : [david.prangnell@example.org] <--- HTTP 404 http://spin-igor.spinnaker:8088/builds/artifacts/30958/build.example.org?job=Promote%2520Docker%2520Image (110ms)
2024-04-25 04:24:32.833 ERROR 1 --- [    handlers-19] c.n.s.orca.q.handler.RunTaskHandler      : [david.prangnell@example.org] Error running GetBuildArtifactsTask for pipeline[01HW9QT70AH1PPYD9SHA0BBDWD]
retrofit.RetrofitError: 404 
	at retrofit.RetrofitError.httpError(RetrofitError.java:40)
	at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:388)
	at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)
	at com.sun.proxy.$Proxy182.getArtifactsWithJobAsQueryParam(Unknown Source)
	at com.netflix.spinnaker.orca.igor.BuildService.getArtifacts(BuildService.java:71)
	at com.netflix.spinnaker.orca.igor.tasks.GetBuildArtifactsTask.tryExecute(GetBuildArtifactsTask.java:44)
	at com.netflix.spinnaker.orca.igor.tasks.GetBuildArtifactsTask.tryExecute(GetBuildArtifactsTask.java:35)
	at com.netflix.spinnaker.orca.igor.tasks.RetryableIgorTask.execute(RetryableIgorTask.java:54)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1$1$1.invoke(RunTaskHandler.kt:166)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1$1$1.invoke(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLoggingContext(RunTaskHandler.kt:471)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$withLoggingContext(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1$1.invoke(RunTaskHandler.kt:122)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1$1.invoke(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.AuthenticationAware$sam$java_util_concurrent_Callable$0.call(AuthenticationAware.kt)
	at com.netflix.spinnaker.security.AuthenticatedRequest.lambda$wrapCallableForPrincipal$0(AuthenticatedRequest.java:272)
	at com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:51)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:121)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:290)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:69)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:46)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:86)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:46)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:96)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:75)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:61)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:279)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$withTask(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.run(RunTaskHandler.kt:119)
	at com.netflix.spinnaker.orca.lock.NoOpRunOnLockAcquired.execute(ExternalLock.kt:191)
	at com.netflix.spinnaker.orca.lock.RetriableLock$LockAndRun.get(RetriableLock.java:128)
	at com.netflix.spinnaker.orca.lock.RetriableLock$LockAndRun.get(RetriableLock.java:103)
	at com.netflix.spinnaker.kork.core.RetrySupport.retry(RetrySupport.java:34)
	at com.netflix.spinnaker.orca.lock.RetriableLock.lock(RetriableLock.java:57)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withLocking(RunTaskHandler.kt:296)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:118)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)
	at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)
	at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:90)
	at com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:72)
	at com.netflix.spinnaker.q.QueueProcessor$callback$1$1.run(QueueProcessor.kt:90)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-25 04:24:32.838  INFO 1 --- [    scheduler-4] com.netflix.spinnaker.q.QueueProcessor   : [] Received message CompleteTask(executionType=pipeline, executionId=01HW9QT70AH1PPYD9SHA0BBDWD, application=okdtest, stageId=01HW9QT79SV5J84H6QDWG01VHZ, taskId=5, status=TERMINAL, originalStatus=TERMINAL)

[dprangnell]

I've found that this problem is present regardless of whether the jenkins job produces an artifact or not. Spinnaker calls the igor endpoint for GetBuildArtifactsTask whether there is an artifact or not. The key problem is if the jenkins job has a space name in it.

[christosarvanitis]

@dprangnell as a workaround until the fix is released you can disable in the orca profile:

        feature:
          igor:
            jobNameAsQueryParameter: false

This change however will affect the artifact retrieval if you have jobs that have a forward / (like multibranch Jenkins projects etc)

[dprangnell]

Thanks, disabling the jobNameAsQueryParameter worked great!