HOWTO:Share Jenkins Artifacts

Why this page

In setting up the Jenkins jobs to test forward and backward compatibility, I ran into a bunch of gotchas. While none were showstoppers, it took a while to work through each little issue like "death by a thousand paper cuts". Now that the jobs are up and running, here is a brain dump of the process. There are various question/answer pages I found that helped a bit with some of this along the way, like StackOverflow, but none of them were quite what I wanted to do so I had to cherry pick tidbits of answers.

This page will use the forward/backward compatibility as an example, although it should apply to nearly any jobs needing to access pieces from another Jenkins job. The rest of this page assumes you are using the Copy Artifacts Plugin and already have that installed and available in Jenkins. The Copy Artifact is something added to the Build actions.

General Jenkins items

These are things that were obvious after the fact, but I had to hit them first before they became obvious. They are not specifically related to Copy Artifacts, but I hit them in the process. These are the sorts of things that move you from User to Power User!

• Exit status - If you are using Execute shell as part of the build process, which all our jobs do, remember that any non-zero exit status from anything causes a Jenkins failure. For the compatibility project this tripped me up because I needed to determine the table type of the random run. I wanted to do that with:

grep data_source RUNDIR/CONFIG | grep -q file
if test $? -eq 0; then
    uri="file:wt"
else
    uri="table:wt"
fi

But because of the exit status problem, I had to do the following. Not a big deal nor a lot of extra code, but an annoyance:

isfile=`grep datasource RUNDIR/CONFIG | grep -c file || exit 0`
if test $isfile -ne 0; then
    uri="file:wt"
else
    uri="table:wt"
fi

• Need to add copying in correct order relative to the shell commands. Most of our Jenkins jobs have two Execute Shell command sections, one for configure and make, and the second to run whatever is specific for this job/test. The Copy Artifact from another Project piece is also a Build Action. But in order to add it into an existing project, you need to delete the second shell command section (assuming you want it in between there). If you want the copy to occur first, you have to delete both. That is, you can only append a new build action, not insert it in between existing ones.
• Everything is relative to the top level workspace. So if you add a Build Action to copy into DEV that goes into <top_level>/DEV. If you want it local to the build, tell it to copy into build_posix/DEV in the Build Action and then the shell commands, likely already in the build_posix directory, can use ./DEV. This is obvious to me in hindsight, but I had to pay attention to what I was doing at the Jenkins level versus the shell level.
• Chicken and Egg Problem. Happily I thought of this before I started and the solution is easy. This only applies if your new jobs needing to copy have circular dependencies. My work for compatibility tested both forward and backward compatibility so every newly added job was testing an executable from another job. You need a successful build without the copying requirement to start the process so that there exists something for another job to copy. Then once it is working between two jobs, you can add in the piece to close the circle. I just left all compatibility components out of the develop job until all old releases were working.

Things Relating to Using Artifacts

• If you are planning to copy an executable from another project, you need to restrict all related jobs to run on the same machine so that they can execute the same binary between jobs. This is only half obvious. The artifacts are stored, as far as I can tell, in a central location. I assumed the copy would occur on the slave machine, and would copy from the last successful build on this particular slave. I still do not really have a mental image of where things need to be between the Jenkins daemon on the master versus the jobs on the slave.
• Pay attention to file permissions if you're using executables. In particular, there is a bug in Copy Artifacts where it loses execute permissions during normal copies. This is discussed in Jenkins-9741. My workaround is just to have the shell command run chmod a+x before I use the executable. There appears to be a fix if you flatten what you copy but I did not pursue that nor even figure out what flatten means.
• It preserves the entire directory structure on the copy. I assumed Unix-style copy, but if you tell it to copy build_posix/wt to DEV, then you get DEV/build_posix/wt, not DEV/wt.
• I naively thought after I installed the plugin that all I had to do was add the action and tell it what file, from what project and where to put it and that would be it! Hahaha No. There are three separate steps to actually copying an artifact successfully from one Jenkins job to another.
  • In the source-job, in the Job Notifications section you need to check the Permission to Copy Artifact box and specify what other projects have permission to copy from this job.
  • In the source-job, in the Post-build Actions you need to add a new action to Archive the Artifacts and specify what file(s) this job should archive.
  • Finally, in the destination-job, in the Build section you need to add the expected new action to Copy Artifacts from another project and fill in the where and what.