Global caching of signed artifacts

[tschulte]

If an application has many dependencies, that do not change often, a huge amount of time on each clean build is wasted signing the same artifacts again and again with the same parameters.

There should be an option to cache already signed artifacts. This may be using a maven/ivy repository. This is somewhat related to #22, in that the cached artifact has the same constraints.

[rafik777]

+1 Really desired feature.
As an alternative: What would you think about allowing parallel signing?

[tschulte]

I am working on this feature at the moment. But parallel signing is possible already. Just enable parallel builds using the gradle parameter "--parallel". Or by defining the property org.gradle parallel=true in either your home-gradle.properties or in the project local gradle properties.

[rafik777]

I'm happy to hear that. I can't wait for this feature!

Off-topic about paralleling:
I'm new in gradle, but according to my undestanding gradle --parallel option, it only works for paralelling mulitple projects building, not paralelling tasks in the same project.
Correct me if I'm wrong.
From my experience I can say that setting parallel option has no influence on summary time of signing jars, even on my multi-core machine.

[tschulte]

You are right, the gradle --parallel does build projects in parallel, but builds one project sequentially. There is however an incubating switch to enable parallel building of tasks within a project (-Dorg.gradle.parallel.intra=true). This allows to execute multiple tasks of one project in parallel, as long as the task uses the annotation Parallelizable.

This is, however, not what the jnlp-plugin uses. The signJars-task uses gpars to sign multiple jars in parallel (https://github.com/tschulte/gradle-jnlp-plugin/blob/0.2.1/gradle-jnlp-plugin/src/main/groovy/de/gliderpilot/gradle/jnlp/SignJarsTask.groovy#L41).

[rafik777]

Thank you for the clarification.
I see indeed (in logs) many signjar commands are executed when I set --parallel (--max-workers=8 which is used to compute threadCount)
But after temporary peek of CPU on my computer (~5-10 sec), next few minutes (~10 minutes) gradle spends on ...nothing: CPU - 1%, memory - normal, disk - low, net - low.

---- After investigation ----
Ok, I know the answer!

• Parallel jar signing works very well (super fast) when I don't use signjar parameter: tsaurl - URL for a timestamp authority for timestamped JAR files
• Freezing on signing when I use tsaurl is caused by /dev/random (linux special file - a blocking pseudorandom number generator) which is used by java (signjar) process.
  More details here (good explanation):
  http://stackoverflow.com/questions/23214082/ant-signjar-task-takes-too-long-to-timestamp

So, I tried to use non blocking alternative: /dev/urandom and it seems to work excellent:

GParsPool.withPool(8) {
  ...
  AntBuilder ant = project.createAntBuilder()
  ant.signjar(jar: jarToSign, alias: keystoreAlias, storepass: keystoreStorepass, keypass: keystoreKeypass, keystore: keystore, tsaurl: 'http://tsa.starfieldtech.com') {
      sysproperty(key: 'java.security.egd', value:'file:/dev/./urandom')
  }
}

But, as you can see I needed to add sysproperty nested parameter to ant signjar task.
I can't achieve this with your present version of jnlp plugin. Now you only allow to pass signJarParams flat map.
Is there a chance you could allow passing nested parameters or overriding singjar task in other ways?

This solution will speed up jar siging significantly on our jenkins server.

[tschulte]

We are using haveged (http://www.irisa.fr/caps/projects/hipsor/) on our CI-Server to ensure /dev/random is always filled with entropy. Since then our builds never stalled.

[rafik777]

Thank you for advice.
After installing haveged, total time of parallel signing was reduced from 10 minutes to 1 minute!
Cool! Thank you again and keep up the good work!

[rafik777]

Is there any progress on this issue?
This is really desired feature for me.

[tschulte]

I must confess I have not worked on this particular issue in some time. On a branch I did try to break up the sign jars task into one task per jar instead of one big task doing all the signing. The reasoning behind this was to allow gradle to do all the heavy lifting -- both parallelization (using the new incubating intra-project parallelization support) and caching (for this I was hoping for https://discuss.gradle.org/t/distributed-cache/101 to be implemented).

But I don't know if this route is the way to go. Dynamically creating tasks depending on the amount of dependency might be a bad idea, because the tasks are created at configuration time, and the dependencies should not be resolved at that time, but without resolution it is not possible to create a task per dependency.

Thinks are pretty busy at the moment for me, so I cannot guarantee I will have much time for this, but at least I pushed this ticket up on my TODO list.

[ChristianCiach]

This probably comes too late for most people, but I've hacked something together:

def rpmTask = tasks.register('rpm', Rpm) {
  dependsOn 'generateJnlp'
    
  packageName project.name.toLowerCase()
  into('/var/www/myproduct') {
    from fileTree(dir: "${buildDir}/jnlp")
    from fileTree(dir: 'webstart')  
  }
  into('/var/www/myproduct/lib') {
    from fileTree(dir: "${buildDir}/signed-jars").files
  }
}

configurations.jnlp.incoming.files.each { file ->
  def signConf = configurations.detachedConfiguration(dependencies.create(files(file)))
  def signTask = tasks.register('sign-' + file.name, de.gliderpilot.gradle.jnlp.SignJarsTask) {
    duplicatesStrategy = 'EXCLUDE'
    from = signConf
    into = project.file("${buildDir}/signed-jars/${file.name}")
    
    // Let's only cache artifacts located inside the gradle cache. These are mostly external dependencies.
    outputs.cacheIf { file.toPath().startsWith gradle.gradleUserHomeDir.toPath() }
    
    doFirst {
      // Hack for AbstractCopyJarsTask.newName(..) to find the corresponding artifact.
      from = configurations.jnlp
    }
  }
  rpmTask.configure { dependsOn signTask } 
}

The idea is to create one task for each file to be signed. This way the Gradle cache can be used to cache the signed jar files. We are using the official build cache node for remote caching.

Because the JNLP-plugin declares an output-folder (not a file), every signed file must be written to its own folder for the caching to work. You have to keep this in mind when building your distribution. As an example you can see my RPM-task above that shows how to flatten the fileTree into a list of files.

The downside of this approach is that this completely breaks any kind of parallelism. The JNLP-plugin usually uses all workers to sign multiple jar files in parallel, but this doesn't work anymore if the plugin is called for each file separately. Unfortunately, Gradle cannot execute multiple tasks of the same project in parallel, even if they are independent from another. This makes my hack above almost useless, because the build now takes forever if there are uncached artifacts. Bummer.