Retry get Bucket on failure if possible

[jnuxadrian]

GoogleUtils.getStorageBucket can throw StorageException
upon failure. This commit wrapps the call with a ExponentialBackOff
logic.

[dtretyakov]

@jnuxadrian, thanks for contribution. Could you please share some details and the stacktrace of the problem which fixes this PR?

[jnuxadrian]

@dtretyakov Do you know of a way to enable only debug logs related to this plugin. Our TeamCity deployment is so massive, we in the past tried this and logs were rotated every 5s.

[jnuxadrian]

[2018-11-07 02:26:50,175] WARN - blish.GoogleArtifactsPublisher - Failed to publish files: com.google.cloud.storage.StorageException: Error getting access token for service account: (enable debug to see stacktrace)
[2018-11-07 02:26:50,176] WARN - jetbrains.buildServer.AGENT - Failed to publish artifacts: jetbrains.buildServer.agent.ArtifactPublishingFailedException: Failed to publish files: Error getting access token for service account: (enable debug to see stacktrace)

[jnuxadrian]

We are using the GoogleRegularFileUploader.kt

Given that there is two calls that could throw com.google.cloud.storage.StorageException

1. bucket.create
2. GoogleUtils.getStorageBucket

We rule out the first one cause we could not find in the build logs anything related to this retry message https://github.com/JetBrains/teamcity-google-storage/blob/master/google-storage-agent/src/main/kotlin/jetbrains/buildServer/artifacts/google/publish/GoogleRegularFileUploader.kt#L53

The second one instead is catch here https://github.com/JetBrains/teamcity-google-storage/blob/master/google-storage-agent/src/main/kotlin/jetbrains/buildServer/artifacts/google/publish/GoogleArtifactsPublisher.kt#L60 and this message match with what we found in the posted logs.

[dtretyakov]

@jnuxadrian, to see debug logs you need to configure logging preset on the build agent. For that modify the %buildAgent%/conf/teamcity-agent-log4j.xml file where you need to uncomment DEBUG priority in the corresponding section like that:

<category name="jetbrains.buildServer">
    <priority value="DEBUG"/>
    <appender-ref ref="ROLL"/>
  </category>

Then in the teamcity-agent.log file will be stored the complete stacktrace of this error.

AFAIK GoogleUtils.getStorageBucket could not throw the mentioned IOException "Error getting access token for service account:", so I suspect that something unpredictable has happend during access token refresh during files uploading, but it's unclear what was the cause of that: https://github.com/googleapis/google-auth-library-java/pull/206/files

If you could enable debug logging on the build agents and could reproduce it, it will help us to investigate the problem.

Another possible way is to enable signed URL option and check whether it prevents such problems.

[jnuxadrian]

@dtretyakov Thanks for the log suggestion. Still I think is worthy to consider retry that call to google, it will make the plugin more reliable, and you already have that in place for creating the object.

If you look at the gist of logs provided you will see that StorageException is not handled by GoogleRegularFileUploader.

Don't you think that adding a retry to an external service call would make sense, or do you think the problem is happening in some other place ?