PackageTar actions used by assemble rules don't hit the RBE cache

[lolski]

Grakn Labs uses the assemble rules heavily and the fact that PackageTar (which is invoked by the assemble rules) does not get cached in RBE causes the CI to run slower than it should.

Effectively, at the time this issue is created, running bazel build //... would take 10-15 minutes even with RBE on. Ideally, bazel build //... should run within, say 2-3 minutes.

Follow the Discord conversation with BuildBuddy regarding the fact that PackageTar is the culprit here: https://discordapp.com/channels/665254494820368395/730453118713593869/746043881489170472

Ganesh (Grakn Labs):
we've got a job that does bazel build //... --config=rbe. we whould expect subsequent run of that job would be quick, however they're still taking 10-14 minutes. do you know why is this? https://app.circleci.com/pipelines/github/graknlabs/grakn/1486/workflows/de640472-0ce1-4290-8c19-72a8ee633d29/jobs/33137

Siggi (BuildBuddy):
it looks like the biggest contributing factor (90%+ of the time) here is that these bazel-distribution rules (https://github.com/graknlabs/bazel-distribution) don't ever seem to ever hit cache, so all of the tar packaging and bundling happens on every execution - even if it's fully cached

you can reproduce this locally when taking buildbuddy completely out of the picture by running a bazel build with a disk cache, then clean, then running the build again - you'll see that all of the PackageTar actions don't hit cache and are executed every time

when executed remotely, there's additional overhead associated with uploading and downloading these tar files and docker images to and from the cache as they're pretty large

you can debug the cache misses with this tutorial
https://docs.bazel.build/versions/3.4.0/remote-execution-caching-debug.html#troubleshooting-cache-hits