Image pull issue at high invocation rates

[aditya2803]

Describe the bug
I am running a 3 node cluster (1 master and 2 worker nodes). At high invocation rates, I get the following error on one of the worker nodes, for all requests that get sent to that node, and as a result more than 95% requests fail on that node:

time="2022-12-24T07:43:53.953774832-06:00" level=error msg="coordinator failed to start VM" error="Failed to get/pull image: context deadline exceeded" image="[ghcr.io/aditya2803/json:v23](http://ghcr.io/aditya2803/json:v23)" vmID=715
time="2022-12-24T07:43:53.953851537-06:00" level=error msg="failed to start VM" error="Failed to get/pull image: context deadline exceeded"

At lower invocation rates, and even with single-node clusters, I do not see this error. I am using images from the GitHub container registry. How can I solve this issue?

To Reproduce
Set up a 3 node cluster, and try deploying a function at very high invocation rates. This way, there is a lot of demand for pulling images simultaneously. On one of the nodes, you would start seeing errors like above.

Expected behavior
There should be no error regarding the pulling of image from the registry. Also, I believe that once the images are accessed for the first time, they get stored locally. So I am not sure why this is happening.

Logs

The vhive.stdout logs get filled with the below message

time="2022-12-24T07:43:53.953774832-06:00" level=error msg="coordinator failed to start VM" error="Failed to get/pull image: context deadline exceeded" image="[ghcr.io/aditya2803/json:v23](http://ghcr.io/aditya2803/json:v23)" vmID=715
time="2022-12-24T07:43:53.953851537-06:00" level=error msg="failed to start VM" error="Failed to get/pull image: context deadline exceeded"

Notes
I have tested this on bare-metal setups and virtualised setups.

[ustiugov]

hi @aditya2803, this is expected behavior. Aproduction system would not have pull rates as most of the image layers are cached. To address this issue you need to mimic a long-running system, you should modify your experiment driver scripts to pre-pull the required images.

[aditya2803]

Thanks for the quick reply. If I understood you correctly, I should move the "getImage(ctx context.Context, imageName string) (*containerd.Image, error)" function outside of the process of starting the new microVM? So before each invocation, I need to getImage first, and then start the invocation?

[ustiugov]

what you can do is:

• invoke all functions sequentially in a pre-phase
• apply the load and measure response times

it's not the vHive infra problem, it's the experiment driver problem.

[aditya2803]

Thanks for your reply. However, given a multi-node cluster, I suspect that complete serialisation of requests would not be possible, right? For instance there would always be chances of some concurrent requests from worker nodes. What could be done in this case?

[ustiugov]

you need to distinguish modeling the "systems warmup" and "measurement phases". The warmup can be done in any convenient way whereas the measurement phase needs to model the production environment. So far, if I understood correctly, your problem is in the warmup phase, which is not the problem for vHive. However, if there are such problems in the measurement phase, let's discuss.

[aditya2803]

So I think by "systems warmup" you mean all activities done prior to the actual function invocation. If this is true, I suppose image pull too would be part of this phase. Thus, I should try to pre-pull the images before the function invocation? (Sort of removing them from the critical path ?)