Support for nvidia-container-toolkit and docker 19.03

[haggalin]

From nvidia-container-toolkit github page:

Note that with the release of Docker 19.03, usage of nvidia-docker2 packages are deprecated since NVIDIA GPUs are now natively supported as devices in the Docker runtime.

I can't upgrade to docker 19.03 with the new nvidia-container-toolkit because adding gpus to the container is not supported from the portainer UI with this new toolkit.

With the new nvidia-container-toolkit the way to run containers with gpu access is:
docker run --gpus all nvidia/cuda:9.0-base nvidia-smi

With nvidia-docker2 it used to be:
docker run --runtime=nvidia nvidia/cuda:9.0-base nvidia-smi
which was easy to do from the portainer UI under the "Runtime & Resources" tab.

So the feature request is to support --gpus options from the portainer UI with the new nvidia-container-toolkit and docker 19.03.

An idea is to have it available under "Runtime & Resources" -> "Resources" tab where you can already configure other resource options.

Here are some examples of using the --gpus option taken from the nvidia-container-toolkit github page.

#### Test nvidia-smi with the latest official CUDA image
$ docker run --gpus all nvidia/cuda:9.0-base nvidia-smi

# Start a GPU enabled container on two GPUs
$ docker run --gpus 2 nvidia/cuda:9.0-base nvidia-smi

# Starting a GPU enabled container on specific GPUs
$ docker run --gpus '"device=1,2"' nvidia/cuda:9.0-base nvidia-smi
$ docker run --gpus '"device=UUID-ABCDEF,1"' nvidia/cuda:9.0-base nvidia-smi

# Specifying a capability (graphics, compute, ...) for my container
# Note this is rarely if ever used this way
$ docker run --gpus all,capabilities=utility nvidia/cuda:9.0-base nvidia-smi

Edit: There is a workaround which is listed at the bottom of the nvidia-container-toolkit github page s.t. using --runtime option still works. They note that this won't be supported in the future.

[zocker-160]

the --runtime nvidia option does only work, if you have nvidia-docker2 installed, but this package is deprecated and won't be updated in the future

so I second this request, GPUs won't be usable with Portainer otherwise

[romansavrulin]

Hello! Any news on this?

[estimadarocha]

Hello! Any news on this?

any update?

[mimxrt]

Are there any plans on implementing this at all? Please let us know - it is a crucial feature and the deprecated nvidia-docker2 is not going to work forever. We might have to look into alternatives. Quote:

Note that with the release of Docker 19.03, usage of nvidia-docker2 packages are deprecated since NVIDIA GPUs are now natively supported as devices in the Docker runtime.

(Source: https://github.com/NVIDIA/nvidia-docker)

[ghost]

Hey there everyone, in response to the interest on this I have this assigned to me as a normal priority task. Once I finish work on my current high priority task this week I will focus on this 👍

[piwi3910]

any news on this?

[georgechang]

@RonB I believe the nvidia-container-runtime is still based on the nvidia-docker2 packages which is what was mentioned as being deprecated. What's needed for the new native integration is instead of setting nvidia as the runtime is to add flags for --gpus.

[RonB]

@georgechang ok sorry, was just happy that I got it working. And I am definately not an expert.
Despite being deprecated, is there a reason not to use this, seems to work like a charm.

[KeenJelly]

Any update?

[ncresswell]

Not scheduled for development until later this year (post July). From: KeenJelly <notifications@github.com> Sent: Tuesday, 14 April 2020 7:43 PM To: portainer/portainer <portainer@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [portainer/portainer] Support for nvidia-container-toolkit and docker 19.03 (#3143) Any update? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_3143-23issuecomment-2D613279526&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=j6xlBrq94UENMgmA5smhw6QuvUMKf5OWsWs0pZds4Gs&s=o4R8NiSu_dqR0DdB9XkSGXL5A8XLwTpLIZNMrNFimp0&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFQ2XFI6GDRNHNEKKMTQVKTRMQHZNANCNFSM4IUZC3LA&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=j6xlBrq94UENMgmA5smhw6QuvUMKf5OWsWs0pZds4Gs&s=s3hoCUBlgAUZJFPRs2WjPwmrFs_kk6CEwUeHXQK2JSM&e=>.

[ne0ark]

How are you guys handling this outside of portainer?

[zocker-160]

@ne0ark actually pretty simple, all you have to do is to add the --gpus all argument to your docker run command

[ne0ark]

@ne0ark actually pretty simple, all you have to do is add the --gpus all argument to your docker run command

Thanks I just got it installed with Nvidia runtime. It’s much better route.

[mimxrt]

@ne0ark Do you mean nvidia-docker2? How is it a better route? It works well but it will not be supported in the future.

Frankly, I don't understand why there is no option to manually add command line parameters to docker run commands in Portainer - it would be the most trivial solution. Anyway, just make sure you have a backup plan to using Portainer once nvidia-docker2 stops working!

[ncresswell]

This question on command line parameters has been asked dozens of times. We are talking to the docker API, not using the cli, so its not as simple as passing a cli flag through; for each cli option you would want to add, we must parse and determine the correct api syntax to send instead. Thats not trivial. We wont be adding the ability to specify random cli options anytime soon sorry.

[deviantony]

Required changes

Container-creation view

Introduce a new input to support the GPUs option in the Runtime & Resources tab

Container-edition view

Support the same input as the one in the container creation view.

Container-details view

Introduce a new entry in the Container details showing the value associated to GPUs only if it was set when the container was created.

[chiptus]

Implementation plan

1. Add new fields to “create container form” and send the info to docker
2. Parse and Add new fields to “edit container form” and send the info to docker
3. Parse and show it in “container info”

The problem will be translating from portainer to docker API.

Docker API v1.40+ expects a field called “DeviceRequests” which is an array of objects. Each object has the shape of:

{
    Driver: "nvidia",
    Count: -1,
    DeviceIDs: ["0", "1", "GPU-fef8089b-4820-abfc-e83e-94318197576e"],
    Capabilities: [["gpu", "nvidia", "compute"]],
    Options: { property1: "string", property2: "string" },
  }

I’m not 100% sure how it needs to be filled. I expect that Options can be an empty object (or nil), caps should stay the same and DeviceIDs can be a list of ids or an array with one slot which is set to “all” or “none”.

If that’s the case then we can have a switch that enables this feature (otherwise we send DeviceRequests as an empty array. if enabled the user can switch between a list of device ids, or just all.

This is only supported from v1.40, so we should hide it for earlier versions

How can we test this? @deviantony @yi-portainer

[ncresswell]

Dont forget, we also need to support the use of gpu’s in kube..

[deviantony]

Feature request for the equivalent in Kubernetes: #4640

[estimadarocha]

is there any trick to see gpus on Container-edition view or Container-creation view?

[deviantony]

@estimadarocha we have a work in progress on this via #4791

We're currently reviewing it but if you want to have a look at it you can use the following development/preview image: portainerci/portainer:pr4791

[estimadarocha]

I will try and see how it looks like... Thanks @deviantony

[deviantony]

To everyone following this item, we have a first piece of work available for this via #4791

We'd be happy to get some feedback on it for those of you that can give it a go (this is a development build and should not be used in production environments): portainerci/portainer:pr4791

[bobarune]

Moved my plex install over to docker and decided to use Portainer to manage it. Once I got everything working with just CPU transcoding, I decided to install the nvidia drivers and the nvidia docker container toolkit to re-enable HW transcoding.

To support that, I moved my portainer docker instance to portainerci/portainer:pr4791 and wanted to report my findings. I was originally on the latest of 2.1.1. I only found a few issues, but it appears to be working. Creating the plex container on 2.1.1 may be the cause of one of the issues. Playing a 10bit H265 MKV in chrome triggers a transcode and it shows it is using hardware transcoding which is good.

The first issue happened when I attempted to use my pre-existing plex container. I turned on the "Enable GPU" under "Runtime & Resources". I had capabilities set to compute, utility, and video. Upon clicking "Deploy the container", I got an error that said "Cannot read property 'push' of null". I assume this error might be from re-using the container as once I re-created it, I was able to deploy without error. Realized this could be Chrome vs Firefox, but will be able to check later.

The second issue is just a display issue. The drop down for capabilities is visually under the Resources Memory bars. This appears to only be a Chrome issue as Firefox displays without issue. Chome: 89.0.4389.58

OS details:
OS: Ubuntu 20.04.2 LTS x86_64
Host: MS-7A34 1.0
Kernel: 5.4.0-65-generic
Uptime: 44 mins
Packages: 1189 (dpkg), 6 (snap)
Shell: bash 5.0.17
CPU: AMD Ryzen 7 1700 (16) @ 3.000GHz
GPU: NVIDIA GeForce GTX 1060 3GB
Memory: 685MiB / 32128MiB

[deviantony]

Hi @bobarune, thanks for the feedback ! I've passed it to our development team via #4791 (comment) and we'll continue working on this.

[Skoal262]

Any updates?? I have been fallowing this for quite some time, my plex container stopped working. I had to disable the old nvidia runtime and without gpu support i have to rely on cpu transcoding.

[oramirite]

Hey everyone. Happy to find this work-in-progress version of GPU compatibility. I've recently pivoted the role of my systems into some very render-heavy workfloads and GPU support for my container ecosystem is now a must.

The portainerci/portainer:pr4791 image seems to work great for me. I also had the display issues @bobarune mentioned with the dropdowns but other than that, all good. I am using this to spin up Unreal Engine containers that render to specific displays and being able to do that exclusively from Portainer is fun :)

It would be really awesome for this to merge soon so I could have dark mode AND GPU capabilities!

[oramirite]

After a little more testing, I found that the following features don't work at all on this image, even on a totally fresh install of Docker + Portainer:

• entering the App Templates section shows the error "Failure: Unable to retrieve templates". Nothing in the logs.
• entering the Registries section shows the error: "Failure: Unable to retrieve DockerHub details". Nothing in the logs.

Hopefully I'm not just pointing out something obvious.

[dfldylan]

Hello, everyone
I found that this PR #4791 was based on version 2.1 and was too old to merge into the main branch version 2.11. And it still has some missing features. I refactored it based on version 2.11.1, and add some extra features.

[huib-portainer]

You can give it a try by using the image portainerci/portainer:pr6872.
Note that this is a development build and should not be used in a production environment.

This works when using a new Portainer instance on a Docker standalone environment:

Using all GPUs

Results in

To use specific GPUs, configure it in the Environments

Shows as

And can be selected when deploying the container

[joshua-portainer]

Closing this issue as this work has been included in our 2.15 release, which will be releasing next week.