<p>Ok so i finally made it work.</p>

<p>It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..</p>

CC BY-SA 4.0
2019-11-08T14:56:34.463Z
2019-11-08T14:56:34.463Z
lifeguru42
<p>I have setup a kubernetes node with a Nvidia Tesla k80 and followed <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="nofollow noreferrer">this tutorial</a> to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.</p>

<p>I have managed to install the nvidia daemonsets and I can now see the following pods: </p>

<pre><code>nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84
</code></pre>

<p>The problem is that even while using the recommended image <code>nvidia/cuda:10.0-runtime-ubuntu18.04</code> I still can't find the Nvidia drivers inside my pod:</p>

<pre><code>root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src
</code></pre>

<p>But the tutorial mentions:</p>

<blockquote>
  <p>CUDA libraries and debug utilities are made available inside the container at <code>/usr/local/nvidia/lib64</code> and <code>/usr/local/nvidia/bin</code>, respectively.</p>
</blockquote>

<p>I have also tried to test if CUDA was working through <code>torch.cuda.is_available()</code> but I get False as a return value.</p>

<p>Many thanks in advance for your help.</p>

CC BY-SA 4.0
2020-05-09T15:58:17.750Z
https://stackoverflow.com/q/58766003/2777965
030
<kubernetes><google-cloud-platform><google-compute-engine>
GKE - GPU nvidia - cuda drivers dont work
lifeguru42
# GKE - GPU nvidia - cuda drivers dont work

(Asked by: username_0 on 2019-11-08)

I have setup a kubernetes node with a Nvidia Tesla k80 and followed this tutorial to try to run a pytorch docker image with Nvidia drivers and CUDA drivers working.
I have managed to install the nvidia daemonsets and I can now see the following pods: 
nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84

The problem is that even while using the recommended image nvidia/cuda:10.0-runtime-ubuntu18.04 I still can't find the Nvidia drivers inside my pod:
root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src

But the tutorial mentions:

CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.

I have also tried to test if CUDA was working through torch.cuda.is_available() but I get False as a return value.
Many thanks in advance for your help.

**Tags:** kubernetes, google-cloud-platform, google-compute-engine

> **Comments:**
> **username_1**: https://stackoverflow.com/q/58766003/2777965  

---

## 1 answers

### Answer by username_0 on 2019-11-08. Score: 1 | Accepted answer

Ok so i finally made it work.

It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..


---

