<p>I am trying to use the GPU enabled version of the guppy_basecaller on an HPC cluster.  I am requesting a node that has 2 GPUs and am requesting 1 of the two GPUs.  So, I am trying to set the <code>-x cuda:--</code> flag.  What I was thinking this needed was the device number, ie if the allocation gave me a node and had the <code>CUDA_VISIBLE_DEVICES=</code> flag set, I would use <code>-x cuda:0</code> or <code>-x cuda:1</code> depending on which GPU I was assigned.  When I set it to 0 or auto, it seems to work when I have a whole.  When I set it to 1 when I am given a job with that device id flag set, the program errors out.  Am I thinking about the flag wrong?  The documents I have found on this are a bit sparse.</p>

<pre><code>CUDA_VISIBLE_DEVICES=1
working on run id  8138
working on subset  0

git er done ...
ONT Guppy basecalling software version 3.0.3+7e7b7d0
config file:        /groups/ju_lab/softwares/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
model file:         /groups/ju_lab/softwares/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
input path:         ./
save path:          ./
chunk size:         1000
chunks per runner:  1000
records per file:   4000
num basecallers:    4
gpu device:         cuda:1
kernel path:
runners per device: 4

[guppy/error] main: Could not create CUDA device 1: CUDA error at /builds/ofan/ont_core_cpp/ont_core/basecall_nn/nn_caller_cuda.
cpp:651: CUDA_ERROR_INVALID_DEVICE
[guppy/warning] main: An error occurred in the basecaller. Aborting.
<span class="math-container">```</span>
</code></pre>

CC BY-SA 4.0
2019-05-11T21:36:06.863Z
I should have mentioned, I am trying to request 1/2 of a node because as far as I can tell guppy can't use both GPUs.  And probably should have mentioned this is on a PBS/Torque cluster using Centos 7.  And this is my command to call guppy: `guppy_basecaller -i ./ --save_path ./ --flowcell FLO-MIN106 --kit SQK-LSK109 --gpu_runners_per_device 4 -r --min_qscore 7.0 -x cuda:${CUDA_VISIBLE_DEVICES} --disable_pings`
Bob
CC BY-SA 4.0
2019-05-11T22:39:27.113Z
New information.  Requesting a full node with 2 gpus, both `-x cuda:0` and `-x cuda:1` work.  But, setting the device to 1 does not result in the second gpu actually being utilized as measure by watching nvidia-smi gpu activity. So I am not sure what that flag means.  What I really need to do is specify which GPU device to use so that I don't kill other users jobs.  In SLURM with cgroups, this would not be a problem, but on this PBS/Torque cluster, I can get access to both even though I am requesting one.
Bob
CC BY-SA 4.0
2019-05-11T22:59:58.623Z
OK, solved I think.  I believe `-x cuda:--` is looking for the item in the list given by `CUDA_VISIBLE_DEVICES` rather than the actual device id.
Bob
CC BY-SA 4.0
2019-05-13T12:04:42.457Z
Welcome to the site. You can [edit] the question to include the relevant code into the question. Also to mark as solved you can (and are encouraged) to post your solution as an answer (and then mark it as the accepted one). This way others, like me we, can learn from you :D
llrs
CC BY-SA 4.0
2019-05-21T12:30:00.080Z
Did using `-x auto` not find the device you wanted?
Michael Hall
CC BY-SA 4.0
2019-06-21T14:45:30.157Z
please where did you get the software of guppy_basecaller fro Centos 7? thanks
archengzi
<minion>
using guppy_basecaller on node with 2 GPUs
# using guppy_basecaller on node with 2 GPUs

(Asked by: username_0 on 2019-05-11)

I am trying to use the GPU enabled version of the guppy_basecaller on an HPC cluster.  I am requesting a node that has 2 GPUs and am requesting 1 of the two GPUs.  So, I am trying to set the -x cuda:-- flag.  What I was thinking this needed was the device number, ie if the allocation gave me a node and had the CUDA_VISIBLE_DEVICES= flag set, I would use -x cuda:0 or -x cuda:1 depending on which GPU I was assigned.  When I set it to 0 or auto, it seems to work when I have a whole.  When I set it to 1 when I am given a job with that device id flag set, the program errors out.  Am I thinking about the flag wrong?  The documents I have found on this are a bit sparse.
CUDA_VISIBLE_DEVICES=1
working on run id  8138
working on subset  0

git er done ...
ONT Guppy basecalling software version 3.0.3+7e7b7d0
config file:        /groups/ju_lab/softwares/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
model file:         /groups/ju_lab/softwares/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
input path:         ./
save path:          ./
chunk size:         1000
chunks per runner:  1000
records per file:   4000
num basecallers:    4
gpu device:         cuda:1
kernel path:
runners per device: 4

[guppy/error] main: Could not create CUDA device 1: CUDA error at /builds/ofan/ont_core_cpp/ont_core/basecall_nn/nn_caller_cuda.
cpp:651: CUDA_ERROR_INVALID_DEVICE
[guppy/warning] main: An error occurred in the basecaller. Aborting.
```

**Tags:** minion
> **Comments:**
> **username_0**: I should have mentioned, I am trying to request 1/2 of a node because as far as I can tell guppy can't use both GPUs.  And probably should have mentioned this is on a PBS/Torque cluster using Centos 7.  And this is my command to call guppy: `guppy_basecaller -i ./ --save_path ./ --flowcell FLO-MIN106 --kit SQK-LSK109 --gpu_runners_per_device 4 -r --min_qscore 7.0 -x cuda:${CUDA_VISIBLE_DEVICES} --disable_pings`  
> **username_0**: New information.  Requesting a full node with 2 gpus, both `-x cuda:0` and `-x cuda:1` work.  But, setting the device to 1 does not result in the second gpu actually being utilized as measure by watching nvidia-smi gpu activity. So I am not sure what that flag means.  What I really need to do is specify which GPU device to use so that I don't kill other users jobs.  In SLURM with cgroups, this would not be a problem, but on this PBS/Torque cluster, I can get access to both even though I am requesting one.  
> **username_0**: OK, solved I think.  I believe `-x cuda:--` is looking for the item in the list given by `CUDA_VISIBLE_DEVICES` rather than the actual device id.  
> **username_1**: Welcome to the site. You can [edit] the question to include the relevant code into the question. Also to mark as solved you can (and are encouraged) to post your solution as an answer (and then mark it as the accepted one). This way others, like me we, can learn from you :D  
> **username_2**: Did using `-x auto` not find the device you wanted?  
> **username_3**: please where did you get the software of guppy_basecaller fro Centos 7? thanks  

---

## 0 answers

