<p>Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that <code>train_model_config_A</code>, <code>train_model_config_B</code>, etc.</p>

<p>Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding <code>train_model_config_&lt;blah&gt;</code> script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.</p>

<p>You might not need to install the slave directly on the GPU cluster if the <code>train_model_config_&lt;blah&gt;</code> scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.</p>

CC BY-SA 3.0
2018-02-25T08:12:43.343Z
2018-02-25T08:12:43.343Z
Dan Cornilescu
<p>With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)</p>

<p>But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc) </p>

CC BY-SA 3.0
2017-09-14T07:52:32.413Z
You have a GPU cluster at the moment?
030
CC BY-SA 3.0
2017-09-14T08:00:28.213Z
please elaborate your question better - I tend to answer yes
Ta Mu
CC BY-SA 3.0
2017-09-14T08:20:01.527Z
`But what if your "client" environment is a GPU cluster` Is it a hypothesis or is there a real GPU cluster?
030
CC BY-SA 3.0
2017-09-14T08:25:17.267Z
real, indeed it is
Ta Mu
CC BY-SA 3.0
2017-09-14T08:38:04.287Z
GPU cluster = https://en.wikipedia.org/wiki/GPU_cluster?
030
CC BY-SA 3.0
2017-09-14T08:38:17.090Z
`model trainings in different configurations` Could you give examples?
030
CC BY-SA 3.0
2017-09-14T08:51:29.470Z
Yes. Many configurations means just using different tools/frameworks, or different sets of input data.
Ta Mu
CC BY-SA 3.0
2017-09-14T08:56:01.687Z
But what exactly is the problem? It's just a resource, it shouldn't matter if it's a GPU cluster or a self-driving car you run tests on. You just have to schedule it's use.
Jakub Kania
CC BY-SA 3.0
2017-09-20T06:23:11.253Z
@JakubKania: this should be an answer IMHO.
Dan Cornilescu
CC BY-SA 3.0
2017-09-20T09:00:16.687Z
@DanCornilescu To qualify as a full asnwer it would have to be expanded much. That comment is only fishing for more details.
Jakub Kania
<continuous-integration><cloud>
CI/CD with a GPU cluster
Ta Mu
# CI/CD with a GPU cluster

(Asked by: username_0 on 2017-09-14)

With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)

But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc)

**Tags:** continuous-integration, cloud

> **Comments:**
> **username_1**: You have a GPU cluster at the moment?  
> **username_0**: please elaborate your question better - I tend to answer yes  
> **username_1**: `But what if your "client" environment is a GPU cluster` Is it a hypothesis or is there a real GPU cluster?  
> **username_0**: real, indeed it is  
> **username_1**: GPU cluster = https://en.wikipedia.org/wiki/GPU_cluster?  
> **username_1**: `model trainings in different configurations` Could you give examples?  
> **username_0**: Yes. Many configurations means just using different tools/frameworks, or different sets of input data.  
> **username_2**: But what exactly is the problem? It's just a resource, it shouldn't matter if it's a GPU cluster or a self-driving car you run tests on. You just have to schedule it's use.  
> **username_3**: @JakubKania: this should be an answer IMHO.  
> **username_2**: @DanCornilescu To qualify as a full asnwer it would have to be expanded much. That comment is only fishing for more details.  

---

## 1 answers

### Answer by username_3 on 2018-02-25. Score: 2

Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that `train_model_config_A`, `train_model_config_B`, etc.

Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding `train_model_config_` script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.

You might not need to install the slave directly on the GPU cluster if the `train_model_config_` scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.


---

