Q: tensorflow on multiple GPUs (very odd behaviour) I have a machine with multiple GPU units and want to run many Tensorflow model training processes on them, one process per gpu. In order to do so I write simple script which uses python Process from muliprocessing package and manage runs in simple way as follows:
pool = []
gpu_num = 0
for param in params:
    p = Process(target=run_training, args=param)
    p.start()
    pool.append(p)

    # suspend if the processes are too many
    wait = True
    while wait:
        time.sleep(5)
        for p in pool:
            alive = p.is_alive()
            if not alive:
                pool.remove(p)
        if len(pool) < concurrent_runs:
            wait = False
    gpu_num = (gpu_num + 1) % 13

In order to make TF process run only on selected GPU unit number gnu_num I set that only one card is visible for process and then run the model using direct assignment with uf.device:
os.environ["CUDA_VISIBLE_DEVICES"] = "{}".format(gpu_device)
with tf.device("/gpu:" + str(gnu_num)):
    #tf code here...

In addition I set allow_soft_placement=True.
The machine contains 13 cards, but for test purposes I start with 3 training processes at the same time. This means that 3 models trains on three different cards and after one finishes then new training starts on next card (as first snippet suggests). And, as I write in description, I get next strange results:

As anyone can see, the first several processes finished successfully and next models scheduled just in time. But the last one (card #6) did't actually work (zero kernel utilization) and then all three processes finished at the same time (cards #4,5,6) and after that nothing else happened, EXCEPT some small fluctuations in utilization at the last GPU#12. It's really odd.
I made several attempts and they all looks similar (processes stops working and then immediately card#12 shows fluctuations).
The logs of the program itself didn't show any sort of exception or resource exhaustion/OutOfMemoryError, it is just stops. First several runs finish training successfully and that is all. I also notice that from logs is seems like TF still assigns some variables to CPU even I try to avoid this using direct device placement.
I also check CPU/RAM usage during training. It is high enough, but nothing extraordinary:

So it is all really frustrating. The question is - does anybody has the clue what is going on here? What can be done to fix the situation or even to make some steps to detect additional part of puzzle?
tensorflow-gpu==1.4
