  Deep Learning (DL) algorithms have become the {\em de facto} choice for data
analysis. Several DL implementations -- primarily limited to a single compute
node -- such as Caffe, TensorFlow, Theano and Torch have become readily
available. Distributed DL implementations capable of execution on large scale
systems are becoming important to address the computational needs of large data
produced by scientific simulations and experiments. Yet, the adoption of
distributed DL implementations faces significant impediments: 1) most
implementations require DL analysts to modify their code significantly -- which
is a show-stopper, 2) several distributed DL implementations are geared towards
cloud computing systems -- which is inadequate for execution on massively
parallel systems such as supercomputers.
  This work addresses each of these problems. We provide a distributed memory
DL implementation by incorporating required changes in the TensorFlow runtime
itself. This dramatically reduces the entry barrier for using a distributed
TensorFlow implementation. We use Message Passing Interface (MPI) -- which
provides performance portability, especially since MPI specific changes are
abstracted from users. Lastly -- and arguably most importantly -- we make our
implementation available for broader use, under the umbrella of Machine
Learning Toolkit for Extreme Scale (MaTEx) at {\texttt
http://hpc.pnl.gov/matex}. We refer to our implementation as MaTEx-TensorFlow.
