  This paper reduces the cost of DNNs training by decreasing the amount of data
movement across heterogeneous architectures composed of several GPUs and
multicore CPU devices. In particular, this paper proposes an algorithm to
dynamically adapt the data representation format of network weights during
training. This algorithm drives a compression procedure that reduces data size
before sending them over the parallel system. We run an extensive evaluation
campaign considering several up-to-date deep neural network models and two
high-end parallel architectures composed of multiple GPUs and CPU multicore
chips. Our solution achieves average performance improvements from 6.18\% up to
11.91\%.
