  Large neural networks require enormous computational clusters of machines.
Model-parallel training, when the model architecture is partitioned
sequentially between workers, is a popular approach for training modern models.
Information compression can be applied to decrease workers communication time,
as it is often a bottleneck in such systems. This work explores how
simultaneous compression of activations and gradients in model-parallel
distributed training setup affects convergence. We analyze compression methods
such as quantization and TopK compression, and also experiment with error
compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error
feedback approach. We conduct experiments on image classification and language
model fine-tuning tasks. Our findings demonstrate that gradients require milder
compression rates than activations. We observe that $K=10\%$ is the lowest TopK
compression level, which does not harm model convergence severely. Experiments
also show that models trained with TopK perform well only when compression is
also applied during inference. We find that error feedback techniques do not
improve model-parallel training compared to plain compression, but allow model
inference without compression with almost no quality drop. Finally, when
applied with the AQ-SGD approach, TopK stronger than with $ K=30\%$ worsens
model performance significantly.
