  Existing Data Parallel (DP) trainings for deep neural networks (DNNs) often
experience limited scalability in speedup due to substantial communication
overheads. While Overlapping technique can mitigate such problem by paralleling
communication and computation in DP, its effectiveness is constrained by the
high communication-to-computation ratios (CCR) of DP training tasks. Gradient
compression (GC) is a promising technique to obtain lower CCR by reducing
communication volume directly. However, it is challenging to obtain real
performance improvement by applying GC into Overlapping because of (1) severe
performance penalties in traditional GCs caused by high compression overhead
and (2) decline of Overlapping benefit owing to the possible data dependency in
GC schemes. In this paper, we propose COVAP, a novel GC scheme designing a new
coarse-grained filter, makes the compression overhead close to zero. COVAP
ensures an almost complete overlap of communication and computation by
employing adaptive compression ratios and tensor sharding tailored to specific
training tasks. COVAP also adopts an improved error feedback mechanism to
maintain training accuracy. Experiments are conducted on Alibaba Cloud ECS
instances with different DNNs of real-world applications. The results
illustrate that COVAP outperforms existent GC schemes in time-to-solution by
1.92x-15.39x and exhibits near-linear scaling. Furthermore, COVAP achieves best
scalability under experiments on four different cluster sizes.
