But unfortunately, this didn't work at all. The inference result was still a mess.
Now the inference works very well! And the training speed become a little bit faster than Averaging-Gradients-Training, for the Averaging Operation needs to wait multi gradients from multi GPUs.