  Batching is an essential technique to improve computation efficiency in deep
learning frameworks. While batch processing for models with static feed-forward
computation graphs is straightforward to implement, batching for dynamic
computation graphs such as syntax trees or social network graphs is challenging
due to variable computation graph structure across samples. Through simulation
and analysis of a Tree-LSTM model, we show the key trade-off between graph
analysis time and batching effectiveness in dynamic batching. Based on this
finding, we propose a dynamic batching method as an extension to MXNet Gluon's
just-in-time compilation (JIT) framework. We show empirically that our method
yields up to 6.25 times speed-up on a common dynamic workload, a tree-LSTM
model for the semantic relatedness task.
