  Sequence-to-Sequence (S2S) neural text generation models, especially the
pre-trained ones (e.g., BART and T5), have exhibited compelling performance on
various natural language generation tasks. However, the black-box nature of
these models limits their application in tasks where specific rules (e.g.,
controllable constraints, prior knowledge) need to be executed. Previous works
either design specific model structure (e.g., Copy Mechanism corresponding to
the rule "the generated output should include certain words in the source
input") or implement specialized inference algorithm (e.g., Constrained Beam
Search) to execute particular rules through the text generation. These methods
require careful design case-by-case and are difficult to support multiple rules
concurrently. In this paper, we propose a novel module named Neural
Rule-Execution Tracking Machine that can be equipped into various
transformer-based generators to leverage multiple rules simultaneously to guide
the neural generation model for superior generation performance in a unified
and scalable way. Extensive experimental results on several benchmarks verify
the effectiveness of our proposed model in both controllable and general text
generation.
