  Sequence-to-Sequence (S2S) models have achieved remarkable success on various
text generation tasks. However, learning complex structures with S2S models
remains challenging as external neural modules and additional lexicons are
often supplemented to predict non-textual outputs. We present a systematic
study of S2S modeling using contained decoding on four core tasks:
part-of-speech tagging, named entity recognition, constituency and dependency
parsing, to develop efficient exploitation methods costing zero extra
parameters. In particular, 3 lexically diverse linearization schemas and
corresponding constrained decoding methods are designed and evaluated.
Experiments show that although more lexicalized schemas yield longer output
sequences that require heavier training, their sequences being closer to
natural language makes them easier to learn. Moreover, S2S models using our
constrained decoding outperform other S2S approaches using external resources.
Our best models perform better than or comparably to the state-of-the-art for
all 4 tasks, lighting a promise for S2S models to generate non-sequential
structures.
