  We present a generative model for complex free-form structures such as
stroke-based drawing tasks. While previous approaches rely on sequence-based
models for drawings of basic objects or handwritten text, we propose a model
that treats drawings as a collection of strokes that can be composed into
complex structures such as diagrams (e.g., flow-charts). At the core of the
approach lies a novel autoencoder that projects variable-length strokes into a
latent space of fixed dimension. This representation space allows a relational
model, operating in latent space, to better capture the relationship between
strokes and to predict subsequent strokes. We demonstrate qualitatively and
quantitatively that our proposed approach is able to model the appearance of
individual strokes, as well as the compositional structure of larger diagram
drawings. Our approach is suitable for interactive use cases such as
auto-completing diagrams. We make code and models publicly available at
https://eth-ait.github.io/cose.
