  Technical support problems are often long and complex. They typically contain
user descriptions of the problem, the setup, and steps for attempted
resolution. Often they also contain various non-natural language text elements
like outputs of commands, snippets of code, error messages or stack traces.
These elements contain potentially crucial information for problem resolution.
However, they cannot be correctly parsed by tools designed for natural
language. In this paper, we address the problem of segmentation for technical
support questions. We formulate the problem as a sequence labelling task, and
study the performance of state of the art approaches. We compare this against
an intuitive contextual sentence-level classification baseline, and a state of
the art supervised text-segmentation approach. We also introduce a novel
component of combining contextual embeddings from multiple language models
pre-trained on different data sources, which achieves a marked improvement over
using embeddings from a single pre-trained language model. Finally, we also
demonstrate the usefulness of such segmentation with improvements on the
downstream task of answer retrieval.
