  Experts in various fields routinely perform methodical writing tasks to plan,
organize, and report their work. From a clinician writing a differential
diagnosis for a patient, to a teacher writing a lesson plan for students, these
tasks are pervasive, requiring to methodically generate structured long-form
output for a given input. We develop a typology of methodical tasks structured
in the form of a task objective, procedure, input, and output, and introduce
DoLoMiTes, a novel benchmark with specifications for 519 such tasks elicited
from hundreds of experts from across 25 fields. Our benchmark further contains
specific instantiations of methodical tasks with concrete input and output
examples (1,857 in total) which we obtain by collecting expert revisions of up
to 10 model-generated examples of each task. We use these examples to evaluate
contemporary language models highlighting that automating methodical tasks is a
challenging long-form generation problem, as it requires performing complex
inferences, while drawing upon the given context as well as domain knowledge.
