  Creating high-quality scientific figures can be time-consuming and
challenging, even though sketching ideas on paper is relatively easy.
Furthermore, recreating existing figures that are not stored in formats
preserving semantic information is equally complex. To tackle this problem, we
introduce DeTikZify, a novel multimodal language model that automatically
synthesizes scientific figures as semantics-preserving TikZ graphics programs
based on sketches and existing figures. To achieve this, we create three new
datasets: DaTikZv2, the largest TikZ dataset to date, containing over 360k
human-created TikZ graphics; SketchFig, a dataset that pairs hand-drawn
sketches with their corresponding scientific figures; and SciCap++, a
collection of diverse scientific figures and associated metadata. We train
DeTikZify on SciCap++ and DaTikZv2, along with synthetically generated sketches
learned from SketchFig. We also introduce an MCTS-based inference algorithm
that enables DeTikZify to iteratively refine its outputs without the need for
additional training. Through both automatic and human evaluation, we
demonstrate that DeTikZify outperforms commercial Claude 3 and GPT-4V in
synthesizing TikZ programs, with the MCTS algorithm effectively boosting its
performance. We make our code, models, and datasets publicly available.
