  Automatic extraction of information from publications is key to making
scientific knowledge machine readable at a large scale. The extracted
information can, for example, facilitate academic search, decision making, and
knowledge graph construction. An important type of information not covered by
existing approaches is hyperparameters. In this paper, we formalize and tackle
hyperparameter information extraction (HyperPIE) as an entity recognition and
relation extraction task. We create a labeled data set covering publications
from a variety of computer science disciplines. Using this data set, we train
and evaluate BERT-based fine-tuned models as well as five large language
models: GPT-3.5, GALACTICA, Falcon, Vicuna, and WizardLM. For fine-tuned
models, we develop a relation extraction approach that achieves an improvement
of 29% F1 over a state-of-the-art baseline. For large language models, we
develop an approach leveraging YAML output for structured data extraction,
which achieves an average improvement of 5.5% F1 in entity recognition over
using JSON. With our best performing model we extract hyperparameter
information from a large number of unannotated papers, and analyze patterns
across disciplines. All our data and source code is publicly available at
https://github.com/IllDepence/hyperpie
