  The MineRL BASALT competition has served to catalyze advances in learning
from human feedback through four hard-to-specify tasks in Minecraft, such as
create and photograph a waterfall. Given the completion of two years of BASALT
competitions, we offer to the community a formalized benchmark through the
BASALT Evaluation and Demonstrations Dataset (BEDD), which serves as a resource
for algorithm development and performance assessment. BEDD consists of a
collection of 26 million image-action pairs from nearly 14,000 videos of human
players completing the BASALT tasks in Minecraft. It also includes over 3,000
dense pairwise human evaluations of human and algorithmic agents. These
comparisons serve as a fixed, preliminary leaderboard for evaluating
newly-developed algorithms. To enable this comparison, we present a streamlined
codebase for benchmarking new algorithms against the leaderboard. In addition
to presenting these datasets, we conduct a detailed analysis of the data from
both datasets to guide algorithm development and evaluation. The released code
and data are available at https://github.com/minerllabs/basalt-benchmark .
