  Fruit harvesting poses a significant labor and financial burden for the
industry, highlighting the critical need for advancements in robotic harvesting
solutions. Machine vision-based fruit detection has been recognized as a
crucial component for robust identification of fruits to guide robotic
manipulation. Despite considerable progress in leveraging deep learning and
machine learning techniques for fruit detection, a common shortfall is the
inability to swiftly extend the developed models across different orchards
and/or various fruit species. Additionally, the limited availability of
pertinent data further compounds these challenges. In this work, we introduce
MetaFruit, the largest publicly available multi-class fruit dataset, comprising
4,248 images and 248,015 manually labeled instances across diverse U.S.
orchards. Furthermore, this study proposes an innovative open-set fruit
detection system leveraging advanced Vision Foundation Models (VFMs) for fruit
detection that can adeptly identify a wide array of fruit types under varying
orchard conditions. This system not only demonstrates remarkable adaptability
in learning from minimal data through few-shot learning but also shows the
ability to interpret human instructions for subtle detection tasks. The
performance of the developed foundation model is comprehensively evaluated
using several metrics, which outperforms the existing state-of-the-art
algorithms in both our MetaFruit dataset and other open-sourced fruit datasets,
thereby setting a new benchmark in the field of agricultural technology and
robotic harvesting. The MetaFruit dataset and detection framework are
open-sourced to foster future research in vision-based fruit harvesting,
marking a significant stride toward addressing the urgent needs of the
agricultural sector.
