  We present a novel fruit counting pipeline that combines deep segmentation,
frame to frame tracking, and 3D localization to accurately count visible fruits
across a sequence of images. Our pipeline works on image streams from a
monocular camera, both in natural light, as well as with controlled
illumination at night. We first train a Fully Convolutional Network (FCN) and
segment video frame images into fruit and non-fruit pixels. We then track
fruits across frames using the Hungarian Algorithm where the objective cost is
determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In
order to correct the estimated count from tracking process, we combine tracking
results with a Structure from Motion (SfM) algorithm to calculate relative 3D
locations and size estimates to reject outliers and double counted fruit
tracks. We evaluate our algorithm by comparing with ground-truth
human-annotated visual counts. Our results demonstrate that our pipeline is
able to accurately and reliably count fruits across image sequences, and the
correction step can significantly improve the counting accuracy and robustness.
Although discussed in the context of fruit counting, our work can extend to
detection, tracking, and counting of a variety of other stationary features of
interest such as leaf-spots, wilt, and blossom.
