  Large-scale crop yield estimation is, in part, made possible due to the
availability of remote sensing data allowing for the continuous monitoring of
crops throughout their growth cycle. Having this information allows
stakeholders the ability to make real-time decisions to maximize yield
potential. Although various models exist that predict yield from remote sensing
data, there currently does not exist an approach that can estimate yield for
multiple crops simultaneously, and thus leads to more accurate predictions. A
model that predicts the yield of multiple crops and concurrently considers the
interaction between multiple crop yields. We propose a new convolutional neural
network model called YieldNet which utilizes a novel deep learning framework
that uses transfer learning between corn and soybean yield predictions by
sharing the weights of the backbone feature extractor. Additionally, to
consider the multi-target response variable, we propose a new loss function. We
conduct our experiment using data from 1,132 counties for corn and 1,076
counties for soybean across the United States. Numerical results demonstrate
that our proposed method accurately predicts corn and soybean yield from one to
four months before the harvest with a MAE being 8.74% and 8.70% of the average
yield, respectively, and is competitive to other state-of-the-art approaches.
