  An in-season early crop yield forecast before harvest can benefit the farmers
to improve the production and enable various agencies to devise plans
accordingly. We introduce a reliable and inexpensive method to predict crop
yields from publicly available satellite imagery. The proposed method works
directly on raw satellite imagery without the need to extract any hand-crafted
features or perform dimensionality reduction on the images. The approach
implicitly models the relevance of the different steps in the growing season
and the various bands in the satellite imagery. We evaluate the proposed
approach on tehsil (block) level wheat predictions across several states in
India and demonstrate that it outperforms existing methods by over 50\%. We
also show that incorporating additional contextual information such as the
location of farmlands, water bodies, and urban areas helps in improving the
yield estimates.
