  With the increasing penetration of renewable power sources such as wind and
solar, accurate short-term, nowcasting renewable power prediction is becoming
increasingly important. This paper investigates the multi-modal (MM) learning
and end-to-end (E2E) learning for nowcasting renewable power as an intermediate
to energy management systems. MM combines features from all-sky imagery and
meteorological sensor data as two modalities to predict renewable power
generation that otherwise could not be combined effectively. The combined,
predicted values are then input to a differentiable optimal power flow (OPF)
formulation simulating the energy management. For the first time, MM is
combined with E2E training of the model that minimises the expected total
system cost. The case study tests the proposed methodology on the real sky and
meteorological data from the Netherlands. In our study, the proposed MM-E2E
model reduced system cost by 30% compared to uni-modal baselines.
