  We generalize Cremmer-Julia 1st order action of $N=8$ supergravity with
manifest $E_{7(7)}$ symmetry to cases of $N=6$ with manifest $SO^*(12)$ and
$N=5$ with manifest $SU(1,5)$ duality symmetries. These U dualities belong to
groups of type E7 which do not admit a symmetric bilinear invariant for vector
fields. Therefore the 2d order classical action derived from the one with
manifest E7 type duality has a ghost vector field which, under appropriate
boundary conditions, decouples. We show that when classical $N\geq 5$
supergravities are deformed by a candidate UV divergence the ghost field does
not decouple. Therefore we argue that U duality and supersymmetry suggest an
explanation of the mysterious cancellation of UV infinities at $L=4$, $N=5$ in
d=4. The same reasoning implies that, in absence of duality and supersymmetry
anomalies, which still require a better understanding, $N\geq 5$ perturbative
supergravities may be UV finite at higher-loops.
