  We develop a microscopic theory for the charge ordering (CO) transitions in
the spinels AlV$_2$O$_4$ and LiV$_2$O$_4$ (under pressure). The high degeneracy
of CO states is lifted by a coupling to the rhombohedral lattice deformations
which favors transition to a CO state with inequivalent V(1) and V(2) sites
forming Kagom\'e and trigonal planes respectively. We construct an extended
Hubbard type model including a deformation potential which is treated in
unrestricted Hartree Fock approximation and describes correctly the observed
first-order CO transition. We also discuss the influence of associated orbital
order. Furthermore we suggest that due to different band fillings AlV$_2$O$_4$
should remain metallic while LiV$_2$O$_4$ under pressure should become a
semiconductor when charge disproportionation sets in.
