  The full U-duality symmetry of toroidally compactified M-theory can only be
displayed by allowing non-rectangular tori with expectation values of the gauge
fields. We construct an E_d(Z) U-duality invariant mass formula incorporating
non-vanishing gauge backgrounds of the M-theory three-form C. We interpret this
mass formula from the point of view of the Matrix gauge theory, and identify
the coupling of the three-form to the gauge theory as a topological theta term,
in agreement with earlier conjectures. We give a derivation of this fact from
D-brane analysis, and obtain the Matrix gauge theory description of other gauge
backgrounds allowed by the Discrete Light-Cone Quantization. We further show
that the conjectured extended U-duality symmetry of Matrix theory on T^d in the
Discrete Light-Cone Quantization has an implementation as an action of
E_{d+1}(Z) on the BPS spectrum. Some implications for the proper interpretation
of the rank N of the Matrix gauge theory are discussed.
