  Quantized vortices stunningly illustrate the coherent nature of a superfluid
Bose condensate of alkali atoms. Introducing an optical lattice depletes this
coherence. Consequently, novel vortex physics may emerge in an experiment on a
harmonically trapped gas in the presence of a rotating optical lattice. The
most dramatic effects would occur in proximity to the Mott state, an
interaction dominated insulator with a fixed integer number of particles per
site. We model such a rotating gas, showing that the lattice-induced spatial
profile of the superfluid density drives a gross rearrangement of vortices. For
example, instead of the uniform vortex lattices commonly seen in experiments,
we find parameters for which the vortices all sit at a fixed distance from the
center of the trap, forming a ring. Similarly, they can coalesce at the center,
forming a giant vortex. We find that the properties of this system are
hysteretic, even far from the Mott state. We explain this hysteresis in terms
of vortex pinning, commensurability between vortex density and pinning site
density, and energy barriers against changing the number of vortices. Finally,
we model time-of-flight expansion, demonstrating the experimental observability
of our predictions.
