  We present an extensive quantum Monte Carlo study of the N\'eel-valence bond
solid (VBS) phase transition on rectangular and honeycomb lattice SU($N$)
antiferromagnets in sign problem free models. We find that in contrast to the
honeycomb lattice and previously studied square lattice systems, on the
rectangular lattice for small $N$ a first order N\'eel-VBS transition is
realized. On increasing $N\geq 4$, we observe that the transition becomes
continuous and with the {\em same} universal exponents as found on the
honeycomb and square lattices (studied here for $N=5,7,10$), providing strong
support for a deconfined quantum critical point. Combining our new results with
previous numerical and analytical studies we present a general phase diagram of
the stability of $\mathbb{CP}^{N-1}$ fixed points with $q$-monopoles.
