  Motivated by near-term experiments with ultracold alkaline-earth atoms
confined to optical lattices, we establish numerically and analytically the
phase diagram of two-leg SU($N$) spin ladders. Two-leg ladders provide a rich
and highly non-trivial extension of the single chain case on the way towards
the relatively little explored two dimensional situation. Focusing on the
experimentally relevant limit of one fermion per site, antiferromagnetic
exchange interactions, and $2\leq N \leq 6$, we show that the phase diagrams as
a function of the interchain (rung) to intrachain (leg) coupling ratio
$J_\perp/J_\Vert$ strongly differ for even vs. odd $N$. For even $N=4$ and 6,
we demonstrate that the phase diagram consists of a single valence bond crystal
(VBC) with a spatial period of $N/2$ rungs. For odd $N=3$ and 5, we find
surprisingly rich phase diagrams exhibiting three distinct phases. For weak
rung coupling, we obtain a VBC with a spatial period of $N$ rungs, whereas for
strong coupling we obtain a critical phase related to the case of a single
chain. In addition, we encounter intermediate phases for odd $N$, albeit of a
different nature for $N=3$ as compared to $N=5$. For $N=3$, we find a novel
gapless intermediate phase with $J_\perp$-dependent incommensurate spatial
fluctuations in a sizeable region of the phase diagram. For $N=5$, there are
strong indications for a narrow potentially gapped intermediate phase, whose
nature is not entirely clear.
