  We examine how the average of double-winding Wilson loops depends on the
number of color $N$ in the $SU(N)$ Yang-Mills theory. In the case where the two
loops $C_1$ and $C_2$ are identical, we derive the exact operator relation
which relates the double-winding Wilson loop operator in the fundamental
representation to that in the higher dimensional representations depending on
$N$. By taking the average of the relation, we find that the
difference-of-areas law for the area law falloff recently claimed for $N=2$ is
excluded for $N \geq 3$, provided that the string tension obeys the Casimir
scaling for the higher representations. In the case where the two loops are
distinct, we argue that the area law follows a novel law $(N - 3)A_1/(N-1)+A_2$
with $A_1$ and $A_2 (A_1<A_2)$ being the minimal areas spanned respectively by
the loops $C_1$ and $C_2$, which is neither sum-of-areas ($A_1+A_2$) nor
difference-of-areas ($A_2 - A_1$) law when ($N\geq3$). Indeed, this behavior
can be confirmed in the two-dimensional $SU(N)$ Yang-Mills theory exactly.
