  We study rigid surface operators in the $N=4$ supersymmetric Yang-Mills
theories with gauge groups $SO(n)$ and $Sp(2n)$. Using maps $X_S$ and $Y_S$
between these two theories, Wyllard made explicit proposals for how the
$S$-duality map should act on certain subclasses of surface operators. We study
the maps $X_S$ and $Y_S$ further and simplify the construction of symbol
invariant of rigid surface operators by a convenient trick. By consistency
checks, we recover and extend the $S$-duality maps proposed by Wyllard. We find
new subclasses of rigid surface operators related by $S$-duality. We try to
explain the exceptions of $S$-duality maps. We also discuss the extension of
the techniques used in the $B_n/C_n$ theories to the $D_n$ theories.
