  We study the gauge invariant 't Hooft operator in canonical formalism for
Yang-Mills theory as well as the $\mathcal{N} =4 $ super-Yang-Mills theory with
the gauge group $ U(N) $. It is shown that the spectrum of the 't Hooft
operator labeled by the arbitrary irreducible representation of the gauge group
is the same as the spectrum of the Wilson operator labeled by the same
representation. So it is possible to construct a unitary operator $ S $ making
the two kinds of loop operators transformed into each other. S-duality
transformation could be realized by the operator $ S $. We compute the
supersymmetry variations of the loop operators with the fermionic couplings
turned off. The result is consistent with the expectation that the action of $
S $ should make supercharges transform with a $ U(1)_{Y} $ phase.
