  The ${\cal N}=4$ higher spin generators for general superspin $s$ in terms of
oscillators in the matrix generalization of $AdS_3$ Vasiliev higher spin theory
at nonzero $\mu$ (which is equivalent to the 't Hooft-like coupling constant
$\lambda$) were found previously. In this paper, by computing the
(anti)commutators between these ${\cal N}=4$ higher spin generators for low
spins $s_1$ and $s_2$ ($s_1+s_2 \leq 11$) explicitly, we determine the complete
${\cal N}=4$ higher spin algebra for generic $\mu$. The three kinds of
structure constants contain the linear combination of two different generalized
hypergeometric functions. These structure constants remain the same under the
transformation $\mu \leftrightarrow (1-\mu)$ up to signs. We have checked that
the above ${\cal N}=4$ higher spin algebra contains the ${\cal N}=2$ higher
spin algebra, as a subalgebra, found by Fradkin and Linetsky some time ago.
