  In this paper we study the group theoretic structures of colored HOMFLY
polynomials in a specific limit. The group structures arise in the perturbative
expansion of $SU(N)$ Chern-Simons Wilson loops, while the limit is $N
\rightarrow 0$. The result of the paper is twofold. First, we explain the
emergence of Kadomsev-Petviashvily (KP) $\tau$-functions. This result is an
extension of what we did in arXiv:1805.02761, where a symbolic correspondence
between KP equations and group factors was established. In this paper we prove
that integrability of the colored Alexander polynomial is due to it's relation
to soliton $\tau$-functions. Mainly, the colored Alexander polynomial is
embedded in the action of the KP generating function on the soliton
$\tau$-function. Secondly, we use this correspondence to provide a rather
simple combinatoric description of the group factors in term of Young diagrams,
which is otherwise described in terms of chord diagrams, where no simple
description is known. This is a first step providing an explicit description of
the group theoretic data of Wilson loops, which would effectively reduce them
to a purely topological quantity, mainly to a collection of Vassiliev
invariants.
