  In this paper, we compare a certain field arising from the pro-$p$ outer
Galois representation associated to a once-punctured CM elliptic curve over an
imaginary quadratic field $K$ with the maximal pro-$p$ Galois extension of the
mod-$p$ ray class field $K(p)$ of $K$ unramified outside $p$. We prove that
these two fields coincide for every prime $p$ which satisfies certain
assumptions, assuming an analogue of the Deligne-Ihara conjecture. This may be
regarded as an analogue of a result of Sharifi on the kernel of the pro-$p$
outer Galois representation associated to the projective line minus three
points.
