  Let $E$ be an elliptic curve over $\mathbb{Q}$, $p$ an odd prime number and
$n$ a positive integer. In this article, we investigate the ideal class group
$\mathrm{Cl}(\mathbb{Q}(E[p^n]))$ of the $p^n$-division field
$\mathbb{Q}(E[p^n])$ of $E$. We introduce a certain subgroup
$E(\mathbb{Q})_{\mathrm{ur},p^n}$ of $E(\mathbb{Q})$ and study the $p$-adic
valuation of the class number $\#\mathrm{Cl}(\mathbb{Q}(E[p^n]))$. In addition,
when $n = 1$, we further study $\mathrm{Cl}(\mathbb{Q}(E[p]))$ as a
$\mathrm{Gal}(\mathbb{Q}(E[p])/\mathbb{Q})$- module. More precisely, we study
the semi-simplification $(\mathrm{Cl}(\mathbb{Q}(E[p]))\otimes
\mathbb{Z}_p)^{\mathrm{ss}}$ of $\mathrm{Cl}(\mathbb{Q}(E[p]))\otimes
\mathbb{Z}_p$ as a
$\mathbb{Z}_p[\mathrm{Gal}(\mathbb{Q}(E[p])/\mathbb{Q})]$-module. We obtain a
lower bound of the multiplicity of the $E[p]$-component in the
semi-simplification when $E[p]$ is an irreducible
$\mathrm{Gal}(\mathbb{Q}(E[p])/\mathbb{Q})$-module.
