  Given a topological dynamical system $\Sigma = (X, \sigma)$, where $X$ is a
compact Hausdorff space and $\sigma$ a homeomorphism of $X$, we introduce the
associated Banach $^*$-algebra crossed product $\ell^1 (\Sigma)$ and analyse
its ideal structure. This algebra is the Banach algebra most naturally
associated with the dynamical system, and it has a richer structure than its
well studied $C^*$-envelope, as becomes evident from the possible existence of
non-self-adjoint closed ideals. This paper initiates the study of these
algebras and links their ideal structure to the topological dynamics. It is
determined when exactly the algebra is simple, or prime, and when there exists
a non-self-adjoint closed ideal. In addition, a structure theorem is obtained
for the case when $X$ consists of one finite orbit, and the algebra is shown to
be Hermitian if $X$ is finite. The key to these results lies in analysing the
commutant of $C(X)$ in the algebra, which can be shown to be a maximal abelian
subalgebra with non-zero intersection with each non-zero closed ideal.
