  We introduce and study families of finite index subgroups of the modular
group that generalize the congruence subgroups. Such groups, termed
$\phi$-congruence subgroups, are obtained by reducing homomorphisms $\phi$ from
the modular group into a linear algebraic group modulo integers. In particular,
we examine two families of examples, arising on the one hand from a map into a
quasi-unipotent group, and on the other hand from maps into symplectic groups
of degree four. In the quasi-unipotent case we also provide a detailed
discussion of the corresponding modular forms, using the fact that the tower of
curves in this case contains the tower of isogenies over the elliptic curve
$y^2=x^3-1728$ defined by the commutator subgroup of the modular group.
