Representation theory is a subject I want to like (it can be fun finding the representations of a group), but it's hard for me to see it as a subject that arises naturally or why it is important. I can think of two mathematical reasons for studying it:


The character table of a group is packs a lot of information about the group and is concise.
It is practically/computationally nice to have explicit matrices that model a group.


But there must certainly be deeper things that I am missing. I can understand why one would want to study group actions (the axioms for a group beg you to think of elements as operators), but why look at group actions on vector spaces? Is it because linear algebra is so easy/well-known (when compared to just modules, say)?

I am also told that representation theory is important in quantum mechanics. For example, physics should be $\mathrm{SO}(3)$ invariant and when we represent this on a Hilbert space of wave-functions, we are led to information about angular momentum. But this seems to only trivially invoke representation theory since we already start with a subgroup of $\mathrm{GL}(n)$ and then extend it to act on wave functions by $\psi(x,t) \mapsto \psi(Ax,t)$ for $A$ in $\mathrm{SO}(n)$.

This Wikipedia article on particle physics and representation theory claims that if our physical system has $G$ as a symmetry group, then there is a correspondence between particles and representations of $G$. I'm not sure if I understand this correspondence since it seems to be saying that if we act an element of G on a state that corresponds to some particle, then this new state also corresponds to the same particle. So a particle is an orbit of the $G$ action? Anyone know of good sources that talk about this?

One comment about your sentence "this seems to only trivially invoke representation theory". It might be surprising, but such obvious representations are actually the source of interesting mathematics, and a lot of effort of representation theorists is devoted to studying them.

More precisely: start with a group (in your example $SO(n)$) acting on a space $X$ (in your example $\mathbb R^n$), and look at the space of functions on $X$ (let me write it $\mathcal F(X)$; in a careful treatment, one would have to think about whether we wanted continuous, smooth, $L^2$, or some other kind of functions, but I will suppress that kind of technical consideration).

Then, as you observe, there is a natural representation of $G$ on $\mathcal F(X)$.

You are right that from a certain point of view this seems trivial, because the representation is obvious.  Unlike when one first learns rep'n theory of finite groups, where one devotes a lot of effort to constructing reps., in this context, the rep. stares you in the face.

So how can this be interesting?

Well, the representation $\mathcal F(X)$ will almost never be irreducible. How does this representation decompose?

Suddenly we are looking at a hard representation theoretic problem.  


First, we have to work out the list of irreps of $G$ (which is much
like what one does in a first course on rep'n theory of finite groups).
Second, we have to figure out how $\mathcal F(X)$ decomposes, which involves representation theory (among other things, you have to develop methods for investigating this sort of question), and also often a lot of analysis (because typically $\mathcal F(X)$ will be infinite dimensional, and may be a Hilbert space, or have some other similar sort of topological vector space structure which should be incorporated into the picture). 


I don't think I should say too much more here, but I will just give some illustrative examples:


If $ X = G = S^1$ (the circle group, say thought of as $\mathbb R/\mathbb Z$)
acting on itself by addition, then the solution to the problem of decomposing
$\mathcal F(S^1)$ is the theory of Fourier series.  (Note that a function on $S^1$ is the
same as a periodic function on $\mathbb R$.)
If $ X = G = \mathbb R$, with $G$ acting on itself by addition, then
the solution to the above question (how does $\mathcal F(\mathbb R)$ decompose under the
action of $\mathbb R$) is the theory of the Fourier transform.
If $ X = S^2$ and $G = SO(3)$ acting on $X$ via rotations, then decomposing $\mathcal F(S^2)$ into irreducible representations gives the theory of spherical harmonics.
(This is an important example in quantum mechanics; it comes up for example in the theory of the hydrogen atom, when one has a spherical symmetry because the electron orbits the nucleus, which one thinks of as the centre of the sphere.)
If $ X = SL_2(\mathbb R)/SL_2(\mathbb Z)$ (this is the quotient of a Lie group by a discrete subgroup, so is naturally a manifold, in this case of dimension 3),
with $G = SL_2(\mathbb R)$ acting by left multiplication, then the problem of decomposing $\mathcal F(X)$ leads to the theory of modular forms and Maass forms, and is the first example in the more general theory of automorphic forms.


Added: Looking over the other answers, I see that this is an elaboration on  AD.'s answer.
