  We examine the interplay of symmetry and topological order in $2+1$
dimensional topological phases of matter. We present a definition of the \it
topological symmetry \rm group, which characterizes the symmetry of the
emergent topological quantum numbers of a topological phase, and we describe
its relation with the microscopic symmetry of the underlying physical system.
We derive a general framework to characterize and classify symmetry
fractionalization in topological phases, including phases that are non-Abelian
and symmetries that permute the quasiparticle types and/or are anti-unitary. We
develop a theory of extrinsic defects (fluxes) associated with elements of the
symmetry group, which provides a general classification of symmetry-enriched
topological phases derived from a topological phase of matter $\mathcal{C}$
with symmetry group $G$. The algebraic theory of the defects, known as a
$G$-crossed braided tensor category $\mathcal{C}_{G}^{\times}$, allows one to
compute many properties, such as the number of topologically distinct types of
defects associated with each group element, their fusion rules, quantum
dimensions, zero modes, braiding exchange transformations, a generalized
Verlinde formula for the defects, and modular transformations of the
$G$-crossed extensions of topological phases. We also examine the promotion of
the global symmetry to a local gauge invariance, wherein the extrinsic
$G$-defects are turned into deconfined quasiparticle excitations, which results
in a different topological phase $(\mathcal{C}_{G}^{\times})^{G}$. A number of
instructive and/or physically relevant examples are studied in detail.
