  In this paper, we consider spectral approximation of fractional differential
equations (FDEs). A main ingredient of our approach is to define a new class of
generalized Jacobi functions (GJFs), which is intrinsically related to
fractional calculus, and can serve as natural basis functions for properly
designed spectral methods for FDEs. We establish spectral approximation results
for these GJFs in weighted Sobolev spaces involving fractional derivatives. We
construct efficient GJF-Petrov-Galerkin methods for a class of prototypical
fractional initial value problems (FIVPs) and fractional boundary value
problems (FBVPs) of general order, and show that with an appropriate choice of
the parameters in GJFs, the resulted linear systems can be sparse and
well-conditioned. Moreover, we derive error estimates with convergence rate
only depending on the smoothness of data, so truly spectral accuracy can be
attained if the data are smooth enough. The idea and results presented in this
paper will be useful to deal with more general FDEs associated with
Riemann-Liouville or Caputo fractional derivatives.
