  To our knowledge, this paper is the first attempt to consider the stable
solution and extremal solution for the fractional $p$-Laplacian equation:
$(-\Delta)_p^s u= \lambda f(u),\; u> 0 ~\text{in}~\Omega;\; u=0\;\text{in}~
\mathbb{R}^N\setminus\Omega$, where $p>1$, $s\in (0,1)$, $N>sp$, $\lambda>0$
and $\Omega$ is a bounded domain with continuous boundary. We first construct
the notion of stable solution, and then we prove that when $f$ is of class
$C^1$, nondecreasing and such that $f(0)>0$ and $\underset{t\to
\infty}{\lim}\frac{f(t)}{t^{p-1}}=\infty$, there exists an extremal parameter
$\lambda^*\in (0, \infty)$ such that a bounded minimal solution $u_\lambda$
exists if $\lambda\in (0, \lambda^*)$, and no bounded solution exists if
$\lambda>\lambda^*$, no $W_0^{s,p}(\Omega)$ solution exists if in addition
$f(t)^{\frac{1}{p-1}}$ is convex. Moreover, this family of minimal solutions
are stable, and nondecreasing in $\lambda$, therefore the extremal function
$u^*:=\underset{\lambda\to\lambda^*}{\lim}u_\lambda$ exists.
  For the regularity of the extremal function, we first show the
$L^r$-estimates for the equation $(-\Delta)_p^su=g$ with $g\in W_0^{s,
p}(\Omega)^*\cap L^q(\Omega)$, $q\ge 1$. When $f$ is a power-like nonlinearity,
we derive the $W_0^{s,p}(\Omega)$ regularity of $u^*$ in all dimension and
$L^{\infty}(\Omega)$ regularity of $u^*$ in some low dimensions. For more
general nonlinearities, when $f$ is class of $C^2$ and such that some convexity
assumptions, then $u^*\in W_0^{s,p}(\Omega)$ if $N<sp(1+\frac{p}{p-1})$ and
$u^*\in L^{\infty}(\Omega)$ if $N<\frac{sp^2}{p-1}$. Furthermore, when the
limit $\tau=\underset{t\to\infty}{\lim}\frac{f(t)f''(t)}{f'(t)^2}$ exists and
$\tau>\frac{p-2}{p-1}$, the results above can be improved as: $u^*\in
W_0^{s,p}(\Omega)$ for all dimensions, and $u^*\in L^{\infty}(\Omega)$ if
$N<sp+\frac{4sp}{p-1}$.
