  We present a first physical application of Tsallis' generalized entropy to
the thermodynamics of self-gravitating systems. The stellar system confined in
a spherical cavity of radius $r_e$ exhibits an instability, so-called
gravothermal catastrophe, which has been originally investigated by Antonov
(1962) and Lynden-Bell & Wood (1968) on the basis of the maximum entropy
principle for the phase-space distribution function. In contrast to previous
analyses using the Boltzmann-Gibbs entropy, we apply the Tsallis-type
generalized entropy to seek the equilibrium criteria. Then the distribution
function of Vlassov-Poisson system can be reduced to the stellar polytrope
system. Evaluating the second variation of Tsallis entropy and solving the zero
eigenvalue problem explicitly, we find that the gravothermal instability
appears in cases with polytrope index $n>5$. The critical point characterizing
the onset of instability are obtained, which exactly matches with the results
derived from the standard turning-point analysis. The results give an important
suggestion that the Tsallis entropy is indeed applicable and viable to the
long-range nature of the self-gravitating system.
