  Planetesimal formation is one of the most important unsolved problems in
planet formation theory. In particular, rocky planetesimal formation is
difficult because silicate dust grains are easily broken when they collide.
Recently, it has been proposed that they can grow as porous aggregates when
their monomer radius is smaller than $\sim$ 10 nm, which can also avoid the
radial drift toward the central star. However, the stability of a layer
composed of such porous silicate dust aggregates has not been investigated.
Therefore, we investigate the gravitational instability of this dust layer. To
evaluate the disk stability, we calculate Toomre's stability parameter $Q$, for
which we need to evaluate the equilibrium random velocity of dust aggregates.
We calculate the equilibrium random velocity considering gravitational
scattering and collisions between dust aggregates, drag by mean flow of gas,
stirring by gas turbulence, and gravitational scattering by gas density
fluctuation due to turbulence. We derive the condition of the gravitational
instability using the disk mass, dust-to-gas ratio, turbulent strength, orbital
radius, and dust monomer radius. We find that, for the minimum mass solar
nebula model at 1 au, the dust layer becomes gravitationally unstable when the
turbulent strength $\alpha\lesssim10^{-5}$. If the dust-to-gas ratio is
increased twice, the gravitational instability occurs for
$\alpha\lesssim10^{-4}$. We also find that the dust layer is more unstable in
disks with larger mass, higher dust-to-gas ratio, and weaker turbulent
strength, at larger orbital radius, and with a larger monomer radius.
