  We show how to construct an N=1 superconformal vertex algebra (SCVA) from any
Riemannian manifold. When the Riemannian manifold has special holonomy groups,
we discuss the extended supersymmetry. When the manifold is complex or
K\"{a}hler, we also generalize the construction to obtain N=2 SCVA's. We study
the BRST cohomology groups of the topological vertex algebras obtained by the
$A$ twist and the $B$ twist from these N=2 SCVA's. We show that for one of
them, the BRST cohomologies are isomorphic to $H^*(M, \Lambda^*(T^*M))$ and
$H^*(M, \Lambda^*(TM))$ respectively. This provides a mathematical formulation
of the $A$ theory and $B$ theory in physics literature. The connection with
elliptic genera is also discussed. Furthermore, when the manifold is
hyperk\"{a}hler, we generalize our constructions to obtain N=4 SCVA's. A
heuristic relationship with super loop space is also discussed.
