  This is a continuation of our work to understand vertex operator algebras
using the geometric properties of varieties attached to vertex operator
algebras. For a class of vertex operator algebras including affine vertex
operator algebras associated to a finite dimensional simple Lie algebra
$\mathfrak{g}$, we describe their varieties of semi-conformal vectors by some
matrix equations. These matrix equations are too complicated to be solved for
us. However, for affine vertex operator algebras associated to the simple Lie
algebra $\mathfrak{g}$, we find the adjoint group $G$ of $\mathfrak{g}$ acts on
the corresponding varieties by a natural way, which implies that such varieties
should be described more clearly by studying the corresponding $G$-orbit
structures. Based on above methods for general cases, as an example,
considering affine vertex operator algebras associated to the Lie algebra
$\mathfrak{sl}_2(\mathbb{C})$, we shall give the decompositions of $G$-orbits
of varieties of their semi-conformal vectors according to different levels. Our
results imply that such orbit structures depends on the levels of affine vertex
operator algebras associated to a finite dimensional simple Lie algebra
$\mathfrak{g}$
