  The electron beam-plasma system is ubiquitous in the space plasma
environment. Here, using a Darwin particle-in-cell method, the excitation of
electrostatic and whistler instabilities by a gyrating electron beam is studied
in support of recent laboratory experiments. It is assumed that the total
plasma frequency $\omega_{pe}$ is larger than the electron cyclotron frequency
$\Omega_e$. The fast-growing electrostatic beam-mode waves saturate in a few
plasma oscillations by slowing down and relaxing the electron beam parallel to
the background magnetic field. Upon their saturation, the finite amplitude
electrostatic beam-mode waves can resonate with the tail of the background
thermal electrons and accelerate them to the beam parallel velocity. The
slower-growing whistler waves are excited in primarily two resonance modes: (a)
through Landau resonance due to the inverted slope of the beam electrons in the
parallel velocity; (b) through cyclotron resonance by scattering electrons to
both lower pitch angles and smaller energies. It is demonstrated that, for a
field-aligned beam, the whistler instability can be suppressed by the
electrostatic instability due to a faster energy transfer rate between beam
electrons and the electrostatic waves. Such a competition of growth between
whistler and electrostatic waves depends on the ratio of
$\omega_{pe}/\Omega_e$. In terms of wave propagation, beam-generated
electrostatic waves are confined to the beam region whereas beam-generated
whistler waves transport energy away from the beam.
