  Protoplanetary discs may become dynamically unstable due to structure induced
by an embedded giant planet. In this thesis, I discuss the stability of such
systems and explore the consequence of instability on planetary migration. I
begin with non-self-gravitating, low viscosity discs and show that giant
planets induce shocks inside its co-orbital region, leading to a profile
unstable to vortex formation around a potential vorticity minimum. This
instability is commonly known as the vortex or Rossby wave instability.
Vortex-planet interaction lead to episodic phases of migration, which can be
understood in the framework of type III migration. I then examine the effect of
disc self-gravity on gap stability. The linear theory of the Rossby wave
instability is extended to include disc gravity, which shows that self-gravity
is effective at stabilising the vortex instability at small azimuthal
wavenumber. This is consistent with the observation that more vortices develop
with increasing disc mass in hydrodynamic simulations. Vortices in
self-gravitating discs also resist merging, and is most simply understood as
pair-vortices undergoing mutual horsehoe turns upon encounter. I show that in
sufficiently massive discs vortex modes are suppressed. Instead, global spiral
instabilities develop which are associated with a potential vorticity maximum
at the gap edge. These edge modes can be physically understood as a result of
unstable interaction between the gap edge and the exterior disc through
gravity. I show the spiral arms can provide a positive torque on the planet,
leading to fast migration outwards. I confirm the above results, obtained from
razor-thin disc models, persist in three-dimensions.
