  We study the effect of disc self-gravity on vortex-forming instabilities
associated with gaps opened by a Saturn mass planet in a protoplanetary disc.
It is shown analytically and confirmed through linear calculations that vortex
modes with low azimuthal mode number $,m,$ are stabilised by increasing
self-gravity if the basic state is fixed. However, linear calculations show
that the combined effect of self-gravity through the background and through the
linear response shifts the most unstable vortex mode to higher $m.$ Nonlinear
hydrodynamic simulations of planetary gaps show more vortices develop with
increasing strength of self-gravity. For sufficiently large disc mass the
vortex instability is suppressed and replaced by a new global instability,
consistent with analytical expectations. In the nonlinear regime, vortex
merging is increasingly delayed as the disc mass increases and multiple
vortices may persist until the end of simulations. With self-gravity, the
post-merger vortex is localised in azimuth and has similar structure to a
Kida-like vortex. This is unlike the case without self-gravity where vortices
merge to form a single vortex extended in azimuth. We also performed a series
of supplementary simulations of co-orbital Kida-like vortices and found that
self-gravity enables such vortices to execute horseshoe turns upon encountering
each other. As a result vortex merging is avoided on time-scales where it would
occur without self-gravity. Thus we suggest that mutual repulsion of
self-gravitating vortices in a rotating flow is responsible for the delayed
vortex merging above. The effect of self-gravity on vortex-induced migration is
briefly discussed. We found that when self-gravity is included, the
vortex-induced type III migration of Lin & Papaloizou (2010) is delayed but the
extent of migration is unchanged.
