  How the quiescent galaxies evolve with redshift and the factors that impact
their evolution are still debated. It is still unclear what the dominant
mechanisms of passive galaxy growth are and what role is played by the
environment in shaping their evolutionary paths over cosmic time. Our aim is to
study the mass-size relation (MSR) of the quiescent population and to
understand how the environment shapes the MSR at intermediate redshift. We used
the VIPERS, a large spectroscopic survey of $\sim$90~000 galaxies in the
redshift range $0.5\le z \le 1.2$. We selected a mass-complete sample of 4786
passive galaxies based on the NUVrK diagram and refined it using the $D_n4000$
spectral index to study the MSR of the passive population over $0.5\le z \le
0.9$. The impact of the environment on the MSR and on the growth of the
quiescent population is studied through the density contrast. The slope and the
intercept of the MSR, $\alpha=0.62\pm 0.04$ and $\textrm{log}(A)=0.52\pm 0.01$,
agree well with values from the literature at the same redshift. The intercept
decreases with redshift, $R_e(z)=8.20\times (1+z)^{-1.70}$, while the slope
remains roughly constant, and the same trend is observed in the low-density
(LD) and high-density (HD) environments. We find that the average size of the
quiescent population in the LD and HD environments are identical within
$3\sigma$ and this result is robust against a change in the definition of the
LD and HD environments or a change in the selection of quiescent galaxies. In
the LD and HD environments, $\sim$30 and $\sim$40% of the population have
experienced a minor merger process between $0.5\le z \le 0.9$. However, minor
mergers account only for 30 to 40% of the size evolution in this redshift
range, the remaining evolution likely being due to the progenitor bias.
