  The layered material SrCu$_2$(BO$_3$)$_2$ has long been studied because of
its fascinating physics in a magnetic field and under pressure. Many of its
properties are remarkably well described by the Shastry-Sutherland model (SSM)
- a two-dimensional frustrated spin system. However, the extent of the
intermediate plaquette phase discovered in SrCu$_2$(BO$_3$)$_2$ under pressure
is significantly smaller than predicted in theory, which is likely due to the
weak interlayer coupling that is present in the material but neglected in the
model. Using state-of-the-art tensor network methods we study the SSM with a
weak interlayer coupling and show that the intermediate plaquette phase is
destabilized already at a smaller value around $J''/J\sim0.04-0.05$ than
previously predicted from series expansion. Based on our phase diagram we
estimate the effective interlayer coupling in SrCu$_2$(BO$_3$)$_2$ to be around
$J''/J\sim0.027$ at ambient pressure.
