  Let $d \geq 4$ be a natural number and let $A$ be a finite, non-empty subset
of $\mathbb{R}^d$ such that $A$ is not contained in a translate of a
hyperplane. In this setting, we show that \[ |A-A| \geq \bigg(2d - 2 +
\frac{1}{d-1} \bigg) |A| - O_{d}(|A|^{1- \delta}), \] for some absolute
constant $\delta>0$ that only depends on $d$. This provides a sharp main term,
consequently answering questions of Ruzsa and Stanchescu up to an
$O_{d}(|A|^{1- \delta})$ error term. We also prove new lower bounds for
restricted type difference sets and asymmetric sumsets in $\mathbb{R}^d$.
