  Bias elimination and recent probing studies attempt to remove specific
information from embedding spaces. Here it is important to remove as much of
the target information as possible, while preserving any other information
present. INLP is a popular recent method which removes specific information
through iterative nullspace projections. Multiple iterations, however, increase
the risk that information other than the target is negatively affected. We
introduce two methods that find a single targeted projection: Mean Projection
(MP, more efficient) and Tukey Median Projection (TMP, with theoretical
guarantees). Our comparison between MP and INLP shows that (1) one MP
projection removes linear separability based on the target and (2) MP has less
impact on the overall space. Further analysis shows that applying random
projections after MP leads to the same overall effects on the embedding space
as the multiple projections of INLP. Applying one targeted (MP) projection
hence is methodologically cleaner than applying multiple (INLP) projections
that introduce random effects.
