  Embedding based models have been the state of the art in collaborative
filtering for over a decade. Traditionally, the dot product or higher order
equivalents have been used to combine two or more embeddings, e.g., most
notably in matrix factorization. In recent years, it was suggested to replace
the dot product with a learned similarity e.g. using a multilayer perceptron
(MLP). This approach is often referred to as neural collaborative filtering
(NCF). In this work, we revisit the experiments of the NCF paper that
popularized learned similarities using MLPs. First, we show that with a proper
hyperparameter selection, a simple dot product substantially outperforms the
proposed learned similarities. Second, while a MLP can in theory approximate
any function, we show that it is non-trivial to learn a dot product with an
MLP. Finally, we discuss practical issues that arise when applying MLP based
similarities and show that MLPs are too costly to use for item recommendation
in production environments while dot products allow to apply very efficient
retrieval algorithms. We conclude that MLPs should be used with care as
embedding combiner and that dot products might be a better default choice.
