  In this note we discuss a common misconception, namely that embeddings are
always used to reduce the dimensionality of the item space. We show that when
we measure dimensionality in terms of information entropy then the embedding of
sparse probability distributions, that can be used to represent sparse features
or data, may or not reduce the dimensionality of the item space. However, the
embeddings do provide a different and often more meaningful representation of
the items for a particular task at hand. Also, we give upper bounds and more
precise guidelines for choosing the embedding dimension.
