  Discrete and continuous representations of content (e.g., of language or
images) have interesting properties to be explored for the understanding of or
reasoning with this content by machines. This position paper puts forward our
opinion on the role of discrete and continuous representations and their
processing in the deep learning field. Current neural network models compute
continuous-valued data. Information is compressed into dense, distributed
embeddings. By stark contrast, humans use discrete symbols in their
communication with language. Such symbols represent a compressed version of the
world that derives its meaning from shared contextual information.
Additionally, human reasoning involves symbol manipulation at a cognitive
level, which facilitates abstract reasoning, the composition of knowledge and
understanding, generalization and efficient learning. Motivated by these
insights, in this paper we argue that combining discrete and continuous
representations and their processing will be essential to build systems that
exhibit a general form of intelligence. We suggest and discuss several avenues
that could improve current neural networks with the inclusion of discrete
elements to combine the advantages of both types of representations.
