  A common assumption about neural networks is that they can learn an
appropriate internal representations on their own, see e.g. end-to-end
learning. In this work we challenge this assumption. We consider two simple
tasks and show that the state-of-the-art training algorithm fails, although the
model itself is able to represent an appropriate solution. We will demonstrate
that encouraging an appropriate internal representation allows the same model
to solve these tasks. While we do not claim that it is impossible to solve
these tasks by other means (such as neural networks with more layers), our
results illustrate that integration of domain knowledge in form of a desired
internal representation may improve the generalization ability of neural
networks.
