  Deep convolutional network has been the state-of-the-art approach for a wide
variety of tasks over the last few years. Its successes have, in many cases,
turned it into the default model in quite a few domains. In this work, we will
demonstrate that convolutional networks have limitations that may, in some
cases, hinder it from learning properties of the data, which are easily
recognizable by traditional, less demanding, models. To this end, we present a
series of competitive analysis studies on image recognition and text analysis
tasks, for which convolutional networks are known to provide state-of-the-art
results. In our studies, we inject a truth-revealing signal, indiscernible for
the network, thus hitting time and again the network's blind spots. The signal
does not impair the network's existing performances, but it does provide an
opportunity for a significant performance boost by models that can capture it.
The various forms of the carefully designed signals shed a light on the
strengths and weaknesses of convolutional network, which may provide insights
for both theoreticians that study the power of deep architectures, and for
practitioners that consider applying convolutional networks to the task at
hand.
