  The world is fundamentally compositional, so it is natural to think of visual
recognition as the recognition of basic visually primitives that are composed
according to well-defined rules. This strategy allows us to recognize unseen
complex concepts from simple visual primitives. However, the current trend in
visual recognition follows a data greedy approach where huge amounts of data
are required to learn models for any desired visual concept. In this paper, we
build on the compositionality principle and develop an "algebra" to compose
classifiers for complex visual concepts. To this end, we learn neural network
modules to perform boolean algebra operations on simple visual classifiers.
Since these modules form a complete functional set, a classifier for any
complex visual concept defined as a boolean expression of primitives can be
obtained by recursively applying the learned modules, even if we do not have a
single training sample. As our experiments show, using such a framework, we can
compose classifiers for complex visual concepts outperforming standard
baselines on two well-known visual recognition benchmarks. Finally, we present
a qualitative analysis of our method and its properties.
