  We present a new approach to modeling visual attributes. Prior work casts
attributes in a similar role as objects, learning a latent representation where
properties (e.g., sliced) are recognized by classifiers much in the way objects
(e.g., apple) are. However, this common approach fails to separate the
attributes observed during training from the objects with which they are
composed, making it ineffectual when encountering new attribute-object
compositions. Instead, we propose to model attributes as operators. Our
approach learns a semantic embedding that explicitly factors out attributes
from their accompanying objects, and also benefits from novel regularizers
expressing attribute operators' effects (e.g., blunt should undo the effects of
sharp). Not only does our approach align conceptually with the linguistic role
of attributes as modifiers, but it also generalizes to recognize unseen
compositions of objects and attributes. We validate our approach on two
challenging datasets and demonstrate significant improvements over the
state-of-the-art. In addition, we show that not only can our model recognize
unseen compositions robustly in an open-world setting, it can also generalize
to compositions where objects themselves were unseen during training.
