  We present a novel deep zero-shot learning (ZSL) model for inferencing
human-object-interaction with verb-object (VO) query. While the previous
two-stream ZSL approaches only use the semantic/textual information to be fed
into the query stream, we seek to incorporate and embed the semantics into the
visual representation stream as well. Our approach is powered by
Semantics-to-Space (S2S) architecture where semantics derived from the residing
objects are embedded into a spatial space of the visual stream. This
architecture allows the co-capturing of the semantic attributes of the human
and the objects along with their location/size/silhouette information. To
validate, we have constructed a new dataset, Verb-Transferability 60 (VT60).
VT60 provides 60 different VO pairs with overlapping verbs tailored for testing
two-stream ZSL approaches with VO query. Experimental evaluations show that our
approach not only outperforms the state-of-the-art, but also shows the
capability of consistently improving performance regardless of which ZSL
baseline architecture is used.
