  We address the task of annotating images with semantic tuples. Solving this
problem requires an algorithm which is able to deal with hundreds of classes
for each argument of the tuple. In such contexts, data sparsity becomes a key
challenge, as there will be a large number of classes for which only a few
examples are available. We propose handling this by incorporating feature
representations of both the inputs (images) and outputs (argument classes) into
a factorized log-linear model, and exploiting the flexibility of scoring
functions based on bilinear forms. Experiments show that integrating feature
representations of the outputs in the structured prediction model leads to
better overall predictions. We also conclude that the best output
representation is specific for each type of argument.
