---
abstract: 'The recent progress on image recognition and language modeling is making automatic description of image content a reality. However, stylized, non-factual aspects of the written description are missing from the current systems. One such style is descriptions with emotions, which is commonplace in everyday communication, and influences decision-making and interpersonal relationships. We design a system to describe an image with emotions, and present a model that automatically generates captions with positive or negative sentiments. We propose a novel switching recurrent neural network with word-level regularization, which is able to produce emotional image captions using only 2000+ training sentences containing sentiments. We evaluate the captions with different automatic and crowd-sourcing metrics. Our model compares favourably in common quality metrics for image captioning. In 84.6% of cases the generated positive captions were judged as being at least as descriptive as the factual captions. Of these positive captions 88% were confirmed by the crowd-sourced workers as having the appropriate sentiment.'
author:
- |
    Alexander Mathews$^*$, Lexing Xie$^{*\dag}$, Xuming He$^{\dag *}$\
    $^*$The Australian National University, $^\dag$NICTA\
    alex.mathews@anu.edu.au, lexing.xie@anu.edu.au, xuming.he@nicta.com.au
title: 'SentiCap: Generating Image Descriptions with Sentiments'
---

Introduction {#sec:intro}
============

Automatically describing an image by generating a coherent sentence unifies two core challenges in artificial intelligence – vision and language. Despite being a difficult problem, the research community has recently made headway into this area, thanks to large labeled datasets, and progresses in learning expressive neural network models. In addition to composing a factual description about the objects, scene, and their interactions in an image, there are richer variations in language, often referred to as styles [@crystal1969investigating]. Take emotion, for example, it is such a common phenomena in our day-to-day communications that over half of text accompanying online pictures contains an emoji (a graphical alphabet for emotions) [@instagram2015emoji]. How well emotions are expressed and understood [influences]{} decision-making [@lerner2015emotion] – from the mundane (e.g., making a restaurant menu appealing) to major (e.g., choosing a political leader in elections). Recognizing [sentiment and opinions]{} from written communications has been an active research topic for the past decade [@pang2008opinion; @socherrecursive], the synthesis of text with sentiment that is relevant to a given image is still an open problem. In Figure \[fig:intro\], [each]{} image is described with a factual caption, and with positive or negative emotion[, respectively]{}. One may argue that the descriptions with sentiments are more likely to pique interest about the subject being pictured (the dog and the motocycle), or about their background settings (interaction with the dog at home, or how the motocycle came about).

![Example images with neural, positive () and negative () captions, by crowd workers in MSCOCO dataset [@chen2015microsoft] and this work (Section \[sec:mturk\]). []{data-label="fig:intro"}](fig/intro_example){width=".4\textwidth"}

In this paper, we describe a method, called SentiCap, to generate image captions with sentiments. We build upon the CNN+RNN (Convolution Neural Network + Recurrent Neural Network) recipe that has seen many recent successes [@donahue2015long; @Karpathy2015CVPR; @mao2014deep; @vinyals2015show; @xu2015show]. In particular, we propose a switching Recurrent Neural Network (RNN) model to represent sentiments. This model consists of two parallel RNNs – one represents a general background language model; another specialises in descriptions with sentiments. We design a novel word-level regularizer, so as to emphasize the sentiment words during training [and to optimally combine the two RNN streams]{} (Section \[sec:model\]). We have gathered a new dataset of several thousand captions with positive and negative sentiments by re-writing factual descriptions (Section \[sec:mturk\]). [Trained on 2000+ sentimental captions and 413K neutral captions, our switching RNN out-performs a range of heuristic and learned baselines in the number of emotional captions generated, and in a [[[variety]{}]{}]{} of subjective and human evaluation metrics. In particular SentiCap has the highest number of success in placing at least one sentiment word into the caption, 88% positive (or 72% negative) captions are perceived by crowd workers as more positive (or negative) than the factual caption, with a similar descriptiveness rating. ]{}

Related Work
============

[Recent]{} advances in visual recognition have made “an image is a thousand words” much closer to reality, [largely due to the advances in Convolutional Neural Networks (CNN) [@Simonyan2015; @Szegedy_2015_CVPR]]{}. [A related topic also advancing rapidly is image captioning, where most early systems were]{} [based on similarity retrieval using objects and attributes [@farhadi2010every; @kulkarni2011baby; @hodosh2013framing; @Gupta2012a], and assembling sentence fragments such as object-action-scene [@farhadi2010every], subject-verb-object [@rohrbach2013translating], object-attribute-prepositions [@kulkarni2011baby] or global image properties such as scene and lighting [@Nwogu2011].]{} [Recent systems model richer language structure, such as formulating a integer linear program to map visual elements to the parse tree of a sentence [@kuznetsova2014treetalk], or embedding  [@Xu2015] video and compositional semantics into a joint space.]{}

Word-level language models [such as RNNs [@mikolov2011strategies; @sutskever2011generating] and maximum-entropy (max-ent) language models [@mikolov2011strategies] have improved with the aid of]{} significantly larger datasets and more computing power. Several research teams independently proposed image captioning systems that combine CNN-based image representation and [such]{} language models. Fang [et al.]{}  used a cascade of word detectors from images and a [max-ent]{} model. The Show and Tell [@vinyals2015show] system used an RNN as the language model, seeded by CNN image features. Xu [et al.]{}  estimated spatial attention as a latent variable, to make the Show and Tell system aware of local image information. Karpathy and Li[ ]{} used an [RNN to generate a sentence from the alignment between objects and words]{}. [Other work has]{} [employed multi-layer RNNs [@Chen_2015_CVPR; @donahue2015long]]{} [for image captioning.]{} [Most]{} RNN-based multimodal language models [use]{} the Long Short Term Memory (LSTM) unit that preserves long-term information [and prevents]{} overfitting [@hochreiter1997long]. We adopt one of the competitive systems [@vinyals2015show] – CNN+RNN with LSTM units as our basic multimodal sentence generation engine, due to its simplicity and computational efficiency.

[Researchers have modeled]{} how [an]{} image is presented, and what kind of response it is likely to elicit from viewers, such as analyzing the aesthetics and emotion in images [@murray2012ava; @joshi2011aesthetics]. More recently, the Visual SentiBank [@borth2013sentibank] system constructed a catalogue of Adjective-Noun-Pairs (ANPs) that are frequently used to describe online images. We build upon Visual SentiBank to construct sentiment vocabulary, but to the best of our knowledge, no existing work tries to compose image descriptions with desired sentiments. [Identifying sentiment in text is an active area of research[ [@pang2008opinion; @socherrecursive]]{}. Several [teams]{} [@Nakagawa2010; @Mcdonald2011] [designed sentence models with latent variables representing the sentiment.]{} [Our work focuses on]{} generating sentences and not explicitly modelling sentiment using hidden variables.]{}

Describing an Image with Sentiments {#sec:model}
===================================

Given an image $I$ and its $D_x$-dimensional visual feature ${\mathbf{x}}\in{{\mathbb R}}^{D_x}$, our goal is to generate a sequence of words (i.e. a caption) ${\mathbf{Y}}=\{{\mathbf{y}}_1,\cdots,{\mathbf{y}}_T\}$ to describe the image with a specific style, such as expressing sentiment. Here ${\mathbf{y}}_t\in\{0,1\}^V$ is 1-of-V encoded indicator vector for the $t^{th}$ word; $V$ is the size of the vocabulary; and $T$ is the length of the caption.

We assume [[[that]{}]{}]{} sentence generation involves two underlying mechanisms, one of which focuses on the factual description of the image while the other describes the image content with sentiments. [We formulate such caption generation process using a switching multi-modal language model, which sequentially generates words in a sentence.]{} [Formally]{}, we introduce a binary [sentiment]{} variable $s_t\in\{0,1\}$ for every word ${\mathbf{y}}_t$ to indicate which mechanism is used. At each time step $t$, our model produces the probability of ${\mathbf{y}}_t$ and the current sentiment variable $s_t$ given the image feature ${\mathbf{x}}$ and the previous words ${\mathbf{y}}_{1:t-1}$, denoted by $p({\mathbf{y}}_t,s_t|{\mathbf{x}},\mathbf{y}_{1:t-1})$. We generate the word probability by marginalizing [over]{} the sentiment variable $s_t$: $$\resizebox{0.25\columnwidth}{!}{$
p({\mathbf{y}}_t|{\mathbf{x}},{\mathbf{y}}_{1:t-1})$}=\sum_{s_t}\resizebox{0.525\columnwidth}{!}{$p({\mathbf{y}}_t|s_t,{\mathbf{x}},{\mathbf{y}}_{1:t-1})p(s_t|{\mathbf{x}},{\mathbf{y}}_{1:t-1})
$}
\label{eq:bayesrule_s}
{\vspace{-0.mm}}$$ Here $p({\mathbf{y}}_t|s_t,\cdot)$ is the caption model conditioned on the sentiment variable and [$p(s_t|\cdot)$ is the probability of the word sentiment]{}. The rest of this section will introduce these components [and model learning]{} in detail.

Switching RNNs for Sentiment Captions {#ssec:sentimodel}
-------------------------------------

We adopt a joint CNN+RNN architecture [@vinyals2015show] in the conditional caption model. Our full model combines two CNN+RNNs running in parallel: one capturing the [factual word generation (referred to as the background language model)]{}, the other specializing in words with sentiment. [The full model is a switching RNN, in which the variable $s_t$ functions as a switching gate.]{} This model design aims to learn sentiments well, despite data sparsity – using only a small dataset [of]{} image description with sentiments (Section \[sec:mturk\]), with the help from millions of image-sentence pairs that factually describe pictures [@chen2015microsoft].

Each RNN stream [consists of]{} a series of LSTM units. Formally, we denote the $D$-dimensional hidden state of an LSTM as ${\mathbf{h}}_t \in {{\mathbb R}}^D$, its memory cell as ${\mathbf{c}}_t\in {{\mathbb R}}^D$, the input, output, forget gates as ${\mathbf{i}}_t,~{\mathbf{o}}_t,~{\mathbf{f}}_t \in {{\mathbb R}}^D$, [respectively.]{} [Let $k$ indicate which RNN stream it is,]{} the LSTM can be implemented as: $$\begin{aligned}
&\begin{pmatrix} {\mathbf{i}}_t^k \\ {\mathbf{f}}_t^k \\ {\mathbf{o}}_t^k \\ {\mathbf{g}}_t^k \end{pmatrix}  = 
\begin{pmatrix} \sigma \\ \sigma \\ \sigma \\\tanh \end{pmatrix} \mathlarger{{\mathbf{T}}}^k{}
\begin{pmatrix} {\mathbf{E}}^k{\mathbf{y}}_{t-1} \\ {\mathbf{h}}^k_{t-1} \end{pmatrix}
\label{eq:lstm}\\
&{\mathbf{c}}_t^k = {\mathbf{f}}_t^k \odot {\mathbf{c}}_{t-1}^k + {\mathbf{i}}_t^k \odot {\mathbf{g}}_t^k, \quad
{\mathbf{h}}_t^k = {\mathbf{o}}_t^k \odot {{\mathbf{c}}_t^k}. \nonumber
{\vspace{-0.mm}}\end{aligned}$$ Here $\sigma(\chi)$ is the sigmoid function $1/(1+e^{-\chi})$; $tanh$ is the hyperbolic tangent function; [${\mathbf{T}}^k\in{{\mathbb R}}^{4D\times2D}$]{} is a set of learned weights; ${\mathbf{g}}^k_t\in{{\mathbb R}}^D$ is the input to the memory cell; [${\mathbf{E}}^k \in {{\mathbb R}}^{D\times V}$]{} is a learned embedding matrix [in model $k$]{}, and ${\mathbf{E}}^k{\mathbf{y}}_t$ is the embedding vector of the word ${\mathbf{y}}_t$.

To incorporate image information, we use an image representation $\hat{\mathbf{x}}={\mathbf{W}}_x{\mathbf{x}}$ as the word embedding ${\mathbf{Ey}}_0$ when $t=1$, where ${\mathbf{x}}$ is a high-dimensional image feature extracted from a convolutional neural network [@Simonyan2015], and ${\mathbf{W}}_x$ is a learned embedding matrix. Note that the LSTM hidden state ${\mathbf{h}}_t^k$ summarizes ${\mathbf{y}}_{1:t-1}$ and ${\mathbf{x}}$. The conditional probability of the output caption words depends on the hidden state of the corresponding LSTM, $$\begin{aligned}
p({\mathbf{y}}_t | s_t=k, {\mathbf{x}}, {\mathbf{y}}_{1:t-1}) &\propto \exp({\mathbf{W}}_y^{k} {\mathbf{h}}_t^k) {\vspace{-0.mm}}\end{aligned}$$ where ${\mathbf{W}}_y^k \in {{\mathbb R}}^{D\times V}$ is a set of learned output weights.

[The]{} sentiment switching model generates the probability of switching [between the two RNN streams]{} at each time $t$, [with]{} a single layer network taking the hidden states of both RNNs as input: $$\begin{aligned}
p(s_t=1 | {\mathbf{x}}, {\mathbf{y}}_{1:t-1}) &= \sigma({\mathbf{W}}_s[{\mathbf{h}}^0_t;{\mathbf{h}}^1_t])\label{eq:switch}
{\vspace{-0.mm}}\end{aligned}$$ where ${\mathbf{W}}_s$ is the weight matrix for the hidden states.

[An illustration of this sentiment switching model is in Figure \[fig:rnn\].]{} In summary, the parameter set for each RNN ($k=\{0,1\}$) is [$\Theta^k=\{{\mathbf{T}}^k, {\mathbf{W}}_y^k, {\mathbf{E}}^k, {\mathbf{W}}_x^k\}$]{}, and that of the switching RNN is [$\Theta=\Theta^0\cup\Theta^1\cup {\mathbf{W}}_s$]{}. We have tried including ${\mathbf{x}}$ for learning $p(s_t|{\mathbf{x}}, {\mathbf{y}}_{1:t-1})$ but found no benefit.

![Illustration of the switching RNN model for captions with sentiment. Lines with diamonds denote projections with learned weights. LSTM cells are described in Eq \[eq:lstm\]. $\gamma_t^0$ and $\gamma_t^1$ are probabilities of sentiment switch defined in Eq (\[eq:switch\]) and act as gating functions for the two streams. []{data-label="fig:rnn"}](fig/parrallel_rnn){width=".35\textwidth"}

Learning the Switching RNN Model
--------------------------------

[One of the key challenges is to design a learning scheme for $p(s_t | {\mathbf{x}}, {\mathbf{y}}_{1:t-1})$ and two CNN+RNN components. We take a two-stage learning approach to estimate the parameters $\Theta$ in our switching RNN model based on a large dataset with factual captions and a small set with sentiment captions.]{}

**Learning a background multi-modal RNN.** We first train a CNN+RNN with a large dataset of image and caption pairs, denoted as $\mathcal{D}^0=\{({\mathbf{x}}_0^i,{\mathbf{y}}_0^i)\}_{i=1}^N$. $\Theta^0$ are learned by minimizing the negative log-likelihood of the caption words given images, [ $$\label{eqn:baselearn}
\resizebox{0.26\columnwidth}{!}{$   
    L^0(\Theta^0,\mathcal{D}^0)
= -$}\sum_i\sum_t \resizebox{0.5\columnwidth}{!}{$\log p({\mathbf{y}}^i_{0,t}|s_t=0,{\mathbf{x}}^i_0,{\mathbf{y}}^i_{0,1:t-1} ).$}
{\vspace{-0.mm}}$$ ]{}

**Learning from captions with sentiments.** [Based on the pre-trained CNN+RNN in Eq , we then learn the switching RNN using a small image caption dataset with a specific sentiment polarity, denoted as $\mathcal{D}=\{({\mathbf{x}}^i,{\mathbf{y}}^i,\eta^i)\}_{i=1}^M$, $M\ll N$. Here $\eta_t^i \in [0, 1]$ is the sentiment strength of the $t^{th}$ word in the $i$-th training sentence,]{} [being either positive or negative as specified in the training data.]{}

[We design a new training objective function to use word-level sentiment information for learning $\Theta^1$ and the switching weights ${\mathbf{W}}_s$, while keeping the pre-learned $\Theta^0$ fixed. For clarity, we denote the sentiment probability as: $$\begin{aligned}
\gamma^{0}_t = p(s_t = 0|{\mathbf{x}},{\mathbf{y}}_{1:t-1}), \quad\gamma^{1}_t = 1-\gamma^{0}_t;  \label{eq:switch}
{\vspace{-0.mm}}\end{aligned}$$ and the log likelihood of generating a new word ${\mathbf{y}}_t$ given image and word histories $({\mathbf{x}},{\mathbf{y}}_{1:t-1})$ as $L_t(\Theta,{\mathbf{x}},{\mathbf{y}})$, which can be written as (cf. Eq ), $$\begin{aligned}
L_t&(\Theta,{\mathbf{x}},{\mathbf{y}})=  \log p({\mathbf{y}}_t | {\mathbf{x}}, {\mathbf{y}}_{1:t-1})= \\ &\log [\gamma^0_t p({\mathbf{y}}_t|s_t=0,{\mathbf{x}},{\mathbf{y}}_{-t} ) + 
 \gamma^1_t p({\mathbf{y}}_t|s_t=1,{\mathbf{x}},{\mathbf{y}}_{-t} )].\nonumber   \end{aligned}$$ The overall learning objective function for incorporating word sentiment is a combination of a weighted log likelihood and the cross-entropy between $\gamma_t$ and [$\eta_t$]{}, $$\begin{aligned}
{\cal L}(\Theta,\mathcal{D}) &= -\sum_i\sum_t (1+ \lambda_\eta \eta^i_t ) 
[L_t(\Theta,{\mathbf{x}}^i,{\mathbf{y}}^i)  \label{eq:giantr}
\\ &+ \lambda_\gamma (\eta_t^i \log\gamma^{1,i}_t + (1-\eta_t^i) \log\gamma^{0,i}_t)  ]  + R(\Theta),\nonumber\\
R(\Theta)=&\frac{\lambda_\theta}{2}\|\Theta^1 - \Theta^0\|^2 
\label{eq:modreg}\end{aligned}$$ where $\lambda_\eta$ and $\lambda_\gamma$ are weight parameters, and $R(\Theta)$ is the regularization term with weight parameter $\lambda_\theta$. Intuitively,]{} when $\eta_t > 0$, i.e. the training sentence encounters [a sentiment word]{}, the likelihood weighting factor $\lambda_\eta \eta^i_t$ [increases the importance of $L_t$ in the overall likelihood; at the same time, the cross-entropy term $\lambda_\gamma (\eta_t^i \log\gamma^{1,i}_t + (1-\eta_t^i) \log\gamma^{0,i}_t) $ encourage switching variable $\gamma^1_t$ to be $>0$, emphasizing the new model. The regularized training finds a trade-off between the [data]{} likelihood and L2 difference between the current and base RNN, [and is one of the most competitive approaches in domain transfer [@schweikert2008empirical]]{}.]{}

[[**[Settings for model learning.]{}**]{}]{} We use stochastic gradient descent with backpropagation on mini-batches to optimize the RNNs. We apply dropout to the input of each step, which is either the image embedding $\hat{\mathbf{x}}$ [for $t=1$]{} or the word embedding ${\mathbf{E}}^k{\mathbf{y}}_{t-1}$ and the hidden output ${\mathbf{h}}^k_{t-1}$ from time $t-1$, for both the background and sentiment streams $k=0,1$.

We learn models for positive and negative sentiments separately, due to the observation that either sentiment could be valid for the majority of images (Section \[sec:mturk\]). [We initialize $\Theta^1$ as $\Theta^0$ and use the following gradient of to minimize ${\cal L}(\Theta,\mathcal{D})$ with respect to $\Theta^1$ and ${\mathbf{W}}_s$, holding $\Theta^0$ fixed. $$\begin{aligned}
{\frac{\partial \cal L}{\partial \Theta}} = &-\sum_i\sum_t (1+ \lambda_\eta \eta^i_t ) [{\frac{\partial L_t}{\partial \Theta}} \nonumber\\&+ \lambda_\gamma ( \frac{\eta_t^i}{\gamma^{1,i}_t}{\frac{\partial \gamma^{1,i}_t}{\partial \Theta}} + \frac{1-\eta_t^i}{\gamma^{0,i}_t} {\frac{\partial \gamma^{0,i}_t}{\partial \Theta}}) ] +  {\frac{\partial R(\Theta)}{\partial \Theta}}
{\vspace{-0.mm}}\end{aligned}$$ ]{}Here ${\frac{\partial L_t}{\partial \Theta}},{\frac{\partial \gamma^{0,i}_t}{\partial \Theta}},\text{ and }{\frac{\partial \gamma^{1,i}_t}{\partial \Theta}}$ are computed through differentiating across Equations (\[eq:bayesrule\_s\])–(\[eq:switch\]). During training, we set $\eta_t=1$ when word ${\mathbf{y}}_t$ is part of an ANP with the target sentiment polarity, otherwise $\eta_t=0$. We also include a default L2-norm regularization for neural network tuning $|\Theta|^2$ with a small weight ($10^{-8}$). We automatically search for the hyperparameters $\lambda_\theta$, $\lambda_\eta$ and $\lambda_\gamma$ on a validation set using Whetlab [@snoek2012practical].

An Image Caption Dataset with Sentiments {#sec:mturk}
========================================

![image](fig/gt_rating){width=".9\textwidth"}

In order to learn the association between images and captions with sentiments, we build a novel dataset of image-caption pairs where the caption both describes an image, and also convey the desired sentiment. [We summarize the new dataset, and the crowd-sourcing task to collect image-sentiment caption data. More details of the data collection process are included in the suplementary.]{}

There are many ways a photo could evoke emotions. In this work, we focus on creating a collection and learning sentiments [*from an objective viewer*]{} who does not know the back story outside of the photo – a setting also used by recent collections of objectively descriptive image captions [@chen2015microsoft; @hodosh2013framing].

[[**[Dataset construction.]{}**]{}]{} We design a crowd-sourcing task to collect such objectively described emotional image captions. This is done in a caption re-writing task based upon objective captions from MSCOCO [@chen2015microsoft] by asking Amazon Mechanical Turk (AMT) workers to choose among ANPs of the desired sentiment, and incorporate one or more of them into any one of the five existing captions. Detailed design of the AMT task is in the appendix.

The set of candidate ANPs required for this task is collected from the captions for a large sets of online images. We expand the Visual SentiBank [@borth2013sentibank] vocabulary with a set of ANPs from the YFCC100M image captions [@thomee2015yfcc100m] [as]{} the overlap between the original SentiBank ANPs and the MSCOCO images is insuffcient. We keep ANPs with non-trival frequency and a clear positive or negative sentiment, when rated in the same way as SentiBank. This gives us 1,027 ANPs with a positive emotion, 436 with negative emotions. We collect at least 3 positive and 3 negative captions per image. Figure \[fig:mturk\_eval\](a) contains one example image and its respective positive and negative caption written by AMT workers. We release the list of ANPs and the captions in the online appendix.

[[**[Quality validation.]{}**]{}]{} We validate the quality of the resulting captions with another [two-question]{} AMT task as detailed in the suppliment. This validation is done on [124 images with 3 neutral captions from MSCOCO]{}, and images with 3 positive and 3 negative captions from our dataset. We first ask AMT workers to rate the descriptiveness of a caption for a given image on a four-point scale [@hodosh2013framing; @vinyals2015show]. The [*descriptiveness*]{} column in Figure \[fig:mturk\_eval\](b), shows that the measure for objective descriptiveness tend to decrease when the caption contains additional sentiment. Ratings for the positive captions ([Pos]{}) have a small decrease (by 0.08, or one-tenth of the standard deviation), while those for the negative captions ([Neg]{}) have a significant decrease (by 0.73), likely because the notion of negativity is diverse.

[We also ask whether the sentiment of the sentence matches the image. Each rating task is completed by 3 different AMT workers.]{} In the [*correct sentiment*]{} column of Figure \[fig:mturk\_eval\](b), we record the number of votes each caption received for bearing a sentiment that matches the image. We can see that the vast majority of the captions are unanimously considered emotionally appropriate ($94\%$, or 315/335 for [Pos]{}; $82\%$, or 250/305 for [Neg]{}). Among the captions with less than unanimous votes received, most of them (20 for [Pos]{} and 49 for [Neg]{}) still have majority agreement for having the correct sentiment, which is on par with the level of noise (16 for [Coco]{} captions).

Experiments {#sec:exp}
===========

                    [sen]{}%   [B-1]{}   [B-2]{}   [B-3]{}   [B-4]{}   [Rouge]{}$_L$   [Meteor]{}   [Cide]{}$_r$   [Senti]{}   [Desc]{}        [DescCmp]{}
  -- -------------- ---------- --------- --------- --------- --------- --------------- ------------ -------------- ----------- --------------- -------------
     CNN+RNN        1.0        48.7      28.1      17.0      10.7      36.6            15.3         55.6           –           2.90$\pm$0.90   –
     ANP-Replace    90.3       48.2      27.8      16.4      10.1      36.6            16.5         55.2           84.8%       2.89$\pm$0.92   95.0%
     ANP-Scoring    90.3       48.3      27.9      16.6      10.1      36.5            16.6         55.4           84.8%       2.86$\pm$0.96   95.3%
     RNN-Transfer   86.5       49.3      29.5      17.9      10.9      37.2            17.0         54.1           84.2%       2.73$\pm$0.96   76.2%
     SentiCap       93.2       49.1      29.1      17.5      10.8      36.5            16.8         54.4           88.4%       2.86$\pm$0.97   84.6%
     CNN+RNN        0.8        47.6      27.5      16.3      9.8       36.1            15.0         54.6           –           2.81$\pm$0.94   –
     ANP-Replace    85.5       48.1      28.8      17.7      10.9      36.3            16.0         56.5           61.4%       2.51$\pm$0.93   73.7%
     ANP-Scoring    85.5       47.9      28.7      17.7      11.1      36.2            16.0         57.1           64.5%       2.52$\pm$0.94   76.0%
     RNN-Transfer   73.4       47.8      29.0      18.7      12.1      36.7            16.2         55.9           68.1%       2.52$\pm$0.96   70.3%
     SentiCap       97.4       50.0      31.2      20.3      13.1      37.9            16.8         61.8           72.5%       2.40$\pm$0.89   65.0%

[[**[Implementation details.]{}**]{}]{} We implement RNNs with LSTM units using the Theano package [@BastienTheano2012]. Our implementation of CNN+RNN reproduces caption generation performance in recent work [@Karpathy2015CVPR]. The visual input to the switching RNN is 4096-dimensional feature vector from the second last layer of the Oxford VGG CNN [@Simonyan2015]. These features are [linearly embedded into a $D=512$ dimensional space]{}. Our word embeddings ${\mathbf{Ey}}$ are 512 dimensions and the hidden state ${\mathbf{h}}$ and memory cell ${\mathbf{c}}$ of the LSTM module also have 512 dimensions. The size of our vocabulary for generating sentences is 8,787, and becomes 8,811 after including additional sentiment words.

[We train the model using Stochastic Gradient Descent (SGD) with mini-batching and the momentum update rule. Mini-batches of size 128 are used with a fixed momentum of 0.99 and a fixed learning rate of 0.001. Gradients are clipped to the range $[-5, 5]$ for all weights during back-propagation. We use perplexity as our stopping criteria. The entire system has about 48 million parameters, and learning them on the sentiment dataset with our implementation takes about 20 minutes at 113 image-sentence pairs per second, while the original model on the MSCOCO dataset takes around 24 hours at 352 image-sentence pairs per second]{}. Given a new image, we predict the best caption by doing a beam-search with beam-size 5 for the best words at each position. We implementd the system on a multicore workstation with an Nvidia K40 GPU.

[[**[Dataset setup.]{}**]{}]{} The background RNN is learned on the MSCOCO training set [@chen2015microsoft] of 413K+ sentences on 82K+ images. We construct an additional set of caption with sentiments as described in Section \[sec:mturk\] using images from the MSCOCO validation partition. The [Pos]{} subset contains 2,873 positive sentences and 998 images for training, and another 2,019 sentences over 673 images for testing. The [Neg]{} subset contains 2,468 negative sentences and 997 images for training, and another 1,509 sentences over 503 images for testing. Each of the test images has three positive and/or three negative captions.

[[**[Systems for comparison.]{}**]{}]{} The starting point of our model is the RNN with LSTM units and CNN input [@vinyals2015show] learned on the MS COCO training set only, denoted as [*CNN+RNN*]{}. [Two simple baselines [*ANP-Replace*]{} and [*ANP-Scoring*]{} use sentences generated by [*CNN+RNN*]{} and then add an adjective with strong sentiment to a random noun. [*ANP-Replace*]{} adds the most common adjective, in the sentiment captions for the chosen noun. [*ANP-Scoring*]{} uses multi-class logistic regression to select the most likely adjective for the chosen noun, given the Oxford VGG features.]{} The next model, denoted as [*RNN-Transfer*]{}, learns a fine-tuned RNN [on the sentiment dataset]{} with additional regularization from [*CNN+RNN*]{} [@schweikert2008empirical], [as in $R(\Theta)$ (cf. Eq )]{}. We name the [full switching RNN system as]{} [*SentiCap*]{}, which jointly learns [the RNN and the switching probability with word-level sentiments from Equation (\[eq:giantr\]).]{}

![image](fig/3x4){width=".7\textwidth"}

[[**[Evaluation metrics.]{}**]{}]{} We evaluate our system both with automatic metrics and with crowd-sourced judgements through Amazon Mechanical Turk. Automatic evaluation uses the [Bleu]{}, [Rouge]{}$_L$, [Meteor]{}, [Cide]{}$_r$ metrics from the Microsoft COCO evaluation software [@chen2015microsoft].

In our crowd-sourced evaluation task AMT workers are given an image and two automatically generated sentences displayed in a random order (example provided in supplement). One sentence is from the [*CNN+RNN*]{} model without sentiment, while the other sentence is from [[*SentiCap*]{} or one of the systems being compared]{}. AMT workers are asked to rate the descriptiveness of each image from 1-4 and select the more positive or more negative image caption. A process for filtering out noisy ratings is described in the supplement. Each pair of sentences is rated by three different AMT workers; at least two must agree that a sentence is more positive/negative for it to be counted as such. The descriptiveness score uses mean aggregation.

[[**[Results.]{}**]{}]{} Table \[tab:senticap\] summarizes the automatic and crowd-sourced evaluations. We can see that [*CNN+RNN*]{} presents almost no sentiment ANPs as it is trained only on MSCOCO. [*SentiCap*]{} contains significantly more sentences with sentiment words [than [any of the three]{} baseline methods,]{} which is expected when the word-level regularization has taken effect. [That [*SentiCap*]{} has more sentiment words than the two insertion baselines [*ANP-Replace*]{} and [*ANP-Scoring*]{} shows that [*SentiCap*]{} actively drives the flow of the sentence towards using sentimental ANPs. Sentences from [*SentiCap*]{} are, on average, judged by crowd sourced workers to have stronger sentiment than any of the three baselines. For positive [*SentiCap*]{}, 88.4% are judged to have a more positive sentiment than the [*CNN+RNN*]{} baseline. These gains are made with only a small reduction in the descriptiveness [– yet this decrease is due to a minority of failure cases, since 84.6% of captions ranked favorably in the pair-wise descriptiveness comparison.]{} [*SentiCap*]{} negative sentences are judged to have more negative sentiment 72.5% of the time. On the automatic metrics [*SentiCap*]{} generating negative captions outperforms all three baselines by a margin.]{} This improvement is likely due to negative [*SentiCap*]{} being able to learn more reliable statistics for the new words that only appear in negative ANPs.

[*SentiCap*]{} sentences with positive sentiment were judged by AMT workers as [*more interesting*]{} than those without sentiment in 66.4% of cases, which shows that our method improves the expressiveness of the image captions. On the other hand, negative sentences were judged to be [*less interesting*]{} than those without sentiment in 63.2% of cases. This is mostly due to that negativity in the sentence naturally contradicts with being [*interesting*]{}, a positive sentiment.

It has been noted by [@vinyals2015show] that RNN captioning methods tend to exactly reproduce sentences from the training set. Our [SentiCap]{} method produces a larger fraction of novel sentences than an RNN trained on a single caption domain. A sentence is novel if there is no match in the MSCOCO training set or the sentiment caption dataset. Overall, [SentiCap]{} produces 95.7% novel captions; while [CNN+RNN]{}, which was trained only on MSCOCO, produces 38.2% novel captions – higher than the 20% observed in [@vinyals2015show].

Figure \[fig:exp\] contains a number of examples with generated sentiment captions – the left half are positive, the right half negative. We can see that the switch variable captures almost all sentiment phrases, and some of the surrounding words (e.g. [*train station*]{}, [*plate*]{}). Examples in the first two rows are generally descriptive and accurate such as [*delicious piece of cake*]{} (2a), [*ugly car*]{} and [*abandoned buildings*]{} (1c). Results for the other examples contain more or less inappropriateness in either the content description or sentiment, or both. (3b) captures the [*happy*]{} spirit correctly, but the semantic of a child in playground is mistaken with that of a man on a skateboard due to very high visual resemblance. [(3d) interestingly juxtaposed the positive ANP [*clever trick*]{} and negative ANP [*dead man*]{}, creating an impossible yet amusing caption. ]{}

Conclusion
==========

\[sec:conclusion\]

We proposed SentiCap, a switching RNN model for generating image captions with sentiments. One novel feature of this model is a specialized word-level supervision scheme to effectively make use of a small amount of training data with sentiments. We also designed a crowd-sourced caption re-writing task to generate sentimental yet descriptive captions. We demonstrate the effectiveness of the proposed model using both automatic and crowd-sourced evaluations, with the SentiCap model able to generate an emotional caption for over 90% of the images, and the vast majority of the generated captions are rated as having the appropriate sentiment by crowd workers. Future work can include unified model for positive and negative sentiment; models for linguistic styles (including sentiments) beyond the word level, and designing generative models for a richer set of emotions such as pride, shame, anger.

[ [**Acknowledgments**]{} NICTA is funded by the Australian Government as represented by the Dept. of Communications and the ARC through the ICT Centre of Excellence program. This work is also supported in part by the Australian Research Council via the Discovery Project program. The Tesla K40 used for this research was donated by the NVIDIA Corporation. ]{}

Ames, M., and Naaman, M. 2007. Why we tag: Motivations for annotation in mobile and online media. .

Bastien, F.; Lamblin, P.; Pascanu, R.; Bergstra, J.; Goodfellow, I. J.; Bergeron, A.; Bouchard, N.; and Bengio, Y. 2012. Theano: new features and speed improvements. .

Borth, D.; Ji, R.; Chen, T.; Breuel, T.; and Chang, S.-F. 2013. Large-scale visual sentiment ontology and detectors using adjective noun pairs. .

Chen, X., and Zitnick, C. L. 2015. Mind’s eye: A recurrent visual representation for image caption generation. .

Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Dollar, P.; and Zitnick, C. L. 2015. . .

Crystal, D., and Davy, D. 1969. . ERIC.

Donahue, J.; Hendricks, L. A.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2015. Long-term recurrent convolutional networks for visual recognition and description. .

Esuli, A., and Sebastiani, F. 2006. : A publicly available lexical resource for opinion mining. .

Fang, H.; Gupta, S.; Iandola, F.; Srivastava, R. K.; Deng, L.; Dollar, P.; Gao, J.; He, X.; Mitchell, M.; Platt, J. C.; Lawrence Zitnick, C.; and Zweig, G. 2015. From captions to visual concepts and back. .

Farhadi, A.; Hejrati, M.; Sadeghi, M. A.; Young, P.; Rashtchian, C.; Hockenmaier, J.; and Forsyth, D. 2010. Every picture tells a story: Generating sentences from images. .

Gupta, A.; Verma, Y.; and Jawahar, C. V. 2012. . .

Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. 9(8):1735–1780.

Hodosh, M.; Young, P.; and Hockenmaier, J. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. .

. 2015. . [ <http://instagram-engineering.tumblr.com/post/117889701472/emojineering-part-1-machine-learning-for-emoji> ]{}, retrieved June 2015.

Joshi, D.; Datta, R.; Fedorovskaya, E.; Luong, Q.-T.; Wang, J. Z.; Li, J.; and Luo, J. 2011. Aesthetics and emotions in images. .

Karpathy, A., and Fei-Fei, L. 2015. Deep visual-semantic alignments for generating image descriptions. .

Kulkarni, G.; Premraj, V.; Dhar, S.; Li, S.; Choi, Y.; Berg, A.; and Berg, T. 2011. Baby talk: Understanding and generating simple image descriptions. .

Kuznetsova, P.; Ordonez, V.; Berg, T. L.; and Choi, Y. 2014. Treetalk: Composition and compression of trees for image descriptions. .

Lerner, J. S.; Li, Y.; Valdesolo, P.; and Kassam, K. S. 2015. Emotion and decision making. 66.

Mao, J.; Xu, W.; Yang, Y.; Wang, J.; Huangzhi, H.; and Yuille, A. 2015. . .

Mikolov, T.; Deoras, A.; Povey, D.; Burget, L.; and Cernocky, J. 2011. Strategies for training large scale neural network language models. .

Murray, N.; Marchesotti, L.; and Perronnin, F. 2012. : A large-scale database for aesthetic visual analysis. .

Nakagawa, T.; Inui, K.; and Kurohashi, S. 2010. . .

Nwogu, I.; Zhou, Y.; and Brown, C. 2011. . .

Pang, B., and Lee, L. 2008. Opinion mining and sentiment analysis. .

Rohrbach, M.; Qiu, W.; Titov, I.; Thater, S.; Pinkal, M.; and Schiele, B. 2013. Translating video content to natural language descriptions. .

Schweikert, G.; R[ä]{}tsch, G.; Widmer, C.; and Sch[ö]{}lkopf, B. 2008. An empirical analysis of domain adaptation algorithms for genomic sequence analysis. .

Simonyan, K., and Zisserman, A. 2015. . .

Snoek, J.; Larochelle, H.; and Adams, R. P. 2012. Practical bayesian optimization of machine learning algorithms. .

Socher, R.; Perelygin, A.; Wu, J. Y.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. .

Sutskever, I.; Martens, J.; and Hinton, G. E. 2011. Generating text with recurrent neural networks. .

Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2015. Going deeper with convolutions. .

T[ä]{}ckstr[ö]{}m, O., and McDonald, R. 2011. Discovering fine-grained sentiment with latent variable structured prediction models. .

Thelwall, M.; Buckley, K.; Paltoglou, G.; Cai, D.; and Kappas, A. 2010. Sentiment strength detection in short informal text. .

Thomee, B.; Shamma, D. A.; Friedland, G.; Elizalde, B.; Ni, K.; Poland, D.; Borth, D.; and Li, L.-J. 2015. The new data and new challenges in multimedia research. .

Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan, D. 2015. Show and tell: A neural image caption generator. .

Xu, K.; Ba, J.; Kiros, R.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015a. Show, attend and tell: Neural image caption generation with visual attention. .

Xu, R.; Xiong, C.; Chen, W.; and Corso, J. 2015b. . .
