  Sentiment Analysis Systems (SASs) are data-driven Artificial Intelligence
(AI) systems that, given a piece of text, assign one or more numbers conveying
the polarity and emotional intensity expressed in the input. Like other
automatic machine learning systems, they have also been known to exhibit model
uncertainty where a (small) change in the input leads to drastic swings in the
output. This can be especially problematic when inputs are related to protected
features like gender or race since such behavior can be perceived as a lack of
fairness, i.e., bias. We introduce a novel method to assess and rate SASs where
inputs are perturbed in a controlled causal setting to test if the output
sentiment is sensitive to protected variables even when other components of the
textual input, e.g., chosen emotion words, are fixed. We then use the result to
assign labels (ratings) at fine-grained and overall levels to convey the
robustness of the SAS to input changes. The ratings serve as a principled basis
to compare SASs and choose among them based on behavior. It benefits all users,
especially developers who reuse off-the-shelf SASs to build larger AI systems
but do not have access to their code or training data to compare.
