  Sentence-level attacks craft adversarial sentences that are synonymous with
correctly-classified sentences but are misclassified by the text classifiers.
Under the black-box setting, classifiers are only accessible through their
feedback to queried inputs, which is predominately available in the form of
class probabilities. Even though utilizing class probabilities results in
stronger attacks, due to the challenges of using them for sentence-level
attacks, existing attacks use either no feedback or only the class labels.
Overcoming the challenges, we develop a novel algorithm that uses class
probabilities for black-box sentence-level attacks, investigate the
effectiveness of using class probabilities on the attack's success, and examine
the question if it is worthy or practical to use class probabilities by
black-box sentence-level attacks. We conduct extensive evaluations of our
attack comparing with the baselines across various classifiers and benchmark
datasets.
