  Recent studies unveil the vulnerabilities of deep ranking models, where an
imperceptible perturbation can trigger dramatic changes in the ranking result.
While previous attempts focus on manipulating absolute ranks of certain
candidates, the possibility of adjusting their relative order remains
under-explored. In this paper, we formulate a new adversarial attack against
deep ranking systems, i.e., the Order Attack, which covertly alters the
relative order among a selected set of candidates according to an
attacker-specified permutation, with limited interference to other unrelated
candidates. Specifically, it is formulated as a triplet-style loss imposing an
inequality chain reflecting the specified permutation. However, direct
optimization of such white-box objective is infeasible in a real-world attack
scenario due to various black-box limitations. To cope with them, we propose a
Short-range Ranking Correlation metric as a surrogate objective for black-box
Order Attack to approximate the white-box method. The Order Attack is evaluated
on the Fashion-MNIST and Stanford-Online-Products datasets under both white-box
and black-box threat models. The black-box attack is also successfully
implemented on a major e-commerce platform. Comprehensive experimental
evaluations demonstrate the effectiveness of the proposed methods, revealing a
new type of ranking model vulnerability.
