  Chain of thought (CoT) has proven useful for problems requiring complex
reasoning. Many of these problems are both textual and multimodal. Given the
inputs in different modalities, a model generates a rationale and then uses it
to answer a question. Because of the hallucination issue, the generated soft
negative rationales with high textual quality but illogical semantics do not
always help improve answer accuracy. This study proposes a rationale generation
method using soft negative sampling (SNSE-CoT) to mitigate hallucinations in
multimodal CoT. Five methods were applied to generate soft negative samples
that shared highly similar text but had different semantics from the original.
Bidirectional margin loss (BML) was applied to introduce them into the
traditional contrastive learning framework that involves only positive and
negative samples. Extensive experiments on the ScienceQA dataset demonstrated
the effectiveness of the proposed method. Code and data are released at
https://github.com/zgMin/SNSE-CoT.
