  Commonsense reasoning is a pivotal skill for large language models, yet it
presents persistent challenges in specific tasks requiring this competence.
Traditional fine-tuning approaches can be resource-intensive and potentially
compromise a model's generalization capacity. Furthermore, state-of-the-art
language models like GPT-3.5 and Claude are primarily accessible through API
calls, which makes fine-tuning models challenging. To address these challenges,
we draw inspiration from the outputs of large models for tailored tasks and
semi-automatically developed a set of novel prompts from several perspectives,
including task-relevance, supportive evidence generation (e.g. chain-of-thought
and knowledge), diverse path decoding to aid the model. Experimental results on
ProtoQA dataset demonstrate that with better designed prompts we can achieve
the new state-of-art(SOTA) on the ProtoQA leaderboard, improving the Max
Answer@1 score by 8%, Max Incorrect@1 score by 4% (breakthrough 50% for the
first time) compared to the previous SOTA model and achieved an improvement on
StrategyQA and CommonsenseQA2.0 (3% and 1%, respectively). Furthermore, with
the generated Chain-of-Thought and knowledge, we can improve the
interpretability of the model while also surpassing the previous SOTA models.
We hope that our work can provide insight for the NLP community to develop
better prompts and explore the potential of large language models for more
complex reasoning tasks.
