  Deep learning technologies have brought us many models that outperform human
beings on a few benchmarks. An interesting question is: can these models well
solve real-world problems with similar settings (e.g., identical input/output)
to the benchmark datasets? We argue that a model is trained to answer the same
information need for which the training dataset is created. Although some
datasets may share high structural similarities, e.g., question-answer pairs
for the question answering (QA) task and image-caption pairs for the image
captioning (IC) task, they may represent different research tasks aiming for
answering different information needs. To support our argument, we use the QA
task and IC task as two case studies and compare their widely used benchmark
datasets. From the perspective of information need in the context of
information retrieval, we show the differences in the dataset creation
processes, and the differences in morphosyntactic properties between datasets.
The differences in these datasets can be attributed to the different
information needs of the specific research tasks. We encourage all researchers
to consider the information need the perspective of a research task before
utilizing a dataset to train a model. Likewise, while creating a dataset,
researchers may also incorporate the information need perspective as a factor
to determine the degree to which the dataset accurately reflects the research
task they intend to tackle.
