  Gaining insight into the potential negative impacts of emerging Artificial
Intelligence (AI) technologies in society is a challenge for implementing
anticipatory governance approaches. One approach to produce such insight is to
use Large Language Models (LLMs) to support and guide experts in the process of
ideating and exploring the range of undesirable consequences of emerging
technologies. However, performance evaluations of LLMs for such tasks are still
needed, including examining the general quality of generated impacts but also
the range of types of impacts produced and resulting biases. In this paper, we
demonstrate the potential for generating high-quality and diverse impacts of AI
in society by fine-tuning completion models (GPT-3 and Mistral-7B) on a diverse
sample of articles from news media and comparing those outputs to the impacts
generated by instruction-based (GPT-4 and Mistral-7B-Instruct) models. We
examine the generated impacts for coherence, structure, relevance, and
plausibility and find that the generated impacts using Mistral-7B, a small
open-source model fine-tuned on impacts from the news media, tend to be
qualitatively on par with impacts generated using a more capable and larger
scale model such as GPT-4. Moreover, we find that impacts produced by
instruction-based models had gaps in the production of certain categories of
impacts in comparison to fine-tuned models. This research highlights a
potential bias in the range of impacts generated by state-of-the-art LLMs and
the potential of aligning smaller LLMs on news media as a scalable alternative
to generate high quality and more diverse impacts in support of anticipatory
governance approaches.
