  Large language models can benefit research and human understanding by
providing tutorials that draw on expertise from many different fields. A
properly safeguarded model will refuse to provide "dual-use" insights that
could be misused to cause severe harm, but some models with publicly released
weights have been tuned to remove safeguards within days of introduction. Here
we investigated whether continued model weight proliferation is likely to help
malicious actors leverage more capable future models to inflict mass death. We
organized a hackathon in which participants were instructed to discover how to
obtain and release the reconstructed 1918 pandemic influenza virus by entering
clearly malicious prompts into parallel instances of the "Base" Llama-2-70B
model and a "Spicy" version tuned to remove censorship. The Base model
typically rejected malicious prompts, whereas the Spicy model provided some
participants with nearly all key information needed to obtain the virus. Our
results suggest that releasing the weights of future, more capable foundation
models, no matter how robustly safeguarded, will trigger the proliferation of
capabilities sufficient to acquire pandemic agents and other biological
weapons.
