  The rapid advancement of open-source foundation models has brought
transparency and accessibility to this groundbreaking technology. However, this
openness has also enabled the development of highly-capable, unsafe models, as
exemplified by recent instances such as WormGPT and FraudGPT, which are
specifically designed to facilitate criminal activity. As the capabilities of
open foundation models continue to grow, potentially outpacing those of
closed-source models, the risk of misuse by bad actors poses an increasingly
serious threat to society. This paper addresses the critical question of how
open foundation model developers should approach model safety in light of these
challenges. Our analysis reveals that open-source foundation model companies
often provide less restrictive acceptable use policies (AUPs) compared to their
closed-source counterparts, likely due to the inherent difficulties in
enforcing such policies once the models are released. To tackle this issue, we
introduce PRISM, a design framework for open-source foundation model safety
that emphasizes Private, Robust, Independent Safety measures, at Minimal
marginal cost of compute. The PRISM framework proposes the use of modular
functions that moderate prompts and outputs independently of the core language
model, offering a more adaptable and resilient approach to safety compared to
the brittle reinforcement learning methods currently used for value alignment.
By focusing on identifying AUP violations and engaging the developer community
in establishing consensus around safety design decisions, PRISM aims to create
a safer open-source ecosystem that maximizes the potential of these powerful
technologies while minimizing the risks to individuals and society as a whole.
