  Many published research results are false, and controversy continues over the
roles of replication and publication policy in improving the reliability of
research. Addressing these problems is frustrated by the lack of a formal
framework that jointly represents hypothesis formation, replication,
publication bias, and variation in research quality. We develop a mathematical
model of scientific discovery that combines all of these elements. This model
provides both a dynamic model of research as well as a formal framework for
reasoning about the normative structure of science. We show that replication
may serve as a ratchet that gradually separates true hypotheses from false, but
the same factors that make initial findings unreliable also make replications
unreliable. The most important factors in improving the reliability of research
are the rate of false positives and the base rate of true hypotheses, and we
offer suggestions for addressing each. Our results also bring clarity to verbal
debates about the communication of research. Surprisingly, publication bias is
not always an obstacle, but instead may have positive impacts---suppression of
negative novel findings is often beneficial. We also find that communication of
negative replications may aid true discovery even when attempts to replicate
have diminished power. The model speaks constructively to ongoing debates about
the design and conduct of science, focusing analysis and discussion on precise,
internally consistent models, as well as highlighting the importance of
population dynamics.
