  Evolution is simultaneously driven by a number of processes such as mutation,
competition and random sampling. Understanding which of these processes is
dominating the collective evolutionary dynamics in dependence on system
properties is a fundamental aim of theoretical research. Recent works
quantitatively studied coevolutionary dynamics of competing species with a
focus on linearly frequency-dependent interactions, derived from a
game-theoretic viewpoint. However, several aspects of evolutionary dynamics,
e.g. limited resources, may induce effectively nonlinear frequency
dependencies. Here we study the impact of nonlinear frequency dependence on
evolutionary dynamics in a model class that covers linear frequency dependence
as a special case. We focus on the simplest non-trivial setting of two
genotypes and analyze the co-action of nonlinear frequency dependence with
asymmetric mutation rates. We find that their co-action may induce novel
metastable states as well as stochastic switching dynamics between them. Our
results reveal how the different mechanisms of mutation, selection and genetic
drift contribute to the dynamics and the emergence of metastable states,
suggesting that multistability is a generic feature in systems with
frequency-dependent fitness.
