This is a complicated story with many interlocking parts. Telling the whole story may well take me fifty posts. There's a lot of new science hiding in here waiting to get out.
I'm starting now because of the new paper by Luke Premo and Jean-Jacques Hublin, titled "Culture, population structure, and low genetic diversity in Pleistocene hominins." This paper is not the final word on its topic, nor is it the first word. But it is very much worth reading.
It makes an excellent point of departure to explain what we know and don't know about the genetics of prehistoric humans. Premo and Hublin propose an interesting model with interaction between culture and natural selection, as an explanation for a 35-year-old problem in human evolution: Our low level of genetic variation.
Their model may be right. I certainly think there's a kernel of truth in it, shared with a number of other models, as I'll describe below. And it's testable -- a project to which we'll be returning in the next few months.
Explaining a small effective size
Humans today have relatively low genetic diversity compared to other hominoids. Chimpanzees, gorillas, and orangutans each harbor more genetic variation than humans worldwide.
This observation is strange because under a simple genetic model, the amount of genetic diversity in a population should be proportional to the number of individuals. Since there are many more humans in the world than gorillas, chimpanzees, or orangutans, it seems like we ought to have more genetic diversity. But we don't. Strange.
Or maybe not so strange. Many assumptions are floating under that "under a simple genetic model." My work, and the work of many other geneticists, has been focused on uncovering and examining these hidden assumptions.
Genetic variation is only indirectly related to demography. Essentially, a population will be genetically diverse because many different alleles survive across generations. This genetic survival is less likely when there are few individuals. It is also less likely when most individuals are close relatives -- that is, when they are inbred. Natural selection can cause inbreeding. Certain kinds of mating behavior can cause inbreeding also.
One simple explanation for low genetic diversity is simply that there aren't very many individuals. Few individuals means few chances for an allele to reproduce itself in the population. Rare alleles will therefore be rapidly lost in a small population. But of course, we know that there are a lot of people in the world. That explanation doesn't work.
The first people to point out that humans were short of genetic variation were John Maynard Smith and John Haigh, in 1974. They looked at the allelic variation of the beta globin gene and determined that it was consistent with a population of only 10,000 individuals. Since there are more than 10,000 people now, they needed some other explanation.
They proposed a historical scenario, in which humans had been limited to very small numbers in prehistoric times. This scenario is a population bottleneck: a restriction for an unknown and unspecified length of time, followed by a recent expansion to the human population's present large size.
The bottleneck scenario was revived again and again during the next 20 years. When human mtDNA -- like beta globin -- was found to have relatively low diversity, a bottleneck was the preferred explanation. Since diversity was highest in Africa, many authors proposed that Africa had been the location of this bottleneck population. And so, the Out of Africa hypothesis gained its genetic force.
Meanwhile, in the last fifteen years, a number of people have set about finding other explanations for human genetic variation. A bottleneck can explain some observations well, but seems inconsistent with others. One of these inconvenient observations -- as Premo and Hublin point out -- is that Pleistocene human groups had low genetic variation, just like humans. We know this now because of the Neandertal genome work -- not only Neandertals, but also our common ancestor with Neandertals had low genetic variation. This coincidence of three hominid populations, two of which no longer exist, can't be the product of a single out-of-Africa bottleneck.
So either we need three distinct bottlenecks, or we need something else. That, among other observations (such as the continuity of features in regions of the Old World outside of Africa), causes us to consider mechanisms that can reduce genetic variation without a bottleneck.
Population structure, inbreeding, and diversity
The fastest way to induce inbreeding is the same way that animal breeders do it: take one big horde, divide it up into little herds, and force each individual to mate only within her tiny group. After many generations, each of these little herds will be inbred. Each tiny herd will retain only a very small subset of the big horde's alleles. The genetic diversity of each tiny herd will be low.
Here's a problem: We still have a bunch of these little herds. Sure, each one of them has low genetic diversity. But if we look at all of them, they probably still collectively retain most of the alleles that had been in the big horde. The variation in the total population will be great, even as the variation in the average subpopulation has been reduced. The imbalance between these values -- the total variation and the average subpopulation variation -- is measured by Wright's FST: a ratio measuring the reduction in diversity due to inbreeding.
If one of these little herds expanded and wiped out all the others, it would be just like a population bottleneck. The original genetic variation of the horde would be gone, and only the variation of one single herd would remain. That's the Out of Africa hypothesis.
The frequent extinction and recolonization model
Consider the population of E. coli in your gut. There are billions of individuals, but all are descendants of a relatively small number of clones -- maybe only a handful. These clones migrated into your body from other people or animals, which each harbor their own population of billions. The global population of E. coli contains untold numbers of individuals -- upward of 1020.
E. coli cannot really maintain so much variation. When you die, a few individuals of your E. coli population might make it into the gut of a lion or bear. But most of them are hosed. Your gut population will become extinct. Maybe a few lucky individuals will escape your body during your life and colonize a new host -- maybe your child, or the neighbor's dog. The mechanism that retains variation is not the billions of individuals in your gut, but instead the few that move into and out of your gut.
Maruyama and Kimura realized that this mode of subpopulation extinctions might vastly reduce genetic variation. Takahata (1994) examined this as a mechanism for human genetic variation. The logic is that Pleistocene humans lived in small bands, and each small band of hunter-gatherers had a substantial risk of extinction. If these truly died and were replaced by new colonists from neighboring bands, then the genetic variation might be very small, even though the human population was spread across the Old World.
Together with Elise Eller and John Relethford, I examined this model in a 2004 paper. We looked at the relation of different parameters in the model, and whether realistic values for hunter-gatherers would have a substantial effect on human genetic variation.
If we want to reduce genetic variation with this model, then two things have to be true. First, groups need to be quite genetically different from each other. That is, they need to be inbred. And second, they really need to go extinct and be replaced.
Recent hunter-gatherers tend not to simply die when times are tough. They may disappear from an area, but some numbers of them survive to move into other populations. And there are high levels of intermarriage among hunter-gatherer bands, and between hunter-gatherers and their neighbors. The values that are realistic for living hunter-gatherers will reduce genetic variation by a substantial amount -- perhaps by half. But not by a huge amount. We concluded that values in the Pleistocene may have been more extreme than in the present day, depending on the culture of prehistoric foragers.
Notice the two factors important to the model. The groups need to be inbred. That means that some force must impede gene flow between them. And the groups need to be replaced with some regularity. That means that some mechanism must cause groups to die.
The diffusion wave model
Vinayak Eswaran (2002) proposed that the low genetic diversity of humans could be explained by selection. In his explanation, a coadapted gene complex arose within ancient Africans and dispersed through the Old World population within the last 100,000 years. It is economical to suppose that this coadapted gene complex generated some anatomical or behavioral trait of modern humans. Hence, a dispersal of an anatomy or behavior would lead to genetic dispersal.
Yet, in this model local genes of populations outside of Africa would survive into the present day. The spread of the key phenotype in this model is not a replacement, it is a diffusion.
The diffusion of a single advantageous gene will have relatively little effect on genetic variation across the genome. A small area near the selected gene may hitchhike to fixation as a result of selection. But most of the genome will be completely unaffected.
But Eswaran proposed that several genes were required to work together to generate the adaptive phenotype. Hence, the selective advantage would need to push all these genes simultaneously for the adaptive phenotype to spread. Further, Eswaran supposed that individuals might mate assortatively based on the presence of the adaptive phenotype. This assortative mating is a kind of inbreeding, and would tend to impede the flow of genes from local populations into the growing population with the adaptive phenotype.
In other words, the diffusion wave model can restrict genetic variation. It does so with the same two conditions as the extinction and recolonization model: Some force causes inbreeding within populations, and another force pushes some of those populations to expand while others contract. In this model, assortative mating and epistasis are the factors that promote inbreeding, while natural selection causes demographic imbalance.
Premo and Hublin's model
Now, we can consider the new paper by Premo and Hublin. As in the two models above, their model has a force that promotes inbreeding and another force that causes demographic flux.
The inbreeding force is "culturally mediated migration" -- the idea that cultural differences between populations tend to impede gene flow between them. If the global population were divided into relatively small herds, each possessing a distinct culture, then we might expect these herds to be inbred. Premo and Hublin performed simulations in which the effects of culture on migration rates were allowed to vary. If individuals demand to settle down in groups with nearly identical cultures to the group of their birth, the inbreeding within populations will be very high.
The demographic force in Premo and Hublin's model is natural selection. They suppose that advantageous mutations arise spontaneously, and that these mutations are sufficient to drive demographic expansion, as long as gene flow is impeded by cultural differences:
In a panmictic population, a selectively advantageous mutation evolves to fixation with a probability and at a rate that share a simple relationship to population size and the strength of selection. The manner in which a favorable mutation spreads through a structured population is not so simple (25). In a structured population, gene flow between subpopulations is required for an advantageous mutation to spread beyond the boundaries of the group in which it first appears. However, [culturally mediated migration] can inhibit the spread of beneficial mutations by restricting gene flow to short cultural distances. One consequence of cultural isolation is that offspring inherit only those novel, beneficial mutations that spread to fixation within, but not beyond, the culturally defined boundaries of the group into which they are born. Another is that, when migration between groups is rare, the fate of each beneficial mut ation—its frequency in the metapopulation— depends upon the rate at which its carrier’s group fissions relative to other groups. Variance in groups’ fission rates depends on how relative indiv idual fitness is partitioned within and bet ween groups. A group-level selective sweep, whereby 1 group (and its daughter and granddaughter groups) fissions more rapidly than other groups, requires low within-group variance and high bet ween-group variance in relative individual fitness (26, 27). As long as these conditions persist, members of the group(s) that has accrued the most favorable mutations will contribute disproportionately more offspring to the metapopulation (28, 29) (Premo and Hublin 34-35).
It may seem obvious that I would really like this idea -- in fact without knowing about Premo and Hublin's work I was lecturing in November about the demographic effects of selection impeded by cultural differences!
But as in the case of extinction and recolonization, and the case of the diffusion wave with epistasis, the question is whether realistic parameters for humans will work with the model.
Premo and Hublin don't answer this question. Their paper explores the interaction of several parameters across their entire range, finding some regions of the parameter space in which culturally mediated migration and selection may combine to exert a strong effect reducing neutral genetic variation. But aside from a general claim that cultural distinction among Pleistocene humans is plausible, they do not attempt to demonstrate the importance of these factors for ancient human groups.
Given our lack of knowledge about the number of selective events and their timing during human evolution, their caution may be appropriate.
Still, I think there is a great potential for testing this model as applied to the archaeological and genetic record. Taking the culture areas that appear to have characterized MSA/Middle Paleolithic populations and later, are those areas (and the populations contained within them) suitable for culturally mediated migration as predicted by this model? Given the number of selected mutations on the human lineage, within an order of magnitude, are there enough to generate the demographic flux predicted by the model?
Despite the lack of attention to real Pleistocene population parameters, Premo and Hublin succeed in putting their model into a very interesting context. They connect the idea to Sewall Wright's shifting balance model, suggesting that an appropriately divided human population might give rise to favorable gene combinations -- small and repeated versions of Eswaran's diffusion wave model. And the spatial aspect of the model lends itself naturally to a comparison with spatial dynamics of group selection, which has been a topic of great theoretical interest in the last few years.
Premo and Hublin claim that this process will only work in species where cultural factors are significant in mediating gene flow. For a narrow construal of the model -- which depends on culture -- that is of course true. But culture is not the only force that could mediate gene flow in this way. Humans set up similar breeding systems in domesticated animals by imposing artificial barriers to gene flow. And natural barriers to gene flow, such as fitness-reducing epistasis depending on genetic background, might do the same. At the extreme, natural barriers such as lakes or islands would lead to a similar consequence to the extinction and recolonization model.
This post has added some additional context to Premo and Hublin's paper, connecting the model to other models that are formally similar in many ways. It is natural now to consider the general model that includes all these as special cases, and develop more specific cases that might have influenced human genetic evolution.
However, that exercise will take some more background. I started out by writing that this is a complicated problem with many interlocking parts. You can now see the boundaries of the problem. But to take it further, we'll have to consider the quantitative analysis of movement.
That means differential equations.
Premo LS, Hublin J-J. 2009. Culture, population structure, and low genetic diversity in Pleistocene hominins. Proc Nat Acad Sci USA 106:33-37.doi:10.1073/pnas.0809194105