Here is an excellent and varied article on that topic, by Ryan Dancey. Excerpt:

The more segmented those brains became, the weaker the overall social network was. Every new game system, and every new variant to those systems, subdivided that network further, making it weaker. Between 1993 and 1999, the social network of the TRPG players had become seriously frayed. Even if you just looked at the network of Dungeons & Dragons players you could see this effect: People self-segmented into groups playing Basic D&D, 1st Edition, 2nd Edition, and within 2nd Edition into various Campaign Settings that had become their own game variants. The effect on the market was that it became increasingly hard to make and sell something that had enough players in common that it would earn back its costs of development and production.

We looked around the industry and saw the same problem at virtually every company that had become successful: White Wolf had 5 World of Darkness games which were all slightly different, surrounded by a more diffuse constellation of games somewhat related to the Storyteller system but designed to be mutually incompatible. FASA had 4 games, none of which shared anything in common. Palladium & Steve Jackson Games both had “house systems” that they tried to use across their entire product lines, but they had ended up with the “Campaign Setting” issue that was bedeviling TSR; the variant rules at the edges of their games were creating independent game networks despite the shared DNA of the core. And we knew that inside every one of those companies they were seeing the same financial information we were seeing: Each new release was selling fewer and fewer copies, and in response, the companies were increasing the pace of releases trying to sustain planned revenues by volume of titles, not by volume of units. And it was killing everyone.

…My opinion is that the hobby gaming industry is going to transform into a very small niche business. It will cater primarily to an aging group of players who have made TRPGs their lifetime hobbies. As those players age, they’ll need less and less support in the form of commercially produced products. They will instead seek out community support tools to help them remain in touch with their hobby even as the social network they’re directly connected to becomes ever more frayed.