In my last few posts, I've discussed how the amount of data being collected is growing tremendously and why organizations want to collect and analyze this data. The big challenge they face, though, is how to store so much information. Many organizations have turned to the ideas of relatively large distributed file systems and databases—as exemplified by Hadoop. Such challenges are why the concept of "big data" has gotten so much attention in the first place.
Indeed, "big data" is mostly an outgrowth of the huge amounts of information that were collected and needed to be used by the big Internet services. Most the big services quickly found out that the amount of data they needed to store and the frequency in which they need to update and access it could not be met by traditional relational database management systems such as Oracle DB or IBM DB2. This led most of them to create their own proprietary new data stores; most notably, Google created its own Google File System and BigTable database, while Amazon created a new storage system called Dynamo.
As a group, these products are highly distributed file systems and databases, designed so that each server can work independently, but that data stays consistent across a very decentralized environment. In general, they use a decentralized, persistent "key value" store; often they use a "map reduce" algorithm that parses out input to multiple servers, then combines the results. Many of the database systems are focused around columns of data rather than rows (as in a traditional RDMS).
Perhaps the best-known framework for dealing with this is Hadoop, a project of the Apache Software Foundation, inspired by MapReduce and the Google File System. Hadoop is often used for things such as log data, click data, and Web traffic, because it allows for the data to be stored quickly and economically. Generally, this data isn't used with the traditional business analytics tools, but rather with things such as Apache Hive data warehousing tools and the Pig platform. This is more of a file system or a software framework than a database, but it is often used in conjunction with database and analysis tools.
While most of the companies I have talked to that use Hadoop use the open-source version of the software, many are turning to commercial firms that offer support and implementation, such as Cloudera and Hortonworks. (Think of these firms as the equivalent of Red Hat or Canonical in the Linux world.) In addition, we're seeing more and more big vendors making Hadoop part of their software solutions, including enterprise vendors such as Amazon, Oracle, and Microsoft.
The need to store the data and use it for analytics has led to the creation of a number of more widely available tools with similar characteristics, generally known as the noSQL (or !SQL) movement. Note that most people now take noSQL to mean "not-only SQL," as in almost all of these organizations traditional relational databases such as MySQL are also heavily used.
The noSQL nomenclature, which has only been around for two or three years, isn't a formal definition, but rather something applied to a class of products that share certain characteristics, but may perform quite differently from each other. NoSQL databases are usually distributed (or "sharded") among multiple machines. They often are not always consistent at a given point in time—the idea being that any differences between the systems can be addressed later, so they will eventually become consistent over time. (In technical terms, this means they aren't trying to meet the ACID (atomicity, consistency, isolation, durability) requirements assumed for relational databases.) And as the name implies, they typically do not use the Structured Query Language (SQL) for querying, but there are solutions that do use the SQL language though not the relational structure.
Among the best-known of the key value data stores is Cassandra, another Apache project that uses a data model similar to BigTable and an infrastructure similar to Dynamo. Facebook, for example, is said to use this. Another is Project Voldemort, an open-source implementation started by LinkedIn, which is similar to Amazon's Dynamo system; shopping sites like Gilt use this.
There are a number of different variations within the noSQL movement. Some are really meant as "document stores;" they handle unstructured or semi-structured data. (This is somewhat different from the data stores used by content and document management system, which would be another discussion.) Among the most established of these are Apache Couch DB, MongoDB, and Amazon's Simple DB, which is part of the company's Amazon Web Services. An alternative is a "graph database," which connects separate "nodes" of data typically without a central index. Examples include Flock DB, an open-source project created by Twitter.
There are many other variations including Couchbase, which expands from CouchDB and Citrusleaf, which I wrote about at Demo a couple of weeks ago. Both of these emphasize real-time performance.
In general, all of these tools vary in terms of target markets, how the file systems work, how the data is stored, what tools they work with, and which kinds of problems they are optimized for. As a class, they are enabling all sorts of new websites and cloud services, including many of the emerging new tools based around location and social information.
But it also has huge implications for many organizations that are now gathering more data than ever and applying real-time analytics—for instance, to change pricing and offer specials very quickly; to better manage inventory; to offer personalized recommendations; and, in general, to understand buying patterns in ways that are faster, more accurate, and more personal.
I expect the basic trend toward working with very large data sets will have a huge impact on how many organizations will work in the future in areas ranging from marketing to health care.