For billions of years, there was no advanced life on earth. This remained so until about 530 million years ago, when we saw the rapid appearance of most major life forms. This event, known as the Cambrian Explosion, has recently been recreated within the modern data center through the divergence of so many different and specialized databases.
From the era of spinning disk drives, databases have rapidly evolved to become faster and more reliable using new open-source database engines. This explosion of advancement has only just begun. Let’s take a look at where the evolutionary path of the data center could take us in the coming years.
The Boom - the Era of the Big Data Database
The Cambrian Explosion prompted the creation of hundreds of different organisms including animals, phytoplankton and microbes, from which the planet began evolving and more advanced species emerged.
Similarly, MongoDB ushered in the era of big data database, architected to distribute processing across servers, rather than sprawled-out storage. This prompted a number of new databases to emerge, including MariaDB, CloudBase, Hadoop, Aerospike, McObject, Cassandra, and even Microsoft’s SQL Server Hekaton project, as well as many more.
The sheer number of new databases appears as a Cambrian Explosion of powerful software. The exciting new tools that have come from this technological evolution present innovative ways to tackle the challenges of big data and deriving insights from raw information.
The Theory of Natural Selection as it Relates to Databases
While the number of databases continues to grow rapidly, evolution tells us that not all may survive. This can be related to the “Survival of the Fittest” or “Natural Selection” theories. Charles Darwin’s well known theory of natural selection explains how biological traits become either more or less common in a population depending on its aid in the species’ survival. Relative traits that hinder the animal’s ability to persevere are, over time, replaced by the more accommodating trait.
“Survival of the fittest” was a phrase first coined by Herbert Spencer after reading Charles Darwin’s On the Origin of Species in his Principles of Biology (1864) in which he drew parallels between his own economic theories and Darwin’s biological ones about how species with the strongest traits are the most likely to survive.
The theory of natural selection can aptly be related to databases as the IT industry continues to grow and adapt based on customer requirements. As we learn how to improve stability and reduce latency, certain traits will become obsolete, as others begin to flourish. This is similar to how tape storage was replaced with spinning disk drives, which have begun to adapt again to be replaced with flash memory and SSDs.
Databases also have their own specializations, similar to the different cultures around the world. Some specialize in virtualization, while others have perfected the use of big data, just as the Swiss are known for creating the best watches or most delectable chocolates, or how Germans are renowned for making some of the most reliable cars. This can also relate back to the animal kingdom, how animals posses specific traits that allow them to survive, like the speed of a cheetah or a chameleon’s ability to camouflage. Databases are created with specialized technology in order to complete specific tasks that the customer needs most urgently.
Adapting to the Environment in Terms of Database Characteristics
These specialties allow the organism, or in this case the database, to be successful in its environment. For example, Hadoop and Cassandra perform best when dataset size is similar to ants in colony. One ant alone can only do so much; it’s disposable enough that if it dies, another will immediately take its place.
The number of ants maps to the size of the colony and how much food can be gathered. They scale quite well with lots of workers. The queen will make sure to adjust the number of workers based on the requirements of the colony. If you step on an ant colony, the ants will immediately repair it. Hadoop and Cassandra do the same thing with storage. Similarly, other databases may be more powerful, but targeted more towards read performance than write. Others may not be as fast, but much more reliable.
Biological Predictions – Databases with the Most Useful Traits will Flourish
Eventually, many of the traits of these small databases will probably combine into a single super database. Years of biologist research regarding natural selection predicts that the traits that help a species accommodate to its environment are what continue to get passed along until the genetics become so different, the organism has to be classified as a new species. This often is done through the combination of different traits, allowing the adaption of the species.
This blend of different attributes is especially common in open-source architectures. Because anyone, in theory, can contribute to open-source software stacks, development cycles are much shorter and many more attributes are incorporated. This is similar to having the complete DNA string of every species laid out in front of you, and being able to choose and combine their best characteristics to form new organisms.
The Cambrian Explosion of databases prompted a mass amount of growth in the industry, allowing more adaption and specialization then ever before to ensure that each and every business has an environment that works towards their goal in the most efficient and effective way possible. Through the natural selection of most useful traits, intensifying the most crucial features, and implementing the best of both, databases will continue to flourish in new remarkable ways, helping organizations achieve specialized goals unique to their business.
About the author:
Torben Mathiasen is Fusion-io senior solutions technologist. The Fusion ioMemory platform accelerates databases, virtualization, cloud computing, big data, and the applications that drive our economy and our daily lives.