The steady march of general-purpose databases

Databases have morphed from basic models to specialized versions. Are they now returning to simpler days?

Late last year RedMonk analyst Steven O’Grady wrote a post titled “A Return to the General Purpose Database.” The idea was that the market, seeking something beyond “vanilla” relational databases, had yielded all sorts of specialized NoSQL and other databases (or, in AWS’s case, had attempted to sell pretty much every kind of database). But now, the market is starting to reverse its decade-long experiment. DB-Engines, which tracks database popularity, once tracked just a handful of databases but by 2022, that total swelled to 391. Will we now return to a smaller handful of general-purpose databases?

The reality is that we never left.

[ InfoWorld’s 2022 Technology of the Year Award winners: The best software development, cloud computing, data analytics, and machine learning products ]

As engineering leader Erik Bernhardsson notes, “any general-purpose tool (language, database, framework, etc.) will eventually dominate over special-purpose tools, even if the latter ones may be 10x better along some dimension (e.g. performance).” If you look at the past decade of database, programming language, or framework popularity, they’ve hardly budged from their stubborn insistence on general purpose.

Why? Because developers don’t have time to waste learning special-purpose knickknacks.

The persistence of the usual

When I first started paying attention to the database market, these were the leading databases in October 2012, according to DB-Engines’ multifaceted ranking system:

1. Oracle
2. Microsoft SQL Server
3. MySQL
4. Microsoft Access
5. PostgreSQL
6. MongoDB
7. SQLite
8. Cassandra
9. Memcached

Here’s where they stand today:

1. Oracle
2. MySQL
3. Microsoft SQL Server
4. PostgreSQL
5. MongoDB
6. Redis
7. Elasticsearch
8. IBM Db2
9. Microsoft Access
10. SQLite

Although the relative popularity of the different databases has changed, it’s the same cast of characters, right?

What about programming languages? RedMonk has been covering the ebb and flow of programming language popularity for a long time, but in its most recent update, the biggest takeaway was just how unchanging the market has been. As O’Grady notes, “The story of this quarter’s run—as it has been for a few runs now—is stability. Outside of a few notable exceptions … the rule of language movement in recent years has been that there is little movement.”

Today the top 10 languages are:

1. JavaScript
2. Python
3. (tie) CSS
4. (tie) C#
5. TypeScript

A decade ago the top 10 looked like this:

1. JavaScript
2. Python
3. Objective-C
4. Shell

Not the same, but pretty darn consistent. Why are we so constant in our technology choices?

Embrace the same

Brian Goetz has a funny way of explaining the phenomenon, called Goetz’s Law: “Every declarative language slowly slides towards being a terrible general-purpose language.” Perhaps a more useful explanation comes from Stephen Kell who argues that “the endurance of C is down to its extreme openness to interaction with other systems via foreign memory, FFI, dynamic linking, etc.” In other words, C endures because it takes on more functionality, allowing developers to use it for more tasks.

That’s good, but I like Timothy Wolodzko’s explanation even more: “As an industry, we're biased toward general-purpose tools [because it’s] easier to hire devs, they are already widely adopted (because being general purpose), often have better documentation, are better maintained, and can be expected to live longer.” Some of this merely describes the results of network effects, but how general purpose enables those network effects is the more interesting observation.

Similarly, one commenter on Bernhardsson’s post suggests, “It's not about general versus specialized” but rather “about what tool has the ability to evolve. That tool will dominate.” This is certainly the case for general-purpose programming languages like Java: They have thrived precisely because they’ve changed to embrace new industry demands. Likewise for databases. O’Grady argues that “many datastores that were once specialized are becoming less so,” even as mainstays like MySQL take on features innovated by NoSQL databases.

Does this mean we’re left with a big pot of sameness? To a degree, perhaps, but Python is never going to be C is never going to be Rust, just as Oracle is never going to be MongoDB is never going to be DB2. Even general-purpose languages, frameworks, and databases have unique capabilities, which is why we have top 10 lists for databases, languages, and frameworks and not top 1 lists.

Enterprises and the developers within them will continue to demand real choice in their technology decisions. But not too much choice, as I recently detailed. General purpose works because it increases developer productivity (learn a few tools, not a million) and enterprise agility (talent becomes easier to hire, among other things).

By all means, make a hobby of that shiny new programming language. Just don’t try to make a career out of it.