μέγας λογοθέτης · A collection of thoughts


Relational vs MongoDB: Webscale!

Intro

Ever since the very first few MongoDB is Web Scale memes following the boom of NoSQL that I've been confused about why anyone would ever choose this over a relational database1. At the time, however, I wasn't aware of why relational databases came to be - that is, what problem they were attempting to solve - and exactly what problem NoSQL databases, particularly document-oriented ones like MongoDB were trying to solve.

All that came across my attention was that it was the new thing, the in thing, and SQL was the old boring and slow technology that wasn't Web 2.0 enough. Or worse, thought-terminating clichés like "if you have a use for it, use it, if not, don't." Okay but what is its goddamn use then?? "Webscale" isn't a complete argument, or the start of one, or even a term that carries any amount of meaning.

Over the years, as a student, and as a professional, I never came across a complete and coherent argument explaining the actual tradeoffs of one model compared to the other; they usually came from proponents of MongoDB and their anxiety seemed somehow related to "Webscale" and I was just told to work with it; in the back of my mind two vague explanations started living rent-free: people making engineering decisions based on "but it's what all the cool kids are doing!" vibes and/or based on "when my startup blows up we're gonna be Google scale!2" which lol, lmao.

It took some 16 odd years for me to finally understand what the hell and it came as part of a small research project into the history of relational databases - not NoSQL!

Index

Yeah it's a long post.

  1. Background
    1. IBM and the space race
    2. COBOL
    3. The data models
      1. Hierarchical
      2. Network
      3. Relational
  2. MongoDB is Webscale
  3. So what the fuck is my problem with MongoDB?
  4. My central point (finally)

Background

This is the very short version of "what, why and how is relational".

IBM and the space race

Between 1965 and 1970 solutions arose that solved logistical problems during the development of parts for the United States Apollo space program. Guy called Uri Berman, working for IBM, had the brilliant idea to centralize parts and plans management into a single program running on an IBM mainframe with terminals connected to it across the very large industrial park they were producing these parts at.

The key innovation here was moving multiple disconnected implementations by disparate programmers on how their programs stored data into a single central solution that everyone could access. On top of this, Berman was also concerned about the separation of the parts of the program dealing with the data on disk and the parts concerned with user interaction. This latter bit gave us what is probably the world's first query language called DL/1, Data Language 1. The data's representation on disk followed a particular data model which I'll discuss later.

COBOL

Separately, industry and academia representatives joined up in an effort to curb the insane costs of programming - think hundreds of thousands of dollars per program. They formed a group called Committee on Data Systems Languages, CODASYL, which, with the help from the United States Department of Defense, gave us the Common Business Oriented Language, COBOL.

In the mid-60s they decide to extend COBOL to have data querying and processing extensions, culminating in the proposal of a standard which also included a data model different to IBMs'.

The data models

Berman's data model came to be called Hierarchical and CODASYL's Network. They are both graph structures; Hierarchical is representable as a tree and Network as a more general DAG3.

Hierarchical

A database under Berman's model is a forest. Trees always have a root node, queries always start at the root and you have to traverse the tree to find what you're looking for. It was completely incapable of representing many-to-many relations4 and as such a part that belonged to multiple products was repeated throughout all of them.

Another weakness was that finding a part that belonged to a product was straightforward; finding which products had a part was, you guessed it, an awkward for-each equivalent that had to traverse the entire forest.

Network

A database under CODASYL's model did not have either weakness because it included a record type that would relate sets to one another. Now you could write "reverse" queries which were exactly like the "forward" ones, only symmetrical; and it also removed a lot of data duplication from a model at a time where drive space was at a gigantic premium5.

It did, however, introduce a weakness all its own: programmers had to specify the path to the data they wanted exactly. If the schema of the data changed, the programs written against that schema would all be invalidated; by this I don't mean in the "well duh" scenario you might be thinking of - traversing a tree, any tree, doesn't require, say, rewriting a depth-first search; without going into specifics it would have for the CODASYL model - think breaking a contract in any OOP language and how any compiler would yell at you to fix your shit.

Relational

Edgar Frank Codd, also working for IBM, takes a look at this shit and decides he can do way better. He defines a model solidly rooted in Set theory and First-order Logic, an algebra over tuples with some operators and operations over them as axioms, derives more operators and operations and calls it Relational Algebra. Not being very useful on its own apart from being able to model and reason about your data on paper he went a bunch of steps further. Relational calculus, the counterpart, is by him defined as a first-order logic statement over the sets created by the algebra; this lets you reason about the shape of the results you wish to obtain.

Then he proved both the algebra and the calculus were equally expressive and provided a reduction algorithm from the calculus to the algebra! To us mere mortals this is what affords us independence from the path taken to get to a result - a statement written in calculus can be interpreted by the database program which knows the physical disposition of the data and its internal data structures and can rewrite or reorder the implied operations you gave it in order to compare alternatives and pick the cheapest6. This is the coolest fucking shit.

With equivalence, reduction and the derived operators Codd would gift the world a data model that could be optimized to reach an optimal on-disk representation of your data - saving precious drive space - while also providing path-independence to programmers. Others in IBM later implement all of this as a query language with the explicit (and extremely far-sighted7) design goal of being useful to both programmers and the occasional manager who wanted to query the language making it sort-of look like plain english. They named it Structured English Query Language or SEQUEL. Unfortunately the term SEQUEL was trademarked elsewhere so it got shortened to SQL or Structured Query Language.

MongoDB is Webscale

Many years after all of this the techbro is invented and begins blighting the earth. They who invented the ad company Doubleclick - later acquired by Google, beginning its ad torment nexus - start a bunch of startups and observe that they keep reinventing solutions to the horizontal scalability problem; they first tackled it at Doubleclick which at one point was serving somewhere around 400k ads a second in the late '90s internet. They decide, following Google and Amazon's initial forays into horizontal scalability with consumer-type machines and how to handle data - the beginnings of butt8 and NoSQL type shit - to create a database that could handle humongous amounts of data, calling it MongoDB and blighting the earth once more.

MongoDB - among other products, fueled by Web 2.0 furor thanks to Google, Amazon and Facebook - start growing in hype from then on, under the umbrella of the so-called NoSQL movement. SQL was old, boring and slow, and thus, not "Webscale". MongoDB is "Webscale" because it doesn't have joins but SQL databases don't know what JSON is because they're old, stinky and unfashionable.

This one time, however, I'm not gonna blame the techbro for inventing a problem and coming up with a solution for it. I'm blaming the idiots that bought into the hype and followed the marketing implicit in "but Google, Facebook and Amazon are doing it!"

Wait what? What problem?

Yep. The problem they set out to solve was real and MongoDB solves it. It is, to me, a niche problem. To multinational ad corporations they're one of the biggest problems they're facing so to me this is definitionally a niche9.

The problem, and why MongoDB solves it

SQL databases - by which I mean implementations of relational algebra and not the algebra itself - and their predecessors were conceived in an era when singular very powerful machines called mainframes were shared by users connecting to them using terminals which had paltry processing power by comparison. As such they tend to be very fast on a single machine and scale in speed and capabilities with that machine's resources. No matter how powerful any single machine is, it cannot cope with the Internet, in any era of it.

To be more formal: at the root of the "Webscale" term lies a real environment with its own challenges.

The SQL type of thing reduces data to its atoms (or at least atoms as SQL regards them) in order to achieve optimal space savings and ease of querying every which way. Partitioning data in this representation is not trivial and by the time MongoDB was thought of did not have a ready-made solution10. Consider: do you partition by table? By row? But a row describing whatever your unit of data is might involve multiple tables, so where do you draw the line? SQL itself has no notion of these things, and was strictly confined to a machine.

Document stores like MongoDB have a different kind of atom such that is trivial to partition a set of them. Documents - essentially Network Model DAGs without the path issue - are a giant atom. Complete, self contained, and having no references to other documents. This made it very, very easy to add more MongoDB instances running in more computers in order to scale your storage and your processing. Just add some glue to balance your load and find where a particular document is and you're done, easy.

So what the fuck is my problem with MongoDB?

I guess I misplaced my hate. I still hate it tho, #sorrynotsorry

Like I mentioned earlier, realizing that it does solve an actual problem came only recently. In my ignorance, mostly fueled in part by the continual degradation of search engines and mostly by the sheer volume of "MongoDB is Webscale" fools posting about it, I could not understand what problem it solved. All I could see was an entirely new class of problems introduced that were already solved by proven technology.

You see, the fools can't articulate the problem it solves. They can only parrot half (if that) understood talking points they received from others who parrot half understood talking points from someone else; or worse, it's coming from a non-expert11 for whom "big multinational corporations are doing it! It must be good!" counts as an argument (as opposed to being a signifier for a fucking idiot). So whenever I went on my bi-annual tour of the internet to find what real people were saying about MongoDB I could never get a straight answer. Which didn't make me any keener to go to the fundamentals myself. My mistake.

And so I developed a dislike for MongoDB based entirely on the fact that its query language is unfit for humans, does not enforce a schema12, inherits every single problem save one from a 60 year old data model and creates entirely new problems.

The all-new "Webscale" type of problems

Like the tree model, queries along the document are swift, easy; you're working with the tool, as it was designed to work. Queries across document collections, also like the tree model, are beyond awkward and made worse by its awful query language.

Tautological, I know but... if you use it for what it was designed for, it... works. It works well. The moment you want to query across documents or collections you made a mistake. Not in choosing MongoDB for its ease of horizontal scaling; but as a database you can write useful, performant, and easy to understand queries against.

This is my actual problem with MongoDB. The fools proliferated it, aided by the company making the product. And it ends up in places it shouldn't be, which is almost all of them - because you don't have, will never have, and shouldn't even aspire to having the magnitude of the problem MongoDB solves.

By way of defining what places it shouldn't be I'll quote/paraphrase Michael Stonebraker - he was a prolific poster on this subject for decades.

"It is dangerous to use a KV store in a complex application that requires more than just a binary relation. If an application requires multiple fields in a record, then KV stores are probably a bad idea. Not only must the application parse record fields, but also there are no secondary indexes to retrieve other fields by value. Likewise, developers must implement joins or multi-get operations in their application."

He adds:

"...RDBMSs easily emulate KV stores without any changes. If an application needs an embedded DBMS, there are full-featured choices available today, including SQLite and DuckDB."

lol

Further:

"Higher level languages are almost universally preferred to record-at-a-time notations as they require less code and provide greater data independence... But the optimizer remains the hardest part of building a DBMS. We suspect that this engineering burden was a contributing factor to why NoSQL systems originally chose to not support SQL."

Michael Stonebraker & Andrew Pavlo, "What Goes Around Comes Around... And Around...", SIGMOD Record, June 2024

My central point (finally)

I don't credit fools with the ability to articulate or understand the points Stonebraker was making. Engineering is about tradeoffs. You know that, I know that, fools don't.

When you pick MongoDB you're (I hope) solving a narrowly-defined problem - horizontal scalability, particularly the need for speed. Most, I'm certain, do not have this problem.

Using MongoDB to do queries that go against the atom of its data model is using the wrong tool for the job in the same manner that trying to achieve orbit with a bunch of guns is13. Fools cannot accept this - the big players are doing it!!!!

Doing anything across documents or collections is working against the tool. MongoDB, not having a very good query optimizer or any regard for humans with its goddamn awful and very limited query language, makes writing queries a chore, and require you to pull, deserialize, operate on etc. the data in the app, unsolving the original problem Berman was solving. This has cost everyone involved in this a lot of time and money, driven by fools and their anxieties. MongoDB is only partially culpable here.

In short: fools pushing MongoDB into places it doesn't belong forced numerous reimplementations of tools older than some of its users, without the battle-tested hardening. A pointless waste of brainpower in service of capitalism, stemming from listening to those who are rich rather than those who know what the fuck they're talking about14.

That's a lot of words to say "That's Capitalism!".


1

Yes, I'm biased. Show me data that isn't in some way relational and I'll point and say "Behold! A scalar!"

2

The same Google that pioneered some of the tech and later came to regret its costs and walked it back.

3

Dear reader, you have to imagine yourself as a programmer in the '60s. There is no ACID. This is the most barebones shit imaginable - assume nothing aside from what is on the text.

4

Except with a sort of model-compromising hack that added pointers to other trees in addition to pure values.

5

It's like poetry - it rhymes.

6

To find a value the engine might decide it's cheaper to sequentially scan a table, or it might consult an index, or... Point being, you aren't being specific about the how, just the what.

7

If you've ever had to write a MongoDB query of any complexity you'll know why this is of extreme importance. Garbage-ass query language, that.

8

I fucking despise the term "cloud". It's a stellar marketing victory, convincing rubes that were never going to grow big to host their applications on ~* the cloud *~ which, to actual computer touchers, just meant "other people's computers".

9

No, your sicknasty idea that will make a ton of money will not turn your company into one of less than half a dozen multinational corporations with this sort of scale. You don't have this problem and you will never have it. Shut the fuck up. If you do have this problem then none of what I'm writing applies to you because you know what's up. If you don't know what the problem it solves is by this point in the post then you are in the "shut the fuck up" camp for entirely different reason.

10

That I'm aware of at least. I know that around 2006-2009 flickr did not buy into the hype and managed to solve the partitioning issue successfully, proving that MySQL is also "Webscale" with a little elbow grease.

11

Someone without a background in computer science or computer engineering, mostly referring to money counters or managers without subject matter expertise. I give a pass to those autodidacts - a few of which I had the privilege of working with - that have proven themselves. This is not a perspective cast from the top of an ivory tower.

12

In the big '26 you may add a JSONSchema to a collection. Its own snowflake dialect. On top of its slop query language. Like I'm fond of saying, this is a tool that allows itself to be misused.

13

No, it can't be this stupid. You must be explaining it wrong.

14

The fool believes the world is just, and that the rich must have done something to deserve it, so they must be worth listening to. big lol