I think you might be asking the wrong questions. They key questions are Queries per second and the median response size of the query.

For example at reddit (15 years ago) we had 10x more vote traffic than comment traffic, but we only needed two databases to handle votes (technically only one the other was just for redundancy).

But we needed nine comments databases. Mainly because the median query response was so much bigger for comments.

TiDB, 200k inserts per second; 200b per row on average. Bursty insert pattern, e.g., can have 200k inserts for an hour, then almost 0 for days. 8k reads per second on average, mostly reads by primary key. 20 hosts; 16 threads x 128GB RAM, 8TB NVME RAID 10. 60TiB of useful storage with replication factor of 3. Keyset pagination is the key. Also using rocksb for inserts batching. Costs around 20k on ovh

Asking about customers is the wrong question.

1. It's often information that cannot be casually shared for legal reasons (MNPI)

2. A single customer might generate many queries. There have been times where a single one of my employer's customers generates more traffic than most companies will ever reach at peak.

You’re asking the wrong questions. Query complexity and patterns matter. Nobody can answer this for you. You have to do the analysis based on your workload.

I’m not going to pile on with “these are bad questions”.

We serve a few hundred concurrently online users of our web dashboard which visualises data from our ~100k online IoT device fleet.

With respect to our time series systems:

We store around 1M fields (kv pairs) per second, around 10k rows per second i.e. almost a trillion rows per day and almost a hundred trillion new data points per day.

We compress this down to around 1TB of data per day using Clickhouse and store and make available the last 30 days of data for “online analysis”.

We also store all the data we’ve ever ingested in BigQuery for M/L purposes. I shudder to think of how much this costs.

Our queries are all pretty simple and fast, just scanning and aggregating with filtering, all under 100ms.

This costs around 30k USD per month on AWS, not including networking which is too hard for me to separate from our non time series workloads.

Our infrastructure is all python and microservices, using Kafka, and under 1000 cores for the whole pipeline excluding databases.

Our databases are I believe low tens of the largest graviton instances that were available the last time we upgraded.

This is maintained by a backend team of around 15 people, and a total engineering team including hardware, data and software of around 45 people.

I'll bite, just so you get a real answer instead of the very correct but annoying "don't worry about it right now" answers everyone else is going to provide!

We have a rails monolith that sends our master database instance between 2,000 and 10,000 queries per second depending on the time of year. We have a seasonal bike business with more traffic in the summer. 5% of queries are insert/update/delete, the rest read.

mariadb (mysql flavor), all reads and writes sent just to master. Two slaves, one for live failover, the other sitting on a ZFS volume for backup snapshotting sending snapshots off to rsync.net (they are awesome BTW).

We run all our own hardware. The database machines have 512gb of ram and dual EPYC 74F3 24 core processors, backed by a 4 drive raid10 nvme linux software raid volume on top of micron 9300 drives. These machines also house a legacy mongodb cluster (actually a really really nice and easy to maintain key/value store, which is how we use it) on a separate raid volume, an elastic search cluster, and a redis cluster. The redis cluster often is doing 10,000 commands a second on a 20gb db, and the elastic search cluster is a 3tb full text search + geo search database that does about 150 queries a second.

In other words, mysql isn't single tenant here, though it is single tenant on the drives that back our mysql database.

We don't have any caching as it pertains to database queries. yes we shove some expensive to compute data in redis and use that as a cache, but it wouldn't be hitting our database on a cache miss, it would instead recalculate it on the fly from GPS data. I would expect to 3-5x our current traffic before considering caching more seriously, but I'll probably once again just upgrade machines instead. I've been saying this for 15 years....

At the end of 2024 I went on a really fun quest to cut our DB size from 1.4tb down to about 500gb, along with a bunch of query performance improvements (remove unnecessary writes with small refactors, better indexes, dropping unneeded indices, changing from strings to enums in places, etc). I spent about 1 week of very enjoyable and fast paced work to accomplish this while everyone was out christmas break (my day job is now mostly management), and prob would need another 2 weeks to go after the other 30% performance improvements I have in mind.

All this is to serve a daily average of 200-300 http requests per second to our backend, with a mix of website visitors and users of our mobile apps. I've seen 1000rps steady-state peak peak last year and wasn't worried about anything. I wouldn't be surprised if we could get up to 5,000rps to our API with this current setup and a little tuning.

The biggest table by storage and by row count has 300 million rows and I think 150gb including indexes, though I've had a few tables eclipse a billion rows before rearchitecting things. Basically, if you use DB for analytics things get silly, but you can go a long ways before thinking "maybe this should go in its own datastore like clickhouse".

Also, it's not just queries per second, but also row operations per second. mysql is really really fast. We had some hidden performance issues that allowed me to go from 10,000,000 row ops per second down to 200,000 row ops per second right now. This didn't really change any noticable query performance, mysql was cool for some things just doing a ton of full table scans all over the place....

This sounds just wrong.

Why would you use micro service? You are not having teams.

Btw. Your data structure/ bytes per row is missing

5,000 users PER DAY is trivial sauce, you're totally worrying about something ridiculous. Even a crap server with crap code should handle that.

BTW most databases on a decent server could totally handle that 1 million IoT updates per day too. 1 packet per day is nothing. Unless they all come at once. That is also a fairly trivial load, if it's spread out. A small VM could handle that.

You are way off on your understanding of what is a heavy load.

You could load test with something like k6 if you want to find out. Try 'emulating' the requests and average users.

I often test with 5,000 requests per second, 5,000 users per day with 20-30 requests each is several orders of magnitude less load.

Let's do some math.

First, the IOT devices reporting daily. In the absence of further context, I’m going to assume that it doesn’t matter when the devices report, so they can be configured to spread out their load. I’m also going to assume 1kb of data per device, but with an HTTPS API there’s roughly 7kb of overhead that we need to account for when calculating bandwidth. (Source: http://netsekure.org/2010/03/tls-overhead/ . TLS session resumption gets that down to ~300 bytes, but adds implementation complications.)

    $ units -1v '1M req/day' 'req/sec'
        1M req/day = 11.574074 req/sec
    $ units -1v '1M req/day * 8kbyte/req' 'kbyte/sec' # Incoming bandwidth
        1M req/day * 8kbyte/req = 92.592593 kbyte/sec
    $ units -1v '1M req/day * 1kbyte/req' 'years/TB' # Storage
        reciprocal conversion
        1 / (1M req/day * 1kbyte/req) = 2.7379093 years/TB

It looks like our load here is a whopping 12 RPS, and we could handle the traffic on an ISDN line from the 1990s. Data storage is a little trickier; if we can’t compress or delete old data we may have to stop by Best Buy for a new hard drive every half decade or so.

Users can’t be configured to load balance themselves, so we’ll be pessemistic and assume that every single one of them logs in to check their device over their morning coffee. We’ll also assume that every time they do that they want all of the data from the last 3 months, though in practice this could probably be summarized before we send it to them.

    $ units -1v '5000 req/1 hour' 'req/sec'
        5000 req/1 hour = 1.3888889 req/sec
    $ units -1v '5000 req/1 hour * 90 kbyte/req' 'MB/sec' # Outgoing bandwidth
        5000 req/1 hour * 90 kbyte/req = 0.125 MB/sec

For this, we have just under 2 RPS, but our responses are quite a lot bigger so the bandwidth is higher—we probably want to move into the early 2000s and upgrade to a DSL connection. Oh, and we also want to make sure our disk can handle the read load—but conveniently, since we’re just pulling raw data and not asking for any complicated joins, these numbers are actually the same. 2 RPS gives us 500ms per request; since a spinning rust drive pessimistically takes 10ms/seek we only have a ~50-seek budget per request so we probably want to make sure to cluster or partition the table by device to improve data locality. (Doing that increases the write load significantly, though, so maybe we want to upgrade to an SSD or think about a database that’s smart enough to do some kind of periodic bulk rebalancing.)

Oh, I almost forgot, we'll also want to make sure we have disk space to keep track of all of those idle logins who aren’t doing anything:

    $ units -1v '1M records * 1kbyte/rec' 'GB'
        1M records * 1kbyte/rec = 1 GB

Modern computers—for very relative definitions of modern—are fast, if they don’t get bogged down. Based on my numbers, you could probably run your system from a Macbook in your desk drawer; but it would be trivial to add a requirement which would multiply any of these numbers by several orders of magnitude. The problem with trying to compare architectures is that you have to really understand what your needs are, and not only are they going to be difficult to compare to someone else’s setup (small differences can add up fast when multiplied across requests) but it’s also hard to tell how much effort the other system put into optimizing (or, for that matter, how much effort you want to put into optimizing).