System design 101

I’ve seen a certain pattern of thinking in engineering organizations that feels backwards to me but I keep running up against it. Part of the reason it feels backwards to me is probably because I tend to think axiomatically and the pattern of thinking that I’ve observed makes no sense logically. Abstractly the argument almost always involves 3 components of a system (A, B, C). If it makes things simpler you can pretend A = front-end, B = back-end, and C = database. C is usually the bottleneck. The argument in my mind ends up being something of the following form

We know that C only has 100 units of capacity but A and B combined need at least 110 units of capacity from C for the overall system to function properly so we are going to add another component D into the mix and make the system perform as desired

So then the inevitable question is how does D give you -10 units of capacity? D by the way is usually some kind of cache which brings with itself the inevitable issues caches bring, i.e. staleness, saturation/eviction, and cache invalidation. If the access patterns are such that D is not being utilized 100% of the time then you can never achieve -10 units of capacity. Best you can do is some small percentage of that which leaves you still with a capacity deficit and a more complicated system. Adding more components into the system by the way is not going to help because adding more and more components into the system fails to address the fundamental issue which is that C only has 100 units of capacity.

Ok, I’ve explained what seems backwards in these types of arguments so what is the right way to address these problems? The right way is to change your axioms. If A and B need at least 110 units of capacity to work properly then the above configuration is never going to work properly. Something in A and B must give. Either you need to add a queue and start prioritizing requests from A and B to remain within the budget for C or you need to rethink the access patterns entirely and shard C in a way such that you have more copies of C that give you more in exchange for the synchronization and coordination overhead. This by the way is a hard problem and there are entire fields of computer science just dedicated to solving this problem. I’m not trying to glorify academics but if there are entire fields dedicated to a specific problem then maybe that problem is not so trivial and can’t be solved by just adding a cache into the mix.