Software Design and Scalability

There is a common meme in programming circles that small teams are more efficient and productive than large teams but few explanations for why this is the case. This is my take for an explanation.

I don’t think the size of the team is relevant. At its core, the problem is about reification and communication of abstractions. No matter what language is used and what capabilities it has there is no way to get around the analog communication barrier. Macros and mini-DSLs don’t encode the reasons for why the DSL looks like it does and you have to go and ask the original authors. It doesn’t matter what the language is if the implementation is the design there will be scaling issues. So one way to alleviate the scalability issues in any language is to decouple the design from the implementation. Formally specifying the design can communicate the constraints in a given problem domain much better than a handful of macros and overloaded operators.

Zooming out a little bit from code and programming, we can claim humans are just bad at collective decision making when the problem doesn’t fit in a single person’s head. Groups often create incoherent structures and most large engineering organizations look like one of two ways: geological layers of abstractions with teams sticking to their own layer or archipelagos that are far enough to make travel a hard problem. In fact, I don’t think this structure is restricted to just programming and any large enough organization that is solving problems will have a similar structure.