This is mostly advice I wish someone had given me when I was starting out. It probably won’t make much sense until you have dealt with enough code to be sick of it. The usual disclaimers about ranting, warranties, and being a charitable audience stand.
The way most contemporary software is built is structured around local optimums/maximums like Agile and TDD. These are evolutionary answers to the proliferation of software but they’re not global solutions to better software engineering and sustainability because no evolutionary process that proceeds with local optimizations can lead to globally optimal solutions on a complicated landscape like software engineering (it can happen but mostly by accident). History of computing is full of potential global solutions that never gained enough traction for one reason or another. The ones I can think of off the top of my head are smalltalk, lisp, BeOS, xanadu, hoare logic, prolog, refinement calculus, Z notation, etc.
In case anyone is wondering I’m not trying to elevate one approach over another. What we have is what we have and have to make do while trying to improve. My bias is towards more logical and specification oriented approaches to writing software but I’m not claiming it is actually a better way to write software. I have an intuition that it might be but haven’t seen it practiced at scale to form a strong opinion one way or another even if it seems that way.
Now back to regularly scheduled ranting.
Agile sacrifices long term vision for short term gains and TDD optimizes for writing more code to achieve correctness. I don’t think I’m the only that thinks this is backwards. Bugs are correlated with lines of code and TDD forces writing more code so how can it reduce bug counts? If the test code has no bugs then just write the rest of the code in the same style but this is not what happens in practice. Most people use tests and agile planning as crutches to get around having to unambiguously specify their software systems with more high level formalisms.
Kubernetes is another example of a local optimum. Instead of figuring out why software was getting so messy the industry decided that sequestering the complexity behind a distributed control system was the answer. Again, this seems backwards to me because no simplification has been achieved and we’ve just increased the surface area of all systems we have to deal with. Before we had even figured out a proper control system for a fleet of VMs everyone immediately jumped on the container bandwagon and re-invented all the tools that were used in the VM management ecosystem. The only progressive thing with containers was making the tools open source because previous solutions were locked in VMWare’s vaults.
I think most engineers intuitively understand that more software can not lead to simplicity because by its nature software leads to more complexity. Undecidability lurks around every corner and thwarts most attempts at reducing complexity so if we want to sidestep all the issues that Turing completeness leads to then we need to use something other than code to address the issues. If history is a guide then more code is not the answer.
Slightly Less Local Maximums
I don’t have all the answers but what I do know is that I should have started learning about formal methods (both lightweight and otherwise) sooner. Thinking about a system abstractly, intuitively, and from a high level but then specifying it in a formal logical language seems to be net positive from the meager anecdotal evidence I’ve accumulated over the years.
Whenever I’ve approached a new system the bottleneck has never been writing more code. The bottleneck has always been in understanding all the implicit assumptions baked into the system that for one reason or another are essential to its correctness. Pretending that the system and the runtime traces of the system are the specification is why there is always a shortage of programmers. It takes a particular kind of masochist to enjoy reverse engineering a black box from just poking at it with simple linear impulses. If these systems had more formal foundations then there would be no shortage because we could teach people the general formalism and let them figure out the runtime mappings.
Incidentally, this mapping process is something people are good at but machines suck at. We can reason with analogies and partial information much better than probably any other animal but we keep squandering this gift by not filling in the gaps with a general enough formalism that plays to our strengths. Instead, most programmers are tasked with simulating a computer in their head and this is something we are notoriously bad at. Humans are bad at simulating a computer and contemporary programming is all about simulating a computer in one’s head.
So here’s the punchline: if you want to be a good programmer then learn a technology and language agnostic formalism. Logic, statistics, and game theory are probably good bets. As things stand that kind of skill is probably the only thing that is going to survive the coming automation apocalypse because so far no one has figured out a way around rigorous and heuristic thinking.