Too often I see software codebases with horribly convoluted architectures. Sometimes the choices are justified because of legitimate business edge cases or backwards compatibility issues but other times it is basically a lack of intelligence and discipline.
In algebraic topology there is a subfield known as obstruction theory. Obstruction theory is concerned with justifying why certain constructions are impossible by showing the existence of some other object that gets in the way. Software development needs such a theory. If there are no obstructions then there is no excuse for writing horrible software.
I wanted to get a taste of Node.js development on the server side by playing around with a very simple screen scraper. Writing vanilla JS is no fun so I also wanted to combine it with TypeScript to see if it would be as nice for server-side development as it is for client-side development. The prognosis is quite dismal. Most of the Node.js APIs are almost actively resistant to being typed and you are forced to use ‘any’ all over the place. I like to use ‘tsc’ with ‘–noImplicitAny’ but try as I might I couldn’t get the types to flow through properly without major surgery on ‘.d.ts’ files and instead of expending the effort just fell back on ‘any’. The code for my little experiment can be found at davidk01/node-typescript-amazon-price-scraper.
Some time ago I read an article on proggit that made an analogy between compression and clean design. Basically if you see a lot of repetition then you factor out that structure and re-use it the same way a compression algorithm takes out common patterns and re-uses them to more compactly represent some blob of data. Sometimes though it is not the structure or the common patterns that make things hard to understand but instead it is too much generality and indirection. So how do you solve that problem? Continue reading →
All software at the end of the day is written for some purpose. In order to fulfil its purpose the software must be deployed into an environment that can sustain it. Unfortunately this aspect of software development falls into the uncanny valley surrounded by development, infrastructure, and operations. The devops movement was motivated partly by the need to shine light into that uncanny valley. Initially the goal was to just reveal all the dark corners and flush out all the monsters that dwelt there and once the shock had worn off people started to think of ways to keep the monsters at bay. Many practices developed for taming the complexity of software projects where brought into the uncanny valley and currently the aspiring infrastructure and operations engineer does not have a shortage of tools to choose from to help with keeping a consistent fleet. In many ways things have gotten better but in many other ways things are still pretty bad and there is no tool out there that will save you.
If you are using the cloud for you infrastructure then you need to version it the same way you version your code and deployment artifacts. Lately, I’ve been using packer to generate AMIs and I couldn’t be happier with how things are working out. I now have a consistent environment whenever I want to experiment with something and when I discover something worthwhile that I think should be a basic functionality of all environments I need then I check that into the repository that contains the packer templates and regenerate all the AMIs. This means I can go back to any point in time and spin up an environment exactly as it existed at that time. Infrastructure is no longer something special. It is now just another artifact in a software development pipeline and you should treat it that way.
Look at me. I’m awesome. I have this thing that can parse titles, tokenize them, do some probability calculations, and tell me how likely I am to like an article based only on its title. I’m truly marvelous. Continue reading →
So I’ve been doing this long enough and have played with enough tools to give you some pointers on how exactly you should set up your infrastructure if you want it to be as smooth as possible. There are two quotes I go back to every time I’m trying to figure out how to approach an infrastructure problem:
The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs. – Alan Kay
When you do things right, people won’t be sure you’ve done anything at all. – God Entity (Futurama, Godfellas)
To live up to the above two quotes you need a bit of help. That help comes in the form of some really neat software: Continue reading →
Lately the folks behind the agile movement have started to denounce the movement because marketers took what was once pure and simple and commercialized the hell out of it. Now agile is just another waterfall. The same thing is happening with devops. Continue reading →
Suppose you want to write a simple in-memory JSON store with an equally simple socket based protocol. You want this in-memory store to support parallel and consistent reads. By “parallel reads” what I mean is if 10 clients request to read data from the store then no client should be blocking any other client. By “consistent reads” what I mean is when a client requests some data from the store there is absolutely no way that client gets half of the data before a write and half of it after a write and there is also some kind of ordering for reads and writes. In other words, if we have an array “[1,2,3,4]” that corresponds to the key “ints” in our JSON store then the following sequence of events is impossible: Continue reading →