finger exercises: pipes, bytes, and fibers

In which I try to figure out how to pack and unpack bytes over an in-process pipe so that I can use it in some future message framing protocol for a worker pool. There will be a guest appearance by Fiber to simplify the parsing of messages in a non-blocking manner. Continue reading

optimizing spot instance allocation (part deux)

See part 1 for context. This post explains the control/planner strategy I settled on to minimize the cost and make things as stable as possible to mitigate disruptions caused by VMs being re-allocated. Continue reading

good design through abstraction lowering

Many times when I run into buggy software and peek under the hood I notice that the design is convoluted and confused. There are modules with cross-cutting concerns, there is no obvious flow for the inputs and outputs, and there is no set of core abstractions that everything else builds on.

Now compare this to something that is well designed. There is usually a core set of abstractions and associated operations for building up larger computations to accomplish a goal and the rest of the software is built upon those core components. Parser combinators and optimizing compilers are good examples of such software. The flow of the computation is obvious because there are explicit and clear steps that transform high level abstractions and operations into successively lower level ones until everything is expressed by the core set of abstractions. So to understand the whole you just need to understand the core components and how they are combined. There is no reason more general applications can not be built this way. In fact most enterprise architectures with their layered approach are trying to do exactly this but fall short for one reason or another.

refactor by partial evaluation

Some time ago I read an article on proggit that made an analogy between compression and clean design. Basically if you see a lot of repetition then you factor out that structure and re-use it the same way a compression algorithm takes out common patterns and re-uses them to more compactly represent some blob of data. Sometimes though it is not the structure or the common patterns that make things hard to understand but instead it is too much generality and indirection. So how do you solve that problem? Continue reading

continuous integration for infrastructure

If you are using the cloud for you infrastructure then you need to version it the same way you version your code and deployment artifacts. Lately, I’ve been using packer to generate AMIs and I couldn’t be happier with how things are working out. I now have a consistent environment whenever I want to experiment with something and when I discover something worthwhile that I think should be a basic functionality of all environments I need then I check that into the repository that contains the packer templates and regenerate all the AMIs. This means I can go back to any point in time and spin up an environment exactly as it existed at that time. Infrastructure is no longer something special. It is now just another artifact in a software development pipeline and you should treat it that way.

simple monitoring with xinetd

Whenever you are running servers behind load balancers you need to perform health checks for marking servers up and down. If you are lucky then those health checks are built into the application code itself but more often than not you need to hack something together on top of existing legacy application code. In those instances xinetd and a simple script can do the trick. Continue reading

fixing hiring problems

As long as I’m figuring things out here’s another one for you. The Silicon Valley technology worker crunch is a self-inflicted wound. Looking for programmers with X years of experience makes no sense and if this is your criteria for hiring technology workers then you are going to fail miserably. Continue reading

schizophrenic companies

I have finally figured out why all big companies are so schizophrenic. By schizophrenic I mean if you are at a certain level of the hierarchy then all the movements around you seem completely out of whack. You see different groups working against each other instead of cooperatively developing cool technologies and delivering value to the customer. The reason for all of it is what I’m calling the “portfolio theory of management”. Continue reading