System design 101

I’ve seen a certain pattern of thinking in engineering organizations that feels backwards to me but I keep running up against it. Part of the reason it feels backwards to me is probably because I tend to think axiomatically and the pattern of thinking that I’ve observed makes no sense logically. Abstractly the argument almost always involves 3 components of a system (A, B, C). If it makes things simpler you can pretend A = front-end, B = back-end, and C = database. Continue reading

Terraform as a backend (in a compiler sense)

Every time I’ve tried to use terraform proper I have failed. Fundamentally it is a non-composable tool so if you want compostionlity you have to work around its limitations. One way I’ve found to force compositionality is to not rely on any of its high level features. I only use it for managing resource graphs. That’s the only part I think it is good at and that’s the only part that consistently works. If you feed it a resource graph then it will mostly work. This means you’ll have to find some way to generate and feed it the resource graph. There are a few ways to do it and in the past I would have recommended ERB but there is actually a simpler way that just requires generating JSON. You could generate the JSON from anything you are comfortable with that provides the kind of compostionality you expect. I’m going to outline how I use Rake and Ruby to do this for a very basic VPC configuration across several regions and availability zones. Continue reading

Bootstrapping a consul cluster with cloud-init-buddy

Bootstrapping a consul cluster is a non-trivial operation mostly because it requires sequencing the startup of the cluster nodes with the starting of the consul agents. You can’t form a cluster until you know the IP addresses of the nodes to pass to the consul agent for registration and cluster formation. So you first need to start the nodes and then distribute the IP addresses to all the nodes that need to be in the cluster. This is where cloud-init-buddy comes in. Cloud-init-buddy provides the glue for the cluster nodes to discover each other and not worry about sequencing startup and setup. Everything happens in the cloud-init script with all the sequencing and discovery automatically handled by the cluster nodes themselves. Continue reading

Unsolved but tractable problems

The world is full of unsolved but tractable problems. One of my recent projects needed an AWS spot market simulator but all the obvious searches didn’t turn up anything so I went ahead and wrote one for my use cases.

Another but slightly smaller problem was about writing to and rotating logs based on number of lines instead of file size or timestamps and again the obvious searches turned up no results so again I wrote something.

Never be afraid to tackle small and tractable problems. Never let the haters get in the way of making and sharing something because there are people out there that will find your work useful and at the end of the day that’s what counts.

Project checkpoint: AWS spot market simulator

Finally settled on a structure that seems to work. My initial design was doing everything in Ruby by shuffling elements between sets and hashes and this was a little hard to follow. After stepping back from the problem I noticed that I was basically re-inventing database tables very poorly so I shifted everything to using an in-memory SQLite database.

Update: I’ve now written a simulator that can be used for testing bidding and control strategies without having to pay for actual server time.

Continue reading

Project checkpoint: AWS spot market simulator

I’m currently working on a very basic spot market simulator for testing various spot market strategies. I initially thought that this would be pretty simple but it turns out it’s not so simple mostly because programming languages are terrible at expressing transactional semantics of high level state machines. Let me try to elaborate.

Update: I’ve now written a simulator that can be used for testing bidding and control strategies without having to pay for actual server time.

Continue reading

Impulse response programming

In signal processing and filter design there is the notion of a linear system. The nice thing about linear systems is that they can be safely analyzed by just poking and prodding them with an impulse. You push a certain form of signal through the system and observe the output and if for signal s you get output o then by linearity you can conclude that 2s will lead to 2o. Basically, linear systems are nice because you can add and scale inputs and expect the outputs to also add and scale appropriately. The trouble is that the world is not linear and all our intuitions fail us when dealing with non-linear systems. This is especially problematic in software because the scale of the complexity makes the systems we deal with essentially black boxes and the only view we have into them is by poking and prodding them with impulses that we understand and then assuming some kind of linearity. An anecdote from work about Docker and LXC follows. Continue reading

Agile is pigeons dancing on one leg

There are some things that people don’t like to think about. There is no particular reason they don’t like to think about those things. It’s simply because their parents didn’t think those things and through luck and chance that bias was propagated to the next generation. This happens in nature all the time. Just because something is passed on to the next generation doesn’t mean it was useful or adaptive in any way, it could just have been some random structural property that resulted from a confluence of other things. This is very vague but the point I’m trying to get across is that people have a propensity to see patterns where there are none and sometimes they will carry on believing something completely random because random good things happened to them when they believed those things. So through the power of associations those random beliefs were associated with good things even though if you could zoom out and do the proper statistical analysis it would all look like noise. Now couple this with under-constrained systems like we see in software all the time and it is a recipe for a pretty hilarious disaster. Continue reading