There was that one time I used strace and a Ruby script to bypass a really long step in a build pipeline. The idea from a high level was pretty simple: run the build process with strace and see which files were read from and written to. After “profiling” the build with strace the inputs were hashed and used as a key for bundling the outputs.
The core of the script was a utility class and some convenience methods for computing hashes by shelling out to find, tar, and shasum
See part 1 for context. This post explains the control/planner strategy I settled on to minimize the cost and make things as stable as possible to mitigate disruptions caused by VMs being re-allocated. Continue reading →
There is surprisingly little information on how to optimize costs using the AWS spot instance market. There are several services that provide management on top of the spot market if you have an architecture that supports an interruptible workload but very little in the way of how to go about doing it yourself other than surface level advice on setting up autoscaling groups. To remedy the situation here’s an outline of how I’ve solved part of the problem for CI (continuous integration) type of workload Continue reading →