War story: sources, sinks, archives, and SSH sessions

Sometimes the right way to think about programming problems is as a sequence of transformations between sources and sinks. I present some Go code for streaming a tar archive from a remote host to the local filesystem. The standard library has everything we need and we are just going to put it together. Copy and adapt to your own needs as necessary. Continue reading

Caching w/ strace

There was that one time I used strace and a Ruby script to bypass a really long step in a build pipeline. The idea from a high level was pretty simple: run the build process with strace and see which files were read from and written to. After “profiling” the build with strace the inputs were hashed and used as a key for bundling the outputs.

The core of the script was a utility class and some convenience methods for computing hashes by shelling out to find, tar, and shasum

Continue reading