ruby dsl tricks: reifying references

Ruby is great for writing DSLs because it has first class support for two of the most important ingredients of DSLs, contexts and code blocks. With the proper use of instance_eval the same block of code can be evaluated in various contexts to have different kinds of effects but most often what we want to do is evaluate the code block in the “freest” possible context to create an AST (abstract syntax tree). I’m almost certain there is a connection here with initial and terminal algebras in category theory but someone smarter than me will have to chase that analogy. Today I’m just going to demonstrate how to reify references so that we can support cyclic structures in our DSL. Continue reading

war story: caching

There was that one time I used strace and a Ruby script to bypass a really long step in a build pipeline. The trick was figuring out the inputs and outputs by running the process under strace and utilizing the output from strace to compute some hashes. The core of the script was a utility class and some convenience methods for computing hashes by shelling out to find, tar, and shasum Continue reading

finger exercises: pipes, bytes, and fibers

In which I try to figure out how to pack and unpack bytes over an in-process pipe so that I can use it in some future message framing protocol for a worker pool. There will be a guest appearance by Fiber to simplify the parsing of messages in a non-blocking manner. Continue reading

software hygiene: encrypt your secrets

Update: You should use something like Blackbox, git-crypt, or some Rake utilities to do all of this.

Most of my projects have a Rakefile because common tasks should be expressed in code instead of english and Rake is a great way to codify those common tasks. One thing that I have seen developers do is check-in secret tokens into their repositories in plaintext. I have done this as well. It is the simplest thing to do but it is terrible practice so to atone for my past sins and get others to not check-in secret tokens here is some code I now use to handle secret tokens. Adapt to your own workflow accordingly. Continue reading

optimizing spot instance allocation

There is surprisingly little information on how to optimize costs using the AWS spot instance market. There are several services that provide management on top of the spot market if you have an architecture that supports an interruptible workload but very little in the way of how to go about doing it yourself other than surface level advice on setting up autoscaling groups. To remedy the situation here’s an outline of how I’ve solved part of the problem for CI (continuous integration) type of workload Continue reading

How does Ruby do X?

The answer can usually be found with ruby -rdebug or ruby -rtracer. There is also the trick with RUBYOPT. If you are executing something with bundler then you can run RUBYOPT="-rdebug" bundle exec ${command} and you will be dropped in the debugger as usual.

production grade logging

Logging to sockets is better than logging to files. It allows for more flexibility in terms of log rotation and data integrity. I started looking around for examples of this but everything these days when it comes to logging is built for the enterprise. The actual skeleton of what all those enterprise systems are doing is quite simple. In fact it is so simple that you can do it in less than 30 lines of code in most high level languages. Here’s the skeleton for a logging server in Ruby: Continue reading