philosophy of DSL and library design

Do you ever wonder why almost every library out there is a convoluted mess of ill conceived abstractions with ill defined composition semantics? I do all the time and I think I have finally figured out why.

Some of these libraries/DSLs are developed by newbs who are using a language/technology stack in anger and are trying to bring the abstractions of their old tools to the new stack. In the process they learn their way around their new home but produce an abortion that might accidentally become popular or worse yet become an integral part of operational infrastructure. Java is full of such abortions. For some reason Java architects are unwilling to accept that Java is a statically typed language and every enterprise framework out there bolts on an ill-conceived introspection/reflection mechanism powered by XML on top of the JVM. At the end of the day though, whether the library was the result of an angry newb unhappy with his new home or an architect unwilling to accept the nature of the language, the underlying assumption is the same: The language/technology stack as it stands is inadequate, even though it is turing complete.

So forget the above assumption. Start with the fact that you are working with a turing complete language and ask yourself how can you compose together the existing native abstraction of the language to accomplish what you want in a minimally invasive way. A story to demonstrate my point.

I was recently playing around with an RPC mechanism and needed a simple way to parse messages coming over the wire. I was using nio4r and reading bytes asynchronously for performance reasons and so I needed a way to build something that could be paused and resumed. My first stab at the problem was a shitty DSL for building a stack machine that would parse the bytes coming over the wire. I had goto, return, sleep, yield, and a whole bunch of other instructions implemented. All I really wanted was a sequence of instructions that I could pause and resume while consuming bytes over the wire but I was going about it all wrong. After a few days of this nonsense I heard a voice,

“Stop! You’re using ruby which is an extremely malleable language. Why are you building your own shitty language on top of ruby?”

So I went back to the drawing board and cut out the DSL and used fibers instead. Fibers, a.k.a. co-routines, are almost custom made for writing state machines for parsing stream based protocols. All I needed was a little sugar on top of Fiber.yield to accomplish what I wanted. A side benefit of the fiber approach was that I retained the full power of ruby in the DSL itself without my own ill-conceived state machine abstraction getting in the way.

In conclusion. Building abstractions is good, but when a set of abstraction in and of itself can be considered a full interpreter for some shitty language then you’re doing it wrong. The language you are working with is plenty powerful. There is no reason to bolt on another language on top of it.