Hacker | Engineer

I love Ruby. As yet another Java convert, I enjoy Ruby’s expressiveness and simplicity, not to mention Rails. Lately I’ve been playing more with developing domain-specific languages in Ruby, and as Matz pointed out in his RubyConf keynote a couple of weeks ago, Ruby is great for building DSLs. As I start out on my next project, though, I’m beginning to understand why Lisp people go so crazy about macros. Basically, it’s all about the metaprogramming.

I see DSLs as 1st-order metaprogramming. You’re writing code that writes code, but you stop there. Rails does an amazing job of this, most notably in ActiveRecord relations (has_many, belongs_to, etc). As it turns out, many programming problems can be solved more elegantly with metaprogramming, and you generally only need one order of it. Instead of expressing the solution in basic language constructs, you go one level up and use a DSL. Not all problems, however, lend themselves to 1st-order metaprogramming. This is where macros come in: they’re nth-order metaprogramming.

Let’s say I want to design CPUs in FPGAs. I’m doing this for a class next semester, and the language we get to use, VHDL, is terrible at abstraction. So, I’m going to design a DSL for defining processors. Instead of working from the ground up, I’m going to start at the top and work down. Here’s what I’d like to write to build a simple little CPU.

Now, how am I going to get an FPGA from this? First off, I need to generate something more low-level that looks more like VHDL. Then, I’m going to need to translate that into actual VHDL so I can use an FPGA design tool like Quartus to get the end result. Now, how would I go about writing this? Well, I already started a bit, and the first step was a Ruby DSL for generating VHDL. But wait, I’ve got another DSL on top of that for the computer! Now what? Unfortunately, the backend for Ruby DSLs can get a bit messy, so trying to layer them is going to be a significant effort. Let’s take a step back here, though, and look at what I’m doing. What I really want here is 2nd-order metaprogramming. It can be done with two layers of 1st-order metaprogramming, but is there a better way?

Enter Lisp. Now, I’m not very experienced with Lisp yet, but that’s going to change now that I’ve recognized the significance. In a Lisp dialect, I could define my computer quite similarly to the Ruby above. Then, I use macros to decompose the “data” into something else closer to my goal. I can keep doing this until I’ve created a “data” structure that contains the code I want to execute, then I execute it! Layering the metaprogramming becomes a simple process of keeping track of the macros transforming the code, but they’re completely independent and loosely coupled. No more nasty DSL backend. No more struggling with code generation and parsing (How would you handle the “x+y” part of the above in a code-generating DSL?). This project is going to happen, but it’s going to happen in Clojure.

Lisp natively supports nth-order metaprogramming. In fact, from what I can tell, that’s pretty much the only thing it has going for it. The code/data duality of Lisp is somewhat akin to the wave/particle duality we encounter in physics. You can just accept it and move on, but to really understand things you must embrace the duality. This can be a big leap for a procedural programmer who tends toward languages like C, but I see it as the next step in my growth as a software developer. With nth-order metaprogramming, you can define the problem in code and work forward from there toward the solution, using as many layers of metaprogramming as you need. Often this is only one layer (thus Ruby’s awesomeness), but the flexibility to add another layer of metaprogramming when you need it can be well worth the learning curve required to pick up a language like Clojure.