Lately I've been pairing with my good friend, Caleb Gossler, on some atypical Node.js at work. To be honest it's not the most interesting work to talk about. Rather than go into the bland details I thought I'd extrapolate the valuable part of the experience.
We have been taking a more declarative approach to some of the node code we are writing. It merely describes, through functions, how we would like to consume an API. We are setting goals for how to get all of the data we need. Being declarative affords us a lot of flexibility with how the mechanical parts of the application work. We can deal with HTTP caching semantics under the hood, and even isolate mucky setup code. It also allows us to program what amounts to a bunch of separate HTTP requests somewhat procedurally (it's just a single function at the end of the day).
One of the side effects of this is that we ended up with a bevy of simple functions that are related in some way. When you open up a file and take a look, it can be a little daunting. You tend to acclimate to the approach fairly quickly, but the disorientation part will come up again when you are trying to figure out what the application is doing at runtime. Abstractions come with a cost, and it is in your best interest to precompute the costs and find margins for maximizing your savings.
All of these benefits, so far, seem well worth the confusion. Suppose we can deal with the confusion. We'd just be left with up side, no? Well I figured out a way to deal with a portion of the confusion! Don't mind the stacked argument...
My solution, for the purposes of this post, will be somewhat abstract. I'm more interested in why the solution feels like a good fit. Bear with me.
I talked about a bevy of simple functions above. To elaborate a bit, the functions are simplistic. They take a single parameter, and return nothing. The parameter is an object with functions that let you describe how the current function is related to other functions. If you're thirsty for more, I'm telling you: it's really not that interesting. Okay, so you want to see an overly contrived, unhelpful example that badly. Here you go:
function wiggle(x) {
x.qaz('jiggle', {}, jiggle, (jiggler) => jiggler.wiggled === true));
x.default({});
}
function jiggle(x) {
x.qaz('beep', {}, f => f.default('boop'));
x.qaz('wiggle', {}, wiggle, (wiggler) => wiggler.jiggled === true));
x.default({});
}
function foo(bar) {
bar.qaz('wiggle', {}, wiggle);
bar.qaz('jiggle', {}, jiggle);
bar.qaz('free', {}, free => free.default('can\'t catch me!'));
}
See? I told you it wasn't interesting. However, maybe you can see my point about the levels of indirection at play.
Simplistic functions like these are the bread and butter of this abstraction. Since it is declarative, we're not attempting to hijack a callback to do any hardcore procedural or object oriented programming. We are merely using the functions to convey intent, and intent is a helpful starting place for deciding what work you need to actually do or even how to do it.
Imagine that bar in the example above is some context object that translates obscure function calls to HTTP requests. That's how the approach started. However, as I looked at the code more and more, I started to see a pattern emerge. The functions form a hierarchy, which stands to be more complex than our simplistic functions let on.
The solution seemed simple: I could create a new context to pass to these functions, and collect a structured representation of the intent. That way we could capture the essence of what the code was going to be doing. I dubbed this our execution plan. Programming languages are interpreted in a similar way: you describe your intent by writing the code. The compiler parses it and builds an abstract syntax tree, and proceeds to optimize it and perform code generation.
There are a couple of concrete details that are interesting. First, this is Node.js, so you can do things like call .toString() on a function. And so I did. Then I parse for things like function names, parameter names, and even entire function bodies (predicates). They get woven into the structured data to give you meaningful context. I don't actually care much for this part, but I have some ideas in mind for getting around it (maybe more on that another time).
Second, since there is nothing stopping circular function calls in this abstraction, my solution also needed to take this into account (thanks to Caleb for finding this with a real and complicated example). My approach is to walk up the structure looking at the call stack to see if the function finds another occasion where it was already called. If it finds itself, it will flatten the path and indicate that the next sequence of functions is circular, and also detailing which functions are in the sequence.
The result is a set of tooling that gives us some deep introspection into what is going on at development time and at runtime. Not only that, but we now have a more sophisticated structure that we can use to optimize the mechanical code that is doing all of the hard work. Imagine that we are using a distributed cache for HTTP caching, and this structure allows us to batch up our caching logic to decide which HTTP requests are stale and need to be reissued. This is also a convenient way to capture key performance metrics per logical grouping of requests. We can follow the execution plan, capturing cache hits and request/response times, and even errors. What if we just generate execution plans as a form of documentation? Or, if we think back to the abstract syntax tree thing for a moment, what if we could derive a simple DSL for these goals? Let the possibilities soak in for a moment.
Oh yeah, and it's declarative. That's why we could do this in the first place. Caleb came up with this declarative approach, and it has been pretty great. Being able to easily add value to it is reassuring for my brain. The only thing really missing at this point is being able to do it as efficiently as you could in lisp.