Experts, Idiots, and Taste

My nerd rage on the topic of Fibers, co-routines, and compiled-to-js languages in node.js comes from seeing a non-problem solved in a way that makes it worse.

There are two really dangerous rationalizations in software design:

  1. “We need to make it easier for newcomers.”
  2. “Only experts will use this, anyway.”

Tower of Babel of Cards

They’re dangerous because each seems to imply that the other can be disregarded. The worst approach is to vacillate between them: “We need to make (async IO, object cleanup, database queries, whatever) easier for new users, so we’ll add this magic that experts will use to make intuitive APIs.”

It’s a vicious spiral. You can justify adding complexity, because “this is for experts only”, and also exposing the functionality in an obscured way in the name of making it “intuitive” for newcomers. The result is inevitably a towering house of cards. On a somewhat visceral level, I get this gut reaction to a “handing down from the mountain” mentality that is only harmful in the long run for everyone.

A better goal is to make an API such that when an expert uses it expertly, a newcomer can still understand it thoroughly. It requires an API (and more importantly, a culture) that limits experts, and educates newcomers. I don’t know the best way to approach problems to get there, but I’ve found pretty good results from thinking, “I’m probably going to mess this up the next time I touch it, so I’d better make it really obvious.” I usually fail at it, and end up cursing myself later when I rewrite the damn thing yet again.

Passing around a function that gets called later is not that hard. We’ve been doing that in JavaScript forever. Yes, newcomers to the language, especially if they come from a language without closures and first-class functions, have a little bit of a learning curve to get used to this phenomenon. In my opinion, you do them a disservice by trying to shield them from that.

(Passing around function pointers in C is actually a little bit better in some ways, because you can’t define your functions inline, and all the captured data must be passed around explicitly. This is a bit like using named functions instead of nested closures, which is a common low-complexity approach to the alleged callback problem.)

I’ve been writing JavaScript for a while now, and other languages for a few years before I got into JavaScript. I’m not the most expert at it, but I certainly don’t consider myself a newcomer. In that time, I’ve learned something:

We’re All Idiots

All of us, whether experts or newcomers. Well-meaning idiots, perhaps, but idiots. We all write bugs, all the time. That is the normal state of software. There is no abstraction so perfect that a human won’t fuck it up at the first opportunity. But we can try to be better, simpler, to make it easier for our future idiotic selves to figure out what we were thinking in our cleverly idiotic brains, and chop the problems into smaller pieces so that our future selves can have some clues to find the maddeningly idiotic bugs we stuffed in there.

If your code uses step or async or slide or Q, I can read it, and I see a function getting called and passed some arguments. I can look at how you got that function from a require() statement, pull up that module, and look at its definition. These are lightweight abstractions that are highly penetrable. I’ve suggested in the past that everyone should write their own flow control lib, because it’s the best way to really understand how they work, and it’s not that hard. Once you grasp the basic concepts, reading someone else’s flow control utility is pretty easy.

If your code uses a compiled fibers library, or something that transforms the code, then it’s much harder for me to figure out what the heck is going on. The introspection is gone. The languages change. Stuff happens that isn’t present in the code I’m looking at. This is fundamentally different from a library that just passes functions around.

That means: I won’t use your library if there’s any alternative (including writing the exact same functionality myself). It’s nothing personal. I just would rather spend my time solving necessary problems rather than manufactured ones. I simply don’t believe that you’ve put the same level of thinking into your coro util that TC-39 and the v8 team has put into JavaScript, or that Bert and Ryan and Ben and Igor have put into libuv.

Not everyone is the same, and that’s great!! If you find that Fibers solve a problem for you, and they make the code that you write easier for you to manage and design, then that’s awesome. Use them. Understand them. Drop the illusory dichotomy of “experts” and “newcomers”, and realize that they are both idiots, and both you.

Every choice that isn’t completely boring will alienate someone. Pleasing everyone is a recipe for blandness. That’s why we keep the node-core standard lib super small, and encourage variation and repetition and iteration in userland modules.

Absolutes

If your goal is to please me (and other stodgy “javascript is fine, callbacks aren’t hard” types) with a compiled coroutine library that adds magic to the language, you should realize how horribly unlikely that is. I do actually think that this approach is fundamentally flawed. I’ve played with Marcel’s fibers lib a bit, and it’s about the nicest I’ve seen, but it still adds an unacceptable (for me!) amount of magic to programs.

Programmers tend to think in absolutes, perhaps because computers are such absolute things, or maybe just because absolutes are easier and we’re all idiots. I’ve done it in this post, because I’m as lazy as anyone. Maybe you’re the exception, and you actually are smart enough to use Fibers or Streamline wisely and still know what’s going on when you read the code a year later. If the benefit is greater than the cost, then go for it. It’s your house, you decide whether to wear pants or not.

When we are discussing things like this, it’s important to realize that we’re artists discussing matters of taste. If everyone agreed, then it would mean it wasn’t interesting. Maybe we should be less interested in convincing people of stuff, and more interested in just writing programs that do interesting things.