Re: @brixen’s "Is Node Better"

Update: The talk is online at http://blip.tv/jsconf/jsconf2012-brian-ford-6091521. If you weren’t there, you should go watch it before reading this.

As usual, this year’s JSConf did not fail to deliver. The parties, the location, the talks, the food, it was incredible. Chris Williams (and the rest of the JSConf family) really put together an incredible event.

By far, the most controversial talk was Brian Ford’s “Is Node.js Better”. I must admit, I walked in skeptical. Brian Ford seems like a bright and reasonable guy, but is clearly not a noder.

I’m not going to transcribe what he said. If you want that, read his slides and watch the video when it comes out. This is my reaction and response, and says more about me than it does about Brian.

This is based mostly on my meat-brain memory, and some notes that I scribbled. It’s quite possible that I may get some of the particulars wrong. However, rather than wait for the video to come out, I’d like to write this now, since the errors in perception are themselves useful information. If you notice any such error, please let me know.

Of course, there are limits to how deep a speaker can go into a topic in a 30 minute talk, and those limits are much shallower than how deep I can go in this blog post. So, I’d like to frame this response not so much as me-vs-Brian, but rather as just another public part of a longer multi-format conversation. My goal is not to produce more controversy than is useful, but to perhaps produce enough to keep things interesting.

Fair warning, this is a much longer post than most things I write. If you’re reading it, you’re probably interested mostly in the specific node stuff at the end. I’ve opted not to shorten it too drastically, because the topics are in my opinion best served by a thorough exploration.

[citation needed]

A lot of what I’m talking about is discussed in much more rigor elsewhere. This blog post is a synthesis, not a research paper. In addition to Thinking Fast and Slow which Brian mentioned, these are about the most approachable and informative discussions of the subjects that I’ve found:

The Right

Brian started his talk with a very insightful exploration of the nature of controversy. He talked about our fast brain and slow brain, and guaranteed that reading Thinking Fast and Slow would change your life for the better, or he’d refund the cost of the book.

In discussing the title of the talk, he made the point that every value judgement implies a comparison, and invites controversy. If presented with a choice of apples and oranges, and you remark that “oranges are healthy”, then the implication is that apples are not (or at least less) healthy. So, the question “Is Node.js Better” raises the question, “Better than what?”

Storming Brains

He went on to stress the importance of the scientific method, especially in cases where groupthink can grab ahold of a popular intuition. A powerful example of this is the idea of brainstorming, which has come under criticism in recent studies.

Of course, another fascinating point worth mentioning (which was not covered in Brian’s talk) is that the primary source of the criticism is a survey of the results of 22 different studies with different methodologies, done in different environments, on different subjects.

I’m wondering what the differences are between the 4 where brainstorming worked, and the 18 where it didn’t. It’s especially worth noting that 18 instances of brainstorming’s failure vs 4 instances where it was successful, does not imply that brainstorming is “usually” less effective: only that it’s usually less effective in the studies surveyed. If you check the weather in Wisconsin 18 times in the summer, and 4 times in the winter, you might conclude it’s usually tropical.

This survey was reported several times in the Journal of Personality and Social Psychology. More recently, these journal articles were summarized in the New Yorker and other popular magazines. To me, this smacks of “turns out”-ism. It might be that the popular intuition about brainstorming is wrong. It might also be that it’s just incomplete.

Organization

Brian discussed the concept that organizations tend to perpetuate the problems that they were designed to solve. To my mind, this implies that we ought not to try to solve our problems with organizations, but rather, to solve them with chaos and disruption instead, wherever possible. Since it is often not possible, and because “disruption” becomes just another buzzword meaning a specific sort of organization, we should make it our goal not to “solve problems”, but rather to find the problems that are hidden by our current assumptions.

“Solving” a problem, then, becomes more an exploration than a “fix this thing” exercise. An exploratory expedition stops exploring when it returns with an answer. Similarly, organizations that are designed to solve a specific problem (as opposed to organizations that are designed to, say, throw parties or make buttons) should have their own destruction built into their core constitution.

This principle of minimum institution has guided the choices we’ve made in npm and in Node.js, technologically as well as socially. It has shaped the community in subtle ways that are difficult to recognize from the outside. It’s a principle that is close to my heart, and it’s why Node and I have gotten along so well.

Behavioral Science

“Programming is a behavioral science.” Couldn’t agree more. However, the transition from “behavioral science” to concurrency as a way to deal with scarcity of compute resources felt a bit forced to me. From that point of view, everything is behavioral science, because everything we do is done out of some human need (or else we wouldn’t do it.)

I think it’s more precise to say that software development is behavioral science, because every interface is a human interface, and humans are even more unreasonable in their behavior than hard disk platters.

The Wrong

There were a few points where I believe that Brian simply was either incorrect, misinformed, or perpetuating and reacting to the very groupthink that he is ostensibly setting out to move past.

Justifications for Using Node.js

There were a few justifications for using Node that Brian listed:

These are all interesting justifications, though really, the first two are rationalizations for the third. For a talk that started out by diving into the science of hedonics and decision theory (albeit a somewhat Gladwell-esque popularization of it), I was a bit disappointed to see that he didn’t go into more depth on this point.

He said that users call themselves polyglots, so he rejects the claim that a single language is actually relevant. This is a highly specious rebuttal. I can stand, but I still often find it pleasant to sit. It doesn’t have to be necessary in order to be valid in this case, it only needs to be comfortable. If you’ve never worked in a single-language stack, I highly recommend it. Even though you can’t share a lot of code in most situations, it does reduce the cognitive load switching between different parts of the application. This is the same reason that I’ve suggested using semicolons in JavaScript programs that include other non-ASI languages: reducing switching cost is reducing cost.

Brian claimed that the “many JavaScript programmers” argument is invalid, or at least, not very compelling. We ought to figure out the ideal platform, he said, and then provide advocacy, education, and resources to help people adopt it. This also strikes me as a bit weird. I’m not claiming that one should make all their technical decisions based solely on popularity, of course, but it certainly is not irrelevant. There’s something to be said for pragmatism.

People don’t use Node.js because it’s the same language on the client and server. They use Node.js because that language is JavaScript, and JavaScript is fun.

There aren’t more JavaScript programmers because of a lack of education or advocacy in other languages. JavaScript is fun (and unavoidable), and so all programmers use it.

“Node.js is more fun.” This reminds Brian of Ruby a few years ago. It’s important to remember, in the context of hedonics especially, that “fun” is a highly subtle term worthy of study on its own.

The problem with “Justifications for doing X” is that we do things primarily because of our emotions. Even the action of “being rational” is a thing that we choose to do because of a positive feeling attached to our self-image as a “rational person who does smart things”, and the assumption that we’ll get some pleasantly winful reward that feels good. When we really want to do something irrational, we usually have no problem reconciling the conflict with our preferred self-image as a rational individual.

Emotions are not just part of motivation, they are motivation. If you want to know why a person does something, don’t ask them to explain why they do it; try instead to figure out how they feel. It’s a much more challenging question, but it leads to much more interesting information.

About the word “hype”

When we have a good feeling about something (or even, when we just make any choice at all, and don’t quite understand why), our brain begins inventing rationalizations right away. The first one to pattern-match against “X because Y” is likely accepted as the “reason” we’re doing something, regardless of how much the truth of “Y” actually affects the likelihood of “X”.

Try asking a compulsive gambler why he goes to the casino. He might tell you about his “strategy” for roulette. Of course, this is bullshit. Roulette is a deterministically losing game; the more you play, the more you lose. Same with craps, slots, and (unless you’re the best at the table) poker. It’s the feeling of winning that he’s chasing, and the feeling of the chase. If he was truly motivated by the desire for money, there are obviously much better ways of getting it.

When a person shares their justifications for doing something, one of two things typically happens, in the “fast-brain”:

  1. You feel the same way about the thing. The justification pattern-matches against “I should X-that-I-do because Y”, and so you accept the justification as rational, wise, and well-informed. After all, you already do it, so you must agree.
  2. You feel differently about the thing. Or, at least, it’s not a thing you do, and so you have invented some justifications for not doing it. Your brain pattern-matches against “I should X-that-is-stupid because of Y”, and so you reject the justification as irrational, foolish, and ignorant. After all, it’s stupid, so any justification must be misguided.

If you have some vegan friends, and some friends on the paleo diet, and feel like watching some fireworks, ask them in each others company why they eat the way that they do. No matter how polite they try to be, each will subconsiously perceive the statements of the other as an attack, and feel the need to defend their position. If they are wise enough to retreat from needing to convince the other, they’ll most likely at least make a big show about “agreeing to disagree”. (Of course, if they’ve been on this diet for a while, they may have enough practice at these sorts of situations to handle them quite gracefully.)

If X is not just “do this action”, but rather “feel this way”, and you do not already feel that way, and the justification is not enough to incite this feeling (as justifications almost never are), then we label it “hype”, and it raises the bar that we now require to take the thing seriously next time.

Like so many cognitive shortcuts, this is a really good move much of the time. After all, people’s justifications are usually self-delusions as often as well-informed and rational reasons for doing or thinking anything. It makes perfect sense to be extra skeptical when we are at the risk of being influenced by it. If we have to be extra skeptical over and over again, we start to pattern match “X is good” into the hype category. Suddenly, it’s not just that veganism or paleo is not for me, it’s that the diet is a mark of a foolish person.

The net result is that anyone saying anything positive is likely to be labelled a “fanboi”, and their statement called “hype”, no matter what they say. This is a dangerous feedback loop that leads technology communities to stagnation, bitterness, and chest-beating.

Yo Dawg! I heard you like cognitive distortions, so I distorted your cognition so you can distort while you cognit!

The mind is such a wonderful thing! Mistakes in the mistake detection lead to potentially valid statements being discounted because they are presented along with invalid justifications, or presented by a low-status speaker, or are in conflict with already-held beliefs, especially when those already-held beliefs are a part of our Tribal Story.

Even worse, you have situations where we see a few “X is good because Y” justifications, deem them false, and then subconsciously internalize the fact that we gain status in our tribe by applying the “hype” label and discounting it, resulting in spiraling down into the toilet. Politicians and marketers have made a science of getting us to elevate arbitrary ideas into this Tribal Story, blinding us to any disagreement.

Birds fly. Fish swim. Humans make mistakes. It is our Super Power. We all do this.

We almost never change our minds. We are influenced in ways we are incapable of detecting (and will deny!) We are motivated by the behavior of people who look like us superficially, and believe (at least temporarily) literally everything we hear.

There is virtually no limit to the ability of the human mind to find new and creative ways to get out of touch with reality. I don’t want to give the impression that I (or anyone) could delineate all the many subtleties of human cognitive error. Every one is so rich and complicated. Even this discussion of pattern-matching “X-because-Y” is a serious oversimplification.

In the ancestral environment of adaptation, disagreement was often fatal. We’ve gotten a bit better at intelligent disagreement, but we still try unconsciously to come to an all-or-nothing agreement within our own heads, annihilating any “bad” idea entirely, and shrowding any “good” idea in a halo.

Whenever tempted to call something “hype” (or, even, “anti-hype”), I try to remember to ask the following questions:

  1. Is the provided justification evidence, data, or an expression of a feeling? What is the speaker’s experience of the thing being discussed? If it’s evidence, is it reproducible? If it’s data, is it relevant?
  2. How do I feel about the subject? How did I feel about it yesterday? What are my justifications?
  3. Am I tempted to dismiss this? Is it because of bad evidence, a speaker who holds low status in my tribe, or because I disagree with the feeling? If it’s a lack of evidence, what evidence would make me feel the same way? If it’s a feeling I don’t share, what other things do I feel that way about? If it’s a low-status speaker, how would I feel if <person I respect> was saying it?
  4. Am I tempted to accept this? Is it because it agrees with something I already think? What new information does it actually contain?

This is part of the reason why I try to criticise Node.js and npm as harshly and often and publicly as I can. I do think that they’re tremendously useful tools… but how can you believe me if I tell you that they’re perfect, when I clearly know better? And if you can’t believe me when I talk about Node.js, then what good am I?

Better justifications to use Node.js

  1. The IO paradigm is a good fit for your problem, and V8 is fast enough. This is the case for a lot of web software, but definitely not every program. Node.js really does make it pretty easy to write things like IRC bots and crawlers and websites, and is probably not ideal for many compute-intensive applications.
  2. It’s fun to write programs in this fashion. There are time-honored traditions of thinking of data as streams of bytes, and JavaScript is a relatively simple and expressive language for doing this in creative ways.
  3. It’s fun to be a part of making a popular platform better. The community is active and still pretty friendly, and that feels good to be a part of. Newcomers turn into regulars and then celebrities very swiftly. All it takes is a bit of persistence and enthusiasm.
  4. You prefer the simpler approach to asynchronous programming, using a callback/observer model (or actor model for child processes), rather than something like green threads or coroutines that “look” synchronous.

If you’re not writing systems that are well served by nonblocking IO, or an event loop/observer pattern approach, or if you don’t think it’s fun to make programs this way, then you probably shouldn’t use it.

But, even that aversion is worth looking carefully at that, no matter what you decide. V8 is an extremely fast virtual machine, and these days, a lot of programs are IO-bound.

And even if it’s not, we tend to focus on “the right tool for the job”, at the exclusion of “an awesome tool for the job”. The opposite of “right” is “wrong”, but the opposite of “awesome” is “boring”, which is sometimes worse, especially if you’re trying to maximize creativity.

I’m not sure how much research Brian did when he went out looking for justifications for using Node.js, but I don’t think he really got at what actually is driving so many people interested in this thing.

Controversy

So far, I felt like the setup was nice, but the criticism itself was kind of lackluster. We’ve heard it before. Node is hyped. The single-language stack isn’t all it’s cracked up to be. Etc. I was eager for the meat.

No sooner had I thought this, than the next slide said:

Node.js rejects reality

Aha! There it is. That primal “under attack” feeling, the visceral tension in my lower abdomen, the warmth on my face.

When I was younger (and still today when I’m not at my best, I must admit) the temptation was to hit back, or dismiss the message. Fight or flight. I think Brian probably lost a bunch of the audience with this slide. It sure did get my attention.

These days, when I feel that reaction, I try to leverage the added awareness that comes with the adrenaline, to be on the lookout for whatever comes next, because it is full of valuable information, especially if it is highly disagreeable. Beware of the moments when rationality tries to slip away: those are the times you most need it.

As I am somewhat dogmatic about endeavoring to reject reality less, I was very excited to hear what we might be missing.

Sadly, the justification for the “ignores reality” claim was not as pointed as I would have liked. It amounted to:

  1. JavaScript is a garbage collected language. Garbage collection pretends that you have infinite memory, and you don’t.
  2. IO always “actually” blocks somewhere, so the comparison between events and blocking is not valid, since it has to block somewhere.

For sure, garbage collection is tricky, and the interaction between Node.js and the V8 garbage collector could probably be improved. But to call it a lie is a bit silly. Garbage collection doesn’t pretend that you have infinite memory; that’s virtual memory. Garbage collection pretends that you don’t have to free memory in order for it to be re-used.

One of the biggest pain points of non-managed languages is having to explicitly free memory. The biggest pain point in managed languages is having to deal with a garbage collector. There may be some sort of approach to memory management that isn’t collected or manual, but better than either. If so, I don’t know of it, and it doesn’t matter anyway, because that is a language design problem, and Node.js is not about designing a language. If you have a beef with garbage collection, take it up with V8 and TC-39.

As for node being single-threaded, and IO always blocking, that’s kind of confusing to me. Of course, you don’t actually have application-level processing of IO in parallel, since there’s only a single JavaScript thread, but IO definitely is performed in parallel to the degree that the machine can provide, and “nonblocking” is the technical term for the type of IO that node does on sockets. (Nonblocking IO on files is simulated using threads.)

I’d really like more details about what exactly Brian was referring to on this point about IO “always blocking somewhere” and in what way Node.js rejects that reality.

entire ecosystem of tools and libraries must be built

This was a valid concern in 2010. I said at the time that it would not be an issue in a few years. It’s not an issue today.

To an extent, this was by design. A large part of Ryan’s reason for choosing JavaScript was that it is the only popular interpreted language with a suitably powerful VM and no existing IO paradigm. Additionally, to the extent that JavaScript does have a tradition of IO, it’s XHR and the DOM, which are event-based and asynchronous.

Since the birth of Node.js, a lot of libraries have been built. There are 8500 modules on npm. There are bindings to every popular database, sophisticated test frameworks, template libraries, HTTP routers, and so on. It’s easy and fun to write modules, and so people have written modules. This approach is remarkably valuable.

That being said, module discovery and visibility leaves a lot to be desired. npm has grown much faster than Rubyforge or PyPI, much earlier in the life of Node, and so we have hit these problems earlier.

Regarding tools, things are coming along quite nicely.

Post-mortem debugging with MDB just landed in the master branch, and will be in node 0.7.8. (Node v0.6 has had this for a while already.) Mad props to Dave Pacheco for that. Seriously, if you’ve never seen it in action, it’s amazing. In all my years using PHP, I can’t even count the number of times I’d desparately wanted this sort of thing. The zend IDE had some pretty good debugging capabilities, but Dap’s jsstack stuff is truly magical. Call me a fanboi, I don’t care: believe the hype. It’s fantastic. The only sad point is that it’s only available in SmartOS, but it’s not exactly surprising that the OS made by Joyent has some special love for Node.js ;)

The DTrace support in Node is also impressive. Again, not surprising, given the fact that so many DTrace heavyweights work at Joyent, but the go-to DTrace library for node is not written by a Joyent employee. It’s Chris Andrews’ node-dtrace-provider module. Anarchy > curation, yet again.

The depth and breadth of information that can be provided by DTrace, and the great work by Brendan Gregg and others at Joyent to actually massage that data into a format that humans can easily consume, is absolutely phenomenal.

Similar work on run-time analytics and post-mortem debugging is underway at Microsoft, I’m told. Windows users: stay tuned.

As a long-time text-editor-and-shell guy (moved from TextMate to vim a while ago), I don’t really follow the IDE stuff too closely. But, Microsoft, Cloud9, and many others are iterating furiously in this space. Many of them are leveraging their existing work on other JavaScript tooling, so it’s moving quite fast.

The built-in node debugger client is more my style, but node-inspector hooks into the Chrome debug tools, which imo blow away most visual debuggers available. (They even have heap analysis tools!) There are also plugins for Eclipse to provide stepwise debugging and analysis, and a lot of existing Eclipse users are more at home there.

So, it’s not so much that the ecosystem of tools “must be built”, as much as that it is being built, and leverages the existing JavaScript, V8, and system tools that already existed before Node showed up. In many ways, Node.js provides one of the first server-side systems that can fully take advantage of these things in a mostly cross-platform fashion.

Node.js will reproduce the last 15 years of ruby mistakes

Bold claim. One that I was excited to see ample justification for. What are the last 15 years of Ruby mistakes? How can we avoid making those same mistakes? How did ruby fix them? What can we learn from ruby’s experience?

Brian’s main justification of this point was that the cluster module is a repeat of Phusion Passenger.

It’s not entirely clear whether he was referring to TJ’s “cluster” package, or the built-in “cluster” module in v0.6, or the much improved “cluster” module in node v0.7. None of them are a particularly close reproduction of passenger, though I suppose that TJ’s is probably the closest. But it’s not as if any of these are an Apache plugin, or tied to a specific web framework. The analogy is a stretch, to say the least.

What’s even less clear is how Phusion Passenger, a program written in 2009, and widely recognized as the most effective and popular way to deploy the most effective and popular Ruby application framework, can possibly represent a “mistake”, let alone 15 years of mistakes.

I can only conclude that I do not understand what he was trying to say, because otherwise, I cannot make sense of it. Maybe there was something else called “passenger” in the Ruby world once upon a time, which did kernel load balancing, IPC, and not much else. Perhaps this “passenger” was deemed a mistake, and is now gone.

Show me a ruby mistake we’re reproducing, and I’ll make sure it gets fixed.

process concurrency is doomed

Brian declared that process concurrency will never be able to scale adequately. I cannot accept this without data. Instead of data, he gave some hand-wavey assertions about garbage collection getting out of control.

I’ll assume that he was running low on time, and perhaps had to cut out the histograms and demonstrations. So let’s leave that question open.

Of course, spawning one process per request, or one process per IO or timer, is completely unreasonable. That’s why CGI was not adequate, and no one runs a real web platform on bash. However, running a single process per core, and having multiple HTTP servers share the open socket, and then using an event loop for IO and timers, is remarkably effective.

If there is doom on the horizon, don’t hint about it. Show me the evidence.

symmetric errors, chain of evidence

Finally! Something that genuinely sucks a lot in Node.js!

Domains is on the list for v0.8. I don’t know that the first pass at domains will be the final and ultimate fix for this, but it’ll be something to iterate on. This will make errors much more symmetric, and provide a much clearer chain of evidence.

The chain of evidence is also much improved by having post-mortem debugging of production systems. If you haven’t yet, check out Bryan Cantrill’s talk And It All Went Horribly Wrong.

So, this is important to me, it’s important to Node.js users, it’s important to the various companies paying the salaries of the node core team. It’s reasonable to trust it’s high priority.

Problems I Wish He’d Mentioned

If you want to know what’s wrong with node, ask a node-core developer.

These are some of the things that are really problematic, some of them in a pretty deep way. Of course, these are the things I know about, so they’re things that we’re working on for future releases of Node.js. If there are things that you think belong on this list, let us know.

Some of them are very hard, and will require iteration. Some of them are pretty straightforward, and will be addressed soon. Some of them are not super difficult, but just lower priority, and won’t get addressed for a while.

  1. Debugging. See above. It’s our top issue, and it’s coming along. It was mentioned, but less strongly than I would have liked to see.
  2. It is very hard to get visibility into which pending actions are keeping the event loop running. This is a source of subtle errors. The libuv refcount behavior is in the process of being cleaned up, and v0.8 will include a mechanism for seeing exactly which handles are in an open state.
  3. The Stream API, which is by all accounts the core use-case for node, is remarkably inconsistent and hard to extend. It’s a lot nicer in v0.4 and v0.6 than it was in v0.2 and before, but it’s got a ways to go yet. We paper over a lot of the inconsistency in the Stream.pipe() method, but it’s a source of many subtle issues. Node v0.9 will focus primarily on this issue.
  4. It’s hard to find good node modules. Reducing the barrier to entry for node package creation has made it a lot easier for more people to take advantage of. However, when you solve a problem, you open the door for new ones that were hiding behind it. We’ve grown faster than most platforms, which means that we’re encountering the discovery issues sooner. This will be addressed in the coming months with a new npmjs.org website.
  5. The Node.js project lacks a consistent and visible continuous integration system, so performance degradations can occur silently.
  6. Using binary modules is overly difficult, and requires a build toolchain on the install target. Work is underway to improve this, but it’s tricky.

And of course, just the fact that Node.js is very young, and has some bugs. It’ll no doubt get more, as we continue to work on it, but the goal is to change things in ways that make these bugs easier to spot and easier to fix.

It is not sufficient for Node.js to be better than any other platform. It must be so good that it permanently raises expectations in this space. It must continue to impress and delight users. There is so much work yet to do, it’s mind boggling.

Laurels are not very comfortable to rest on.

And finally…

If you’ve gotten this far, congratulations. I try to keep blog posts much shorter than this, but there was a lot to respond to here, and touched on several issues that are very close to my interests.

Brian’s talk was fascinating. Even the mistakes were interesting. When the video comes out, I highly recommend watching it. Go get Thinking, Fast and Slow. I haven’t read it, but I’m familiar with some of Kahneman’s other work in behavioral economics, and I suspect it’s every bit as fantastic as Brian claims.

I got a chance to talk to Brian after the talk, and so hopefully a lot of these points (at least, the specific technical details of what sucks in Node.js and what we’re doing about it) doesn’t come as a surprise to him. He seems like a genuinely nice and thoughtful person.

For a talk that started off with a request that we use science and make conflict useful, I was disappointed that his criticisms of Node were lacking in scientific rigor or constructive calls to action. But as I mentioned earlier, it’s hard to fit it all into a 30 minute talk. I hope this conversation will not end here.

We’re all humans. We all think wrong. We all love, and worry, and try to do interesting things. We all fail, and feel shitty, and try to explain why we’re not a bad person for it. We all succeed, and want to brag about it. We tell ourselves stories, and sometimes those stories are true, but usually they’re only just true enough to get by.

Try to remember this, the next time you’re telling someone how awesome (or over-hyped) Node.js is, or any other thing you have feelings about, and they suggest that you may be a brainwashed idiot. They are just trying to protect something, as are you, and in the course of this protection, our instincts will cause us to fail.

Show that monkey brain who’s boss! Slow down, breathe, and try to remember that this person who disagrees with you is not some big dumb idiot, but most likely a very thoughtful person who’s trying to do their best to create truth and beauty in the world. If you don’t learn what motivates them, how can you ever join forces?

Many thanks to Mikeal Rogers, Bryan Cantrill, Marco Rogers for reading early drafts of this post and providing feedback.