Why No directories.lib in Node (the less snarky, too-long-for-twitter version)

This question comes up in Node occasionally:

Node peoples… is there any way to specify a root directory for my npm package so that I don’t need require(“mypackage/lib/foo”)?

— wycats

In other words, you have a package directory structure like this:

./package.json
./README.md
./test/some-tests.js
./lib/main.js  <-- this is the package.json "main"
./lib/some-other-module.js
./lib/util/a-utility-module.js
./lib/etc....

Now, let’s say that this package’s name is foo. You want to be able to use foo’s some-other-module.js file, outside of foo.

The way to do this in node is do write:

var blah = require('foo/lib/some-other-module.js');

What Yehuda would like to see is:

var blah = require('foo/some-other-module.js');

The “how” here is simple: If you want a module in the root of your package namespace, put the file in the root of your package folder. It’s a 1:1 mapping, so it’s not hard to figure out how to get that result.

But I don’t want to have so many files in the root of my package folder

Ok. So don’t have so many files in the root of your package folder.

If you must export multiple modules from a single package, then you have a choice:

  1. Use an extra 4 characters in your require() statements, or
  2. Just don’t have so many exports.

Many of us would suggest that in general you may want to consider if those things actually are a part of the “foo” package, or if they belong in a dependency, with their own tests, name, version, etc.

I am not personally so die-hard about “one package = one file”. Occasionally there are internal things, and it’s not so bad. But, in general, I tend to be of the opinion that if a module requires its own individual unit tests, it probably should be its own package, if only so that I can think of it as a separate thing and give it the attention it clearly deserves. If I’m intending it to be used outside of the main package, then it most likely needs its own tests, and I’ll package it up and make it a dep.

Note all the weasel words in the above paragraph! There are lots of exceptions to this rule. But they don’t fit on twitter, and in fact, even the weasel words put my response way over 140 characters, and so with the qualifications stripped out, it looks like a capricious religion, unbound to factual realities of the imperfect world in which real software development happens.

The world being imperfect as it is, you may well find yourself in a state where you have multiple exports, and you don’t have the time or inclination to split them up into separate packages. Sometimes the Inappropriate Intimacy smell is less bad than the alternatives, it’s true. Sometimes it’s just a simple business decision; sometimes it’s less important to make it work right, than to make it work right now.

In that case, put "lib/" in your require() statement. It’s not that hard, really.

But WHY, cats?

The argument to require() maps directly to a file path. If it doesn’t start with ./ or ../ and isn’t absolute (ie, starts with a / on unix, or any of a half dozen weird things on windows), then node finds the closest node_modules folder with a thing by that name, and that’s the prefix. But everything after the prefix is just a direct filename.

(Aside: For historical reasons we do make the filename extension optional, which is a deeply regrettable accident of history that I would not repeat, given the chance.)

Let’s look at the benefits, and the costs.

Aesthetics

The first benefit is removing 4 extra chars in some of the lines at the top of some files. This is rather small. Even if you have 100 requires at the top of a module (which is very very many), and every single one of them is diving into some other package’s guts, you’re still talking about pushing the average line length out by 4 chars. It’s not adding extra lines; it’s not pushing the line length out that much.

If you cannot accept that, then put the files in the root of your package. If that is too ugly, well, then weigh the ugliness of file clutter against the ugliness of lines that are 4 whole characters longer.

The fact that this seems ugly is actually great, in my opinion. You usually shouldn’t be writing programs like this. Packages should be exporting an interface that is designed for a purpose, and you should be using them for that purpose. This is basic software design.

Doing an ugly thing should look ugly. There may be reasons to do the ugly thing, of course, and no one is saying that you’re a bad person for doing it, but the ugliness of the code should be a thing that you notice, so that you can avoid ignoring the ugliness of the design. There is a reason we evolved to find plump brightly colored fruit more appealing than wrinkled moldy fruit. There is a reason we evolved to be more able to ignore pleasure than pain.

In other words, your aesthetics are trying to help you. Saying “it’s ugly” is an argument against adding this feature, not for it.

(Alleged) Portability

Let’s say that you want to have something like this:

./lib/node/module.js
./lib/browser/module.js
./lib/ringo/module.js
./lib/narwhal/module.js

Then, I want to say “Node should use lib/node, Ringo should use lib/ringo, and the browser should use lib/browser, so when I do require('foo/module.js'), I get the right one for my environment.

First of all, that’s very trivial to do without making the mapping of require arguments to filenames any more complicated. Create a file like this:

// module.js
switch(getPlatform()) {
  case 'node':
    module.exports = require('./node/module.js');
    break;
  case 'ringo':
    module.exports = require('./ringo/module.js');
    break;
  case 'narwhal':
    module.exports = require('./narwhal/module.js');
    break;
  default:
    module.exports = require('./browser/module.js');
    break;
}

// This can of course be simplified.
// Usually something like this is sufficient:
// if (typeof window === 'object')
//   return module.exports = require('./browser.js');

If you find yourself doing this a lot, maybe ask yourself if you’re using the best abstractions, or maybe trying to make something portable that is fundamentally platform-specific. Maybe try supporting fewer platforms.

Or, maybe, just do the ugly thing, and let it be ugly, and accept that it’s a net benefit in your case. But if your request is "Make platform-specific module behavior forking less ugly”, then we’re back to aesthetics, and my answer is, “Ew. No. That’s gross, and it should look gross.”

Priorities

The cost of adding this feature is that the mapping from your require() statements to the actual file with the code in it, is more complicated.

Yes, it’s “only one extra file to look in”. But “looking in files” isn’t the hard part. The hard part is remembering which packages mapped in which ways, and thus knowing where to follow the train of logic, when you’re juggling a bunch of other peoples’ code in your head for the first time, while trying to debug a problem as fast as possible because it’s preventing actual human beings from doing what they are trying to do, and they give exactly zero fucks about the ugliness of your require() statements.

Adding “one extra file to look in” doubles the number of files to look in, and much more than doubles the cognitive overhead of debugging problems in code you didn’t write.

Debugging problems in code you did write is pretty easy. That’s not what I’m talking about. I’m talking about the case where I’m debugging your code, because I’m using your module in my website, and my website is misbehaving.

If this seems like a trivial complaint, then you have either never been in this situation, or you are smarter than I am. Either way, consider yourself blessed, because it’s a shitty situation, and I’ve been in it, and I’m not smart enough for it to be trivially solved, and I wrote most of the code that did all this shit.

Few remember, because the node community was about 1/100th the size at the time, but npm used to support this feature (and also the “modules” hash, which allowed arbitrary mapping of module names to files, and was significantly worse.) Even fewer people were actually running production sites at the time in Node, and of those that were, most just wrote everything from scratch, because there was much less of an ecosystem to draw on.

Ryan and I were among the people building things in Node at Joyent, and trying to actually Do The Right Thing, and use npm and the community module ecosystem in the way it was intended. The first time we had to debug something like this, where a few modules actually used directories.lib, we both were convinced that it was a terrible idea, and had to go.

Around that time, we started conceiving the module system for Node v0.4, which is what we have now. We moved the node_modules lookup stuff into node core, and stripped out the module hash and directories.lib features from npm. Moving node_modules lookup into core meant that npm could stop using symlinks and shims all over the place to implement isolated deps.

Again, if you think that this cost is not relevant, then I have nothing but respect and blessings for you. Either you’re much better at this stuff than I am, or you’ve suffered less of the hell of other peoples’ bugs. Either way, mad props.

Making the node module system any more complicated than it is will make my life noticeably worse, and I don’t think I am alone in this. As I am in a position to prevent that reduction in Node quality of life, and as that is my job, I will prevent it.

The Node module system is frozen for a reason. It will not change to add new features, ever. It’s done, and not open for discussion.

Addendum:

@izs Would like to hear more about why omitting the filename extension in require statements was a mistake.

— jimmycuadra

Answer:

It quadruples the number of stat(2) calls, makes the 1:1 mapping more vague, and complicates implementation. Slower, greater complexity, etc.

However, the cost of removing this feature is nowhere near the benefit of not having it, so it stays for backwards compatibility. C'est la vie.