Next Phase Montage

npmjs

tl;dr -- Good news!

  • npm, Inc., is being purchased by GitHub.
  • The public registry remains public, free, and as available as ever.
  • npm as you know it continues, and in fact, there is good reason to believe that it'll only get better.
  • I'm still going to be working on npm (but with the luxury of more resources and less stress).
  • I'm really excited about the stuff we're going to do.

It is customary for a founder, in the closing chapter of their startup, to ruminate in a blog post about their incredible journey[1]. When we founded npm, Inc., I knew that I was signing up for a post like this someday. I'm extremely thankful that this is the one I get to write[2].

history and origin

Back in 2009, after too long without a vacation, I quit my job, in hopes of discovering what happens when I untether my creativity.

What happened is I wrote a package manager.

I created npm with the goal of sharing modules in a tiny group of nerdy weirdos who'd decided to write web servers in JavaScript. From that niche beginning, npm grew, slowly but surely, the background project I'd greedily steal away time for.

At the end of 2013, npm hit a rough spot and we had to make a decision. This was the first big bookmark in the npm story, when the project could no longer survive on donations. We made some slide decks and raised an amount of money that seemed enormous at the time, but inevitably got spent really fast.

the part where i talk about startups

Running a company is hard. It is grueling, unglamorous work.

I don't want to make it sound completely terrible. There are definitely a few awesome parts. But once is enough for me.

Throughout this journey, our north star has been the mission that we founded this company with: reduce friction in JavaScript software development.

I have a set of goals that I wrote down back then, and have shared openly with the team. They haven't changed much.

  1. Keep the npm registry running forever (not only for the life of the company).
  2. Be a company that we can all enjoy working at, and do the best work of our careers up until now.
  3. Get a big enough exit that I can quit my job and see what comes out of me a second time.
  4. Share the rewards equitably with the people who got npm to where it is.

There are few unmitigated successes or failures in the real world. But this is a win, and a good one, for me and the team and the entire JavaScript community.

We have made tremendous progress on (1), and that's the thing I'm happiest about in this. As far as (2), there's been ups and downs, to say the least, but the bright spots outshine the dark. I've lost some valued friendships in the process, but made a few as well. On (3), well, I'm still working a jobby job, but I always knew that was a long shot, and "make npm a better package manager" is a job I enjoy. And as for (4), I'm proud of the deals that we've been able to negotiate for the team.

It's not a kajillion billion dollar 10x startup cinderella story, and we've taken our hits, but in the end we've done right by our community, team, and careers, and I'm extremely proud of what we've achieved.

the part where i talk about the company buying us

One of the questions founders get asked a lot is "what might your exit look like?" I always mentioned the big tech companies as possibilities, and GitHub as a sort of "wishful thinking" option.

I've been following GitHub's trajectory closely since they came to Yahoo! to give a talk about git and social coding way back in 2009[3]. It's been a huge part of my life ever since I dove head-first into open source as a lifestyle choice.

When I saw the GitHub Packages beta announcement and demo at GitHub HQ in San Francisco, I remember turning to Shanku Niyogi and clumsily blurting out, "Why aren't you trying to buy us?"

It seemed so maddeningly... obvious. Forget about whose logo is on which webpage, just... if you're going to do this thing, do it right, ffs. This clearly needs to be integrated with the actual registry in a very deep way. "I mean... You see that, right?" (I think he probably did. And if he didn't, then props to him for taking my reaction as flattery or a good idea, rather than condescension.)

What I didn't really expect at the beginning of the acquisition process was how much I'd genuinely like everyone I met at GitHub, starting with my initial conversations with Nat, as well as all the people on the team he's built. As we dug into the technical and strategic plans for how npm would fit into the vision of GitHub moving forward, it became clear that this isn't just a good option for the JavaScript community -- it's significantly better than what npm, Inc., can provide on its own.

There are not many companies that can claim to have the kind of fanatical commitment to open source that GitHub does. In the track record of Nat and the team he's assembled, there's really something special here that I'm thrilled to be a part of.

I've said countless times before that I wouldn't let the registry go someplace that won't take care of it. (See goal (1) above. I've sacrificed years of my life and put a strain on many personal and professional relationships in pursuit of that goal.)

As GitHub has branched out into other aspects of the end-to-end developer community experience, it's natural to see how the JavaScript package management process fits into that story. It's not a loss leader or an experimental add-on or a way to quickly hire a team. Rather, the npm registry is a significant and concrete strategic asset serving GitHub's mission of eliminating transaction costs in software development.

That's important.

the part about what comes next

Today, npm serves over 1.3 million packages to roughly 12 million developers, who download these things 75 billion times a month, and all of this is growing at a rate that ensures these massive numbers will seem small in a few years.

Our commitment to that community is to keep the npm registry free for open source development for the foreseeable future, and continue to improve the npm CLI. At GitHub, npm will have the added support and backing of one of the world's largest companies, behind the world's largest community of developers.

There are some awesome opportunities for improvement in the npm experience, to meaningfully improve life for JS devs in countless large and small ways. We'll be making things more reliable, convenient, and connected for everyone across our vast interdependent JavaScript ecosystem.

For six years, in the grind of a startup, we've had dreams too big to dare hope for. This next chapter is a chance to realize those dreams.

This is the end of "npm, Inc.", the Delaware C Corp. But it's an exciting upgrade for npm.


[1]: Reference joke Our Incredible Journey

[2]: It's also worth noting that GitHub allowed such a long, nuanced, and candid announcement as this one, and didn't push for a watered down corporate version. Cultural alignment is a good sign.

[3]: Check out the date on the post for that talk. Then note the date on this commit. It's like some kind of cryptic message from the past, and it's weirding me out, tbh.

Easy Automatic npm Publishes

One common question from people using npm to publish, especially on CI systems, is how best to automate the process, especially when dealing with multiple branches.

For a while now, I've been using a pattern that takes almost all of the human interaction out of it, since I tend to mess stuff up when I type it with my fingers. This works well for automatic publishes from CI or when publishing manually from a terminal.

I haven't manually typed npm publish in a while, which is a good thing.

First things first, have good tests

I am a huge fan of running tests with 100% test coverage. It isn't a perfect guard against every problem out there, but it does keep me from doing stupid things, like assuming that I know what my program does.

My go-to test library is tap, but you can do this with any testing library that supports code coverage. If it doesn't support code coverage out of the box, you can use nyc to run any Node.js process with coverage tracking.

To use it, run npm i tap -D, and then add this to your scripts section in package.json:

{
  "scripts": {
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

The npm version Command

The npm version command will figure out what the next version should be, edit your package.json file, and even check it into git with a signed tag. The beauty of this is that it also prevents you from bumping the version when you have untracked changes in your git working directory, and provides some hook scripts that can do other things before or after the version bump.

In the scripts section of my package.json file, I add a preversion script that runs my tests:

{
  "scripts": {
    "preversion": "npm test",
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

Now, before I can bump the version, npm will make sure that my tests pass. If the test fails (or if coverage isn't 100%), then it'll fail and the version command fails.

Publishing on Version Change

Bumping the version is fine, but then it's time to share it. Along with preversion, the postversion command does actions after the version is bumped. So, let's hook onto that to publish the package.

{
  "scripts": {
    "postversion": "npm publish",
    "preversion": "npm test",
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

Keeping Git and npm in Sync

That's fine for pushing to npm, but then I have to remember to push the changes to git. (I have many times forgotten to do this, and gotten issues because the code on npm is not in GitHub, which is generally a bad sign.)

Thankfully, npm also gives us a way to hook a script around the publish event, so let's use that:

{
  "scripts": {
    "postpublish": "git push origin --all; git push origin --tags",
    "postversion": "npm publish",
    "preversion": "npm test",
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

This runs two commands. The first pushes all branches, and the second pushes all the tags (including my newly published version tag).

Branches and Dist-Tags

Occasionally, I'll find myself working on some big feature for a new release that is not yet ready for prime time.

In the feature branch, I'll modify the scripts by adding a --tag argument to the npm publish command to put it on a dist-tag other than latest.

{
  "scripts": {
    "postversion": "npm publish --tag=next",
    "postpublish": "git push origin --all; git push origin --tags",
    "preversion": "npm test",
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

Now, I can tell people to run npm install my-module@next to try out the new prerelease version.

On the other side, I might want to land a bugfix or backport a feature for a legacy version. To do that, I create a git branch with the old version, and update package.json to add a legacy tag instead.

{
  "scripts": {
    "postversion": "npm publish --tag=legacy",
    "postpublish": "git push origin --all; git push origin --tags",
    "preversion": "npm test",
    "test": "tap"
  },
  "tap": {
    "check-coverage": true
  }
}

Bonus Points: Sign Your Versions

Git has support for PGP signing tagged commits. To tell npm to take advantage of this, set these two config values:

npm config set sign-git-commit true
npm config set sign-git-tag true

If setting up PGP and hooking it up with Git is too much of a pain, you're not alone. I'm a nerd who's been computering for a very long time, and I can't handle it. Also, I'm always worried about my keys just sitting on my machine in text files, even if they are encrypted with a passphrase. And if they are encrypted with a passphrase, then I have to type it in all the time, and that's just too much work.

I'm a huge fan of Krypton. It stores your PGP and SSH private keys in your mobile device's secure storage vault, and then sends a push notification to allow it to do things with those keys. It's dead-simple to set up, and extremely easy to use, and gives you a hardware second factor for everything that matters.

Of course, and I don't exactly know if this is a bug or a feature, it does mean that whenever I run npm version, between the commit, the signature, the tag, and the two SSH connections, my phone does a lot of buzzing.

Running npm version to Test and Publish

From there, I use the npm version command to do all my publishing. For bug fixes, I run npm version patch. For new features, I run npm version minor. For breaking changes, I run npm version major.

If you use Conventional Commits or similar tools, you could even automate the detection of what kind of version bump it should be, though that's left as an exercise for the reader.

This approach of using npm scripts to automate the process works well with any system where you'll be publishing and committing. Set it up in your next project, and trust your human fingers a little bit less :)

PS: npm Configuration is Very Flexible

You'll note that I did --tag=<whatever> in the publish commands above. You can also configure npm in many other ways. Any configuration value (including tag in the case of npm publish) can be set:

  • explicitly on the command line, like --tag=whatever
  • in the environment, like NPM_CONFIG_TAG=whatever
  • in a .npmrc file in the root of your project, like tag = whatever
  • in a .npmrc file in your home directory
  • in /usr/local/etc/npmrc (or /usr/etc/npmrc on some systems).

This works inheritance-style, so the higher up on that list a setting is, the higher the priority.

For CI/CD systems, this means that you can sometimes set environment variables to control the behavior of npm commands, without having to change the code or inject files into places. If it's easier to control it with a file (for example, checking a .npmrc file into git), then that's also fine.

On Building npm and Hiring a CEO - Founders Talk

I had the opportunity to chat with Adam Stacoviak recently about the journey of creating npm and turning that into npm, Inc., 4 and a half years as CEO, and the transition to my new role as Chief Product Officer. Along the way, we touched on some of the long dark teatimes of the soul one goes through when contemplating handing your company over to a new leader, after pouring your heart into it for so long.

I hope you enjoy it.

Imgur - One hell of a rollercoaster ride

wow.

OSS, Risk, and Compliance

npmjs

I'm going to tell you a story.

There are no villains in this story. Just smart people doing their best, and unfortunately working at cross-purposes through no fault of their own.

The names and places have been changed, but it is a true story. I've heard this story a lot over the years in my role at npm.

Once Upon A Time...

Way back in the late 1900s, the once-successful ACME Corporation was falling behind. Their development of proprietary widgets (on a proprietary software stack) was unable to keep up with the competition, who were leveraging Open Source platforms at a much lower cost.

Many within ACME Corp wanted to adopt the OSS approach, but they were bound by a multitude of contracts and agreements with customers and the regulatory rules of the various countries in which ACME Corp operated.

ACME Corp was in a pickle. Over a barrel. Pickled in a barrel of mixed metaphors, one could say.

Accepting Open Source Software

Luckily, ACME Corp hit on a solution. They joined some of the foundations springing up to provide governance structures for popular OSS projects, and instituted a policy where any employee could use any Open Source code that they liked, provided it was submitted for review by their compliance team.

This allowed them to avoid projects that were abandoned, insecure, or published with an incompatible license. Using a simple form was all it took, their developers could deliver value using the most up to date methods and tools.

Life was good.

Then Life Changed

Shortly after the turn of the 21st century, a series of well-intended solutions to valid problems ended up causing new problems for ACME Corp. All solutions, in solving a problem, reveal new ones.

First, GitHub made it far easier for developers of Open Source to collaborate with one another. This allowed projects to become quite popular without any corporate or nonprofit backing.

Next, Node.js brought JavaScript out of the web browser. Prior to Node, plenty of Server-Side JS platforms had been hacked up as side projects, or funded projects of promising companies. But Node was the first to significantly benefit from GitHub's network effects.

The last piece of this puzzle was an early Node.js contributor, who'd been working in the SSJS space for a while, and decided to write a package manager. He'd seen the importance of package management as a development tool before, and had spent quite a bit of time thinking about how reducing friction makes great things happen.

Impacts

A simple module system and package methodology became ubiquitous. Suddenly JavaScript was easy to share and compose. Instead of JavaScript platforms having to include the kitchen sink, they could be lightweight toolkits with loose coupling between parts.

This reduction in friction enabled what came to be known as the "small modules" movement. A number of prolific Open Source enthusiasts began to conceive of a single file as the default unit of code sharing, instead of a branded platform backed by a foundation.

Meanwhile, back at ACME Corp...

With all this distributed sharing, instead of relying on 2 or 3 well-known OSS platforms with clear governance, web applications came to rely on an interconnected mesh of thousands of tiny modules, created by an unaffiliated horde of hundreds individual contributors.

At ACME Corp, the process has started to creak. Well, not "creak", exactly. More like "break". It's broken.

The compliance team insists on only using modules that pass review. Developers who do write hand-rolled scripts to catalog all of their dependencies for the requisition forms are laughed at.

"2305 modules? You've gotta be kidding me. Use less, or come back next year."

The best devs have moved on to companies with less stringent rules. New developers coming out of school don't even know how to create websites without npm and React and Babel and a zillion of these things.

Today, the battle lines are drawn within ACME Corp, forcing developers to rely on subterfuge. The cost of a security vulnerability or getting sued for violating a license can be in the millions. But failing to ship a website is an existential threat.

When compliance complains that the new continuous delivery system is circumventing their OSS rules, the CTO says "I know, I'm on it", and then quietly ignores it.

And they all lived happily ever after...?

I wish that this was pure fiction.

The approach to compliance in almost every industry has not kept up with the advances in Open Source Software practices. This is a pressing crisis facing some of the biggest software development teams in the world right now.

I believe this problem is solvable, but it is not adequately solved yet.

Most solutions ask an organization to choose between safety and efficiency; but inefficiency is never safe. The only valid approach is to reduce friction for development teams, while also helping compliance teams to do their job. This is the the only way to bring peace to the enterprise.