Going Fast, Frankenstein, and Refactoring

Very soon now, the project that has consumed my days for the last 6 months will go into public beta. It’s too early to pop the champagne, and that’s not what this post is about. However, the newness of this project is worth pointing out for the following reason: We’re already planning a fairly major refactoring effort to take care of a lot of code rot that has crept into the designs.

That’s right. We haven’t released anything, and there’s already enough bloat to justify spending at least 2 weeks cleaning it up.

Of course, due to the pressures of market, and the morale effects of delaying the release, we’re buttoning up what we have now as much as possible, and planning to work on the refactor once the beta is out there. It may actually help a lot, since the gigantic firehose of traffic that Yahoo! points at sites has a way of helping even the most egotistical of developers realize that their clever baby needs some serious surgery.

This post is going somewhere, I swear, but I can already tell it’s going to take me a while to get there. Go get some coffee and a snack and come back. I’ll wait.

Build Fast

In my opinion, the core of good software development methodology comes down to two basic questions that should be asked daily:

  1. What portion of your time is spent focused on the task of creating software? (That includes planning, discussion, and thinking as well as coding, provided that these non-coding activities are focused and aimed at creating software.)
  2. Does it work today? (If you don’t know, then you are full of Fail. Get on that right away.)

I call this the Go Fast method. The goal is to efficiently create software with minimum long-term cost and maximum long-term benefit, without driving away your best talent. (Have I got the attention of the business and product people in the audience? Good. Please stay for the rest of the show.)

I know that someone out there is bound to point out the obvious parallels between my “go fast” methodology and Agile, and maybe they’re right. To the extent that Agile accepts these premises as important, it is correct. But I’m claiming that everything else stands or falls based on how it affects these questions. Stand-ups, planning meetings, org charts, IDEs, frameworks; they’re all only as good as their effect on the principles of speed and stability in development.

If you’re developing software properly, you should be creating software that does something as soon as possible, and keep it up as much of the time as you can. Until there’s something to look at, you’re just spinning your wheels. Nowhere is this more true than on the web. Photoshop comps don’t break in different browsers. Product presentation decks don’t “feel klunky” when pieces load out of order. Grand visions promise to solve all your problems and make you biscuits for breakfast, but they don’t deliver.

You make software for a living? So make software, already. You wanna build a website? What’s stopping you? You don’t have designs yet? Build the parts that you do have. Always be releasing, even if your actual release date is months away.

Build the core, and grow the other pieces up around it. Every minute spent waiting, instead of actually building something, is worse than wasted. When you live in the code day by day, it becomes a part of you. You learn how the pieces work, and the knowledge is automatic. When you get too far from it, it becomes foreign, like driving a car for the first time.

The agilistas call this “shippable code.” In my opinion, that’s a horrible term. The vast majority of code that I’ve seen released into the wild was not even close to what I’d call “shippable,” but it got shipped just the same. Most of the time, the prototypes leading up to the release were significantly worse. Some agilistas are quick to point out that “shippable” doesn’t always really mean “that you would actually ship it.” Sorry, but drivable means that you can drive it, and drinkable means that you can drink it, and flyable means that you can fly it, so if “shippable” doesn’t mean “you can ship it,” then they picked a bad term.

I suspect the ambiguity is intentional on some level. I think the claim that “Agile makes software faster” is a myth. Agile methodologies, used in moderation by a well-balanced team of dedicated and talented individuals, do tend to result in higher quality software, greater management visibility, and a happier bunch of people than any alternative, but that means that the overall development time can be significantly longer. And if you’re not on a well-balanced team of dedicated and talented individuals, then no methodology is going to save that sinking ship, so dust off the CV and bail out ASAP. But I digress, and besides, you already know that I think Agile sucks.

I prefer to call it a running prototype, or an integration environment, or anything else that’s more, well, accurate. “Shippable code” tells overeager managers that they can just press “stop” at any point and ship what they’ve got. Nice dream land, hot shot, but it doesn’t actually work quite like that. Nevertheless, a good integration development environment, where everyone’s code gets placed and which is subjected to regular updates and testing—that’s important, almost as important as CVS and a decent build script, and for a lot of the same reasons. If the team’s code isn’t conversing regularly, then the team may as well not be, either. The best way to prevent blockages is to prevent assumptions; the best way to prevent assumptions is to check them every single day. So: Does it work today?

Completion feels good. Getting something that works feels good. That’s why we got into this business. Because, despite all the pain and frustration and work that goes into developing software, The High here is about as good as it gets.

You don’t get that waiting. You don’t get that talking. You get that when you are immersed in building something.

From a team dynamic perspective, I can’t even find words to express how it makes people gel when we each do our little pieces, and plug them in, and see a site come up with data and styles and behavior and images and everything. It’s a pretty marvelous event that makes everyone even more eager to get to work. If you’re a web developer, don’t wait for your designer or back-end engineer to give you what you need. Make an unstyled page with dummy data, if you have to, but make something on day 1 if possible. Then make a list of what you need. When everyone can see their effect on the product, it’s amazing how fast they tend to deliver.

Going fast and pushing to milestones also forces everyone on the team to prioritize. There may be plenty of features that might be nice to implement, but if speed is a priority, then it forces you all to work on the things that really need to be implemented. By raising red flags whenever something doesn’t work, the whole project stays in sync most of the time, and assumptions never get too far out of whack. But, of course, that’s not the whole story.

Gene Wilder as Victor FrankensteinSoftware development is a lot like building Frankenstien’s monster. You start out with a pile of useless ugly pieces, and try to turn it into something beautiful. Along the way, it’s a monstrosity, and tends to get uglier as you tack new bits onto it. Then it kills people.

In Mary Shelly’s classic, everyone Victor loves is killed by the eloquent and sensitive creature, who was turned evil by his creator’s neglect and hatred. If software could walk and talk, how many developers would still be alive today?

Slow Down

Don’t build Frankenstein’s monster, or at least, don’t figure you’re “mostly done” once it walks and talks, as the strict waterfall process proposes. Going fast is important, and necessary. You can’t fix pure ideas very easily, because it’s hard to see what’s wrong with them. Write the code. But know that it’s going to have to be rewritten, possibly several times. Plan for it, and keep it in the back of your mind.

The hazard of moving quickly is that we tend to go with the easy choice, rather than the good choice. Adding one more method onto the class I’ve already got is easier than creating a whole new class; nevermind that it doesn’t really “fit” with what this type of object is supposed to be doing. Do that once, and it’s probably a lot simpler and clearer than any alternative. Do that 50 times, and you end up with a theological problem, or something else equally pungent.

This will happen. Every time. If you don’t have some downright embarrassing shit-tastic WTF-worthy code lurking in your project—I mean, the kind of thing where you see a bug report go to your coworker, and you say to him, Oh, I’ll take that one, I know that feature, but really, you’re saying Please don’t look at that code, or at least, if you do, please don’t judge me—then you probably weren’t going fast enough. The remarkable thing isn’t that this team has produced a product that “already” needs a cleanup; the remarkable thing is that we all seem to recognize that we need to clean things up, before it’s blown up in any serious way yet. That’s why I’m thankful to be on the team that I am. I’ve heard it said that the difference between a good programmer and a bad programmer is that a good programmer feels pain when he looks at ugly code he’s written, while a bad programmer thinks everything he’s done is great.

I used to be proud of every bit of code I’d written, like a two-year-old who just learned how to use the potty. Then I grew up a little bit, and realized that my shit stinks, and hoped no one would ever see it. I’ve grown up a little more, and realized that everyone shits, so there’s nothing to be ashamed of. I try to work towards a point where my actually shit doesn’t stink at all, but I’m not all that hopeful about that. The fecality of this line of metaphor, while a bit disgusting, is meant to highlight an important point.

You want to create code that is nice to be around? Do your best to build quality in from the start. The more you learn, the better you’ll be at it, especially if you manage to code yourself into some truly awful maintenance nightmares, where you can’t walk away and can’t blame anyone else. But no matter how good you get, it’s still gonna stink most of the time. Everyone writes bad code. So fix it.

Refactoring

Wikipedia says:

Refactoring neither fixes bugs nor adds new functionality. Rather it improves the understandability of the code or changes its internal structure and design, and removes dead code, to make it easier for human maintenance in the future.

To relate to the twin principles of software development, “Go fast” and “Does it work?”, refactoring is not important early on. Surely, doing things as close as possible to The Right Way is usually a big help. But The Right Way almost always flies in the face of the “Go Fast” maxim.

I’ve been privy to a few refactoring projects, I think I can safely say that almost every single one was an utter disaster. Without naming names, here’s how a few turned out:

  1. Lead engineer spent 3 months internally redesigning a feature with the intent of making it more extensible in the future. Released. Numerous bugs and problems. When I left the company, it was still unstable. FAIL
  2. The product has grown into an amalgam of junk, and no one knows how to maintain all the disparate pieces. Can’t add features easily. Needs to be re-architected. [[numerous planning meetings]] OK, management says we have 2 months, so let’s just add another layer on it, change how it looks, and call it a day. Result was even more disorganized and difficult to extend. FAIL
  3. #2, again, with new people driving it, who insisted that the last ones were all wrong. Same result. DOUBLE FAIL
  4. Repeat, again. TRIPLE SUPER DUPER FAIL
  5. Last, but oh, so certainly not least, there were all the refactorings that didn’t even make it to a real execution. The times when the development team was so fed up that they pushed to change things, and were told repeatedly that they could get to that just as soon as the next project got done—there was always a “next project.” It works now, what’s the matter with it? To the best of my knowledge, of three teams where I saw this pattern, (a) one company no longer exists, (b) another company lost all their best talent, and © in the third case, the team dissolved and everyone moved onto other projects (it was at Yahoo!, so it’s not like they were going to go under.) For failing before they’d even gotten started, these refactoring projects are QUADRUPLE UBER KAMEHAMEHA FAIL!!!

I strongly doubt that any of these cases were rare. In fact, I think they’re the norm.

It’s easy to cite bad management, and in a few of the above cases, management was to blame. They put a high premium on getting new features, without recognizing the hidden costs down the road of dealing with the increasing instability. However, the fact that I’ve come to realize is that most developers don’t really understand maintainability, and thus, tend to completely miss the point of what the purpose of refactoring should be. As a result, their attempts at refactoring end up being worse than failures, because they are interpreted as success.

I did get to see one significant refactoring project that I’d consider a success. It was done in an under-the-radar sort of way, simply because it was the only way to add new features to a product that had grown frightfully unwieldy. Also, I’ve seen and done a lot of small scale refactorings that were executed well and increased the quality of the product as a result. (These, as the Wikipedia article mentions, are usually referred to as simply “cleaning up” the code base.)

Refactoring can only be justified in the “Go Fast” methodology on the condition that it makes a more stable product, thus enabling developers to more efficiently create working software. Elegance is not an end in itself. The end goal is a working product, and code is the enemy. The vast majority of software development is about modifying an existing product. The short-term memory of a human is limited and fleeting. The less information that must be held in mind to understand a given section of code, the more quickly code can be modified, the lower the likelihood of introducing bugs with any given change. So, there is, ultimately, only one purpose that Refactoring should serve:

Be more obvious.

Love Your Monster

Inside cover of the original 1918 Frankenstein by Mary Shelly.Victor was so consumed by his desire to create something new that he ended up rushing the job and not thinking ahead. When his creature opened its eyes, he feared it, and fled, and tried to hide from his sin. Shunned from the world, the creature asked Victor to make him a companion. When his request was denied, he proceeded to destroy everything Victor loved.

The moral of the story: Don’t ignore the ugliness you create. It will find you.

Refactoring should make the code simpler. Complexity is the demon that we fight in writing code. It is the enemy. It opens the doors to bugs; it increases the difficulty in fixing problems and adding features; it raises the learning curve for new developers. Sadly, a lot of refactoring seems to make code more complex rather than less.

In my opinion, there are a few guidelines that work well when changing around existing code. These guidelines apply equally well when creating new code, but since it’s harder to see the whole picture in that scenario, mistakes and shortcuts are more forgivable.

Refactoring isn’t about using the coolest object oriented tricks that you just learned. It’s not about making the code “more abstract.” Abstraction is a necessary evil, in my opinion, not a feature. Like Victor, it’s easy to get wrapped up in testing out new discoveries. Most of us got into this job because we like playing with new puzzles, and that’s a great thing. But one of the most satisfying puzzles is to reduce the complexity of an implementation without reducing the problems that it can solve.

I’m a big fan of “comment-driven development”. That is, write out the pseudo-code in comment form first, stubbing out the functions that need to be implemented. It helps to separate the design and implementation phases, since it can be hard to do both simultaneously. As implementation kinks are getting worked out, the design might change slightly, but it helps to have the comments there as a guide to what I was originally thinking. Once a module moves into a more stable place, the comments are a liability. At best, they’re clutter; at worse, misleading.

I like using those comments to clean up the code as well. The process goes like this:

  1. Read through the code, starting at the entry point, and opening each file as it’s referenced.
  2. If a section of code has a shit-ton of comments, then it needs work. If it’s clean, just make sure it makes sense.
  3. Make sure the comments match what the code is doing.
  4. Remove any incorrect comments (comments that are lying.)
  5. If the code isn’t understandable without the comments, then dig in and clean it up. Rename methods, sort things better, etc.
  6. Remove any and all implementation-related comments. Those are dangerous.
  7. Repeat until the file is almost comment free, and understandable.

I’m not so certain on the common refactoring claim that it should not change any existing functionality. Sometimes, this is of vital importance. However, there are plenty of times where the feature set of a bit of code simply needs to be trimmed or revised. For public-facing APIs and products, either the functionality should not be changed, or interfaces should be publicly deprecated long before they are actually removed.

For internal code, or APIs that are only used by a small number of consumers, my favorite approach is to just change it, and then see what breaks. This should never be done in a way that affects customers, of course, but in a development environment, it’s quite appropriate to just change the back-end, and then update the front-end once it’s fixed. Defensive coding is a good practice, but overly defensive coding, where errors are not surfaced at all, can hide problems that may turn up later in unexpected ways.

Don’t bother refactoring something you just wrote. You’ll only make it worse. You have to wait until you don’t remember how it works before you look at it again, or else you won’t be able to appreciate the difficult bits. If you don’t have time for that, then make someone else do it, and tell them to be as harsh as possible.

Ego

As they say, No matter what the problem is, it’s always a people problem. The first and most important step in any refactoring effort is to detach our egos from our code. This is even more of an issue if something you wrote is being refactored by someone else, or worse, if you are refactoring someone else’s code.

I’ve worked hard to develop a pretty good understanding of software development. I’ve found that the technology side is easy—on the human side, I still have a lot to learn, and probably always will. Like a lot of geeks, I didn’t really get into the whole “social” thing in a serious way until I was almost an adult, and even today, I tend to focus on The Project and keep my distance from the rest of the world. So, I suppose I don’t know the best way to handle this part of the problem.

TBL’s classic recommendation to be loose in what you accept, and strict in what you send seems to apply well to this case. On a healthy team, everyone is committed to a successful product, and leaves their egos at home when they come to work. Code is passed around and everyone seeks as much input as possible. On a dysfunctional team, everyone’s sense of self-worth is tied up in the appraisal of their work by others—which has the twin effect of making them overly sensitive to criticism, and overly critical of their teammates.

It’s natural to put up resistance to the prospect of your code being modified. After all, you wrote it that way for a reason. Sit down with the other developer, and try to hash out the best approach. If arbitrary decisions have to be made, explain that the best approach is to make them consistently. Don’t focus on the problems in their code, but rather on the need for consistency and abstraction throughout the product.

If someone won’t listen to reason, frankly, the only solution I’ve ever found is to either leave or wait for them to. Hopefully they’re a contractor or something, so you can just wait, and do your best to isolate and minimize the damage as much as possible in the meantime. If not, then the issue should be escalated, if only because it’s polite to tell your manager that their ship is in danger of sinking.

Oh, right, that book…

Refactoring, by Martin FowlerI read Refactoring about 5 years ago, but I’ve been meaning to pick it up again, since I’ve gotten a fair amount of real-world development experience since then, and have seen my share of dismal failures. But this isn’t supposed to be a book review, so I won’t review it in any detail. It’s a good read, though, and I highly recommend it, even though it is mostly in Java, and seemed (to me at least) to simply highlight a lot of design problems inherent in the Java language that lead inevitably to bloated over-sized code. But that’s a post for another day.

brokencode beautycode ecosystems