blog at izs.me

Category Archives: the business

∞

OpenSocial, Meetings, and Success

It’s a very interesting time for me at Yahoo! right now. I’d been keeping quiet about what’s going on for two big reasons. First, I’ve been too busy working to write about it. Second, it’s secret.

Well, as of 2008 March 25, it was a secret. This press release broke the news a few days ago, so I doubt I’m giving anything away to say that Yahoo! is busy working on implementing a framework for OpenSocial applications that tie into various sorts of Yahoo! data.

Buzz was a very fun project to work on. After games’ mountain of legacy and the brand universe fiascos, it was really great to be able to have a hand in developing a site from scratch and also have enough space and resources to really create quality. (It has its problems, but they’re being worked out, and overall I’m quite proud of the site we built.)

However, the really interesting parts of Buzz are all behind the scenes. The front end was an exercise in the complicated and delicate art of engineering simplicity and eliminating redundant effort. Even in the face of exacting requirements and changing specifications, we ended up releasing a site that performs well and is relatively easy to work on. On some level, I feel like it was my karma baby for some of the complicated monstrosities that I’ve left in my wake as a developer.

But, with Buzz behind me, creating and maintaining yet another website has gotten a bit old. Nothing wrong with that work, but I’m done with it for now. Time to do something else.

OpenSocial and Mash-ups

Douglas Crockford has called mash-ups One of the most exciting features of WEB 2.x, and I couldn’t agree more. The vision of the web from the get-go was a distributed platform for innovation. As client technology and usage advanced, dynamic features followed naturally. The ability to mix-and-match data and functionality into new applications is a very exciting prospect.

However, like so much potential innovation on the web, this effort is severely hampered by the same system that makes it possible. The browser Dom was designed by committee, and the committee didn’t actually meet for a while, and the members of the committee were trying to kill one another. Throughout the browser wars, IE and Netscape introduced features in a frantic arms race to try to shut one another out of the browser market. Many of those features were not really ready for prime time, and many others probably should have been wiped off the whiteboard and never made it into the code.

On the plus side, browsers and web pages were able to innovate much faster than the w3c’s crawl. Much of the time, the “official” specs only included a feature because a few browsers had already implemented it and it was in widespread use. That speed got us to where we are, but lots of important things were sacrificed or forgotten along the way. Among the missing: a proper permission model. Same-domain restriction works to some degree, but it makes mashups trickier and there are still big holes. In particular:

Scripts run with the same access permission as the page that invokes them, even if they’re hosted elsewhere.
XHR calls can’t get data asynchronously from external domains.
The Dom is one big global that all scripts have access to.
Iframes restrict permission, but they’re a pain to work with.

Sadly, this means that mash-ups are faced with the choice of:

Only accessing public data, lest some untrusted third party get ahold of something it shouldn’t have.
Running the risk that a third party might steal some data.

Essentially, mash-ups are thus restricted from really doing anything interesting. PayPal or Amazon or Yahoo—-that is, sites with money and interesting private data—-could never risk exposing their data to a mash-up. Their visibility, and the value of the private data they hold, makes them prime targets for attack. And, since they have money, and visibility, they’re also prime targets for lawsuits if a user was damaged by a mistake.

OpenSocial represents a concerted effort by some very big players on the web to make severely interesting and useful mash-ups not just a reality, but common and relatively easy to create and host. API layers will sit between the developer and the browser, and between the developer and the container page, allowing them to code as if it’s 2099 and the browser is sane. It’s no small task. When Hao asked me to join the team, and told me what they were doing, my initial reaction was Well, it might end up a total failure. But it’ll be one hell of a fun problem to work on. I’m in.

There were hints and rumors that Yahoo was doing something with OpenSocial a while back. (We were.) But, the main reaction to the press release really should be Well, spluh. Yahoo’s all about social; it may have started with indexing the web into searchable categories, but some of its earliest and most popular applications are mail, instant messaging, and groups. Yahoo’s all about open; we invest countless developer-hours into the open source technology that we use. Yahoo has a lot to gain from a more exciting social internet; when the ocean rises, the bigger fish benefit the most.

Warnings Wrapped in Compliments

For a month or so, I’ve been transitioning off the Buzz team and onto the team that’s doing the OpenSocial container stuff. As people found out that I was joining that team, I heard this line a lot from various people: Oh, thank god. They really need a good front-end.

There’s a very flattering element to that, and I’m not trying to brag. Objectively speaking, I know I’m good at what I do. I love doing it, and I work hard at it, and that’s just what happens when you work hard at what you love. No magic at all.

At the risk of sounding like a falsely modest blow hard, though, on a more moment-to-moment basis in my own world, I focus on many of my own failings that others rarely see, and I am always very impressed when someone solves a problem that I was stuck on. I often feel like I’m still new at Yahoo, and on some very insecure level, I’m waiting for them to find out that I’m just playing at this stuff and tell me I’ve fooled the real adults for long enough, and it’s time to go home.

But I am good at what I do. It’s real. These skills are rare, and they’re worth money. When others acknowledge it, though, I kind of feel embarrassed and get all “aw, shucks, twern’t nuttin’” and find myself unable to make eye contact. (A few times, this reaction has evoked a No, seriously! I mean that! which just adds to the effect.)

So, the flip side of that comment… yes yes, I’m a great webdev, blah blah, but say that first part again? It’s a bit like hearing, Oh, thank god they picked you to wrestle that giant raptor. He’s been killing one guy after the next, so I’m really glad that they finally picked someone who can take him down. Why couldn’t it have been That’s great! There’s a lot of great talent on that team, I’m sure you’ll fit right in!

Well, that’s the cards that life deals you sometimes. Impossible problems are the most fun, anyway.

And I’ve found that there really are a lot of talented developers on this team. Some more have joined it recently. I’m really happy to be able to be working in this group.

Meetings

Somehow, I managed to work myself into a role on this team where most of my time is spent in meetings. Some of them were soul-sapping and boring, as meetings can be sometimes, but lately, they’ve seemed to focus more and more on solving front-end problems at an architectural level. After almost every one, someone whips out a phone to take a picture of the whiteboard, lest our valuable work be lost to the eraser.

Additionally, I’ve been attending the YUI team meetings, as they’re interested in knowing what they can do to aid Yahoo’s big bets. Through that, and by virtue of working on a product that promises to push the limits of what the browser can safely do, I’m finding myself consulted because Microsoft is asking Yahoo what we think of the IE 8 beta, and what we’d like to see in it. I’ve been asked to teach a CSS class for the next round of incoming junior/trainee webdevs (Jukus, as they’re called). I’m meeting with other groups and helping to figure out what OpenSocial means for their product’s context, and how they’ll use what we’re building. Today, we’ve got a meeting at the Googleplex.

It certainly looks like success. And the pay is pretty good. The product is fun and open and creative. It’s one puzzle after another.

But, I’m feeling a bit envious of the other developers I talk to who get more time to write code. From Tuesday to Thursday of last week, iCal was booked solid from 11-5 every day, I ended up in ad hoc discussions about things (or dealing with email) until about 7 or 8 each night.

I have tried hard not to become a manager at Yahoo. Perhaps a management path is in my future somewhere, but right now, it just seems like a job I would not enjoy. Programming is a very social activity if you’re doing it right. With a new team and a new product, there’s plenty of architecture and designs to figure out, and I’ll be coding again soon enough.

I hope.

Hey, one more thing…

We have openings for great engineers and webdevs. If any of this sounds like something you’d be interested in, let me know. I’m 100% serious. Javascript and PHP experts, we need you. Yahoo is, in my opinion, one of the best places for talented engineers to work. They generally treat their employees very well, and this is a product that is just full of fascinating puzzles.

Thu 2008-04-03 12:20:41 | 0 notes

∞

Needed: chpass, finger, and pw for the web

It’s been said that the best startups take a popular Unix command and bring it to the web. But there are a few that are poorly represented. I understand that I may be making a bad career move by discussing this openly on a blog, but quite honestly, my desire as a consumer for a satisfying product is enough to risk—-nay, hope—-that someone else makes a million dollars doing this before I get a chance to.

I’m thinking specifically chpass, finger, and pw.

I know what you’re thinking. There have been a few forays into this arena. MySpace, Facebook, and Plaxo come to mind, not to mention whatever else some MBA has stuck to a “social network” this week. (It’s just like a regular duck, but this one swims around the lake and lets you put all your friends in a list! I’ll be rich! The sad thing isn’t that he thinks it; the sad thing is that he just might be right.)

The social-for-the-sake-of-being-social sites tragically miss the point. I have resisted getting a Facebook account for a few years now, and even deleted my MySpace account. They seemed to require a lot of time and effort doing basically nothing, and didn’t give me what I really want.

Managing data is not (necessarily) enjoyable

I hate managing contact lists. The worst contact list, the one that is the hardest to manage, is the one in my head. Every day I get older, I get worse at remembering phone numbers, and I like to know who’s calling me. I like to see your picture when you call, see your real name when you email. I want my email program or my phone to know who you are when I start to type the three letters of your name that I can remember off-hand, even (especially!) if you’re someone I don’t talk to often.

That’s why I shelled out $40 for Missing Sync so that my phone and computer can share an address book. I have an Applescript program sitting on my back burner that will sync any contacts I add in Adium into this same collection, and even look up their contact details from Yahoo’s corporate intranet (since most of the time, they’re work mates.) Automated replication is still not great, but it eases the pain of managing multiple lists.

Facebook and MySpace are software platforms designed around the premise that managing a contact list is fun. And it can be in that 12-23 age range where we attempt to define ourselves and carve out our place in the world through our social connections. That’s a key demographic for advertisers. Good for you, Facebook. But if I wanted to spend all this time managing my friendships, I’d have more of them in real life. Ooh, burn! i mean… hey, wait a second..

Plaxo is actually a pretty good approximation of what I’d like to see, at least on the “managing contacts” side of things. Granted, I’ve been spamvited to their service by half a dozen people I hardly know, which is a classic example of “let’s be viral” gone horribly horribly wrong. But their product offering is pretty close. You get one contact list online, and it syncs with other areas. It’s unfocused since they’ve added “Pulse” (basically an RSS aggregator for your other web profiles), but still pretty good.

However, even Plaxo misses a key point, and makes several fatal flaws. I’m actually talking about a profile and contact management system that is much grander.

DRY — What Changed

In a relational database or data map, the idea is to keep a piece of data in only one place, and store the relationships between entities rather than making multiple copies. Most contact management systems, from a little black book to the cell phone contact list to Outlook to Plaxo, fail to implement this simple principle. Instead of making each node in the network keep track of all the data about all the other nodes in which it is interested, instead let each node control its own data, and store links to the nodes in which it is interested.

In the old days of land lines, the phone book was enough. If you knew someone’s name and city, you could get their phone number and, perhaps, their street address by performing a simple lookup. Each user had the option to control how much information was shared with the public. Until the autodialer came to telemarketing, the abuse rate was limited by the cost of using the technology.

Today, each person has many more pieces of contact information, and the cost of abuse is virtually zero. There is no way in hell that I’d let anyone publish my actual cell phone number, and once an email address is exposed, it’s basically useless. Spam fighting is an arm’s race, and an unfair one even for Google and Yahoo to fight.

Why we need those commands on the web

Back to my original point: chpass, finger, and pw.

chpass

add or change user database information

In other words, manage my info in one place.

finger

The finger utility displays information about the system users.

In other words, look up the information that other have made available.

pw

create, remove, modify & display system users and groups

In other words, specify who has permission to what.

Many large companies have some sort of online system like this. At Yahoo, it’s the almighty Backyard, the corp website that started as a list of email addresses and grew into a full-scale intranet with contact lookup and LDAP access. (It also features conference room booking and documentation searching and plenty of other handy things. But mostly, it’s still all about the mega employee contact list.)

You manage your own profile, and make sure that your numbers and whatnot are up to date. No one else ever has to worry about how to contact you, because it’s all in one place. However, that only works because access to the backyard system is tightly limited to current employees, so abusing the system would entail serious consequences for the abuser’s reputation (and career).

In other words, we have finger and chpass, but pw is being done manually by the HR department, which limits the possible size of the network considerably.

Abuse Prevention is Extremely Non-Trivial

The easier it is to use a networked contact management system, the easier it is to abuse. The more useful it is for you and your friends and associates, the more useful it is for spammers and scammers. Already, we have to keep our email addresses hidden from strangers. Imagine how much worse it would be if a PPC pusher could just e-finger “isaac.schlueter” and have my home address, email, phone number, instant messenger alias, birthday, and photo. Yes this is exactly the sort of information that I’d like to easily share with everyone else.

Everything that has been done so far in the area of email spam, while impressive and necessary, is fundamentally inadequate. As long as it remains profitable for a spammer to send out 100 billion emails every day, it will happen. Any attempts to prevent or avoid this behavior run counter to the incentives of the market; which is to say that it’s a bit like building a dam of sand and expecting to stop the Mississippi. Won’t happen. A bigger dam will take longer, but eventually, they’ll all crumble.

In order to truly divert human behavior, the incentive must be dealt with at the source. Direct attacks against the offenders (ie, shutting down their accounts) are not effective in the long run (they just get new accounts.) Negative incentives, such as putting spammers in jail, are not going to be effective in the long run, because it doesn’t push the cost of spamming up high enough. John Q. Spammer doesn’t think he’ll be the one to get caught, and he’s probably right.

I don’t claim to have this bit of the system figured out, not by a long shot. But I have a few ideas.

IRL

In real life, we meet a lot of people, and many of them can and do annoy us by contacting us in unwanted, if mostly harmless, ways. The foul smelling man who stops babbling for a second to ask me for a quarter. The smiling woman who shoves a tract at me and tells me I’m going to hell. Sadly, the list goes on and on and on.

However:

It’s easy to size someone up quickly, because:
Annoying people build a reputation for being annoying, because:
They’re real people and you can see who they are.

There are still, of course, the violent offender and the con man. However, in real life, direct attacks incur a high degree of risk, due to the chances of being caught or retaliated against, and so law enforcement has a relatively easy time keeping serious criminals in check. And those looking to do you harm by gaining confidence and taking advantage of it, well, they’ll always be around, but they’re pretty rare and the reputational aspects keep them somewhat in check as well.

So…

Entrance into this global open personnel file would require that an account be tied to a single real person, who doesn’t have any other account in the system. Accounts that are not “backed” by some kind of reliable identity are only given some kind of limited provisional access—-perhaps they can email a user through the system, but they cannot get the user’s “real” email address, and users would be able to deny access altogether to unidentified strangers if they chose.

Identification is itself a non-trivial task requiring a high degree of trust from the web site. Even if you know it’s 100% secure, being asked for your date of birth, SSN or passport, and a major credit card is a tall order. Without biometrics, it must come down to discrete bits of information at some point, which can be (and often are) faked.

A rinky-dink fly-by-night startup can’t hope to achieve this level of trust quickly. And, without getting a critical mass of users, the value proposition to new users is a lot tougher.

The company to build this system would need:

A huge base of existing users and preferably their contact details, too.
A strong reputation for protecting user data.
Impressive engineering resources and domain knowledge in the areas of spam protection and social networking.
A serious commitment to open APIs that help the web as a whole.

If it’s not everywhere…

…then it may as well not be anywhere. The goal of this system would be to revolutionize contact management the same way that email and hypertext revolutionized written communication.

Just as email works in any email client, and web pages work can be viewed by any browser, the APIs provided by this system would have to be completely open. Any application must be allowed to interact with them, both to change data and fetch it.

In order for it to work, and really have the effect that I’m talking about, there must be absolutely no lock-in, no up-sell, and reasonably liberal rate limits.

$$$

How does something like this make money? That’s an open question, and a big one, probably part of the reason why I’m still pushing bits in a day job and not out getting VC to build this thing. I also happen to really like what I do at that day job.

Maybe it would have to be something built under the Apache foundation or some other OSS group, and sponsored by donations of capital and resources from some major players in the online social arenas. Maybe there’s some clever way in which smaller users and early adopters get the API for free, but then charge everyone else.

Who could do this? What’s going on now?

OpenID is a great start, but what we really need is an open profile and open contact list system, and OpenID doesn’t provide that.

Google’s Open Social is an interesting product, but the more I read about it, the more I think it’s not quite low-level enough to really deliver on what I’d like to see here. While it promises to expose social data to third-party applications in an API that could be consistent across social websites, it doesn’t fully address the issue of being able to manage contact data in a distributed way.

As I said above, the company to do this will need:

A huge base of existing users and preferably their contact details, too.
A strong reputation for protecting user data.
Impressive engineering resources and domain knowledge in the areas of spam protection and social networking.
A serious commitment to open APIs that help the web as a whole.

Yahoo has all four of these, but that whole China escapade has damaged Yahoo’s reputation in the eyes of many users. Of any company on the web, however, Yahoo has perhaps the most to gain from such a system and a lot of resources and domain knowledge to throw at the problem.

Even if they only share user information when presented with a subpoena, that means that using this system exposes my information to governmental intrusion, which is deeply problematic. In order to be truly trustworthy, a stronger commitment to protecting privacy needs to be in place than just words on a corporate press release. The officers of the company to provide this service should enter into a binding agreement that they will not knowingly expose user information, even in the face of governmental pressure.

Like I said, I don’t have all the answers to this product. But I know that, as a user, I’d be absolutely delighted to see something like this take hold.

Mon 2008-03-10 10:00:38 | 0 notes

Code Ecosystems The Business

∞

Surgical Team or Motley Crew of Adventurers?

I’ve read The Mythical Man-Month lately (at Geoff’s suggestion, among others) and it’s great. I definitely recommend it highly to anyone in the software development field.

In particular, I was struck by the idea of Mills and Brooks that software development could be done more effectively by a “surgical team.” That is, the experienced lead developer writes all or most of the code, and is supported by a team of people to make him more effective. While this is perhaps an effective way to capitalize on the significant difference between excellence and mediocrity in coders, it seems to me that most of the jobs on the team are today either (a) made obsolete by technological advancements, or (b) best done by specialists who float between teams. For example, Yahoo! has some of the best Javascript and CSS “language lawyers” in the world—-but does every team really need one? The need for a full-time toolsmith has been all but completely surpassed by the customizability of workstations and the availability of many high-quality tools for coding and such.

The nature of software development has changed somewhat as well. We aren’t writing assembly code any more; not only do we have high-level languages, in my current project, there are no fewer than 9 different languages in use, 10 if you count regular expressions, which we all use quite frequently. [1] We aren’t all creating the same kind of stuff. Thus, there is a strong need for all of us to be experts, surgeons if you will, in our respective areas. In addition, the non-coding jobs require their own sort of specialized focus.

It occurred to me that this division of labor, where everyone depends on everyone else but everyone has a very different job, is more like a Dungeons and Dragons party than a surgical team. Each member plays a vital but remarkably different role, each owns what they do, and each is the team’s expert in what they do. As a webdev, I get consulted about front-end concerns, and am quick to ask a back-end engineer about things that touch their domain. UI questions are decided by the interaction designer; questions of product features are handled by the product manager, etc. I suspect that this is not unique to software for the web; a similar metaphor exists (or at least, could exist) for application software.

The best D&D parties are a bit formulaic, but that’s simply because a certain distribution of talents tends to be most adaptable to the most different kinds of problems.

Fighter

fighter Not the flashiest or most prestigious role, Fighters have the lowest barrier to entry. They accumulate experience quickly, and don’t have a lot of class bonuses. (Until 3^rd Edition made them into action heroes, that is.) But their simplicity belies a focus and strength that is crucial. The fighter is the guy in the party that protects the more “thinky” types from anything that might get in their way. There can be honor and satisfaction in bashing orcs one at a time.

On a web development team, there are two archetypes that sometimes satisfy this role. The first is the diligent code monkey who might not be the most ambitious or inventive, but is good at getting things done. They are perfectly happy to churn through buglists, knocking down whatever they can. If you show them how to do something, they’ll make it work however it has to through sheer force of will—-not always fast or elegant, but functional. When depended upon properly, code monkeys are a great and valuable asset.

Also in the fighter category are the producers who manage the content of the site, use the admin tools, troll for spammers, and do all the other busywork of web site maintenance. When the programmers have moved on to the next problem, they’re still working. Don’t forget them. Whenever possible, make their lives easier. They are the ones that keep your product shiny, so you can continue to be proud to have the URL sitting there on your resume.

Ranger

Rangers are good at surviving on their own. They shun convention and human society. ranger Often a bit wild themselves, they feel more comfortable in the company of nature. Their chaotic bent can sometimes be a liability in an adventuring party, but their wide-ranging knowledge makes them hard to discount. They are fiercely independent and difficult (if not impossible) to tame.

In a programming team, this is the quintessential hacker. Most mashups are created by a single ranger acting alone. They are great at creating new things, and no browser quirk or network irregularity will stand in their way. On the other hand, they don’t always follow convention, and have a tendency to create overly clever code that is unintelligible to anyone who follows in their wake. Thus, they have to be watched closely lest they venture off.

I’ve learned a lot of great tricks and techniques from the Ranger programmers I know. I’m always a bit envious of their free approach to things, but I guess I’m just too timid to abandon convention completely. Make sure that you understand their code before you try to adapt it to your own ends—-wild animals turn vicious when handled improperly.

Paladin

paladin The paladin is the character class with the most stringent requirements. They must conform to the strictest of moral codes and demanding attribute requirements. As such, they’re rare, and respected. The paladin’s job is twofold. As a warrior subclass, he protects the party members from outside threats. However, he’s also a healer, and a leader. Charismatic and compassionate, he holds the party together by negotiating internal conflicts. Unshakably positive, he nonetheless keeps everyone going with firm conviction in the mission.

The best project managers aspire to paladin-like grace in their roles. They defend the team from the pressures of management and the monotony of GANT and PERT charts, and make sure that everyone is able to work at their top potential. When there’s a problem, they are approachable and always seek a win-win situation. By being consistently honest and righteous, they earn the respect and admiration of those both below and above them on the management totem pole.

When someone leads like a paladin, you want to get your tasks done and meet your deadlines. Not because you’re afraid of the consequences, but simply because their drive and attitude are contagious.

There are some managers that are decidedly un-paladin-like. Few things are as hazardous on a project as leadership that does not inspire confidence and commitment. Attitude is everything in a leader.

Rogue

rogue The rogue is a highly adaptable character. They can pick locks and disarm traps, making them essential in any dungeon-faring party. Surviving in the streets and cities on his wit and stealth, a rogue comes to know the different factions and groups, and learns to navigate the urban waters. He knows how to make a buck, how to make a friend, and how to sell, and where to buy.

The product manager takes on this role for a programming team. They are often the ones who first envision the product and pitch it to management. Often the only member of a programming team with a business background [2], they settle questions of business goals and product requirements. They make the powerpoint presentations that eventually drive the development of a real software product.

The product manager, like a rogue, also sings the song of the party, talking them up when necessary and interacting with foreign powers [3]. If a product requires buy-in from another team, whether it’s a standards group in the same company or a potential outside partner, these are the people who make the calls and settle the deal. Their day is usually full of meetings.

I’ve known some developers who look down on product managers as being less technical. This is a big mistake. Their job is every bit as intellectually rigorous as ours, due to the huge amount of information that they need to be able to call forth on demand. Since they deal with features and requirements that have not even been fully envisioned, and might not exist in any form whatsoever in the market, they are even more in the world of pure thought-stuff than programmers. Make friends with your product manager, and you’ll find that they often do actually know their stuff. And after all, some rogues will stab you in the back if you cross them ;)

Mage

The mage is the general purpose wizard. They are resourceful and possess a broad understanding of all types of magic. While not specialized in their studies, they are deeply theoretical, research oriented, and possess incredible focus and discipline. Their power comes from being able to summon a wide variety of spells, but they tend to be weak and vulnerable in the chaos of melee, since they must prepare spells ahead of time.

Many experienced back-end engineers fit into this role. They tend to prefer structured approaches and strongly typed languages, and produce exceptionally high-quality code. wizard They often have advanced degrees in computer science and are knowledgeable in the intricacies of design patterns and memory management. They excel in areas that are somewhat regular and predictable, where the environment can be controlled, and the assumptions and edge-cases checked accurately.

Dedicated programming wizards are obviously important on any programming team. However, as much as “laymen” tend to see all software as a form of magic, the truth is that mages are actually pretty rare in the development world. They are the core of the “20%” of developers who truly care about what they create and diligently work to improve their abilities. Their craft is important to them, and they approach each problem with care and precision. It is much more than just a way to pay the bills.

While dedication is important, it comes at a price. Many back-end engineers come to get out of touch with the way that “normal” people think and interact with software. Most are pretty aware of this fact; I once knew an engineer who complained that bugzilla’s user interface was obviously designed by a database expert.

Specialist Wizard (illusionist, abjurer, conjurer, etc.)

specialist_wizard These are the wizards that have selected a certain area of magic in which to specialize, at the expense of reduced ability in some other area. They have the same theoretical focus as a mage, but instead of having a broad understanding of every type of magic, they work to gain a much deeper understanding of a single area. When it comes to solving that sort of problem, they are the best—-however, in other areas, they may be lacking.

There are many “specialist wizards” in the programming world. Here at Yahoo!, there are teams that focus 100% of their time on developing PHP, or MySQL, or Apache. (Rasmus comes to mind.) The “Paranoids” team sits in the cubes adjacent to ours, and they work to make sure that all the software at Yahoo! is as bulletproof as possible. The exceptional performance team does research to help us webdevs make our pages load and run faster.

When you have a hammer, the world can begins to look like a nail. A MySQL specialist will be tempted to suggest a database schema to solve any problem, even perhaps if the problem would be better solved by some other method. You may be asking for trouble if you ask them to do something outside their specialization. However, if you have a certain task that you’re certain to need, they can be invaluable. Database optimization is a highly developed science, and it’s not always easy or simple. Good programmers are rarely security experts (or even security knowledgeable!)

If you aren’t likely to have a lot of problems in their specialty, it’s often better to consult with a specialist only if and when they’re necessary. That way a single specialist can service multiple different teams.

Sorcerer

Sorcerers are similar to mages in the types of spells that they can cast, but instead of preparing spells from a spellbook, they simply select from the spells that they know, calling them when needed. As such, they tend to be much more spontaneous. However, they do not cultivate the discipline or depth of understanding of the wizards, and so they are limited in the number of spells that they can learn. The sorcerer excels in areas that are uncharted, and their spontaneous approach to magic allows them to quickly adapt to any situation, no matter how unpredictable.

The sorcerer is the perfect corollary to the webdev, aka the front-end engineer. Web browsers are a constantly moving target, with each behaving a little differently and new versions being released all the time. sorceror While programming patterns and the “right” ways of doing things can be useful, they are often too bulky for Javascript that has to be delivered, parsed, and executed on demand. Essentially, the webdev creates software in an environment where the rules often make no sense; where the code has to work on different platforms with conflicting requirements; where speed and features are both must-haves.

I’ve only recently heard of any colleges that make any serious attempt to teach courses on web development. From what I have heard, the things they teach are pretty bad—-at best out of date, at worst outright harmful. Most professional web developers that I’ve met didn’t learn their craft in school. Like the sorcerer, they simply picked up their craft as they could, and learned techniques by studying “in the wild”; that is, by viewing source and reading websites and trying things out.

I’ve said before, it takes a certain kind of crazy to want to write software in the browser. You have to love the chaos and enjoy pulling out tricks and coming up with new hacks to work around impossibly ridiculous requirements. At the same time, unlike the ranger’s free-wheeling wildness, the team’s sorcerer must have enough discipline to tame the chaos and create software with whatever degree of stability he can, in order to help create a functional and stable site.

It can be a challenge at times to find stability and see the bigger picture and not write code for a single case. Yes, yes, that hack made it work in Safari. Lovely. But it’s also going to be a nightmare to maintain, or you just ruined the accessibility. Trade-offs are always present, and as far as I can tell, only experience and a lot of mistakes can teach you how to choose wisely.

Cleric

Diligent and committed to the ethos of their higher power, the cleric summons heavenly forces to do his bidding. By making himself an instrument of the divine, he can accomplish great things. A cleric’s power comes from their will and common sense (in D&D, this is the Wisdom score), as opposed to the cleverness and wit that Wizards rely on. The cleric has the power to vanquish undead through the force of their convictions, and their healing spells make them a must-have for any party.

cleric In a programming party, this is the role of the QA tech. These noble souls must possess tenacity and diligence far beyond what is normal or sane. They must have the conviction and tenacity to insist that every test case be faithfully executed, even if it’s “impossible” that this-or-that change would have caused a regression. When they see something in the product that doesn’t look right, they must get to the bottom of it, carefully catalog the event, and enter it into the bug queue. They help banish the skeletons by exposing them to the light, no matter where they may try to hide.

Many a programmer, close to a deadline, has remarked with exasperation, Jeez, %$!@# QA keeps making me revisit this thing, I fixed that already!, only to find, when they go through the steps in the bug report, that yes, it is indeed broken.

While they may seem difficult or inefficient or outright stubborn at times, they’re vital. Their wisdom is a balance to the programmers’

wit, and they keep teams from burning up. Budget plenty of time for QA. Start it early. Get someone who knows what they’re doing, and use a proper bug tracking system. Your customers won’t thank you for it, but they’ll certainly come to hate you if you don’t.

Druid

Free from any affiliations or alignment, the druid transcends the simple human existence to become one with the natural world. Unlike the clerics who request their power from a deity, the druid opens themselves up to be nature’s instrument. It is an understatement to say that they know about the wild world; they understand the ways of the animals and plants intuitively and instinctively. Always seeking the way of balance and harmony and simplicity, they do not try to change or restrict the chaos of nature, nor do they rebel against the order that sometimes comes out of that chaos.

druid Designers are called by many names in our industry. UEDs, Visual Designers, User Interface Architects, HCI Specialists, Interaction Designers, IAs. (Indeed, these roles are somewhat different, as the task of designing the user interface has many facets.) Whatever they’re called, their job is to understand the expectations and habits of people they’ve never met, and shape the product into something that will fit the user’s mental model. At the same time, they must understand how those models change in the face of new problems and new solutions, and be able to adapt their software properly.

Like the druid, the UED must be free from any biases that may impede their ability to find the best solution for the user, even if that sometimes means that they may take issue with business objectives. (If the business goals and the user goals cannot both be met, then the product is in real trouble, anyhow!)

Part of the UED’s task is to map out the competition’s sites, to determine where this new product fits into the current ecosystem and how the target market’s expectations may have already been set. They must sometimes balance the “ideal” way of doing something with the “common” way of doing it, simply because users have been trained in a certain way.

More Similarities

A lot of the same characteristics that make up a good party are also true of a good programming team.

A team should have enough members to get the job done, and no more.
Each member should know his role, and enjoy doing it
Balance is key. Each part of the task should be accounted for, and no one area should have more resources than is needed
Druids make so-so healers; UEDs are usually pretty good at QA. A really good sorcerer might be able to handle the magic support if it’s not too intensive; most webdevs can usually throw some kind of back-end support together. But, if a certain area of the project is significant, then it’s worth having someone who knows it best.
Multiclassing can make a character more diverse, but they won’t increase in levels as quickly.
Treasure should be distributed fairly.
If you have more than one mage (or sorcerer, or cleric, or druid, or whatever), they should either divide responsibilities or one should be in charge. Same for the roles on a programming team.
It’s hard for a fighter to know how good a wizard is, or vice versa. It takes one to know one. Make sure your team helps interview new hires.
Raising levels and carefully assigning skill points will lead to a more powerful character. We min-max in real life, which is why we do it in RPGs.

Like most of the essays in The Mythical Man-Month, the fundamental principles of the surgical team are valid, even if the specific roles are mostly out of date. Within each class on the adventuring party, you can sometimes see a combination of surgeon and co-pilot; sometimes the co-pilot is unnecessary, or there are two surgeons who divide up the work.

The basic idea of thinking about the programming organization in a creative way is also worthwhile, even if only as a mental exercise. I’m sure that there are other metaphors that would work equally well.

In general, I think that the motley crew of adventurers is a lot more fun than the surgical team.

Disclaimer

All images copyright © 2007 Wizards of the Coast. Please don’t sue me, WotC! We love you! I’ve taken some liberties with the images used. The Rogue is actually a Bard. The Specialist Wizard is some kind of psionicist. The Ranger is actually a Scout. I picked more for style than accuracy.

Footnotes

[1] MySQL, PHP, Javascript, HTML, CSS, Bash, Perl, XML, yinst, regular expressions. Actually, 11 if you count Apache config directives. [2] Unless the webdev is a business school drop-out, that is. [3] Some teams, of course, have a dedicated business development person, or “BizDev” who handles outside relationships so that the product manager can focus on product development. I realize that I’m blurring the distinction, but I don’t think it’s a big deal.

Mon 2007-12-03 12:10:17 | 0 notes

Code Ecosystems The Business

∞

Agile Scrum Sucks (but so do the alternatives)

Software development is not like manufacturing.

In manufacturing, you build to the same specifications over and over again. In software development, you create a new thing each time, and the specifications typically change along the way.

In manufacturing, 100 factories can generally produce 100 times as many things as 1 factory. Adding a new factory means that more things will be produced. 20 programmers are far less productive than 5. Adding another programmer often means that the project will be delayed even further.

In manufacturing, you can gradually tune your methods based on past experience, and become more efficient over time. In software development, past problems generally have little if anything to do with future problems. The best you can hope for is to find and learn very broad abstract efficiency boosters like “best practices.”

Manufacturing and software development are similar in what I think of as the “expert effect.” Generally speaking, for a given task and a given group of people, one of them will be best at that task. If that expert is given control of that task, then they will tend to do it well; if they are forced to take input from everyone else in the group who isn’t as good as them, then the task will tend to suffer, if only from the time required to evaluate the input and decide not to use it. Management is one such task, perhaps the most important one.

Most people I know seem to accept these premises. That is, unless you’re an Agile Scrum zealot. Then you may think that:

Specifications change, but that’s OK, because it’s “iterative design”.
You can predict how long something will take, and your predictions will get better over time.
Tasks can be broken down based on the time required, and can be assigned to any “qualified” person.
Programming experts will be happier if they do project management, and are qualified to do so.

This is all 100% bullshit. In the real world:

Specifications change because it’s virtually impossible to design something that’s never been done if you can’t see it as you go along. The specs stop changing when the team puts their collective foot down to management and the customer, and that’s when “research” stops and “development” begins. The time at which this occurs is flexible, but if you’re targeting a fixed release date, it’d better be soon! If you don’t build in enough time for research, then the product will not be satisfactory.
Your estimates suck. And they’ll keep sucking, no matter what you do. You’re probably around 10-15% accurate, and if you work your ass off, you can get up to a 20-25% accuracy rate. Estimates that aren’t even remotely accurate are a waste of everyone’s time.
I’m not the only one, I’m sure, who finds that at least 60% of my planned tasks for any given sprint don’t get accomplished, either because a) they depend on something else that wasn’t evident at the planning meeting, b) they just weren’t as high priority as something that came up since the planning meeting, or c) my time estimates suck, and so I budgeted way less time than required for a few of my tasks.
No one likes project planning meetings. But, since project managers typically aren’t programmers dealing with the software each day, they need our input. If you’re a product manager, trust me, your team will absolutely love you if you can get these chores down a half hour or less. Streamline every part of the meeting that requires their attention. Focus their time on what you actually need them to weigh in on. Have estimates prepared, and ask them to critique them. Wrapping it in silly games won’t help—-that just makes it take longer. Programmers hate project management. So make them do as little of it as possible. Trust me, if they wanted to be project managers, they wouldn’t have degrees in computer science. Let them get back to work. There’s no worse blocker than having to stop coding for a meeting. Like torture or invasive surgery, it’d better be necessary, otherwise it’s all kinds of wrong. (The “5 levels of estimates” is the Dread Cthulhu of time-wasters.)
Prototypes don’t pay the bills. If you create a bunch of dummy-functional UI screens, you haven’t created software, you’ve created a demo. A prototype is an important first step, a valuable communication tool. But you can’t sell a demo, which means you can’t create demos forever. Once the “research” phase is over, demos are a liability, because they tend to obscure missing functionality.

There are a few things that scrum gets right:

When it comes to estimating, often the crowd is smarter than the individual. “Planning poker,” though overly goofy and time-consuming, does a pretty good job. But it takes forever, and while I’m waiting to show my card, I’m only thinking, I’d really rather be coding right now. I’m not a project manager (that’s part of the problem with Agile Scrum) but I know there simply must be a better way.
A daily meeting is crucial if there’s more than one person on the team. It’s still best if everyone can physically see the people that they’re working with from where they sit, but that’s not always possible. Even in that case, forcing everyone to stand up for a few minutes each morning and tell everyone what they’re up to is a good way to make sure that the team doesn’t get sidetracked or splintered. It keeps programmers from “going dark” and coming back a week later with something that doesn’t mesh with the rest of the product.

It seems that some Scrum Masters are die-hard zealots, and others follow a more “pick and choose” approach. The Agile methodology explicitly encourages project managers to use what works, and customize the process to meet their needs.

In other words, Agile Scrum makes a lot of ridiculous assumptions, and suggests a set of practices—-some harmless, some obvious, some useful, and others specifically counter-productive—-and then says, customize to your team/product/environment. If your project fails or misses a deadline, well, you didn’t do it right, or the team wasn’t committed, or the management was unreasonable. Anything at all to avoid placing blame on the methodology.

Anyone else see the similarities to a faith healer?

Ultimately, a successful project that uses Agile Scrum is successful for the same reason that any other project is successful:

The management managed, and got out of the workers’ way.
The workers were capable of doing what they were called on to do, and happy doing it.
The product vision was something the customer wanted
No one went dark. (That goes for upper management, as well!)

On my current team at Yahoo!, we use Agile Scrum, at least in name. Ostensibly, Yahoo! officially endorses the use of Agile Scrum. In my experience, successful use of agile amounts to this:

Every week or two, the team gets together for an hour to talk about where the project is at.
Every day, the team spends 15 minutes in the morning with each person telling what they’re up to.
Throughout the day, everyone is in constant communication via IM, email, and in person.
Put your tasks in Bugzilla, and mark them done when they’re done.
Most important: Management’s vision of the product is something that the team is happy building, and the team is selected for their expertise in the kinds of problems that the product raises.

No one likes to create crap. No one likes to be surprised with new requirements in the 11th hour. No one likes to be held responsible for something they’re no good at.

Everyone loves to be a part of a quality product. Everyone likes to have visibility into their own future. Everyone likes to be praised and respected for doing what they love.

By taking credit for the success of projects that use it, Agile Scrum glosses over these fundamental principles and tries to turn the team into a bunch of part-time project managers. Anything that deviates from the plan is derided as “cowboy coding” or “waterfall process.” Granted, these two paradigms do tend to standardize bad practices, and they do little to encourage communication, but they also admit important aspects of the software development world:

Actual coding, typing characters into an editor, is best done alone without distractions. You think with your own brain only. All coding is, at some level, “cowboy coding.”
If piece A depends on piece B, then it makes no sense to start working on A until B is done. It makes even less sense to QA a half-working early pre-alpha dev version of the product, or worse yet, file bugs against it. Of course it’s broken. Don’t distract your developers telling them what they already know.

Chances are wildly against getting any software product out on time or under budget. My heart goes out to project managers and executives who are tasked with controlling the chaos. I understand why many of them might grasp at ideologies that promise to solve the problems, but Scrum rarely helps the odds.

If you’re doing software development right, you’re probably doing Agile wrong.

Mon 2007-11-12 10:00:38 | 0 notes

The Business

∞

What Makes a Good Web developer, and how can you tell?

Luckily, I’ve never been a manager with people who report up to me. I’ve been a tech lead on a few teams, and quite frankly, I find that I prefer to be doing the coding than directing it. Managing is about 10% more money for 100% more work that is 100% less fun. No thanks.

Nevertheless, it’s always a good idea to take part in the hiring process, and smart managers get their team involved. I’ve sat in on a lot of interviews, and talked and thought a lot about interview processes. However, despite all the complications inherent in the interview process, it is remarkably difficult to hire web developers.

A lot of great web developers have very odd educational history. Querying is hard.
Everyone knows what keywords to put in their resume.
Some really horrible web developers would be great programmers, but are not of the mindset to deal with in-browser development.

Today, I looked through a stack of resumes from upcoming college grads with my current manager since he’s got a requisition open for a front-end engineer. In the interest of full disclosure on this blog, here’s what I was looking for:

Education - Lots of topics

I’d rather see a list of minors a mile long than a single major that has anything to do with the subject. The world’s top expert on distributed systems and service oriented architectures probably can’t do much else. When you have a hammer, the world looks like a nail, as they say.

On the other hand, you have a candidate who started out in graphic design, and then decided that he liked math better, and then took some classes on computer science, and finally started making web pages, and decided that he’d like to do that professionally. Most of the best web developers that I’ve met have bounced around a lot in their studies, and only settled on web development because they found it satisfying.

Web development is a very open and strange discipline, and people who are self-directed and cross-disciplinary tend to excel in it.

On the other hand, no education at all is also not necessarily a bad thing. In this case, it didn’t apply, since the resumes were all for upcoming college grads.

Extra Activities, Other Information, etc. - Play an Instrument

It seems like a lot of web developers have some kind of musical inkling. Either they play an instrument, make electronic mixes, do amateur production for local bands, run a radio show or podcast, something. Music seems to be a common theme.

Some of us are terrible at it. For example, I’m a bad-to-moderate guitarist. But we all seem to enjoy it.

Put it on your resume. It matters.

Objective - If it’s not compelling, skip it.

I saw a few objectives that read like carbon copies of one another: An entry level position in a software company. Or this one: A job where I can put my education to use. Jeezus, why not just put A soul-crushing career kissing the ass of my dipshit manager while I contemplate burning the building down?

On the other hand, I saw one that said Remake the web in a groovier image. His name got on the callback list. A lot of them didn’t have an objective section, which is perfectly acceptable if it’s obvious.

Ultimately, every hiring decision is an exercise in risk management. The hiring manager has to ask himself, Does it put my project more or less at risk to hire this person? Adding a new team member is a big risk. They might be a drag, and bring down the productivity of the rest of the team. On the other hand, you need people on your team, and not having them can be an even bigger drag. On an emotional level, hiring decisions come down to a question of Will I enjoy the experience of working with this person, or not? There are a lot of really smart people out there, and some of them are a chore to work with.

The resume is a validation key designed to get you in a particular door. The more it shows off what you’re really like and what you’re really into, the more likely it is that you’ll walk through a door and like what you find.

Mon 2007-11-05 10:00:59 | 0 notes