I’ve been making computer games in some form or another for just over 25 years now. At the very beginning, as a hobby (passion) and completely by myself (although not for lack of trying to get some of my friends involved). In the late 90s, when I finally left academia and started making games professionally, teams were still relatively small, with a total of around 10-15 people per team. As we all know, budgets and scopes kept growing, and so did team sizes. At its peak, the largest team I worked at had around 200 people. That’s when I decided to go indie and started Power of Two Games, which was obviously just two of us. Finally, now as Snappy Touch, I’ve gone full circle: It’s just me again.

Development tools and hardware have changed quite a bit from the times I was writing in straight Z80 assembly and saving the programs to a tape. But I’ve also changed and learned a lot during all those years developing games, and even though I’m writing games by myself again, I’m doing things very differently from how I did them back at the start.

One thing that I’ve always done is to question everything. Why should I do things a certain way? Why is that the “accepted” way of doing something? And not surprisingly, at each step of the way, I’ve changed my development style to match my situation (often in ways that went against the “common wisdom”).

When it comes to solo development, I rely heavily on the concept of goals and iterations at multiple levels:

• Immediate (< 1 minute): Prioritizing the ideas going through my mind. Writing tests. Writing code.
• Short Range (< few hours): Tasks that move the project forward in some way.
• Mid Range (< 2 weeks): “Stories” that define a self-contained, significant part of the project.
• Long Range (full project, several months): Ship date, beta testing, etc.

It turns out, I use a different set of tools to help me manage the items at each level of the development cycle.

Immediate

These are the actions I take and complete in less than a minute or two. Most of them are writing tests (with UnitTest++, of course), writing code to make those tests pass, refactor code, and check it into Subversion. I have my Subversion repository hosted on Dreamhost and I access it through SSH so it’s secure, accessible from anywhere with an internet connection (or 3G signal and HandyLight), offsite, and easy to backup. And because Subversion works great offline (unlike Perforce), it’s not a problem to work without connection to the repository for a while.

I also need to manage my minute-to-minute thoughts, write down ideas and reprioritize them when the time comes. When I’m “in the zone”, I get way more ideas than I can execute with my fingers: This function needs to be moved to a different file, I really should be compacting that data over there, who was stupid enough to name this file this way?, that variable shouldn’t be cached, etc. If I don’t write things down, I will either forget them, or I’ll stress for hours until I finally get around to doing them. I could also do things as I think about them, but then I would be chasing a rabbit down a neverending hole and wouldn’t get any work done (I’m sure anybody who’s gotten lost browsing web pages can identify with that).

I used to do this the old-fashioned way, simply with paper and pencil (like Bob described in his blog post). However, I found that physical paper and pencil was just too limiting: I can type a lot faster than I can write, switching to writing requires moving away from the keyboard, and, most importantly, I need to bring the notes with me everywhere and it’s very difficult to rearrange, sort, or coalesce them in any way.

So instead, my tool of choice these days is JustNotes. It’s perfect for jotting down thoughts in a matter of seconds without even interrupting my train of thought. I have JustNotes bound to a global key, so in the middle of typing a line of code, I can press that key, enter whatever I’m thinking about, press the key again, and finish the line of code. All in 4-5 seconds. Don’t laugh: The fact that I can do that in just a few seconds without moving my hands away from the keyboard means I can use it any time without much penalty. It’s amazing how many things I jot down that I wouldn’t do otherwise.

Short Range

To manage tasks up to a few hours in length, I use Trac. Trac is a fantastic issue-tracking tool: It’s free, it’s fast, it’s simple, and it’s configurable. In the past I’ve used anything from spreadsheets, to Bugzilla, to publisher-owned bug databases, and nothing comes close to Trac for my needs. It also scales very well to teams of more than one person (although it might not be good enough for hundreds of people).

Just like Subversion, I have Trac hosted externally, through my web hosting company. Sometimes it’s frustrating if I need to access it and the server is down, but again, the convenience of having it off site makes it well-worth it.

Any task that requires more than 10-15 minutes goes in Trac, and then I can easily prioritize tasks depending on their importance. Usually, if an item has been on my instant queue in JustNotes for about a day and I haven’t gotten around to it, it either gets deleted or it gets moved into a full item in Trac. The only way progress happens is by ticking off Trac tasks. In the end, my projects live and die by Trac.

Medium Range

User stories (to borrow terminology from Scrum/XP) are visible, relatively self-contained features of the final project. They’re made up of several tasks, and usually take a few days to a few weeks to implement. A group of user stories make up a full iteration (sprint) of the game, which is usually between one and two weeks long. Sometimes user stories are complex enough (add a replay feature visible on the web site) that the full iteration is just the one user story.

I keep track of these stories in Trac as well. Trac is both an issue-tracking system and a wiki, so the wiki part is perfect to keep these user stories. In addition to that, I label tasks as belonging o a particular iteration. That allows me to separate what needs to be done for this iteration, from other tasks that I added for the future. At the start of each iteration, I decide on the user stories and either generate new tasks or label existing tasks as due for this iteration.

The wiki in Trac is extremely valuable for all sorts of other things: game design ideas, general brainstorming, gathering reference material, etc.

Trac ends up being the perfect mid-range vision of my project.

Long Range

User stories and tasks in Trac aren’t enough to cover a project that is potentially 3-4 months long. I need something that helps me with the longer view, otherwise I find that things creep up on me without realizing it because I’m so focused on the short and mid-range items.

The best tool I’ve found so far is very low-tech: A printable month-per-page calendar covering the full length of the project. Right now I’m shooting for a November release of my current project, so I printed August, September, October, and November and pinned them to the corkboard on my office. It’s amazing the sense of urgency that seeing your ship date gives you. You realize that you only have a handful of weeks before shipping and makes it much easier to prioritize tasks (and chop off features or save them for an update).

I realize this long-term, calendar view isn’t very useful if you don’t have a set release date and you want to continue chipping at your game until it’s ready. But even if your release date is flexible, having this long-term view can help you keep budgets in perspective and manage them accordingly.

Finally, for an extra bit of motivation (or maybe this falls in the category of excessive pressure), I just started using a countdown widget for the Mac OS Dashboard. Just in case the calendar view wasn’t enough, here’s a countdown (down to the second) of the time left until release.

Speaking of which, I think it’s time I get back to work. Only 102 days left!

You’d think that things would slow down around Christmas time over here. And that I would have lots of time to catch up and write about all the things I keep jotting down in my overflowing “to write” list. Right? Unfortunately that’s not the case.

I do want to catch up and share my iPhone development experience with everybody, but it’s hard to make time, even during the holidays. To make things worse, I have a pretty hard deadline I set myself to have my iPhone app ready by early February.

It’s funny, I’ve worked more hours in the last year than any other year during my career, but it has never felt like work. Quite the opposite actually, and I find myself doing “work” instead of other things that used to be more fun and having a blast with it. For example, my game-playing time has gone waaay down. If I have an hour that I can do anything, I think “oh, maybe I can play a bit of World of Warcraft now” followed by “but it would be so much fun to finally add this other feature…” I let you figure out which one ends up winning in the end. If it wasn’t because of my weekly WoW session with Jim, Joey, and Michael, I wouldn’t make any progress in the game!

Not only am I enjoying what I do immensely, but it’s very easy to spend lots of time doing it because I get to do it from the comfort of my own home. I never have to worry about getting to work by a particular time, so I can get in my morning runs or go grocery shopping when the stores are empty, I have all my stuff around me, my work area has an incredible view overlooking the whole of Mission Valley, and my kitchen is stocked up with all my favorite teas. It’s really easy to spend hours and hours working. A bit too easy actually.

Burnout is the evil flip side of fun work. Even with incredibly rewarding and fun work, doing something for many hours a day can burn me out pretty easily. The symptoms are obvious: I start to lose interest, I’m not as productive, my mind wanders, and I find all sorts of other things I’d rather be doing. Fortunately, I can recognize those symptoms early on, so just backing off a bit, or taking an afternoon off, allows me to walk the fine line between work and burnout and keeps me productive.

Initially I was a bit concerned about the thought of working full time from home. Would I miss the interaction with coworkers and the sparks and new ideas from seeing all sorts of stuff around me in a company. It turns out that it wasn’t anything to be worried about. Thanks to the Internet, it’s very easy to reach out and connect with other people. I regularly share my progress with some friends and family members, and I always enjoy hearing their comments and reactions (even the not so positive ones–especially those actually). I also meet in a semi-regularly basis with my friend Dave, who is doing all the graphic design for my app (and he’s the mastermind behind the Power of Two logo).  Those meetings are always very inspiring, and usually result in all sorts of new ideas flying around and a flurry of activity in the days following.

Having non-work related hobbies really helps to keep me from spending too much time on work. Training for half marathons (I keep saying that one of these days I’ll do a full marathon–we’ll see about that), or just getting back in shape to do a century on my bike require quite a time commitment and keep me in shape and in top mental shape.

I suppose this is exactly the feeling that some companies try to encourage in their employees. They hire people who are really passionate about their work, give them the means and the freedom to do it, and hope they pour their heart into it. They just have to hope that their employees are able to keep themselves from burning out. That might be easier if you’re forced to work from an office environment, totally separate from home. On the other hand, being away from your family for longs periods of time eventually takes a toll on happiness and morale.

That’s a totally different beast from the kind of crunch that is rampant in the games industry. I define crunch as having to mandate long work hours (either explicitly or through peer pressure–which is more sublte and infinitely more evil). In one case, you’re so passionate about what you’re doing that you can’t wait to go back to work on Monday morning. In the other case, you’re still stuck to your desk come Monday morning because you were forced to work the weekend and get a build ready for some marketing demo. One is a very rewarding life, the other is soul-draining. One results in happy, ultra-productive people and the other degenerates into a death march, people leaving, and projects being even more delayed.

So, even with the February deadline looming, I want to take a few more hours these next few days and get back into the wrting swing of things. Besides, with the app coming up so soon, I should start thinking about announcing it soon! First I need to wrap up the Game Developer Magazine column due on Monday, but after that, I’ll try to start posting regular updates.

Until then, happy holidays to everybody. I hope you’re taking some time off, or at least doing what you really love.

High Moon Studios is an unusual company in the games industry. We’re applying agile methodologies for all of our development. My team in particular is using both Scrum (an agile management methodology) and Extreme Programming (an agile engineering methodology). And yes, that means we’re doing pair programming, test-driven development, and all the other often controversial practices. I expect that in a few years, these practices will be a lot more common than they are today.

This article was first published in the 2005 Career Guide issue of Game Developer Magazine.

I lead the R&D team here, and my team’s primary responsibility is to come up with the technology that game teams use for different projects. Nowadays, that means putting a lot of middleware programs through their paces and choosing the ones that suit our needs. But it also means getting down and dirty and writing a lot of code for our engine and tools.

With that in mind, come on and follow me through a typical workday.

8:10 AM

I roll in to work on my bicycle like I do every day. Even though I’m an early bird, Jim, a programmer in my team, arrived a few minutes earlier and is already at his desk.

I quickly catch up with my email. I also notice that the PCLint pass on our codebase last night caught a couple of minor warnings, so I quickly fix those and check them in.

8:20 AM

Today is Tuesday and our two-week iteration ends on Friday. An iteration consists of a fixed period (usually two weeks in our case) during which the team commits to deliver a set of functionality described through customer stories. The customer (in our case the other internal teams in our company) creates and prioritizes a set of stories. My team then breaks down those stories into tasks and estimates how long they will take to complete (tasks vary between 1 and 8 hours).

Jim and I walk over to the “war room” (the room with all the task cards corresponding to user stories pinned to the wall) and we choose the task that reads “Blend ragdoll and animation,” which is part of the user story listed as “Throwing a rigid body at a character and seeing a hit reaction.” The task was originally estimated at 3 hours, but now that we know that the ragdoll system is a bit more complicated than we thought, we re-estimate it at 4 hours.

8:23 AM

In addition to our own personal desk areas, we have pair-programming stations in the R&D lab, with two monitors, two keyboards, two mice, two chairs, and plenty of room for two people. All production code is written by pairs of programmers. We grab a station and start working on the task.

Since we’re using test-driven development, we first write a very simple unit test for what we want to do, and only then write the code to make it pass. In this case, our first test checks that we create a blender object without inputs and that it produces no output. Then we write the blender class and make the test pass. It’s a tiny step, but it’s a step in the right direction. The whole cycle of write test, write code to make test pass, refactor, takes only 5-10 minutes, and we do it over and over.

8:39 AM

We have implemented a small amount of functionality, and all the code builds and all tests pass, so we check it in source control. This is called continuous integration, and it requires that programmers work on the latest version in source control and check in their own changes many times throughout the day.

8:50 AM

Other people from the team roll in, grab other pair programming stations, and start going at it. On their way in we have a quick chat and find out what we’re all working on.

Work hard…

9:37 AM

I overhear Joel and Gary discussing how they’re going to test something that requires updating the physics simulations. I just did that a couple of days ago so I jump in the discussion. It turns out they need to do something that is almost the same as what I already did, so I point them to what I wrote and they will modify it to suit their needs.

10:05 AM

Things are moving along very nicely. We’ve checked in four times already this morning. When a pair gets really going, they might check in as many as 20 or more times in a single day. At this rate we might be done sooner than the four hours we had estimated.

10:14 AM

We have the daily Scrum meeting at 10:15, so we head over to the war room. Scrum meetings are very short, standup meetings with the whole team (eight people plus Brian, our producer). We quickly go around the room discussing what people are doing to get everybody up to speed.

10:23 AM

During the meeting the topic of how we’re loading physics assets came up. So we return to the pair programming area and have a quick discussion with everybody involved. We draw some quick UML charts on a whiteboard, think about how the data is going to be passed around, and after ten minutes we reach an agreement and go back to work on our tasks.

11:15 AM

We got the blending working correctly. All the unit tests pass, although we haven’t implemented it in the demo yet. There’s another task card for that. We check the code in right away. The code definitely can stand to improve in a few places because we were just concentrating on getting things working, so we spend some time refactoring. We have lots of unit tests, so we’re confident our refactoring isn’t introducing any bugs.

11:56 AM

The code is now in a much better state. We check it in and wait a few minutes for the build server to report that everything built correctly and all tests passed. During that time we talk about what the next task should be.

12:05 PM

We have nice shoulder-high surf today, so I’m going out surfing at lunch with several of my teammates. If it’s not surfing, it’s a basketball game, cycling, running, or even yoga. If all else fails, there’s always a game of Guild Wars with the rest of the High Moon clan.

1:10 PM

Back at my desk I eat a quick lunch while I catch up on my email (which I haven’t read since this morning because I’ve been at the pair programming station). I read an email from a middleware developer in response to one of our queries earlier in the day and fire back a quick response.

1:25 PM

Sean stops by my desk. He’s ready to go back to work, but the programmer he was pairing with in the morning got pulled to work on some last-minute issues with Darkwatch, which is about to go gold and has priority over anything we’re doing.

Sean quickly brings me up to speed on what they had been working on that morning, which was to display the exact memory usage for our physics system. I remember them mentioning that in this morning’s Scrum meeting. I also worked on the physics system last week, so I’m pretty familiar with the code. In a couple of minutes we’re already making progress.

… rock hard

2:07 PM

After writing several unit tests and implementing some functionality, we’re ready to add the memory display to the demo program. A couple of minutes later, we have a display in the demo indicating how much memory the physics system is using, and we see it go up and down as we add and remove rigid bodies from the demo.

But something is wrong. When we remove rigid bodies, the memory doesn’t come down to the same level it was before. We exit the demo and we see a long memory leak dump. First things first, we check in our changes, and then we dive in and look for the code that is leaking memory. It’s probably not caused by the code we just wrote, but we have “collective code ownership,” which means that everybody is expected to fix anything that needs fixing anywhere.

2:12 PM

The build just broke! The build server detected a failed build and notified us through a system tray application. I bring up the latest build log and I see that one of the unit tests failed in release mode. Tyson, who is sitting at the station next to ours, says “Oh yeah, I know what that is. I’ll fix it right now.” In less than 30 seconds he makes the change, and checks it in. A few minutes later, the build system reports a passing build and everything is back to normal.

2:17 PM

We found the memory leak. It was a misuse of reference counting. We first wrote a unit test that showed it failing, and then we fixed in the physics library. We check in our code.

2:18 PM

We go to the war room and grab the next task. This one has to do with being able to expose different variables and functions on the demo to tweak them through a GUI. We sign up for the task and start working on it.

3:43 PM

We’re totally in the zone. We’ve been writing tests and code like crazy and this task is going really well. We’ve done three check-ins for this task alone in the last hour.

4:12 PM

Another pair is discussing how to handle errors for some particular case. This is an important topic and it should be done consistently across all the code, so we have a quick discussion about it involving most of the team. Five minutes later we’ve made a decision and we all resume working on what we were doing before.

5:40 PM

We do our last check in for the day. We’re almost done with the task, but not quite there yet. Even though we could stay another hour and try to finish it, we’re both quite tired by now and we’re starting to not think as clearly and make some mistakes. We can wrap this up tomorrow morning as soon as we get in. The important part is that we got to a state where we could check in.

We have a rule that says nobody can check in code and leave. You have to wait for the build server to build the code successfully and pass all the tests. We keep build times short, so that usually means hanging around an extra 4-5 minutes. If anything breaks, you need to fix it or revert what you did, but there’s no excuse to leave with a broken build.

While I’m waiting for the build server to finish, I check the pile of email that accumulated in my inbox during the day.

5:44 PM

After a few minutes we get the green go ahead from the build server. Today was a pretty productive day, and at this pace we will definitely complete all the user stories by Friday. The demo we’re putting together is also starting to look very cool.

One of the things that agile development, and especially pair programming, does is to make each day very intense. There are no little breaks to read email, check a web site, or just goof around. We get a lot accomplished in a work day, but we can’t keep that pace for a long time, so it’s important to call it quits and go home. That leaves me with time at home to read technical books, prototype different ideas, or work on side projects in addition to unwinding, spending time with my family, and enjoying other hobbies.

I hop on my bike and head home with a big smile on my face.

Pair programming really needs to be experienced to be fully appreciated. Just a few years ago, I loved my single office and I was completely against the idea of spending all my time programming with somebody else sitting at the same computer. Today I advocated using pair programming at work and I gladly gave up my office to work in a pair-programming lab alongside the whole team. Funny how things change.

For the last few months we’ve been doing pair programming at work. And not just pair programming, but full extreme programming with test-driven development, continuous integration, collective code ownership, customer-driven stories, team co-location, etc. It has been a fascinating experience and it has been working out better than I even imagined.

I felt that all the extreme programming books I had read just touched on the subject but were light on the specifics. What’s more, it’s one thing to read about it, and something completely different to actually do it. Once we started doing it, there were some very basic questions that immediately popped up: What do we do with an odd number of people? How often should we rotate pairs? How are pairs formed? How often should we be switching drivers? Is it normal to be exhausted after several hours of pairing? etc, etc, etc. With questions like that in mind, I decided to pick up a copy of Pair Programing Illuminated. I was hoping it would provide me with the tips and experience of veteran pair-programmers and jump-start our experience.

Unfortunately, it failed in that respect.

Pair programming really needs to be experienced to be fully appreciated. Just a few years ago, I loved my single office and I was completely against the idea of spending all my time programming with somebody else sitting at the same computer. Today I advocated using pair programming at work and I gladly gave up my office to work in a pair-programming lab alongside the whole team. Funny how things change.

First I started to warm up to the idea by observing how useful it was to sit down with someone and work with them on a piece of code, either debugging it or working through the initial design. Then it finally hit me when I realized that most of the problems that we have as programmers are not due to lack of skill or technology, but rather to lack of communication. Two programmers who communicate well and work very closely with each other will get a lot further than two people who shut themselves in their office, refuse to talk, and are hostile about anybody touching their code. When you only have a couple of people, it might not matter. But as teams grow and we have eight, ten, or more programmers, then communication becomes a much bigger issue.

But pair programming goes much further then just increased communication. It spreads knowledge through the team like wildfire. And that knowledge is not just of the code you’re writing, but also programming languages, design approaches, standards, tools, debugging techniques, etc.

Pair programming is also a key step towards having collective code ownership: code that everybody feels is his or her own and is willing to modify whenever it’s necessary. I’ve said it many times and I’ll say it again: the quality of a code base is directly related to how easy it is to modify. The more people feel that they know what’s going on, the more willing they will be to modify it whenever it’s necessary (and it’s much better to modify it in small steps than to take a huge hit and try to do a huge modification at once). The main ingredient of a healthy, easily modifiable codebase is unit tests, but that’s a whole other story.

Another consequence of pair programming is the reduction of sloppy or bizarre or just plain wrong code. It’s a lot harder for horrible code to happen when two people are at the computer writing something. Whenever the driver starts getting lazy and writing sloppy code, the other person immediately asks “What the hell are you doing? Shouldn’t we be doing this instead?” That’s usually enough to spur the driver into doing the right thing. Otherwise, he can always pass that keyboard and say “Fine, you do it.” I’ve been on both the giving and the receiving end of that situation, and it really works. In the end, you end up with a much better code base.

Finally, a consequence of pair programming that should not be underestimated is that it really fosters team spirit. Sometimes it’s hard to see that we’re all in the same team working towards the same goal. Extreme programming helps a lot in this regard, but pair programming helps a huge amount to get close and personal with your teammates and get to know them a lot better.

All right, going back to the book, where does it fit in? It has a fairly narrow focus, but somehow it tries to appeal to many people. In the introduction, they claim they are targeting developers who are thinking of trying pair programming (or convincing their bosses to do so), developers currently doing it, managers, QA, and even educators! I’m afraid that they over-reached a bit and spread themselves too thin on the ground.

The book is divided in four parts:

Part 1: Benefits of pair programming and how to sell it to your bosses and organization. This is the best-developed part, and this is where the strength of the book really is. Unfortunately, that’s not what I was looking for since we had already sold the idea of pair programming, but it could be useful for people hoping to roll it out in their organizations.

Of most interest are the studies that make a business case for pair programming and show how it really is not a wasteful use of resources. Just recently I was telling a friend how I was doing pair programming at work and his first comment was “Wow! I can’t imagine us doing that for budget/time-constraint reasons, but that sounds interesting.” But the whole point of pair programming is that it’s supposed to be better in the long run (and “long run” can be as short as a few months, and certainly less than one project cycle). This part will give you ammunition to debate this point.

Part 2: Getting started. This is what I really wanted to read a whole book about. It covers some things like the work environment and pair rotation. Unfortunately, it’s way too short and elementary. It covers a few of the basics and quickly moves on. This leaves the door wide open for another book on pair programming. If anybody knows of one that covers this topic, let me know.

Part 3: Tips and tricks. This section is completely forgettable. The book spends almost 100 pages (out of a slim total 260 pages) discussing different pair personalities: introvert-extrovert, introvert-introvert, gender and race issues, etc. Frankly, I didn’t get a single thing out of this section, which is very disappointing.

Parts 4 and 5: Miscellaneous stuff and case studies. I guess the book had to be bulked up a bit more, so it dedicates a chapter to extreme programming (go read Extreme Programming Explained instead) and another to Collaborative Software Process. Not much else of interest here either.

Given the lack of literature in pair programming, the book is definitely worth a quick read if you’re thinking of rolling out pair programming in your company. Part 1 will give you some good arguments to discuss with your managers and ease their fears. Otherwise, if you’re already doing pair programming there really isn’t much of value. It’s maybe worth a quick leafing through, but that’s about it.

In any case, if you haven’t tried pair programming, I really encourage you to give it a try. You don’t need to make it a sanctioned activity in your company. As long as you have somebody else willing to try it, you can both try the experiment of doing pair programming with each other working on both your tasks. You might be surprised that you manage to do both sets of tasks in the same amount of time it would have taken before, but with better quality, and, most importantly, you had a really good time in the process.

Exactly a year ago today, I put up the first article on Games from Within. It was a review of Tom DeMarco’s book Slack. I thought it would make for a nice, symmetrical bookend to wrap the year up with a review for another book by DeMarco: Waltzing with Bears.

As the subtitle indicates, Waltzing with Bears deals with managing risk in software development projects. Managing risk, not reducing risk, or removing risk. Do you think that low risk or even no risk is a good thing? Think again. One of the central points of the book is that a project with no risk is not worth doing. Yes, you read that correctly. Intrigued? Go and read the book right now.

I have to admit that the idea caught me by surprise as well. Maybe because I was used to the high-risk environment of game development, I always thought the best thing you could do for a project was to minimize the risk. But, as the authors point out, a project with no risk is not worth doing. It’s not going to enrich you in any way, and anybody can do it anyway. Their point is that risk and benefits are tied together. A project with a certain amount of risk will come with a certain amount of benefits.

Now that I’ve gotten quite a bit into playing poker, I see it very much like the analysis of expectation in poker. You want to bet when there’s a positive expected value, that is, when the potential benefits you’ll get from the bet outweigh the potential loss of money. As with poker, you can choose little risk and little benefit situations, or you can aim for high risk but huge potential payoffs (especially in no-limit poker). What you want to avoid are dead-end situations with high risk and very little payoff.

In particular, the authors single out death march projects as being of extremely low value.

Sounds strangely familiar, doesn’t it? The book then goes on to say:

Next time you’re deep into your third 60+ hour week in a row, think about those statements again and try to look at the big picture of your project in a new light. That just adds more fuel to the fire to my arguments against extended overtime and death march projects in general.

There are some things you need to watch out for if you’re going to start directly managing risk in your organization. The biggest problem you might encounter is that to manage risk you need to be brutally honest about the current situation of the project and possible negative outcomes. In a company where risk management is not a usual activity, it may come across as being overly negative, and you might scare away developers or even upper management. It can be particularly dangerous to do in a company with a very macho, “can-do” attitude (also described in the book as testosterone-based decision making), which is unfortunately all too familiar in the games industry.

One of the tools used throughout the book to describe projects and risks is probability curves. This emphasizes how things are not black and white. You don’t necessarily answer the question “when is the project going to be completed?” with a firm date, but with a probability curve. Looking at it tells you how likely the project is of being completed by a particular date (assuming the analysis was done correctly). The more “noise” and uncertainty there is in the project, the wider the curve is going to be. Unfortunately, it seems that it’s describing the games industry to a T again, because we tend to have a lot of uncertainty in our projects with moving technology targets and a somewhat whimsical market.

Another very interesting observation brought up in Waltzing with Bears is the concept of “early delivery.” Of all the games you worked on, how many of them were delivered early? Yeah, that’s what I figured. Same thing with me. Not a single one. Early delivery is seen as somewhat of a taboo. As if you didn’t work hard enough or you could have done a better job otherwise. So clearly, companies will never deliver a project earlier than the commit date. At best, they’ll do it by the exact deadline. At worst, the project will slip. As long as the two only possible outcomes are “on time” or “late,” then projects are very likely to keep being late. That’s some food for thought.

Later in the book, in the chapter about risk mitigation strategies, the authors make one very astute observation that I’ve seen repeated over and over in the projects I’ve worked on: projects that finish late are almost always projects that started too late. Truer words were never spoken. There are often very good reasons why a project can’t get started when it should (helping out another project, lack of funding, etc), but those weeks or months lost early on would be precious towards the end.

Perhaps it’s because I have a hopeless breadth-first personality, but I hate when books try to cover every single insignificant detail from the very beginning before giving you the big picture. I would much rather get a couple of passes, each building on what I learned in the previous one. That’s exactly what Waltzing with Bears does, with the first 50 pages dealing with an introduction and motivation to risk management, and the remaining 140 pages covering the same material in more depth and giving it a more advanced treatment. More books should learn from that organization.

Just go ahead and read the book and then pass it on to your manager. It’s short and well worth the read. The important thing is that somebody in your project should be actively managing risk and balancing how much risk you take versus what the potential benefits will be. Otherwise you might run into some very ugly situations that might lead to the cancellation of the project itself.

May 2005 be everything you hope for, and may your projects be managed wisely, with just the right amount of risk. Happy holidays to everybody!

That would be a funny anachronism if it weren’t still so true. OK, so game companies are not asking potential candidates to risk death every day, but they surely are asking them to give up their lives to the company.

I really wish I had saved some of the truly horrible job listings that I have seen over the last few years to contrast it to the Pony Express one. Instead, the “best” one I was able to find on short notice was one with the following requirements:

In itself, there’s nothing really wrong with those requirements. Heck, I consider myself to be a good fit based on that description. Unfortunately, this is more what the job listing is really looking for:

In this second part I argue that long hours in game development are not only something that could be avoided, but that they’re actually detrimental to the project.

Are long hours inevitable?

I want to start by debunking fourth myths:

Myth #1: There are lots of young kids out there willing to take our jobs.

That’s very true. However, most of those “kids” have no idea what the job is really about (hint, we’re not playing games all day long), and they’re hopelessly underqualified. So if a company decides to replace its staff with lots of young, inexperienced developers right out of college, well, that’s their choice. I doubt they’ll manage to create a decent game, and they surely won’t be able to make a second game with the same staff after burning everybody out. Ironically, this comes after the announcement that EA made in response to ea_spouse’s blog that they were planning to do 75% of their hires from developers straight out of college.

The modern version of this myth is offshore development. In some circles it has become quite a paranoia. It hasn’t hit us hard in the games industry yet, although art is the area seeing most jobs going abroad. I don’t want to get into an offshore debate here, but suffice to say that the jobs easiest to replace are those that can be done fairly mechanically, which are in turn the ones that can be done best during major crunch without a clear head.

Myth #2: If you’re getting paid enough, you should be willing to work any amount of time.

So maybe this one is actually true. If someone is willing to pay me enough in one year that so that I don’t need to worry about money for the rest of my life, then sure, they can have my undivided attention 24/7 for a full 365 days. No questions asked. Most of the time though, we’re just talking about a few measly tens of thousands of dollars extra. Good money to be sure, but it doesn’t even come close to the value of spending time with my family, being outdoors, or just doing different activities.

So no, getting paid a good amount doesn’t entitle the company to my full devotion.

Myth #3: If the company pays for overtime, then everything is OK.

Paying for overtime might be a good start, not because of the money (see myth #2), but because it would discourage companies from applying crunch time at the drop of a hat. Maybe they would start looking into alternatives (cutting features, delaying the release, changing design, etc).

But the point is, even if all the extra time is payed for, it won’t make people any more productive or make them do better-quality work. It’ll just make it so some of them don’t complaint so much. In the end though, most people would agree that receiving a lot of money and not having time to spend it is no way to live. It might be an OK short-term situation, but certainly not something to keep up indefinitely.

Myth #4: We’re doing games because we love it, so we have to put up with any hours.

It’s true that most people are working in the games industry because they love what they do. Game development has been my hobby for as long as I can remember, and I’m currently as close to my dream job as I’m ever going to get. That doesn’t mean it’s not a job though. We’re professionals working for a company and getting paid to do a specific job. In the same way, I love cycling but it doesn’t mean I want to do it 12 hours a day every day (going out with the club on an hilly 80-mile Saturday ride is plenty, thank you

At the same time, it doesn’t mean that we’re counting the minutes before we can punch out at 5PM. Sometimes what we’re doing is fascinating and we want to spend some extra time. As long as people are still rested and motivated, that’s great.

So I claim that there’s nothing different about the games industry that makes long hours inevitable. The only remaining, and most important, question is whether long hours are actually useful and help ship a better product.

Are long hours useful?

I’m convinced that developing the technology for a game is like running a marathon. I really think that the analogy carries very far. Notice also that I qualified it as applying to the development of the technology, not the game as a whole. Design and art have some of that touchy-feely creative mystique that can be very bursty. Heck, some of the best art in the history of humankind has been done while under the heavy influence of drugs and totally altered psychological states. So for all I know sleep deprivation actually helps. I’m going to limit myself to talking about technology development, which is what I’m familiar with.

Just like in a marathon, you start out a project knowing that you have to accomplish something (ship a game), in a rough timeframe (2-3 years, or whatever the plan is). It’s true that in game development there are more unknowns and changes along the way (although there can be quite a few in a marathon as well—unexpected hills, cramps, dehydration, etc), but the same general principle applies.

The way game development seems to work is by snoozing through the initial gunshot. Instead of settling into an early even pace, we just doze off at the start line or do some stretching. Then, after a while, we start to leisurely walk down the course. We don’t worry because we’ve promised that we’ll hit the first timed spot by minute 30. As the clock rolls around to minute 25, we realize that we aren’t going to make it there by the time we promised at this pace, so we sprint for a couple of miles and manage to make it there in quite a heroic sprint. Mission accomplished for the moment. We sit down. Take a break. After all, we worked very hard. Then we repeat the same cycle, but each time less walking and more sprinting. Soon we start not meeting the times we wanted. Eventually, we’re faced with a situation that in order to complete the race in the time we wanted, we would have to sprint like hell for the rest of the course. So we actually go ahead and try it (knowing perfectly well that we’ve never been able to sprint like that for an extended period of time). Of course, we get really tired. We need to stop. We move back our estimates. We try it again. Eventually, if we’re lucky, we manage to make it to the finish line. Much later than we had hoped for, but we’re extremely proud of how much effort we put into it. “I gave it 120% percent in the last 5 miles, man!”

OK, enough of that story. The point is that the best marathons are run at a very constant pace. People who go too slow (or too hard) early on end up paying for it in the end.

Fine. So “crunch time” doesn’t work for marathons, but what about for games? Does the snooze-and-sprint schedule pay off in game development?

From personal experience, I have to say the answer is absolutely not. I can totally feel how productivity starts dropping for me after 35-40 hours of work per week. There’s still some value in working about 60 hours because I will get more work done than in just 40 hours, but only about 10-20% more because the amount of work completed doesn’t scale linearly. After about 80-100 hours, productivity doesn’t just get slower, but it actually starts going down. Mistakes, bad decisions, and lousy quality quickly overwhelm any benefits from the extra hours.

So, is working 60-hour weeks best from a company point of view? Not really. If I try to keep the same schedule a second week in a row, productivity is much lower, and it drops off much more quickly. By the third week, we’re already in negative productivity territory after about 40-50 hours. The insane schedule from previous weeks takes its toll really quickly and it’s clearly felt. This is a rough plot of productivity vs. work hours. Keep in mind that I pulled the numbers out of thin air and they’re just based on personal experience.

We also need to consider other consequences apart from pure productivity gains. We’re not machines (I’m not anyway). The third week in a row of crunch I’m basically working at half efficiency, and I really dislike that. I can’t think clearly. I see myself doing sloppy work. I just want to get done with stuff. All those things don’t make me any happier to have to spend countless hours at the office and take a big bite out of morale.

Am I alone feeling this way? Apparently not. Plenty of other people seem to agree. It seems pretty well documented that productivity plummets very quickly after a certain amount of hours. This quote from the c2 Wiki summarizes it all for me:

The story that brought work hours home to me was one Ward tells. The C++ team was working nights and weekends, the Smalltalk team went home (tired) at 5 and rested all weekend. Management wasn’t happy with the results of either team, and pressured the Smalltalkers to work more hours. Ward’s response, “We would if it would help.”

Similar conclusions were reached in studies quoted in Tom DeMarco’s classic Peopleware and Slack books.

When I can work around 40 hours per week, I come in every day to work energized and looking forward to the day. I get tons accomplished. And when I get home, I actually spend quite a bit of time learning about related topics, working on different projects, or just letting my mind wander and do free associations. I tend to have most creative breakthroughs when I’m relaxing and doing something unrelated to work: riding my bike, showering, or even falling asleep (never go to bed without a notebook on the nightstand!). None of that happens as soon as the work hours start to increase. It might be a very small short-term gain, but a large long-term loss.

In conclusion, a short, focused, period of crunch time to reach a particular goal is probably fine and can be used to good effect. Anything longer than a week or two, and it starts having very negative consequences. In any case, you also need to make sure people get good rest afterwards, otherwise they won’t be able to go back to working at full efficiency.

Some people will argue that a two-week crunch every so often is good for the morale. I disagree with that. Who is going to be more proud of what they accomplished, the team who crunched for two weeks, or the one who didn’t have to do any crunch at all and met the same goals?

How to avoid long hours?

All this should be completely common sense, but I’ll reiterate here anyway.

• Don’t assume long hours are inevitable. Do what you can to avoid them.
• Don’t waste time. Don’t re-write things you don’t have to. Don’t spend half the day talking by the coffee machine. Don’t play Call of Duty for 3 hours at lunch.
• Don’t procrastinate. An hour of work now is just as precious and valuable as an hour before going gold. Do what you have to do now.
• Prioritize your tasks (or get your manager/boss/designer to prioritize them for you). Be ready to drop the least essential tasks if you start running out of time.
• Work with a sense of urgency. Large amount of pressure are no good and they tend to paralyze people or make them work in fear and make mistakes. On the other hand, no pressure at all tends to encourage people to take detours, and spend forever to get something perfect (as opposed to good enough to do what it needs to do). A small amount of pressure is perfect to keep the pace up.
• Use agile development. It’s a great way to keep re-prioritizing your goals and hit moving targets.
• Work hard for 40 hours. Play hard by going home at the end of the day. Maintain a sustainable pace.

Conclusion

The most important thing to keep in mind is not to do something you’re not happy with. Game development is a fascinating activity, but it doesn’t mean you have to give up the rest of your life to it. Anybody should be able to work in games and keep a reasonable life outside of work. If things are not reasonable at your company, bring up the subject and try to change it. Show them this article or some of the links mentioned here. If all fails, there are other companies out there that would be happy to have you.

This article has been a long time coming. It should be abundantly clear from past articles I’ve written (and from my rants if you know me in person) that I feel very strongly about quality of life issues in the games industry. It pains me to see rampant overtime be commonplace, and the truly ironic part is, I’m convinced it doesn’t help the final game any. As a matter of fact, it probably makes people be less productive and makes the game suffer for it. Ah, but they crunched some impressive hours. They have something they can feel proud of.

In the meanwhile, people are finally starting to wake up and say enough is enough. As most people are probably already aware (after hitting Slashdot and all the major game news sites), a few days ago the brutally honest blog entry from ea_spouse caught everyone by surprise. To throw fuel in the fire, another great entry by EA ex-employee, Joe Stratiff, quickly followed. Judging by the amount of attention they’re getting, people are clearly very interested.

It has since then made it to the national news, hitting the New York Times, the San Francisco Mercury, and many other national newspapers. I suspect the effects of these developments will be felt for the new few months with other follow ups in Game Developer Magazine and GDC presentations. There is even talk of a game development union. Whatever the direct results, those blog entires already accomplished a lot: raising the awareness of the working conditions of the industry. Putting an ugly truth in front of people’s faces.

The state of the industry

Those blog entries only dealt with EA. What about the rest of the games industry? EA is the major player in the games industry. They are the largest game development company, and they are one of the main publishers. In other words, they’re the big bully in the playground. If they do something, chances are it’s going to affect other people. Even worse, if they do something and it ends up being successful, other companies are quickly going to follow suit, or at least use it as a justification for their decisions.

But unfortunately rampant overtime and major crunch mode are not limited to EA. Most game development companies expect some level of crunch, usually about 65-80 hour weeks for up to a couple of months at the time. That’s not a one-time crunch either, but repeated multiple times during the development cycle.. What is worse, a lot of companies operate in a constant “crunch” mode, with 60-hour weeks being the norm. You can get all the detailed stats and lots of insightful commentary from the IGDA quality of life whitepaper.

As is often the case, variety is good, and variety within companies in the games industry means that it is possible to find different working conditions depending of where you go. There are four types of companies with respect to crunch mode:

1. On one end of the spectrum, we have companies like EA that just require overtime as a matter of fact. To be fair, not all EA studios operate under that mode. Apparently the one mentioned in the blogs was the LA studio. Several people are reporting that the situation is very different in EA Canada for example.
2. Some others will only require overtime when the project is in trouble of not meeting a milestone or a delivery, but of course, to anybody know knows how projects are run, that means that the project is in crunch for most of its development cycle.
3. Going further away from that, we have companies that don’t require overtime, although they certainly encourage it from their employees.
4. Finally, on the other extreme, some other companies either actively discourage people from working ridiculous hours, or they’ll accept it if somebody really wants to do it, but won’t encourage it in any way.

If there is a silver lining to all this mess is the fact that this is bringing companies in the fourth category to the limelight. IGDA has identified several companies, including Blue Fang Games and Firaxis, as companies that respect their employees’ lives and actively try to avoid crunch or overtime. Blue Fang has even been proudly advertising that fact in their job listings. I can only hope that more companies will stand out and identify themselves as being in the same category.

Perhaps that’s even something that IGDA can mediate, with game companies submitting their candidacy for the “quality of life IGDA seal of approval”, and IGDA can then send someone to verify their claims and add them to a publicly-available list of companies that avoid extended crunch time and meet all the quality of life requirements.

Finding a good company

Unfortunately, even though companies like that exist, they’re still far from the norm. You really have to dig deep to find companies in categories 3 and 4. Until IGDA creates an extended list, how do you go about finding one?

Word of mouth is a good approach. Network, find someone who knows someone who works there. Ask them about their work conditions, their typical work-week, how late people leave the office. You’ll find that most people are extremely open about it. One thing to watch out for is information that is several levels removed: If the roommate of their friend’s coworker had some bad things to say about a company, it might be greatly exaggerated by the time that information reaches you. Try to go with first-hand, or at most second-hand accounts.

Also, take anything that an ex-employee says with a grain of salt. Don’t ignore it by any means, just don’t base your whole decision on their opinion. People leave companies for all sorts of different reasons, and it often colors their opinion of the company as a whole.

If you don’t know anybody who works there directly, try asking in some of the industry forums. For example, if you’re already in the games industry, you might want to check out The Chaos Engine. They have a whole forum called “About Company X” just for people to ask opinions about different companies. They have quite a large membership, so it’s very likely you’ll find someone who is currently working at that same company. But the usual dose of salt applies: The forum uses a semi-anonymous format, so anybody is free to say whatever they want without consequences.

Finally, assuming you liked what you heard so far about a company, you went ahead and applied there. If all went well and you got called in for an in-person interview, this is your time to be pro-active about it. You will meet a lot of people during the interview. They’ll ask you a lot of questions, but they’ll also let you ask a lot of questions as well (after all, interviews are a two-way street–you’re trying to find out as much about them as they are about you). Make sure you come prepared with a lot of questions, but make sure that one or two of them are aimed to find out more about their crunch habits. Some good warm-up questions are “What is your typical work week?” or “What is the usual time for people to arrive to work and leave for home?”. If you get fuzzy answers to that, or you suspect there’s more than what they’re saying, ask them straight out: “When was the last time you crunched?” “How long was it and how many hours a week was it?”.

Make sure you ask those questions to everybody you talk to. You’ll be surprised how different people will answer the same questions sometimes. Don’t be shy about asking those questions too. Any employer who thinks less of you for asking that probably won’t try to avoid crunch time, so you probably didn’t want to work there anyway.

I’ve already noticed that people are asking about crunch time a lot more frequently during phone interviews since the whole EA Spouse blog hit the news. If nothing else, that is already doing some good to the industry.

Why does it happen?

So, why is crunch time rampant in this industry? Is there something inherent to games that requires bursts of activity? Some people will argue that games are a creative activity, just like movies, hit or miss, entertainment industry, yadda, yadda, yadda. That might be true for the design part, but there’s no excuse in the engineering part.

The ugly truth is that crunch is simply a self-fulfilling prophecy. It happens because people walk into game projects fully expecting to spend half of it crunching. That also means they can goof off for the first half. Call it procrastination. Call it human nature. Call it whatever you want, but that’s really what happens. It’s ingrained in the game industry culture due to its cottage industry roots. About 20-30 years ago, games were all about two kids spending crazy hours in somebody’s basement. For some reason, the industry is having a hard time letting go of that image, and that’s hurting everybody: the people suffering those hours, and the projects suffering those burned-out, tired people.

The worst part is that it really is so ingrained in the culture, that not only do people expect it, but they feel proud of it after it’s done (if they forget about the 20 pounds they gained during that time, the fact that their kids are two years older than they remember, and that they need to go to the divorce court next Wednesday). I am disgusted every time I read a postmortem in Game Developer Magazine or Gamasutra that lists “team pulled together and worked insane hours during months on end” under the “What Went Right” section.

As long as some people feel that way, it’s very difficult to break out of that mode. It’s very hard to walk out of the building when the rest of the team is in “hero” crunch mode, even if you know you’re not doing anybody a favor by staying more hours in the building. That’s particularly true when it’s your boss who thinks that crunch time is a natural part of the process and is essential to ship a quality game. Trying to keep sane hours can easily cost you raises, promotions, or even your job.

The million-dollar questions are, is crunch time really inevitable and does it really help a product? I’ll tackle those with my unpopular opinions and theories in the second part of this article.

No plan survives first contact with the enemy. In game development, detailed milestones, complex schedules, and careful planning often go out the window as soon as the project starts. Agile development provides a set of techniques to steer the project in the right direction and embrace change. Is your game not shaping up to be as fun as you thought? Has a game come out with features that you must match to remain competitive? Has your code degenerated into an unmanageable mess?

This talk discusses how agile development can help in all those scenarios. In particular we look at methodologies like XP and Scrum, and techniques such as test-driven development and pair programming.

Slides from the Montréal Game Summit 2004 talk (pdf format)

Yeee-haaa! Get your compiler and debugger ready and saddle up, pardner. We got some codin’ to do.

Cowboy coders

There’s no denying it: The games industry is full of cowboy coders. Sure, other industries have their share as well, but something about game development seems to attract them like flies to honey. Maybe it’s living dangerously under the pressure of milestones, maybe it’s being able to surf the bleeding edge of technology, and maybe it’s because it’s the last refuge left for the rogue programmer. Whatever the reason, chances are you’ve worked with a cowboy coder or you’re one yourself (although that’s more doubtful if you’re reading this of your own will).

No, a cowboy coder doesn’t live in a ranch in Texas and wear a funny hat. A cowboy coder is a certain type of programmer who prefers to work in isolation, sets his own rules, and usually dives straight into a problem without much thinking or design (hence the “shooting from the hip” expression).

But not everything about the typical cowboy coder is negative. He is usually very talented; he knows it, and he makes sure everybody around him knows it too. He usually has very little respect for everybody around him, and he’ll try to work by himself, with his own rules and schedules. He can sometimes do great feats and can handle some of the most complicated code in this side of the Mississipi. Just don’t expect anybody else to touch his code. Not only will he be horrified by the thought, but nobody else has enough “skillz” to be able to understand his beautiful code.

To him, speed is life. Getting things working as quickly as possible is all it matters, so he’ll be able to really quickly put enormous amount of code together that actually do something. Just don’t expect much in the way of maintainability, reliability, tests, standard-compliance or any of those boring things. It’s working, so what’s the problem, right?

Does it sound familiar yet? Unfortunately, I suspect it does.

Hero culture

So, why is the games industry so packed with cowboy coders? The analogy of flies and honey makes it sound too nice. I was going to use worms and rotting carcasses, but then I would really show my bias. I really need to step back a second. Are cowboy coders born or developed? It’s the old nature vs. nurture question. I’m going to stick my neck out and say it’s 90% nurture in this case. There might be a certain predisposition based on personality or character, but it’s mostly what the person is used to, what is comfortable doing, and what everybody lets him get away with.

Unfortunately, a predominant company culture in the games industry is the “hero culture,” which seems to be the perfect breeding ground for those cowboys out there. A hero culture is one that values individual displays of effort above anything else. It’s a very human thing to do. Think about it, who got the poem written about him? The hero who rescued the princess from the clutches of the evil dragon in a bloody fight in front of the whole kingdom, or the one who quietly and efficiently destroyed the dragon eggs years before, ensuring many years of peace in the kingdom? Go figure. It’s the same thing with many companies in the games industry. Individuals who are willing to commit body and soul to a project (or at least stay long enough in the office to appear to be doing lots of work) are the ones who get rewarded and promoted.

While some might argue that such a culture provides a healthy dose of commitment, team building, and competition, it usually has very different effects. A hero culture fosters too much of a “can do” attitude. It’s a great thing to stay positive and be willing to do things. However, taking the “can do” attitude to the extreme leads to overcommitment and either to spectacular failures or an incredible victory against all odds (that happens to crash as soon as you look away, but it works, right?). Look at the track record of canceled, over time and over budget games, and then at how many shipped games that need patches and you can decide for yourself if that’s a prevalent attitude in the industry.

One of the other consequences of a hero culture is the lack of progress visibility. A programmer takes on an impossible task and disappears in his cubicle for a few weeks. He spends there nights and weekends (but no mornings or early afternoons) and when asked he reports that he’s 90% done. Only the day before the milestone is due he admits he’s having some problems getting that last bit done. Before you know it the task really needs another three weeks to be completed. I’m sure the publisher is going to appreciate such short notice.

Tasks will also be consistently estimated (or guessed is more like it) way shorter than they should be because nobody is learning from past mistakes and everybody continues to provide ridiculous estimates. All of this leads to unrealistic expectations from the company, which gets passed along as promises to the publisher. From this to a canceled project is a very small step.

An interesting social dynamic created by this culture is that since the hero culture values individual effort above all else, it ends up preventing the team from gelling and working together. Instead, it remains a group of individuals more or less working on the same project. Sometimes less than more. Let’s not even go into burn out, what it does to individuals, and the cost of staff turnover.

If things are really this terrible, how did we ever get anything accomplished in this industry? Here’s the rub: A culture like that probably worked fine ten years ago. Actually, it might have been the ideal culture for a company then. Very small teams and small budgets. Dealing with complexity and large teams wasn’t as much of an issue as conquering the technology and packing stuff into tiny amounts of memory. Priorities were different.

The future and now

What about now? Is that an acceptable culture? Are cowboy coders the ideal game developer? My personal answer is a resounding no (like you haven’t guessed that by now). I really think that priorities have shifted and now there’s no room in the industry for cowboy coders or the hero culture.

A recent Game Developer Magazine article about Everquest had a very telling message. Everquest, by its nature, has to be always up, and there’s no room for screw ups and heroics. Here’s an excerpt from the article:

That sounds like they’re choosing productivity over flash and individual effort and heroics. If we think of Everquest as the way many game projects are going to go in the future, that sounds like the death knell for cowboy coders.

Steve McConnell, one of my favorite writers, has a great chapter in After the Gold Rush (the second edition is titled Professional Software Development) about programmer personalities and heroics. Fortunately, Gamasutra has that chapter online. In particular, the closing thoughts are quite revealing:

It really sounds like it’s time to start thinking of different approaches to game development. Fortunately, some companies are already dealing with this, and hopefully that influence will slowly propagate to the rest of the industry.

What if you have such a talented cowboy coder that he’s single-handedly moving the company forward? Someone as talented as, say, John Carmack. Is that still something to be avoided? If he really, truly is that talented and the difference between his productivity and everybody else’s is a factor of 100, then you’d probably do well to keep him on the payroll. Watch out for the proverbial bus that leaves you without your superstar though. On the other hand, don’t ask me to provide funding for a company with an unproven “superstar” like that. I’d rather take a solid team any day of the week.

In the end, it’s time for all the cowboy coders to realize that their time is over. Civilization is advancing even towards the game industry and team-based cultures are needed. The cowboy way of life is no longer advantageous. They should either adapt to a modern development approach, or head towards the new frontier of handheld games and ride into the sunset towards the wild territories.

When starting a project, it is always a good idea to reflect very critically on your past projects. Think about what went right, and especially what could have been done better, and come up with a plan of attack for the new project. That’s the idea behind project postmortems. To adapt the popular saying, companies that do not conduct some form of postmortem are doomed to repeat the same mistakes.

However, that might not be enough. Every project is different; if they weren’t, we’d be all set, projects would run very smoothly, and I wouldn’t be writing this. Even if you’re just creating a sequel of a franchise game, you’re still going to come across many different issues and challenges you did not encounter before.

This article attempts to extract some common patterns by looking at recent game postmortems. A later article will deal with how to choose the right development methodology for a particular game project given what we find here. This is not by any means intended to be a comprehensive summary of postmortems; rather, I’m just looking at the issues listed under “What went wrong” and trying to look for interesting patterns.

For a more comprehensive study of earlier postmortems, check out the great summary by Simon Larsen on Gamasutra.com Postmortems and his related thesis Playing the Game: Managing Game Development. I don’t agree with everything he says, but it’s a very interesting work anyway. Just the fact that people are finally starting to ask some pointed questions about the overall state of the industry and why so many game development efforts are total disasters is very encouraging. To gather data for this article, I looked through my back issues of Game Developer Magazine starting with April 2004 all the way back to October 2002 (which is where Larsen’s study left off). I excluded postmortems that dealt with subsets of the project (like tools, animation systems, sound systems, etc). Specifically, I selected 13 different postmortems: “Z-Axis’s Aggressive Inline” by Randy Condon (October 2002) “BioWare’s Neverwinter Nights“ by Scott Grieg, Ray Muzyka, James Ohlen, Trent Oster, and Greg Zeschuk (November 2002) “Monolith’s No One Lives Forever 2” by Craig Hubbard (January 2003) “Lost Toys’ Battle Engine Aquila” by Ben Carter (April 2003) “Insomniac Games’ Ratchet & Clank” by Ted Price (June 2003) “Big Huge Games’ Rise of Nations“ by Tim Train and Brian Reynolds (July 2003) “Harmonix’s Amplitude” by Greg LoPiccolo & Alex Rigopulos (August 2003) “Monolith’s TRON 2.0” by Frank Rooke (October 2003) “Relic Entertainment’s Homeworld 2” by Geoffrey Thomas, Stephane Morichere-Matte, and Joshua Mosqueira (November 2003) “Naughty Dog’s Jak II” by Daniel Arey (January 2004) “Totally Games’ Secret Weapons over Normandy” by Morgan Gray (February 2004) “Bizarre Creations’ Project Gotham Racing 2” by Garrett Young, Mario Rodriguez, and Chris Pickford (March 2004) “Ubisoft’s Prince of Persia: The Sands of Time” by Yannis Mallat (April 2004) One word of warning: This is not nearly enough samples to get any sort of representative idea of what the big problems affecting game development really are. It’s also a very self-selected group of projects: not only did they choose to write a public postmortem, but all the projects actually managed to ship a game and were relatively successful. However, what probably skews the results more than anything else is what the authors felt they could write in a public postmortem. Everybody had to be extremely cautious about what to say and how to say it, and I have no doubt that very important problems didn’t even get mentioned. Who would dare publish something along the lines of “our publisher had no clue what it was doing,” or “my boss is completely incompetent but we shipped the game in spite of him.” Still, I’m of the opinion that incomplete data is still better than no data at all in this case, so let’s proceed. Common patterns Resources/time It is a fact of this industry that, id software and a few others notwithstanding, we all have very hard deadlines. Most games commit to shipping by a particular date (Christmas or the Thanksgiving weekend being a couple of industry favorites) and have to do everything in their power to deliver or risk being canceled. It’s not surprising then that the number one problem listed in all the postmortems was that some features were not started until it was too late (Prince of Persia, Project Gotham Racing 2, Tron 2.0, Ratchet and Clank, Battle Engine Aquila, No One Lives Forever 2, Neverwinter Nights, and Aggressive Inline). Just about every aspect of the game was brought up in this respect: technology, tools, design, art, etc. In some cases it meant those features fell short of expectations, but most of the time it severely affected other parts of the development. When crucial tools for content development or key technology pieces become available just a few months before the shipping date you know that something didn’t go right. Two other postmortems (Homeworld 2 and Frequency) listed the flip side of the same situation: not enough resources. At least in that case they identified the lack of resources and tried to deal with it as opposed to just finding out that things were taking longer than expected. That means that 10 out of 13 postmortems brought up a similar problem. I wouldn’t be surprised if the other postmortems also felt the same problem but didn’t write a specific item in the postmortem about it. This is clearly a big problem in the games industry. When managing a project, you have three variables to play with: time, resources, and features; pick any two. Unfortunately it seems that people like to pick all three and deliver in none of them instead. Approval process The next most common issue that was identified as having gone wrong was the lack of an effective internal approval process (Prince of Persia, Homeworld 2, Tron 2.0, Frequency, Ratchet and Clank, Neverwinter Nights, Aggressive Inline). Some of the obvious examples of this problem are sub-par content making it into the game, or technology that appeared to be finished but wasn’t. But there were other manifestations of the same problem: feature creep that ruined the schedule, or overly ambitious designs that were not really feasible. All those are signs that the project was working without sufficient visibility and feedback. If the planned game levels are too ambitious, it should become apparent right away, not two months before beta. If tools are not working as expected, it should be noted right away. Closely tied to this problem was another common complaint: unnecessary rework. Several postmortems cited that as one of the causes for delayed schedules and massive overtime. When content was created on top of a piece of technology that had to be re-written, or based on some design that was changed, all that content had to be re-done from scratch. Nobody’s going to get things right the first time around, but there’s a not-so-fine line between useless rework and an iterative approach. Content pipeline So far I wasn’t too surprised by the patterns that were emerging, but this one caught me totally by surprise. A large number of the postmortems were complaining about inadequacies in their content pipeline (Prince of Persia, Project Gotham Racing 2, Secret Weapons Over Normandy, Jak 2, Ratchet and Clank, Battle Engine Aquila). The issues with the content pipeline ranged from not being able to deal with so many assets, iteration time being too long for the content creators, or not having a fully automated system all the way to the CD/DVD burning process. Interestingly enough, I wasn’t aware that other companies were struggling with this so much when I wrote the article “Optimizing the Content Pipeline” in the April 2004 issue of Game Developer Magazine. This is probably an area that will become more important in the near future and it would pay off for companies to explicitly define their content pipeline and streamline it as much as possible. Large teams Some projects clearly had trouble coordinating the efforts of their team members (Prince of Persia, Project Gotham Racing 2, Jak 2, Neverwinter Nights). Most often this was in the form of rework, as mentioned earlier. It seems that resources were often being put in areas before they were fully ready to go, or, alternatively, some areas didn’t have enough resources before full production started. Interestingly only one postmortem explicitly mentioned communication being an issue (Battle Engine Aquila). I would have expected that to be a much more common complaint. Either my experience is very different from those projects, or it was one of those taboo areas that people were not allowed to write about. As a point of reference, these are the sizes of the development teams that brought up team coordination as an issue: Prince of Persia: 65 (peak, excluding testers) Project Gotham Racing 2: 40 (core team), 102 (peak, including testers) Jak 2: 48 (full time) Neverwinter Nights: 75 (peak), 40 QA, 5 sound, 20 translators. We can’t compare those figures directly against each other (because they all count team size in slightly different ways), but clearly, those are some pretty heavy-weight teams. Most of the teams in the other postmortems were significantly smaller. Is this the way of the future? We better start getting used to it. Crunch time Why does this always come up when I write anything related to game development? It’s a very important topic to me, but I’ve tried putting it aside and only bringing it up a few times. Try as I might, it keeps rearing its ugly head every time someone in the room whispers the words “game development”. Surprisingly, only a few postmortems clearly identified crunch time and employee burnout as a negative aspect they had to go through (Secret Weapons Over Normandy, and No One Lives Forever 2). However, most of the other postmortems acknowledged that there was a fair amount of crunch time involved. The really scary part is that it usually showed up in the “what went right” section, pointing out how dedicated the team was, or how macho they all were that they pulled through it. Until the industry grows out of this basement coder mentality, we’ll continue having the same problems. I’ll spare you the lengthy rant and refer you to the wonderful job the IGDA folks have done putting together the Quality of Life Whitepaper. Draw your own conclusions from that and think about who will be making games five years from now. Other The rest of the issues brought up in the postmortems varied quite a lot. Surprisingly, several projects had trouble with localization. I thought that was a solved problem by now, although it’s probably a lot more complicated in dialogue- and movie-heavy games, or in games where sentences are created on the fly. A few of the other issues mentioned included QA problems (either bad testing or not enough), being side-tracked by demos, have unexpected events come up in the middle of the development cycle, or not having a clear objective for where the game was heading. One of the most insightful comments was brought up in the Battle Engine Aquila postmortem, where they pointed out that their engine was too flexible as a negative point. Having been there, I completely sympathize with that point. Even though it sounds like a great feature on paper, having a completely flexible engine also usually means not having an exceptional engine in any particular front. It also makes narrowing down the design constraints very difficult unless you have a very talented art and design team. Notably absent So far we’ve seen what common points were identified as being problematic. Now let’s have a look at all the interesting things that they didn’t tell us about. Technology/performance Considering the amount of time, effort, and trouble that programmers go through coming up with the most clever algorithms, optimizing inner loops, and trying to out-do each other with fancy technology, hardly anybody mentioned technology or performance being an issue. The only mention in the postmortems was when multiplatform development was involved (Secret Weapons Over Normandy, Battle Engine Aquila). Is it because people were too proud to say their technology let them down, didn’t do what they wanted, and was generally too slow? Or was it because there is too much emphasis placed on technology and performance to start with? Maybe we should concentrate on making our tools crash less frequently and our content pipelines be more robust, and just accept a slightly lower particle count. It will probably make for a better game. Team problems Is this another taboo area that public postmortems can’t touch? Probably. Only one postmortem mentioned having team problems (No One Lives Forever 2), and even then, it only referred to the hiring process and a few bad hires. Maybe everybody else had the perfect team where everybody got along great and worked together in perfect harmony. Suuuuure. I haven’t seen anything even close that mythical beast though. I guess you can’t call your boss’ wife ugly, but they could at least admit to some general problems. Quality and bugs Nobody complained at all about the quality of the code produced. They didn’t complain much about bugs either (other than mentioning bad QA process). I’m afraid this might be one of those things that the games industry takes for granted. I’d like to think that good code quality leads to very few bugs, very little (if any) overtime, and to a much better overall game because features could be tested and experimented easily with until the last moment. I’m convinced that a step in the right direction is to use automation as much as possible to test everything as frequently as possible, from unit tests that get executed every time any code is compiled, gameplay balance tests, random tests looking for crashes, or tests verifying that all objectives can be completed. What does it all mean? By taking a step back and looking at the problems most of these games were facing, we can actually see the cause for all those problems: complexity. I suspect that just a few years ago a lot of the problems would have been technical issues. Now we’re past that. We’re the whale that has difficulty breathing due to its own weight but we don’t yet fully admit it. Games are reaching a point of complexity where they require a huge team, which produces a huge amount of assets, needs to communicate and coordinate its efforts effectively, and create a great game in the end. Oh yeah, did I mention ship in time for Christmas? Edit: Ken Williamson emailed me with feedback about this article and brought up an excellent point that I had managed to completely overlook: Publisher interference. He writes: “The one thing I was surprised that didn’t come up was Publisher interference. By that I mean unreasonable and often misguided demands by publishers during development which derail production schedules and directly result in the crazy hours – and inevitably the nasty crunch times – common to games. I like to call this the “run faster” factor. In my experience these demands are the single most damaging aspect of the monolithic publisher/developer paradigm which the games industry seems to have followed the movie and music industries into. Money realities aside, it’s one of the reasons I think there just must be a better way to structure development.” I completely agree with Ken that publisher interference is a big issue that most teams have to deal with. Changing requirements, feature creep, and trying to copy the latest successful chart-topper are all too common occurrences. Unfortunately, this also falls in the category of taboo subjects that can’t be brought up in a postmortem. Companies have a hard enough time already securing funding and a publisher for their games, so they’re not about to bite the hand that feeds them.