Postmortems: Looking Back, Looking Ahead

To gather data for this article, I looked through my back issues of Game Developer Magazine starting with April 2004 all the way back to October 2002 (which is where Larsen’s study left off). I excluded postmortems that dealt with subsets of the project (like tools, animation systems, sound systems, etc). Specifically, I selected 13 different postmortems:

• “Z-Axis’s Aggressive Inline” by Randy Condon (October 2002)
• “BioWare’s Neverwinter Nights“ by Scott Grieg, Ray Muzyka, James Ohlen, Trent Oster, and Greg Zeschuk (November 2002)
• “Monolith’s No One Lives Forever 2” by Craig Hubbard (January 2003)
• “Lost Toys’ Battle Engine Aquila” by Ben Carter (April 2003)
• “Insomniac Games’ Ratchet & Clank” by Ted Price (June 2003)
• “Big Huge Games’ Rise of Nations“ by Tim Train and Brian Reynolds (July 2003)
• “Harmonix’s Amplitude” by Greg LoPiccolo & Alex Rigopulos (August 2003)
• “Monolith’s TRON 2.0” by Frank Rooke (October 2003)
• “Relic Entertainment’s Homeworld 2” by Geoffrey Thomas, Stephane Morichere-Matte, and Joshua Mosqueira (November 2003)
• “Naughty Dog’s Jak II” by Daniel Arey (January 2004)
• “Totally Games’ Secret Weapons over Normandy” by Morgan Gray (February 2004)
• “Bizarre Creations’ Project Gotham Racing 2” by Garrett Young, Mario Rodriguez, and Chris Pickford (March 2004)
• “Ubisoft’s Prince of Persia: The Sands of Time” by Yannis Mallat (April 2004)

One word of warning: This is not nearly enough samples to get any sort of representative idea of what the big problems affecting game development really are. It’s also a very self-selected group of projects: not only did they choose to write a public postmortem, but all the projects actually managed to ship a game and were relatively successful.

However, what probably skews the results more than anything else is what the authors felt they could write in a public postmortem. Everybody had to be extremely cautious about what to say and how to say it, and I have no doubt that very important problems didn’t even get mentioned. Who would dare publish something along the lines of “our publisher had no clue what it was doing,” or “my boss is completely incompetent but we shipped the game in spite of him.” Still, I’m of the opinion that incomplete data is still better than no data at all in this case, so let’s proceed.

Common patterns

Resources/time

It is a fact of this industry that, id software and a few others notwithstanding, we all have very hard deadlines. Most games commit to shipping by a particular date (Christmas or the Thanksgiving weekend being a couple of industry favorites) and have to do everything in their power to deliver or risk being canceled. It’s not surprising then that the number one problem listed in all the postmortems was that some features were not started until it was too late (Prince of Persia, Project Gotham Racing 2, Tron 2.0, Ratchet and Clank, Battle Engine Aquila, No One Lives Forever 2, Neverwinter Nights, and Aggressive Inline).

Just about every aspect of the game was brought up in this respect: technology, tools, design, art, etc. In some cases it meant those features fell short of expectations, but most of the time it severely affected other parts of the development. When crucial tools for content development or key technology pieces become available just a few months before the shipping date you know that something didn’t go right.

Two other postmortems (Homeworld 2 and Frequency) listed the flip side of the same situation: not enough resources. At least in that case they identified the lack of resources and tried to deal with it as opposed to just finding out that things were taking longer than expected.

That means that 10 out of 13 postmortems brought up a similar problem. I wouldn’t be surprised if the other postmortems also felt the same problem but didn’t write a specific item in the postmortem about it. This is clearly a big problem in the games industry. When managing a project, you have three variables to play with: time, resources, and features; pick any two. Unfortunately it seems that people like to pick all three and deliver in none of them instead.

Approval process

The next most common issue that was identified as having gone wrong was the lack of an effective internal approval process (Prince of Persia, Homeworld 2, Tron 2.0, Frequency, Ratchet and Clank, Neverwinter Nights, Aggressive Inline). Some of the obvious examples of this problem are sub-par content making it into the game, or technology that appeared to be finished but wasn’t. But there were other manifestations of the same problem: feature creep that ruined the schedule, or overly ambitious designs that were not really feasible.

All those are signs that the project was working without sufficient visibility and feedback. If the planned game levels are too ambitious, it should become apparent right away, not two months before beta. If tools are not working as expected, it should be noted right away.

Closely tied to this problem was another common complaint: unnecessary rework. Several postmortems cited that as one of the causes for delayed schedules and massive overtime. When content was created on top of a piece of technology that had to be re-written, or based on some design that was changed, all that content had to be re-done from scratch. Nobody’s going to get things right the first time around, but there’s a not-so-fine line between useless rework and an iterative approach.

Content pipeline

So far I wasn’t too surprised by the patterns that were emerging, but this one caught me totally by surprise. A large number of the postmortems were complaining about inadequacies in their content pipeline (Prince of Persia, Project Gotham Racing 2, Secret Weapons Over Normandy, Jak 2, Ratchet and Clank, Battle Engine Aquila).

The issues with the content pipeline ranged from not being able to deal with so many assets, iteration time being too long for the content creators, or not having a fully automated system all the way to the CD/DVD burning process.

Interestingly enough, I wasn’t aware that other companies were struggling with this so much when I wrote the article “Optimizing the Content Pipeline” in the April 2004 issue of Game Developer Magazine. This is probably an area that will become more important in the near future and it would pay off for companies to explicitly define their content pipeline and streamline it as much as possible.

Large teams

Some projects clearly had trouble coordinating the efforts of their team members (Prince of Persia, Project Gotham Racing 2, Jak 2, Neverwinter Nights). Most often this was in the form of rework, as mentioned earlier. It seems that resources were often being put in areas before they were fully ready to go, or, alternatively, some areas didn’t have enough resources before full production started.

Interestingly only one postmortem explicitly mentioned communication being an issue (Battle Engine Aquila). I would have expected that to be a much more common complaint. Either my experience is very different from those projects, or it was one of those taboo areas that people were not allowed to write about.

As a point of reference, these are the sizes of the development teams that brought up team coordination as an issue:

• Prince of Persia: 65 (peak, excluding testers)
• Project Gotham Racing 2: 40 (core team), 102 (peak, including testers)
• Jak 2: 48 (full time)
• Neverwinter Nights: 75 (peak), 40 QA, 5 sound, 20 translators.

We can’t compare those figures directly against each other (because they all count team size in slightly different ways), but clearly, those are some pretty heavy-weight teams. Most of the teams in the other postmortems were significantly smaller. Is this the way of the future? We better start getting used to it.

Crunch time

Why does this always come up when I write anything related to game development? It’s a very important topic to me, but I’ve tried putting it aside and only bringing it up a few times. Try as I might, it keeps rearing its ugly head every time someone in the room whispers the words “game development”.

Surprisingly, only a few postmortems clearly identified crunch time and employee burnout as a negative aspect they had to go through (Secret Weapons Over Normandy, and No One Lives Forever 2). However, most of the other postmortems acknowledged that there was a fair amount of crunch time involved. The really scary part is that it usually showed up in the “what went right” section, pointing out how dedicated the team was, or how macho they all were that they pulled through it.

Until the industry grows out of this basement coder mentality, we’ll continue having the same problems. I’ll spare you the lengthy rant and refer you to the wonderful job the IGDA folks have done putting together the Quality of Life Whitepaper. Draw your own conclusions from that and think about who will be making games five years from now.

Other

The rest of the issues brought up in the postmortems varied quite a lot. Surprisingly, several projects had trouble with localization. I thought that was a solved problem by now, although it’s probably a lot more complicated in dialogue- and movie-heavy games, or in games where sentences are created on the fly.

A few of the other issues mentioned included QA problems (either bad testing or not enough), being side-tracked by demos, have unexpected events come up in the middle of the development cycle, or not having a clear objective for where the game was heading.

One of the most insightful comments was brought up in the Battle Engine Aquila postmortem, where they pointed out that their engine was too flexible as a negative point. Having been there, I completely sympathize with that point. Even though it sounds like a great feature on paper, having a completely flexible engine also usually means not having an exceptional engine in any particular front. It also makes narrowing down the design constraints very difficult unless you have a very talented art and design team.

Notably absent

So far we’ve seen what common points were identified as being problematic. Now let’s have a look at all the interesting things that they didn’t tell us about.

Technology/performance

Considering the amount of time, effort, and trouble that programmers go through coming up with the most clever algorithms, optimizing inner loops, and trying to out-do each other with fancy technology, hardly anybody mentioned technology or performance being an issue. The only mention in the postmortems was when multiplatform development was involved (Secret Weapons Over Normandy, Battle Engine Aquila).

Is it because people were too proud to say their technology let them down, didn’t do what they wanted, and was generally too slow? Or was it because there is too much emphasis placed on technology and performance to start with? Maybe we should concentrate on making our tools crash less frequently and our content pipelines be more robust, and just accept a slightly lower particle count. It will probably make for a better game.

Team problems

Is this another taboo area that public postmortems can’t touch? Probably. Only one postmortem mentioned having team problems (No One Lives Forever 2), and even then, it only referred to the hiring process and a few bad hires. Maybe everybody else had the perfect team where everybody got along great and worked together in perfect harmony. Suuuuure. I haven’t seen anything even close that mythical beast though. I guess you can’t call your boss’ wife ugly, but they could at least admit to some general problems.

Quality and bugs

Nobody complained at all about the quality of the code produced. They didn’t complain much about bugs either (other than mentioning bad QA process). I’m afraid this might be one of those things that the games industry takes for granted. I’d like to think that good code quality leads to very few bugs, very little (if any) overtime, and to a much better overall game because features could be tested and experimented easily with until the last moment.

I’m convinced that a step in the right direction is to use automation as much as possible to test everything as frequently as possible, from unit tests that get executed every time any code is compiled, gameplay balance tests, random tests looking for crashes, or tests verifying that all objectives can be completed.

What does it all mean?

By taking a step back and looking at the problems most of these games were facing, we can actually see the cause for all those problems: complexity.

I suspect that just a few years ago a lot of the problems would have been technical issues. Now we’re past that. We’re the whale that has difficulty breathing due to its own weight but we don’t yet fully admit it.

Games are reaching a point of complexity where they require a huge team, which produces a huge amount of assets, needs to communicate and coordinate its efforts effectively, and create a great game in the end. Oh yeah, did I mention ship in time for Christmas?

Edit: Ken Williamson emailed me with feedback about this article and brought up an excellent point that I had managed to completely overlook: Publisher interference.

He writes: “The one thing I was surprised that didn’t come up was Publisher interference. By that I mean unreasonable and often misguided demands by publishers during development which derail production schedules and directly result in the crazy hours – and inevitably the nasty crunch times – common to games. I like to call this the “run faster” factor. In my experience these demands are the single most damaging aspect of the monolithic publisher/developer paradigm which the games industry seems to have followed the movie and music industries into. Money realities aside, it’s one of the reasons I think there just must be a better way to structure development.”

I completely agree with Ken that publisher interference is a big issue that most teams have to deal with. Changing requirements, feature creep, and trying to copy the latest successful chart-topper are all too common occurrences. Unfortunately, this also falls in the category of taboo subjects that can’t be brought up in a postmortem. Companies have a hard enough time already securing funding and a publisher for their games, so they’re not about to bite the hand that feeds them.