On paper it looks great:

• Built from the ground up for parallel builds
• Uses a real programming language (Python) instead of overly-complicated XML files
• It allows for discovery of assets to build as other assets are parsed

The big question mark hanging over it was whether it was fast enough. I had done some experiments with Scons and other build systems a few years ago, and Scons totally failed by having ridiculously long incremental build times just checking if anything had changed. That was simply not acceptable. If no data has changed, I want the build system to detect it and come back in a second or less.

Since then, I’ve been told that Scons had finally fixed its dependency checking and it was much faster now. Music to my ears!

So I resurrected the old script I wrote back then to generate a nightmare stress test, downloaded the latest stable version of Scons (1.0.1) and fired it up.

I’m sorry to report that Scons is still as unusable as it was back then. I generated a codebase like the one I used to measure all the build systems (50 libraries, 100 classes each, 15 internal includes, 5 external includes). I built everything once, then ran it again and it took 49 seconds. I have some pretty nice hardware, including a fast SATA hard drive and a 2.8GHz Core2Duo with plenty of RAM, and 49 seconds to say that nothing needs to be changed just doesn’t cut it.

Now to be fair, right now we’re looking at Scons to build our assets, not our code. Maybe the nested project configuration is not particularly realistic, so I tried something simpler: 5000 files in a single library without any dependencies among each other at all. This is a very conservative example for an asset build of a large commercial game.

• It’s only 5000 files. A real game can easily have 10 times as many asset files.
• All the files are tiny (cpp files in this case). Assets can get very large when dealing with textures, sounds, and animations.
• There are zero dependencies among assets in this example. You’ll often have models depending on shaders and textures, levels on game entity definitions, etc.

Building assets doesn’t have to be the instant, no delay kind of build that I require building code. I expect most developers will be building assets locally, tweaking things, rebuilding, and trying them in the game. The faster the build is, the better, but a delay of a few seconds can be acceptable. If Scons can’t handle this build in less than 10 seconds, it’s certainly doomed in a full-scale game.

The result: 37 seconds!

For purely masochistic reasons, I decided to see how it scales, so I threw 20,000 files at it (again, without any dependencies). Incremental build time without any changes: 3 minutes and 46 seconds. That’s over 6 times longer for processing 4 times the number of files, so it doesn’t even scale linearly.

Scons is definitely too slow, and MSBuild has its share of problems. What’s left out there? Any recommendations for good parallel asset building systems? Hopefully something lightweight and easily configurable through a real language. Anything?

Size

Charles and I were priding ourselves in keeping things small and minimal. But truth be told, it’s not like we were keeping track of how many lines of code we had written. Were things as small as we hoped they were?

The most convenient way of counting lines of code that I know is CLOC. It’s an extremely easy to use open source program which counts the lines of code in a codebase, gives very detailed information, strips out whitespace, breaks things down by language, and does just about everything you’d want from a program like that.

Running it on the latest version of our code (not including any 3rd party libraries) produces this:

    1621 text files.    1579 unique files.    3721 files ignored.
-------------------------------------------------------------------------------
Language                    files          blank        comment          code
-------------------------------------------------------------------------------
C++                            485          13577            303          46181
C#                             324           4935            712          22966
C/C++ Header                   407           4153             95          11975
MSBuild scripts                 18              0            126          1490
-------------------------------------------------------------------------------
SUM:                          1234          22665           1236          82612

Almost 60K lines of C++ code seemed very high. At first I thought it was because CLOC was counting files twice: once in their regular location and once in the .svn directory, but apparently it’s already removing all duplicates, so that wasn’t it.

Almost more scary than the amount of C++ code (which is all our runtime and some of our tools) is the amount of C# code. For a language that claims to be of significantly higher level than C++, that’s quite a mouthful of code!

Another surprising count in there is the number of lines with comments. Since we make heavy use of TDD, I really didn’t expect more than a couple dozen lines of code in the whole codebase. Still, I’m kind of proud that we have less than one line of code per file on average :-)

Here’s a more detailed breakdown, with the line count just for our runtime (engine and game):

1089 text files.    1053 unique files.    2338 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            441          11997            245          40943
C/C++ Header                   385           3964             90          11405
-------------------------------------------------------------------------------
SUM:                           826          15961            335          52348

and for our tools:

532 text files.     531 unique files.    1383 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                             324           4935            712          22966
C++                             44           1580             58           5238
MSBuild scripts                 18              0            126           1490
C/C++ Header                    23            199              5            591
-------------------------------------------------------------------------------
SUM:                           409           6714            901          30285

Tests

Then I realized that a good chunk of those were tests. So excluding all directories matching *Tests* gets the following result:

1206 text files.    1187 unique files.    4199 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            283           6464            150          22464
C#                             213           2636            534          12402
C/C++ Header                   380           3824             94          10782
MSBuild scripts                 12              0             84            978
-------------------------------------------------------------------------------
SUM:                           888          12924            862          46626

A bit more than half the C++ code consisted of tests. That’s pretty consistent with my experience with TDD. C# seems to follow a similar percentage as well.

As for the exact number of tests, running a grep for TEST shows all the C++ tests:

C:\pow2>grep -r TEST SweetPea Engine Tools | grep -v svn | wc   2163    3620  221953

And doing the same thing with [Test] brings up all C# tests:

C:\pow2>grep -r \[Test\] SweetPea Engine Tools | grep -v svn | wc   735    1470  52717

That means that our average C++ test is about 11.5 lines long, and C# tests 14.4. Frankly, that sounds rather high. We make heavy use of fixtures whenever possible and each test usually only checks for a single condition (even if it involves a couple check statements). I suppose that number is higher than expected because it probably includes all the lines from #include statements and all the fixtures as part of the average.

* If we only count cpp files, that goes down to 49%

I was curious about that last part of checking a single thing per test, so I ran a grep for the number of CHECK statements in our code:

C:\pow2>grep -r CHECK SweetPea Engine Tools | grep -v svn | wc   3886   15079  399598

That’s 1.8 CHECK statements per TEST, which is about right. Even though we’re checking for a single condition, we’ll often check a couple things about it (i.e. the camera stopped and it reached its final destination).

Build Times

So, given that amount of code, how long does it take to build it? Clearly it depends on your hardware. Since we’re not exactly rolling in money, we don’t have particularly powerful machines. Here at home, I’m using a modest Core 2 Duo E4300 (overclocked to 2.6 GHz) with fast memory and a relatively fast SATA hard drive, so that’s what I used for all my timings.

A full build of our game, plus all the libraries, all the tests, and running all the tests takes exactly 1 minute and 10 seconds. That’s pretty good for two reasons:

• When we work with the game we don’t build and run the unit tests for the engine. We have a separate solution for that. A full build of just the engine, the game, and the game unit tests only takes 43 seconds.
• The game itself is a fairly large project and devenv doesn’t know how to paralellize that build, so it’s only using half the available CPU power for about half the build time.

An incremental build after changing a single cpp file takes slightly over a second (including half a second of unit test execution).

As you can imagine, working with that codebase is a dream come true. Snappy, responsive. Nothing is hard enough that can’t be changed.

Unfortunately that’s where the fairy tale ends. The tools are another story altogether. Our C# tools, with all their unit tests, build in a mere 18 seconds, and the C++ tools in 1 minute and 10 seconds. That’s not too bad, except that it’s a surprisingly large amount of time for the C++ tools since there aren’t that many of them.

Here’s the kicker, doing another build without changing a single thing take 38 seconds. Whoa! We’re doing some C++/CLI trickery and apparently dependency checking is totally broken in VS2005 (either that, or we just don’t know how to set it up right).

Keeping things fast

What’s the secret of a lighting-fast build? Clearly, keeping the code size down is crucial. If your codebase is 2 million lines of code, builds are going to be painful no matter what. But they can be a little less painful with some gentle care.

One of the main build-time killers that we’re avoiding is the use of STL or Boost. Those libraries pull in everything and the kitchen sink, and their heavy use of templates make build and link time go through the roof. No thanks.

Our template use is pretty minimal. We have a couple containers (which I love and I’ll write about it one of these days) and that’s about it.

We’re pretty anal when it comes to keeping physical dependencies to a minimum. We forward declare aggressively, and we only include the headers that are necessary for each cpp file (PC Lint is “kind” of enough to remind us every time we have unnecessary #includes). We’re not using external include guards or #pragma once.

Precompiled headers are either not used, or kept to a minimum. I think the only project that uses them is the game and only for Havok headers. We don’t even have windows.h in a precompiled header (which would be a really bad idea because you’d be putting all the junk in windows.h available to your whole program).

Finally, we are using incremental links whenever possible. I remember a few versions of Visual Studio ago they were pretty broken, but they’re not giving us any problems. The only caveat is that if you modify a static library your program is linking with, it will force a full link. So they’re really only good for modifying the executable itself.

We’re not using any distributed builds. First of all, we don’t have enough computers to make it worthwhile. And second, I had horrible experiences with distributed builds in the past. They would help with a badly structured codebase, at the cost of longer incremental builds and mysterious spurious bad builds. Besides, once they’re in place, they tend to encourage even further disregard for keeping dependencies to a minimum.

How about you?

So, that’s it for the Power of Two codebase. How about you? Want to share your size, build times, or any other data?

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there :-) ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions.

None of that would be an issue except that TDD creates lots of unit tests and encourages almost constant incremental builds. In the past I’ve stressed how important it is to keep them fast. But this week that I was able to really feel what a dramatic difference slow unit tests make.

For many months, our unit tests ran under one second. Each library has its own set of unit tests, and so does the game and each tool. So even though we were writing lots of unit tests, we weren’t always running them all at the same time. There was a definite snappiness to coding: write test, compile, fail, write code, compile, pass, move on. Go, go, go.

As the game project got larger, we started accumulating more unit tests and testing times started creeping up into the second range. At the time, I couldn’t quite put my finger on what had changed, but there was a definite change in feel. Things weren’t quite as snappy.

Then, about a couple of weeks ago, we hit the two second mark and things definitely changed.

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 2.13 seconds.

As the tests were running, my mind would wander, thinking about what test I would write next, or whether the code I had just written would be cache-friendly, or whether we’d do Honey’s or Manhattan for lunch. I would be jerked out of my state of flow. It wasn’t just me, Charles felt it too. The difference in productivity was huge.

The Zones

For those of you laughing at those times and dismissing them as insignificant, you need to keep in mind that our build times are very short:

• An incremental build with just a change in a cpp file takes under a second (including link times).
• A full build of *all* our runtime code, including libraries, is around 50 seconds in a dual-core machine [1].

So adding on 2 seconds to the incremental build time is going from “practically instant” to “OK, I’m waiting…”.

Of course, none of this has anything to do with unit tests in particular. It’s total incremental build times (including compiling, linking, and running unit tests) that matter. An incremental build that takes 3 seconds with no unit tests is just as bad as one that is under one second with 3 second unit tests. They both interrupt the flow the same way.

I’ve often thought about how a company size affects communication and “feel”. It’s close to a step function with logarithmic scale: there’s a distinct jump from a company of 3 to a company of 10, to one of 35, to one of 100+.

I think something similar applies to build times [2]

• 0 to 2 seconds. Programmer stays in the flow. Productivity is way high.
• 2 to 8 seconds. Flow is broken. Definite feel of things being a bit painful and sluggish.
• 8 to 30 seconds. Attention wanders to other parts of the code, email, etc. Multitasking kicks in and overall productivity plummets.
• 30+ seconds. Builds seen as batch processing. After each build is started, some other action is started. Maybe even physically get up, get a drink, start a conversation. Full brain reboot between coding tasks.
• 5+ minutes. Programmers bang their heads against the desk, actively consider the merits of different forms of suicide, or at the very least start polishing their resumes.

The Fix

Having been spoiled by sub-second build times, neither one of us wanted to put up with the broken flow of 2+ second builds, so we decided it was time to fix things [3].

The good news is that our slow build times were solely due to running the unit tests. 2.13 seconds to run only about 1100+ tests is really bad, even on a so-so desktop machine like the ones we can afford. That’s almost 2ms per test! Remember that these are unit tests, not functional tests. So I expect to be able to run several thousands of these under one second. Something was way off.

Things had gotten much worse in the last couple of weeks, so it wasn’t a slow, steady creep up. Chances are we had introduced something very slow recently, which meant it was probably easy to fix.

As a first pass, we turned on the option to fail any test that takes over a certain amount of time in UnitTest++:

RunAllTests(reporter, UnitTest::Test::GetTestList(), NULL, 5); // fail any test over 5ms

That brought about tons of failing test, most of them added recently and creating one particular object in their fixtures. Digging a big deeper, that particular class had an array of 25,000 (yes, that many zeros) objects. Those objects were plain-old-data structures, but had a default constructor. Even though the constructor was simply initializing its members to zero, that was causing a significant slow down when so many objects were created. To make things worse, some of the code I had written was iterating over the array resetting the entries or shifting them up or down, repeatedly calling the constructor. Ouch!

A few minutes and lots of code hacking later, and the object didn’t call a single constructor anymore. Our test times went down by almost half. Victory!

We were already at 1 second, which is in the acceptable range, but since we were here, maybe we could improve things a bit more. After all, 1ms per test is still too slow.

UnitTest++ wasn’t reporting failing tests over 1ms very reliably. Different runs would cause very different results. Things were too spread out or other things in the computer were interfering too much.

About a fifth or so of our tests use Havok. Not with mocks, but directly. They each initialize Havok, create a world, and deal with a simple rigid body, constraint, or something of the sort. I suspected all along that the Havok system initialization and shutdown for each test was probably causing quite a slow down, so I changed it to be initialized only once before we ran any tests.

And that did the trick. With that change we were down to 0.16 seconds. Back to snappy, happy land!

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 0.16 seconds.

In release mode things are even better, but we do most development in debug mode, so that’s the really important one:

> bin\SweetPea.exe -unittestSuccess: 1145 tests passed.Test time: 0.06 seconds.

Perhaps sub-second build times are not realistic for all projects, but I think there’s a lot to be gained by keeping compile, link, and test times as short as possible. For tips on how to keep compile times down, check out these articles I wrote a while ago. If your unit tests are slow and you really can’t speed them up anymore, consider splitting them into two sets: one that is important or recent ones that runs with every build, and another, more comprehensive one that runs in the build server.

Even if you cut down from 20 seconds down to 10, you’ll be making a huge shift in how you’re able to work and iterate on your programming.

[1] We’d love to keep detailed metrics with each build: executable size, number of lines of code, number of tests, etc, etc. Unfortunately, when the rubber hits the road, that ends up in the “nice to have category” and we simply have no resources to spare on a two-man startup.

[2] For a whole whooping statistical population of one since I’m only talking about my experience. Still, I suspect that the same applies to many other programmers by scaling the function by some constant.

[3] Perfect use of agile mentality. Fix things when you need them fixed, not before. But make sure you do fix them when you need them. Otherwise it’s not agile, it’s just lazy.

C++ enums are great in practice, but they’re a bit lacking once you really start flexing them. Don’t get me wrong, they’re definitely a step up from #defines, but you can’t help but wish for more. Type safety is only so-so (enums get promoted to ints silently), you can’t forward declare them so you’re forced to have extra includes, you can’t split up a declaration of enums across multiple files (the compiler would have to do more work, but it would be awesome to be able to create uids for different classes in their own header files instead of a global one). But the most grating of design decisions is their scoping rules.
Take for example a set of enums describing the game flow:

enum GameFlowType
{
    Run,
    Exit,
    Restart,
    Restore,
};

Now functions can return a GameFlowType to indicate that the main loop should do. So far so good.

Except that, look at how it’s used:

GameFlowType DoMainLoop(bool render)
{
    //...
    if (someCondition)
        return Exit;
    return Run;
}

Ouch! We’ve managed to pollute the global namespace with totally generic keywords such as Run and Exit. What if I want to have an enum that controls what action an AI is going to take:

enum AIAction
{
    Enter,
    Exit,
    Stop,
    Walk,
    Run,
};

That won’t even compile because the symbols conflict with the GameFlowType ones. Even worse, it will compile just fine if they don’t happen to be included in the same compilation unit. So you might be fine all along until you write some AI code that has control over the game flow. Oops!
OK, so you can prefix all your enums like this:

enum GameFlowType
{
    GameFlowTypeRun,
    GameFlowTypeExit,
    GameFlowTypeRestart,
    GameFlowTypeRestore,
};

and

enum AIAction{
    AIActionEnter,
    AIActionExit,
    AIActionStop,
    AIActionWalk,
    AIActionRun,
};

That will technically fix the problem, but it’s a bit ugly, and we’re still polluting the global namespace (what if I wanted to have a class called AIActionEnter?).

Another potential solution is to score the enum inside the class that is using them. So we could have:

class AIAction{
public:
    enum AIActionType
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };
};

Much cleaner, do doubt, but also with its share of problems: First of all, it forces you to associate a set of enums with a class, which is not something you always want to do. But the biggest problem is that as soon as you have more than one set of enums per class, they all get promoted to the same scope:

class AIAction
{
public:
    enum AIActionBehavior
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };

    enum AIActionGroup
    {
        None,
        Self,
        Team,
        All,
    };
};

When we see one of those enums in code referred to as AIAction::Self, we really have no idea that it’s referring to that group the action is applied to, as opposed to AIAction::Enter, which is the type of action.
Our solution? Move all enums into a descriptive namespace.

namespace GameFlowType
{
    enum Enum
    {
        Invalid,
        Run,
        Exit,
        Restart,
        Restore,
    };
}

When you have a variable of that enum type, it’s listed as GameFlowType::Enum type, which clearly indicates what it does. And its values are referred to as GameFLowType::Exit. You get the ideal scoping, you’re not tied to any particular class, things are explicit but not verbose, you’re no polluting the global namespace, and it’s doesn’t affect the runtime or compilation times.
Sometimes the best solutions are the simplest ones.

A couple of months ago something unusual happened: The functional test failed but I wasn’t able to reproduce the problem right away. The failure was not a crash, but an object in the world ending up in a different state than expected. That’s always tougher to track down. To make things even more fun, was object was affected changed depending on whether the game was run from the command line or the debugger. Oh, and did I mention it only happened in release mode? I’ve got a baaad feeling about this!

My first instinct was to blame it on Havok. After some investigating, I narrowed it down to one of the enemies that was in mid-air. Sometimes it would fall down, and other times it would stay hovering, as if not affected by gravity. I knew that Havok was supposed to be deterministic, but it’s every programmer’s instinct to always mistrust other people’s code. I should have known better.

I spent a whole day tracking this down. Adding more and more information to the recorded world state to detect the problem as soon as possible. Slowly circling in; silently creeping up on the unsuspecting bug. Eventually, I had my “aha!” moment. I had finally found it.

A member variable, m_alive, in the RobotLogical class was never initialized! That was causing the Havok character controller not to be updated sometimes, causing the discrepancy in object state between the recording pass and the playback pass. What made it even more difficult is that m_alive is a boolean, so any bit pattern other than zero, it’s going to make it true, which is the initial state I expected for the robots. It never failed in debug mode because unused heap memory is filled with patterns like 0xdeadbeef or (the definitely less fun, but more practical) 0xcdcdcdcd, which all make the variable come up as true. Doh!

Yes, C++ sucks because of time-wasters like that, but it didn’t prevent me from feeling really stupid for having sunk a whole day on it.

All the code Charles and I write is done through TDD so we have almost 100% unit-test coverage [2]. How come Test Driven Development didn’t catch that? TDD is not about catching bugs, it’s about designing code. This is the type of error that is only triggered once every thousand times, so running one or two unit tests that cover that path probably won’t trigger it. That’s one of the many reasons end-to-end functional tests are invaluable. Go LeChimp!

Help, LeChimp! Help!

Detecting there’s a problem is good, but if it takes a whole day to fix it, things are going to be pretty painful. The more complicated the game gets, the more difficult life will be.

A good start is to crank up the warning level of your compiler as far as it goes. We’re compiling everything with warning level 4 in VC2005, so we’re already covered in that end [3]. Unfortunately, warning level 4 doesn’t warn us about uninitialized member variables. We need something else.

Fortunately we have a secret weapon in our arsenal we had been ignoring: Lint.

Lint performs static analysis of C++ programs and spews out lots and lots of warnings. Think of it as an extremely pedantic warning level on a C++ compiler. And I mean extremely! It will warn you not only of potential errors and dodgy constructs, but of best practices, and sometimes even of recommendations in books like Effective C++.

As you can imagine, with all those things to check against, Lint is going to find lots to complain about. So taming the output into something readable is a definite challenge.

When it comes to Linting your code, there are a few options available.

Team Edition Compiler

My friend Jim Tilander brought up that there’s a Lint-like tool hidden in the bowels of the Microsoft Platform SDK. Boy, he wasn’t kidding about the hidden part! It’s a whopping 1.4 GB download for a DVD image that then needs to be extracted and installed in order to get to the small free compiler. When will Microsoft adopt a more modular approach?

Once I got past all the installation hurdles, I was pretty excited to give it a test. A free tool is a free tool. And if it has good integration with Visual Studio, so much the better. Strictly speaking, it’s not really a Lint tool, but an optimizing compiler with a special /analyze switch, which performs a lot of the same checks as Lint. But as we know, a rose by any other name…

Right off the bat, it fell flat the moment I started using it. Apparently it doesn’t know how to deal with vcproj files. To work on all the files in a project, I had to feed each cpp file separately. That in itself is only a slight annoyance, but it meant that you also need to pass it all the compiler switches set in the vcproj files: include directories, defines, etc. For a Microsoft product, I really would expect it to understand vcproj files. Oh well. It’s not the end of the world. I can always write a quick script to do that for me.

Unfortunately, the next problem was more serious. There was no global way to control what warnings I wanted reported. I can disable specific warnings in the command line, but if I only care about a few, it gets really cumbersome.

Finally the killer: It simply doesn’t report some of the most important warnings I expect out of Lint, such as uninitialized member variables or variables appearing in incorrect order in the initializer list.

In its defense, it had some decent features I would expect of good Lint tools, like the ability to suppress certain warnings from code or to give extra hits in the code itself. On the other hand, the way to control the behavior of the analyze switch is done with special keywords, which will make other compilers choke, making it harder to write cross-platform code. Somehow, I’m not terribly surprised.

LeChimp’s recommendation: Even for a free tool, it simply doesn’t stand up to any serious use. Skip it.

PC Lint

Discouraged from the few hours I spent in the Team Edition compiler fiasco, I turned to Gimpel Software PC Lint.

Don’t be put off by PC Lint’s archaic web site that seems to be snatched straight out of 1995 [4]. PC Lint has been around forever and it shows. Not just in their web site, but in the program itself. Don’t expect any fancy GUIs, or dialog boxes, or any fancy cyber-amenities developed in the last 20 years.

Still, what it lacks in sophistication and pleasing visuals it more than makes up in functionality: Three minutes, $239, and a 4MB download later, I was ready to go.

In about an hour, I had PC Lint fully integrated with our build server and checking for problems in all of our source code. In just half an hour more, all our code was passing Lint’s rigorous inspection.

PC Lint can be quite intimidating out of the (virtual) box. If you run it on a typical codebase (or even a really high-quality one), it will treat you to megabytes of warning spew. Most of it useless, I have to say. So you have to take some time and tame it. Teach it to be a bit more polite, or more relevant. Or at least not to yell so much.

My preference is to start out with no warnings by starting out the options.lnt file with -e* and then add warning messages that I find useful: uninitalized member variables, unreferenced symbols, gotchas with virtual functions, etc..

Even running Lint with a small number warnings enabled, there were a few places in our code that caused PC Lint to complain but we didn’t want to change (Vec4 really shouldn’t initialize any member variables in its default constructor, or the NonCopyable class shouldn’t have a virtual destructor). In those cases, we can add a few discrete comments directly in the code that tell Lint to be quiet. We know what we’re doing, Lint. Really.

After we got all the code passing the initial set of checks, Charles dug into the Lint options file with a gleeful glint in his eyes and became the Lint-nazi, adding a good couple dozen extra warnings. It only took a few more minutes to get all the code to pass all the new checks. Frankly, the slowest part of the process is reading the thousands of different checks that PC Lint can do and decipher them (there are a couple of them I’m still unsure what exactly they mean).

This is what our options.lnt file looks like today:

-d__debugbreak()=
-e*

+e539  // Did not expect positive indentation from Location
+e549  // Suspicious cast
+e616  // control flows into case/default
+e744  // switch statement has no default
+e750  // local macro 'Symbol' (Location) not referenced
+e751  // local typedef 'Symbol' (Location) not referenced
+e752  // local declarator 'Symbol' (Location) not referenced
+e753  // local struct, union or enum tag 'Symbol' (Location) not referenced
+e764  // switch statement does not have a case
+e766  // Include of header file FileName not used in module String
+e767  // macro 'Symbol' was defined differently in another module
+e773  // Expression-like macro 'Symbol' not parenthesized
+e777  // Testing floats for equality
+e783  // Line does not end with new-line
+e784  // Nul character truncated from string
+e789  // Assigning address of auto variable 'Symbol' to static
+e796  // Conceivable access of out-of-bounds pointer
+e797  // Conceivable creation of out-of-bounds pointer
+e801  // Use of goto is deprecated
+e802  // Conceivably passing a null pointer to function 'Symbol'
+e803  // Conceivable data overrun for function 'Symbol'
+e804  // Conceivable access beyond array for function 'Symbol'
+e806  // Small bit field is signed rather than unsigned
+e814  // useless declaration
+e815  // Arithmetic modification of unsaved pointer
+e817  // Conceivably negative subscript (Integer) in operator 'String'
+e818  // Pointer parameter 'Symbol' (Location) could be declared ptr to const
+e820  // Boolean test of a parenthesized assignment
+e825  // control flows into case/default without -fallthrough comment
+e940  // omitted braces within an initializer
+e954  // Pointer variable 'Symbol' (Location) could be declared as pointing to a const

// TURN THESE ON IF WE CAN FIGURE OUT HOW TO APPLY IT ONLY TO VALUES AND NOT POINTERS!
//+e952  // Parameter 'Symbol' (Location) could be declared const
//+e953  // Variable 'Symbol' (Location) could be declared as const

+e1401 // member symbol 'Symbol' (Location) not initialized by constructor
+e1404 // deleting an object of type 'Symbol' before type is defined
+e1411 // Member with different signature hides virtual member 'Symbol'
+e1413 // function 'Symbol' is returning a temporary via a reference

+e1506 // Call to virtual function 'Symbol' within a constructor or destructor
+e1507 // attempting to 'delete' an array (not a pointer)
+e1509 // base class destructor for class 'Name' is not virtual
+e1511 // Member hides non-virtual member 'Symbol' (Location)
+e1512 // destructor for base class 'Symbol' (Location) is not virtual
+e1516 // Data member hides inherited member 'Symbol' (Location)
+e1520 // Multiple assignment operators for class 'Symbol'
+e1521 // Multiple copy constructors for class 'Symbol'
+e1534 // static variable 'Symbol' found within inline function in header
+e1535 // Exposing low access data through member 'Symbol'
+e1537 // const function returns pointer data member 'Symbol'
+e1538 // base class 'Name' absent from initializer list for copy constructor
+e1539 // member 'Symbol' (Location) not assigned by assignment operator
+e1541 // member 'Symbol' (Location) possibly not initialized by constructor
+e1542 // member 'Symbol' (Location) possibly not initialized
+e1543 // member 'Symbol' (Location) possibly not initialized
+e1544 // value of variable 'Symbol' (Location) indeterminate (order of initialization)
+e1545 // value of variable 'Symbol' used previously to initialize variable 'Symbol' (Location)
+e1547 // Assignment of array to pointer to base class (Context)
+e1552 // Converting pointer to array-of-derived to pointer to base
+e1554 // Direct pointer copy of member 'Symbol' within copy constructor
+e1555 // Direct pointer copy of member 'Symbol' within copy assignment operator
+e1556 // 'new Type(integer)' is suspicious
+e1557 // const member 'Symbol' is not initialized
+e1561 // Reference initialization causes loss of const/volatile integrity

+e1705 // static class member may be accessed by the scoping operator
+e1706 // Declaration with scope operator is unusual within a class
+e1707 // static assumed for String
+e1710 // An implicit 'typename' was assumed
+e1711 // class 'Symbol' (Location) has a virtual function but is not inherited
+e1718 // expression within brackets ignored
+e1719 // assignment operator for class 'Symbol' has non-reference parameter
+e1720 // assignment operator for class 'Symbol' has non-const parameter
+e1724 // Argument to copy constructor for class 'Symbol' should be a const reference
+e1726 // taking address of overloaded function name 'Symbol'
+e1729 // Initializer inversion detected for member 'Symbol'
+e1734 // Had difficulty compiling template function: 'Symbol'
+e1736 // Redundant access specifier (String)

+e1931 // Constructor 'Symbol' can be used for implicit conversions

I originally only had about a dozen warnings enabled, but, as you can see, Charles really went to town with it :-)

To integrate it into our functional test, we just set up a new target in msbuild. For each project, we call it twice: once to generate a lnt file for the project, and once to run it on the files for the project.

<PropertyGroup>
    <LintDir>..\Bin\Lint\</LintDir>
    <Lint>$(LintDir)lint-nt.exe</Lint>
</PropertyGroup>

<ItemGroup>
    <ProjectFiles Exclude="..\Engine\**\*Tests.vcproj"
                  Include="Src\SweetPea.vcproj;..\Engine\**\*.vcproj;"/>
</ItemGroup>

<Target Name="Lint" DependsOnTargets="CleanLint;RunLint"/>

<Target Name="CleanLint">
    <Delete Files="%(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt"/>
</Target>

<Target Name="RunLint" Inputs="@(ProjectFiles)"
        Outputs="@(ProjectFiles -> %(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt)" >
    <Exec Command="$(Lint) -v -os(%(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt) %(ProjectFiles.Identity)"/>
    <Exec WorkingDirectory="%(ProjectFiles.RelativeDir)"
          Command="$(MSBuildProjectDirectory)\$(Lint) -i$(MSBuildProjectDirectory)\$(LintDir) std.lnt %(ProjectFiles.Filename).lnt"/>
</Target>

Whenever PC Lint detects some code that violates one of its rules, it returns an error message, which fails the msbuild target and the whole build is reported as failed by Cruise Control .Net. I love it when things work exactly like you want them to without doing any extra work. To make it even better, because PC Lint can output errors in VC++ format, they’re understood and parsed correctly by msbuild and CCNet, so they show up in the CCNet log like any other error.

PC Lint runs surprisingly fast. It takes about 20 seconds to run on all of our source code. I was almost tempted to add it to the incremental build in the build server, but I really want to keep build times to a minimum there, so it runs once every hour, which seems to be good enough.

If you want to be more hands-on with PC Lint, it’s a breeze to integrate with Visual Studio. Just set up a few custom commands in External Tools and you’re good to go. The Visual Studio Lint options file even details exactly how to set up those commands.

My only minor complaint with PC Lint is the strange behavior of error 766 (Include of header file not used in module). I don’t know what algorithm PC Lint uses to determine that, but it seems not to report some unused headers. That wouldn’t be a big deal, but sometimes just rearranging headers or moving some code around will cause it to recognize a header as unused, triggering a failed build. It’s a very useful warning though, so I’m willing to live with that quirk.

Working with PC Lint is a pleasure. It’s really low level and old school, but that’s precisely what made it so easy to integrate with our code and our build system. It doesn’t need installation programs, databases, registry entries, dll deployments, or anything like that. Just a good-ol’ ini file with options and you’re good to go. We even have it checked it in version control to make it easier to deploy to our development stations and the build server.

PC Lint has the characteristics I want in a tool: Easy to buy, deploy, and integrate into the way I want to work with it.

LeChimp’s recommendation: An absolute must! Worth every penny. Two opposable thumbs up!

Other Lint Alternatives

Lint was originally a Unix utility developed in the late 70s. You would expect that lots of different versions would be available for modern compilers and platforms, but there aren’t that many. Maybe it’s not a really fun project to engage enough Open Source hackers?

I have to admit I haven’t looked at any other alternatives to PC Lint and the Team Edition Compiler that work under Windows. A quick Google search reveals there are a few tools out there. Has anybody tried them? I’d love to hear your experiences with them.

• Splint. Only works on C. Emphasis on security. Open source.
• CodeCheck. Works on C++. Commercial.
• A bunch for Linux/Unix. As much as I’d like to, the reality is that most game development happens under Windows.

Conclusion: LeChimp Rules The Day (Again)

PC Lint should be a requirement for any C++ project. The day you set up a build server (which should be the first day), go ahead and spend the extra hour setting up PC Lint as well. Be safe up front and avoid wasting any time down the line. We should have followed our own advice, but sometimes priorities slip a bit in the rush of getting a whole company off the ground.

Now I can finally sleep better knowing that LeChimp is keeping busy linting my code several times a day.

[1] This is our current broken build sound bite.

[2] There are some things we don’t bother TDDing: main functions, super-high level leaf code, etc.

[3] Of course, you need to have a clean compile in warning level 4 in order to be worth it. That means builds without any warnings or other spew. Incidentally, I do hate the setting to treat warnings as errors because it means you can’t temporarily have warnings while you’re refactoring between checkins, which slows things down.

[5] That was just a wild guess, but it seems it really wasn’t that different in 1996.

Bug Hunting

Unit tests, by their very nature, are limited to a single class or function at the time. There are all sorts of complex interactions between objects and systems that they simply can’t test. Even if you use mock objects and are extremely careful to test all your object interactions, there will be lots of unexpected cases and bugs that crawl out while running the game under real world conditions. Dr. Chaos is alive and well.

Besides, unit tests just test that the code does what you think it should do. So if the algorithm you have in your head is completely wrong, they’ll just make sure that the implementation is as broken as you imagined. Bugs like that might only surface once the code is put in the context of the whole game and interacts with other systems.

Traditionally, game companies rely on QA to uncover and squash most of the bugs created by complex interactions. You know, the type of bug that gets only triggered when the player enter the cave while there’s a blue moon and he spun around in place a seven times. Not only is this is an expensive process, but it’s not even very good at uncovering all the bugs. Most games have millions and millions of combinations of possibilities and interactions, and it’s completely impractical to try to run through all of them by hand.

Our Hero: LeChimp

At Power of Two Games it’s just two of us. So no QA or even interns to play the game endlessly. But, even in preproduction, we can’t afford to ignore those types of bugs. Instead, we enlisted the help of our hero: LeChimp.

LeChimp is our functional test server. It tirelessly runs the game every couple of hours and makes sure it loads and runs without any problem. Sure, ideally it should run more frequently, but LeChimp doubles up as our build server, and we don’t have another good computer to spare (maybe if y’all bought more t-shirts we could afford to buy another cheapo Dell).

Running the game is a good start. It checks that it’s possible to load every level and that nothing crashes. Frankly, that’s a good percentage of what QA does a lot of the time, and a lot of teams would really benefit from having such a simple test and know as soon as a level stops loading. Still, it we can do much better than that.

Monkey business

LeChimp runs the game for a fixed number of frames, and makes sure the game doesn’t crash or hits any asserts. But running the game without any action going on is not very useful, so it runs it with the -monkey switch, which feeds pseudo-random input to the game, as if a monkey were playing the game.

Actually, the input from -monkey is not random at all. I first made it truly random by generating new inputs through std::rand() every frame, but it looked more like a monkey on crack was playing the game and the characters shook violently back and forth and never managed to do anything interesting. Instead, now the monkey input holds the controller sticks and the buttons for some varying time interval, and it looks more like a monkey without its ADD medicine, which is clearly a step up.

When I first heard of this technique at a GDC tutorial a few years ago, I thought it would be pretty useless. How many bugs could feeding random input to the game really uncover? Shouldn’t we try to be doing something more intelligent to mimic how players interact with the game? But since it was really easy to implement I decided to give it a try. Boy, was I in for a treat! Within a few runs, it uncovered several major bugs that nobody hadn’t seen during their runs of the game. Since then, there isn’t a game I work on that doesn’t get treated with some monkey input love..

It’s so simple to implement that if you haven’t done it already, I really encourage you to go and do it right away. Do it over your next lunch break even. I guarantee you’ll be amazed at what it uncovers (or I’ll refund your money for this article :-).

Recording for posterity

In addition to just trying to crash the game, LeChimp records all the inputs and the state of the game at every frame. Then, after running the game for a while, it runs it again, feeding it the recorded input, and verifies that the game is in the same state as it was during the recording session (every enemy is in the same place, every prop has the same orientation, every player has the same health, etc). This verifies that the game is fully deterministic, that is, given the same inputs, it always produces the same output.

We’re planning on some cool features that rely on the game being fully deterministic, so this something very important to us. But even if we weren’t planning on doing anything with it, determinism is an extremely useful feature to have when it comes time to track down bugs. Instead of getting lengthy descriptions of a crash from testers (which somehow always seems impossible to reproduce), they can include the playback file that lead to the crash along with the bug report, allowing us to catch it in the debugger in no time. Of course, it’s not like we have actual testers, but we affectionately think of LeChimp as one, and he’s always very careful to save all his playback files with every functional test run.

Right now we’re just recording the inputs to the game: delta time every frame and controller inputs. Just with that, the game runs exactly the same time in and time out (thanks to Havok for being deterministic!). Things get more complicated as soon as multiple threads are involved, since the exact timing of context switches between threads can affect the output. Some tools out there, like ReplayDirector claim to address this, but I haven’t looked into it very much.

Both the input file and the same state file are opened, written to, and closed every frame. That way there is no data loss if the game crashes unexpectedly and you get all the input leading up to that frame.

As far as checking that the world state is the same, it’s totally an ad-hoc process. We simply pick some of the obvious state and save them to a file: player positions, enemy position, props transforms, etc. If we ever see something get out of sync, we add it to the game state that gets saved and compared so it doesn’t happen in the future.

No waiting around

So far we have LeChimp running the following with every functional test:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name

followed by

sweetpea -playback=functional_test -level=level_name

At 60 Hz, running 10,000 frames is almost three minutes. 10,000 is just a number we pulled out of a hat. The longer you let the monkey loose with the game, the better the chance of uncovering something. Multiply that by the number of levels and sandboxes, and the functional test now takes quite a while to complete.

A good chunk of the time of the functional test is spent rendering each frame and waiting for the vertical sync signal. But nobody is actually looking at the output [2], so why bother?

We added a couple more command line switches to make the game run without rendering or waiting on vsync. To make that really useful, we also added the ability to force the frame time to be a fixed timestep. So we can run the game like this:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name -render=no -vsync=no -timestep=0.01566

The game will cruise through the simulation as fast as possible, often cramming all three minutes of gameplay into 10 seconds or so. Perfect for poor monitor-less LeChimp.

Testing, testing

There’s even more to LeChimp that just monkeying around. It also runs several other functional tests checking some high-level functionality:

• Player attacks. The player character attacks enemies using each different type of attack and verifies that each attack is successful. This is particularly useful when there are a few types of attacks (which themselves are unit tested), but there are many combinations that can be created between attacks and targets.

• Level restart. LeChimp loads a level and then restarts it hundreds of times, checking for crashes and memory leaks.

• Torture chamber. A tiny level in which the hero (in god mode) frantically mows down hundreds of enemies that get immediately respawned. This is a perfect stress test for performance and hardcoded limits.

We’re writing these tests as we go along. Whenever there’s a feature that seems complex enough, or that it relies on other systems, or that it seems to break repeatedly, we take a few minutes, write a new functional test, and throw it to LeChimp to run with all the others.

Functional tests like these are about as high level as it gets. They deal with actions such as “move the player to the right”, “spawn an enemy here”, or “perform special attack XXX”, so it would make sense to implement them in the same way you implement game logic (which in our case is still C++, although we’re considering a switch to Lua in some not very distant future).

Long Live LeChimp

LeChimp has been invaluable battling against Dr. Chaos. Several times I made a refactoring or introduced a new feature, all the unit tests passed, I checked it in, and a little while later LeChimp screams at us that something is wrong. Once we see the functional test fail, it’s usually pretty obvious how to fix it: a memory pool is too small, or a combination of events that causes the player to enter some unexpected state. Fixing it is a matter of writing a unit test, fixing the logic, and checking it in, all in a few minutes.

A few times, however, the problem hasn’t been that obvious. The world gets out of sync, but only in release mode. Running it from the debugger often results in yet a different state. Sounds like some annoying memory overwrite, or perhaps some uninitialized memory. Any programmer who’s had to deal with this before probably has shivers running down his back. Fortunately, LeChimp has a secret weapon in its arsenal to deal with that. But that’s another story and shall be told another time.

Until then, happy holidays, everybody!

[1] Don’t underestimate the power of keeping programmers happy! At a previous company I used to work for, a manager rewarded one of the programmers by buying him a DVD set of a TV show he was really into. That was only about $40, but they had a huge effect on the programmer’s morale and productivity. Talk about well-spent money.

[2] LeChimp doesn’t even have a monitor, although that sucks sometimes. RemoteDesktop is pretty cool, but it locks up the DirectX surfaces in some weird way, and then the graphics renderer refuses to initialize correctly. So we’re forced to use… get this… NetMeeting! With fake phone rings and all! Ring, ring, calling LeChimp…

Of course, we’re far from loaded with money, so keeping expenses to a minimum is definitely a top priority. It’s not coincidence that one of our most popular lunches is sharing a gigantic $5 sub at Manhattan Giant Pizza or the $3.50 Kealani’s chicken teriyaki sandwich (fortunately, they’re actually delicious too!).

Whenever it makes sense, we’ve opted for the most inexpensive options[1]: plastic workbenches from Costco for our desks, lights from Ikea with unpronounceable names, an outdated P3 that became our file server given to us by a friend, or a free scanner/printer donated by my girlfriend.

So when it came time to set up our office tools, we also looked for the most inexpensive solution that met all our needs.

Email

Our requirements for our email solution were:

• Minimize costs. Not just the software, but the hardware needed to run any severs, support, etc.

• Minimize maintenance. We’re our own IT staff, so the less time we have to spend screwing around with severs and updates, the more time we can spend coding and designing the game.

• Reliable. This, after all, is going to be used to send company emails, contracts out to outsourcing companies, responses to publishers, etc. We don’t want to be missing any emails.

• Good search capabilities.

• Available from any location. I guess that implies a web front end.

• Fully integrated with our domain powerof2games.com.

In the past, the companies we had worked at all used Microsoft Exchange and Outlook or the god-awful Lotus Notes. Apart from having to endure the torture of using these day in and day out, they’re quite expensive, so we looked for alternatives.

We could use the email server provided by our web hosting company. It would give us the bare bones functionality that we needed, including some form of web front end. It could work, but after using Gmail for personal email for several years, it felt like a major step backwards.

Since Charles and I were already using Gmail for our personal accounts, we thought of creating a new account for Power of Two Games, but having pow2games@gmail.com in all our business emails doesn’t exactly look very professional.

That’s when we found out about Google Apps for small business. It’s a free service offered by Google that allows integration of Gmail into your domain and use it as your corporate email. That seemed too good to be true. Gmail is hands down the best email web interface I have ever used, so the thought of using it as the company email was extremely tempting.

Setting up Gmail took a bit of messing around with some DNS advanced settings. All email sent to powerof2games.com had to be routed to Google somehow, so we had to set up a custom MX record in our web host and point it to Google’s servers.

We also wanted to access our email through a nice URL like mail.powerof2games.com instead of some long (and hard to remember) default URL going through Google. For that, we had to add a CNAME record pointing the new mail subdomain to ghs.google.com.

Once all that was set up, Gmail was up and running and fully integrated with our Power of Two domain. Not only that, but it was totally independent of our personal Gmail accounts, so we can even have them both open in different tabs in the same browser. Good job, Google!

Calendar

Gmail was just the tip of the iceberg. By going the Google route, we also get Google Calendar for our domain. And Google Calendar totally rules. It’s not like we were planning on scheduling company meetings at all hours of the day (although, technically, since we’re in the same office I guess we’re in a permanent meeting). But Calendar comes in really handy to put down big milestones, remind us to pay the rent, or keep track who we’re having lunch with this week.

Calendar works out of the “box” without any extra set up. The only thing we did was to create a custom subdomain to access it through calendar.powerof2games.com, so we did the same trick as for our mail subdomain with the CNAME entry.

I can even share the Power of Two Calendar with my personal Google Calendar account and see all my events combined in a single place. My whole life in a single web page. Very handy!

Extras

But wait, there’s more. We’re not always working at the office at the same time. Sometimes we’ll work from home for a few hours in the morning before walking down to the office, or maybe I’ll do some work from E Street late in the evening with an oatmeal chocolate chip cookie and some live music. At those times, it’s very helpful to be within easy reach of communication, and since we’re using Gmail, we get Gmail chat for free.

Documents

I’m usually very hesitant of “integrated solutions”. I always prefer to be able to mix and match programs, components, or whatever instead of going the route of vendor lock-in. And the truth is that the different Google components are totally optional, but it’s curious that we’re biting, hook, line, and sinker all the way into Google’s office suite.

I was thinking we would end up using OpenOffice.org, which is a great Microsoft Office clone. But we gave a try to Google Documents and Spreadsheets and we were totally impressed.

Not only has Google Documents reached point as far as features and user interface that they’re usable for real-world situations, but they offer a lot more functionality.

• It’s a web-based app, so we can edit the documents from any computer with web access.

• We can collaborate on the same document at the same time, which came in very handy when we were both working on the Sony Developer Program application document.

• Because it’s a Google product, you can search through the contents of all your documents and spreadsheets very easily.

• Tagging and sorting. You can tag any document with any set of labels and then sort or filter by them. I’m glad the days of hierarchical trees for organizing information are coming to an end.

• Exporting to a variety of formats (MS Office, pdf, etc) comes in very handy when we need to send a document to somebody else who is not using Google.

There are only a few things we wish it did differently, like being able to upload a pdf document (even if it’s just in read-only mode for archiving and searching purposes). The other odd quirk is that we can’t make all documents and spreadsheets shared by default between the two of us. Instead, we need to explicitly share them. Oh well. I guess they have bigger companies than us in mind. I don’t blame them.

It’s very encouraging to see these applications grow and get better week after week. Just a few months ago, Google Spreadsheets couldn’t do plots, but they added that functionality since then. They’ve also improved the whole front end with the tagging and listing of documents.

Voice communication

How the times have changed. When we started Power of Two Games, we didn’t even bother with a phone. Even though we ended up contracting a DSL Internet connection and getting a voice phone line would have been extremely easy, we simply didn’t need it. We were simply using our cell phones for the few times when email communication or chat wasn’t good enough.

But as we started outsourcing art and talking to console manufacturers, we started having the need to do voice conferences and calling internationally. After a couple, very uncomfortable, conference calls using the speakerphone on our cellphone, we decided that we really had to look for a better solution.

For once, Google didn’t have the solution for us (although wait a couple of years and see). I have been using Skype to call internationally for quite a while with great results, so it was natural that we would look at it as our first option. Again, we were pleasantly surprised: not only could we make very inexpensive international calls (with great voice quality), but for a small yearly fee we can get SkypeIn and have anybody call us from a regular phone.

Conclusion

We’re extremely pleased with how Google Apps and Skype have been working for us so far. They fit our needs perfectly and the cost is right. We were able to set things up in about a morning and never have to fuss with it again. How good is that?

I really don’t see why Google Apps wouldn’t work for a larger company. But companies seem to be stuck in the Exchange/Microsoft Office (or some other heavyweight and expensive server solution) choice. I wonder if there’s a good reason for that other than inertia.

We’re certainly happy to have these services available for free and being about to spend our time and money in what really matters: Creating a kick-ass game and getting it out the door.

Notes

[1] There are some things where going rock-bottom cheap doesn’t make any sense though. The $7 desktop speakers we bought online were quite the surprise. I wasn’t expecting top quality sound, but they sounded like those two way radios that security guards use. I didn’t even know they made speakers that bad! Also, we have “splurged” on 24" LCD monitors and comfortable chairs (and even so, always trying to find the cheapest price point for what we want), because those things make a world of difference when you’re using them many hours every day.

Since both Charles and I are tech guys, we knew we could implement our game idea without any problem and make it sing and dance at a silky-smooth 60 fps. But would it be fun? Now that we have these brand-new designer hats, finding that always elusive fun factor is a big concern. The idea of waiting for a couple of months before we could tell if our game idea was fun seemed too risky, so we tackled that problem first and head-on by prototyping.

We could start at the bottom, write some low-level input handling, some graphics rendering, a basic asset pipeline, and all the other usual suspects. The problem is that, even if we try to keep things as simple as possible, it would still be several weeks or even months before we can actually start implementing the game itself. And even when we do, we’ll always be running up against incomplete technology and having to spend time fleshing it out as we try to make a game come out of the other end.

Instead, we decided to start with a prototype. This isn’t supposed to be a “prototype” that eventually morphs into a shipping game, or a prototype that uses the same technology as the production code, or even a prototype that’s used to impress the big-wigs to squeeze some money out of them (ha!). No, all that stuff detracts from the ultimate goal of our prototype. Our approach was very similar to what Chris Hecker and Chaim Gingold described in their GDC presentation. We had the need to answer one very specific question: “Is our game idea fun?” And we wanted the answer as quickly and cheaply as possible. Everything else was secondary.

Prototyping period

And so we dove into a prototyping frenzy. For our first day, we set up shop at the E Street Cafe. We opened up the place, took over a table, hooked up gamepads to our laptops, and hacked away furiously until late in the evening, fueled all the while by cappuccinos, gourmet tea, and pastries [1]. Yum! By the end of the day, we had the basics of the game implemented and even some of the more advanced features. Heck, we even thought it was kind of fun, but that’s because we were tired and we had been staring at Charles’ horrible programmer art all day long. In all honesty, it really was a pretty sucky game, even if the enemy characters were wearing party hats. But hey, that was just end of day one.

Over the next few days, we had to take a break to deal with all the paperwork and logistics involved with setting up the new company and the office, but we resumed prototyping as soon as possible. The plan was to keep hammering away at the prototype until we saw it was clearly fun, or we decided it was hopeless and we had better think of something new (or go back and beg for our jobs back). Charles took it even further and swore not to shave his beard until the prototype was fun, much to his fiancee’s delight (seriously, the only thing the ladies think is hotter than a game programmer is a bearded game programmer).

A week later, things weren’t looking too hot. The game hadn’t gotten much beyond what we had done the first day. Sure, we had a bunch of advanced features in, but it didn’t really make it much more fun. Were we on the wrong path?

The major breakthroughs happened in the last few days. First of all, we removed a key feature that had been there since the very beginning. We really thought it was going to be key to the whole game, but it really made the game too easy and boring. As soon as we removed it, the feel of the game changed completely and things started to look up.

Then, as we were discussing some other advanced moves and features, the idea for a minor feature came up. Since the caffeine hadn’t kicked in yet and I didn’t have the energy to work on the next big feature, I implemented our little idea in about ten minutes. Charles saw it over my shoulder and he started laughing and begging me to check in the code. It was an instant hit! We ended up liking it so much that we decided to expand on with a few features, and before the end of the day, it’s clear that it’s going to become a main gameplay mechanic in the final game. That day we went home knowing we were nearing the end of the prototype face. Good thing because Charles’ beard was starting to look scary!

We tried about seven different captions, but none of them did the sprite justice.

One more day of tweaking the prototype and we officially called it done. We knew we had gotten there when we tweaked a small feature, fired up the game to see how it felt, and then spent 10-15 minutes going crazy with the controller because we were having too much fun. Another good sign was when we would start playing and then yell at each other to look at the crazy things we had just accomplished.

Finally, Charles was able to shave. Thank God.

The prototype

The prototype process was supposed to be as quick and cheap as possible. None of the code written was intended to be kept or extended to go into production code, and performance wasn’t an issue either. So, with that in mind, what did we use for developing our prototype?

C++ and Allegro. Yes, you read it right the first time. Let me say that again: C++ and Allegro.

With all the other choices available of higher-level languages and extensive game development libraries and frameworks, why on earth did we make that choice? Wouldn’t Python with PyGame or C# and the Game Studio Express have been better choices? The honest answer is that I don’t think either one of those options would have allowed us to get the prototype done any faster for several reasons:

• Small scope. The scope and size of the prototype was quite small, so there wasn’t much to be gained by using slightly higher-level language constructs or more complete libraries.

• Allegro. Even though it might seem like an odd choice and be behind the times, it’s precisely the old-school, fully procedural approach that makes Allegro so valuable. There are no frameworks, no enforced class hierarchies… heck, no classes or private members to get in the way. In Allegro, if you want to do something, you call the function and you’re done. Exactly what we needed for the prototype. In retrospect, SDL might have been a better choice since it would have allowed us to use a DirectX surface and do video capture more easily. But Allegro worked like a charm and we have nothing but good things to say about that experience [2].

• Proficiency. Both Charles and I are quite proficient at C++, so even if another language gave slightly faster development time, we more than made up for it by being very familiar with C++ and its libraries. For the prototype we allowed ourselves to use STL recklessly and with wanton disregard for memory access patterns and spurious allocations.  We also didn’t care about memory leaks at the end, so that helped get things up and running a lot faster.  Destructors?  Delete?  Free?  Cache coherency?  How do those equate to fun gameplay?!

It’s no secret that I’m an avid advocate of good software engineering practices for game development. As with anything, good software engineering practices have some very real and important benefits, but those benefits come with a cost. Because of the small scope of the prototype, and the fact that the code wasn’t going to survive more than a week or two, we decided to throw any software engineering practice that would slow us down in the least straight out the window!  Forget about accessors and private scoping; skip any unit tests; bye bye factories, loose coupling, and proper physical dependency management. Welcome to hackfest, check your engineering discipline at the door!

One of the things that made the prototype very successful for us was to concentrate on a single gameplay mechanic. We didn’t try to prototype the whole game, just the one bit we were most concerned about (and that was going to take 80% of the player’s time and attention). That allowed us to deal with a small subset of the game and iterate on it very quickly without getting bogged down by creeping features or dealing with a larger scope.

The prototype shamelessly used the worst programmer art imaginable (Charles + MS Paint == … you can imagine [edit: Charles] TOTALLY AWESOME SWEET). Yes, it looked horrible, but if the game was so much fun with those graphics, just imagine how it’s going to blow people away whenever we add some real art and animations.

Lessons learned

The most valuable lesson we learned was the value of constantly asking ourselves the questions “Is this fun?” and “What can I do now to make it more fun?“  That was actually really crucial. Having such a specific (even if subjective and nebulous) goal allowed us to remain in the right path. Sometimes it was tempting to add feature X or Y, but when we realized that it wouldn’t make the game any more fun, we quickly dropped it and moved to something else.

It’s important not to over-intellectualize the idea of “fun”, though. Sure, most of the fun comes from the choices the player makes and what happens as a result of those choices (even if the choice is to mash the buttons). But don’t underestimate the value of “fun” in having cool, flashy things on the screen, or to have the character jump with a nice ramp-up curve and land with some squishiness to give it that “just right” feel.

It was also very helpful to run the prototype by somebody else towards the end to make sure we weren’t finding it fun because we were simply delirious. Clearly, it has to be someone you can trust and who can understand what the prototype represents and not be put off by the stick figures. It was great to get confirmation from an outside source that it really was getting to be fun. We also got a bunch of feedback that made it a much tighter experience. A lot of it was pretty obvious, but we were too close to the game to be able to see it by ourselves.

As with most prototypes, it’s always very tempting to not throw it away and continue hammering at it until it looks like something that could be shipped. We didn’t really have a choice because of the technologies we went with (allegro is a fully cpu-based renderer), but we could feel the call of the dark side. It really was a good thing that we didn’t even try. Towards the end, the terrible hacked code we’d written was starting to slow our productivity. Code that bad builds up an “engineering debt” that you have to pay off sooner or later or you hit a wall.  Our particular wall came in the form of extending a pre-existing player action in what seemed like a trivial way.  Something like this would have literally taken about 5 minutes at the beginning of the prototype, but by the point we were at it would have taken closer to an hour. Heck, even having to go on for another week would have started to be painful. So in a way, the prototype came with a built-in self-destruct mechanism, which is a good thing because it discouraged us to continue working on it for too long.

Using existing technology is definitely the way to go. That way there’s very little “wasted” time getting up to speed. The fact that we were able to get most of the core mechanics of the game in the first day goes to show how useful is to hit the ground running instead of getting bogged down in how to load bitmaps or create a window.

Additionally, the procedural, non “framework-y”, non-object oriented approach of the API we chose made it much easier to bend to our will and make it work the way *we* wanted, and not the way somebody decided for us ahead of time. The fewer restrictions your prototyping API has, the easier it will be to do all the crazy things that come to mind during the prototyping phase.

The game

Unfortunately, we can’t really talk about the game right now. Just rest assured that we’re working on it at full steam and we’ll announce some details as soon as we can. Clearly, we’re not going to release the prototype since it has a good chance of scaring away any potential publishers in its current state. But if one day the game sells tons of units, maybe we’ll release the prototype as a “behind the scenes” feature so other people can have a glimpse at how things got started and they can laugh at our artistic skills.

Conclusion

There you have it. Now at least we can be confident that the game is going to be pretty fun. Sure, there are lots of aspects of the game that we haven’t prototyped, but we consider those to be lower risk and we’ll figure them as we go along. At least the core mechanic has proven to be fun. Now it’s just a matter of writing the production code and implementing it all. That simple. What could possibly go wrong? ;-)

[1] Money saving tip #1 for starving startups: If you have to pay for a 24-hour Internet connection in a coffee shop and you need to have several computers with Internet access, connect them with an ethernet cable and share the Internet connection from one of them. Ta-dah! :-)

[2] Actually, I’ve been wanting to create a movie of the prototype in action, but I haven’t been able to. It seems that Fraps doesn’t work because it’s not using a DirectX surface, and saving each frame is too slow. Anybody has any suggestions? Don’t get your hopes up though, we’re not uploading a movie of the prototype any time soon, we’re just saving it to show our grandchildren one day.