Testing Code

The purpose of writing unit tests is to verify the code does what it’s supposed to do. How exactly do we go about checking that? It depends on what the code under test does. There are three main things we can test for when writing unit tests:

• Return values. This is the easiest thing to test. We call a function and verify that the return value is what we expect. It can be a simple boolean, or maybe it’s a number resulting from a complex calculation. Either way, it’s simple and easy to test. it doesn’t get any better than this.
• Modified data. Some functions will modify data as a result of being called (for example, filling out a vertex buffer with particle data). Testing this data can be straightforward as long as the outputs are clearly defined. If the function changes data in some global location, then it can be more complicated to test it or even find all the possible places that can be changed. Whenever possible, pass the address of the data to be modified as an input parameter to the functions. That will make them easier to understand and test.
• Object interaction. This is the hardest effect to test. Sometimes calling a function doesn’t return anything or modify any external data directly, and it instead interacts with other objects. We want to test that the interaction happened in the order we expected and with the parameters we expected.

Testing the first two cases is relatively simple, and there’s nothing you need to do beyond what a basic unit testing-framework provides. Call the function and verify values with a CHECK statement. Done. However, testing that an object “talks” with other objects in the correct way is much trickier. That’s what we’ll concentrate on for the rest of the article.

As a side note, when we talk about object interaction, it simply refers to parts of the code calling functions or sending messages to other parts of the code. It doesn’t necessarily imply real objects. Everything we cover here applies as well to plain functions calling other functions.

Before we go any further, let’s look at a simple example of object interaction. We have a game entity factory and we want to test that the function CreateGameEntity() finds the entity template in the dictionary and calls CreateMesh() once per each mesh.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    EntityDictionary dict;
    MeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
    // How do we test it called the correct functions?
}

We can write a test like the one above, but after we call the function CreateGameEntity(), how do we test the right functions were called in response? We can try testing for their results. For example, we could check that the returned entity has the correct number of meshes, but that relies on the mesh factory working correctly, which we’ve probably tested elsewhere, so we’re testing things multiple times. It also means that it needs to physically create some meshes, which can be time consuming or just need more resources than we want for a unit test. Remember that these are unit tests, so we really want to minimize the amount of code that is under test at any one time. Here we only want to test that the entity factory does the right thing, not that the dictionary or the mesh factory work.

Introducing Mocks

To test interactions between objects, we need something that sits between those objects and intercepts all the function calls we care about. At the same time, we want to make sure that the code under test doesn’t need to be changed just to be able to write tests, so this new object needs to look just like the objects the code expects to communicate with.

A mock object is an object that presents the same interface as some other object in the system, but whose only goal is to attach to the code under test and record function calls. This mock object can then be inspected by the test code to verify all the communication happened correctly.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

dict.meshCount = 3;

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);

    CHECK_EQUAL(1, dict.getEntityInfoCallCount);
    CHECK_EQUAL(gameEntityUid, dict.lastEntityUidPassed);
    CHECK_EQUAL(3, meshFactory.createMeshCallCount);
}

This code shows how a mock object helps us test our game entity factory. Notice how there are no real MeshFactory or EntityDictionary objects. Those have been removed from the test completely and replaced with mock versions. Because those mock objects implement the same interface as the objects they’re standing for, the GameEntityFactory doesn’t know that it’s being tested and goes about business as usual.

Here are the mock objects themselves:

struct MockEntityDictionary : public IEntityDictionary
{
    MockEntityDictionary()
        : meshCount(0)
        , lastEntityUidPassed(0)
        , getEntityInfoCallCount(0)
    {}

    void GetEntityInfo(EntityInfo& info, int uid)
    {
        lastEntityUidPassed = uid;
        info.meshCount = meshCount;
        ++getEntityInfoCallCount;
    }

    int meshCount;
    int lastEntityUidPassed;
    int getEntityInfoCallCount;
};

struct MockMeshFactory : public IMeshFactory
{
    MockMeshFactory() : createMeshCallCount(0)
    {}

    Mesh* CreateMesh()
    {
        ++createMeshCallCount;
        return NULL;
    }
};

Notice that they do no real work; they’re just there for bookkeeping purposes. They count how many times functions are called, some parameters, and return whatever values you fed them ahead of time. The fact that we’re setting the meshCount in the dictionary to 3 is how we can then test that the mesh factory is called the correct number of times.

When developers talk about mock objects, they’ll often differentiate between mocks and fakes. Mocks are objects that stand in for a real object, and they are used to verify the interaction between objects. Fakes also stand in for real objects, but they’re there to remove dependencies or speed up tests. For example, you could have a fake object that stands in for the file system and provides data directly from memory, allowing tests to run very quickly and not depend on a particular file layout. All the techniques presented in this article apply both to mocks and fakes, it’s just how you use them that sets them apart from each other.

Mocking Frameworks

The basics of mocking objects are as simple as what we’ve seen. Armed with that knowledge, you can go ahead and test all the object interactions in your code. However, I bet that you’re going to get tired quickly from all that typing every time you create a new mock. The bigger and more complex the object is, the more tedious the operation becomes. That’s where a mocking framework comes in.

A mocking framework lets you create mock objects in a more automated way, with less typing. Different frameworks use different syntax, but at the core they all have two parts to them:
A semi-automatic way of creating a mock object from an existing class or interface.
A way to set up the mock expectations. Expectations are the results you expect to happen as a result of the test: functions called in that object, the order of those calls, or the parameters passed to them.

Once the mock object has been created and its expectations set, you perform the rest of the unit test as usual. If the mock object didn’t receive the correct calls the way you specified in the expectations, the unit test is marked as failed. Otherwise the test passes and everything is good.

GoogleMock

GoogleMock is the free C++ mocking framework provided by Google. It takes a very straightforward implementation approach and offers a set of macros to easily create mocks for your classes, and set up expectations. Because you need to create mocks by hand, there’s still a fair amount of typing involved to create each mock, although they provide a Python script that can generate mocks automatically from from C++ classes. It still relies on your classes inheriting from a virtual interface to hook up the mock object to your code.

This code shows the game entity factory test written with GoogleMock. Keep in mind that in addition to the test code, you still need to create the mock object through the macros provided in the framework.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    EXPECT_CALL(dict, GetEntityInfo())
        .Times(1)
        .WillOnce(Return(EntityInfo(3));

    EXPECT_CALL(meshFactory, CreateMesh())
        .Times(3);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
}

MockItNow

This open-source C++ mocking framework written by Rory Driscoll takes a totally different approach from GoogleMock. Instead of requiring that all your mockable classes inherit from a virtual interface, it uses compiler support to insert some code before each call. This code can then call the mock and return to the test directly, without ever calling the real object.

From a technical point of view, it’s a very slick method of hooking up the mocks, but the main advantage of this approach is that it doesn’t force a virtual interface on classes that don’t need it. It also minimizes typing compared to GoogleMock. The only downside is that it’s very platform-specific implementation, and the version available only supports Intel x86 processors, although it can be re-implemented for PowerPC architectures.

Problems With Mocks

There is no doubt that mocks are a very useful tool. They allow us to test object interactions in our unit tests without involving lots of different classes. In particular, mock frameworks make using mocks even simpler, saving typing and reducing the time we have to spend writing tests. What’s not to like about them?

The first problem with mocks is that they can add extra unnecessary complexity to the code, just for the sake of testing. In particular, I’m referring to the need to have a virtual interface that objects are are going to be mocked inherit from. This is a requirement if you’re writing mocks by hand or using GoogleMock (not so much with MockItNow), and the result is more complicated code: You need to instantiate the correct type, but then you pass around references to the interface type in your code. It’s just ugly and I really resent that using mocks is the only reason those interfaces are there. Obviously, if you need the interface and you’re adding a mock to it afterwards, then there’s no extra complexity added.

If the complexity and ugliness argument doesn’t sway you, try this one: Every unnecessary interface is going to result in an extra indirection through a vtable with the corresponding performance hit. Do you really want to fill up your game code with interfaces just for the sake of testing? Probably not.

But in my mind, there’s another, bigger disadvantage to using mock frameworks. One of the main benefits of unit tests is that they encourages a modular design, with small, independent objects, that can be easily used individually. In other words, unit tests tend to push design away from object interactions and more towards returning values directly or modifying external data.

A mocking framework can make creating mocks so easy, to the point that it doesn’t discourage programmers from creating a mock object any time they think of one. And when you have a good mocking framework, every object looks like a good mock candidate. At that point, your code design is going to start looking more like a tangled web of objects communicating in complex ways, rather than simple functions without any dependencies. You might have saved some typing time, but at what price!

When to Use Mock Frameworks

That doesn’t mean that you shouldn’t use a mocking framework though. A good mocking framework can be a lifesaving tool. Just be very, very careful how you use it.

The case when using a mocking framework is most justified when dealing with existing code that was not written in unit testing in mind. Code that is tangled together, and impossible to use in isolation. Sometimes that’s third-party libraries, and sometimes it’s even (yes, we can admit it) code that we wrote in the past, maybe under a tight deadline, or maybe before we cared much about unit tests. In any case, trying to write unit tests that interface with code not intended to be tested can be extremely challenging. So much so, that a lot of people give up on unit tests completely because they don’t see a way of writing unit tests without a lot of extra effort. A mocking framework can really help in that situation to isolate the new code you’re writing, from the legacy code that was not intended for testing.

Another situation when using a mocking framework is a big win is to use as training wheels to get started with unit tests in your codebase. There’s no need to wait until you start a brand new project with a brand new codebase (how often does that happen anyway?). Instead, you can start testing today and using a good mock framework to help isolate your new code from the existing one. Once you get the ball rolling and write new, testable code, you’ll probably find you don’t need it as much.

Apart from that, my recommendation is to keep your favorite mocking framework ready in your toolbox, but only take it out when you absolutely need it. Otherwise, it’s a bit like using a jackhammer to put a picture nail on the wall. Just because you can do it, it doesn’t mean it’s a good idea.

Keep in mind that these recommendations are aimed at using mock objects in C and C++. If you’re using other languages, especially more dynamic or modern ones, using mock objects is even simpler and without many of the drawbacks. In a lot of other languages, such as Lua, C#, or Python, your code doesn’t have to be modified in any way to insert a mock object. In that case you’re not introducing any extra complexity or performance penalties by using mocks, and none of the earlier objections apply. The only drawback left in that case is the tendency to create complex designs that are heavily interconnected, instead of simple, standalone pieces of code. Use caution and your best judgement and you’ll make the best use of mocks.

This article was originally printed in the June 2009 issue of Game Developer.

The good news is that unit testing is a lot simpler than driving a standard shift, but there are a lot of small details you need to get right to do it successfully. Even after reading about unit testing and being convinced of its benefits, programmers are often not sure how to get started. This month’s column is not going to try to convince you of the many benefits of unit testing (I hope you are already convinced), but rather, describe some very concrete tips to help you get started right away.

Goals of Unit Testing

There are many different reasons to write unit tests:

• Correctness testing. Checking that the code behaves as designed.
• Boundary testing. Checking that the code behaves correctly in odd or boundary situations.
• Regression testing. Checking that the behavior of the code doesn’t change unintentionally over time.
• Performance testing. Checking that the program meets certain minimum performance or memory constraints.
• Platform testing. Checking that the code behaves the same across multiple platforms.
• Design. Tests provide a way to advance the code design and architecture. This is usually referred as Test-Driven Development (TDD).
• Full game or tool testing. Technically this is a functional test, not a unit test anymore because it involves the whole program instead of a small subset of the code, but a lot of the same techniques apply.

Some developers use unit tests only for one of the reasons listed above, while others use many kinds of tests for a variety of different reasons. It’s important to recognize that because there are so many different uses for unit tests, no single solution is going to fit everybody. The ideal setup for some of those situations is going to be slightly different than for others. The basics are the same for all of them though.

When working with unit tests, these are our main goals:

• Spend as little time as possible writing a new test.
• Be notified of failing tests, and see at a glance which ones failed and why.
• Trust our tests. Have them be consistent from run to run and robust in the face of bad code.

Testing Framework

Most of us have created one-off programs in the past to test some particularly complicated code. It’s usually a quick command line program that runs through a bunch of cases and asserts after each one that the results were correct. That’s the most bare-bones way of creating unit tests.

Unfortunately, it’s also a pain and it misses on most of the unit-testing goals described in the previous section: creating a new program just for that is a pain, we have to go out of our way to run the tests, and it usually gets out of date faster than the latest Internet meme. That is in part why a lot of programmers have an initial aversion to writing unit tests.

If you’re considering writing even a small unit test, you should use a unit-testing framework. A unit-testing framework removes all the busy work from writing unit tests and lets you spend your time on the logic of what to test. This doesn’t mean that the framework writes the tests for you. Be very wary of any tool that claims to do that! No, a unit-testing framework is simply a small library that provides all the glue for running unit tests and reporting the results. Sorry, you still need to use your brain and do (some of) the typing.

A quick search will reveal plenty of unit-testing frameworks to choose from for your language of choice, and most of them are free and open source so you can rely on them and modify them to suit your needs. For C/C++ and game development, I strongly recommend starting with UnitTest++. Charles Nicholson and I wrote that framework a few years ago specifically with games and consoles in mind. Many game teams have adopted it for their games and tools, and it has been used on lots of different game platforms including current and last generation consoles, Windows, Linux, and Mac PCs, and even on the iPhone. In most situations, it should be a straight drop-in to your project and you’re up and running.

If you end up using a different testing framework, or even if you roll your own, the techniques described here still apply, even if the syntax is slightly different.

Hello Tests

Writing your first test is easy as pie:

#include <UnitTest++.h>

TEST(MyFirstTest)
{
    int a = 4;
    CHECK(a == 4);
}

To run it you need to add the following line to your executable somewhere. We’ll talk more about the physical organization of tests in a moment.

int failedCount = UnitTest::RunAllTests();

Done! Easy, wasn’t it? When you compile and run the program you should see the following output:

Success: 1 test passed.
Test time: 0.00 seconds.

Let’s add a failing test:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK(a == 4);
}

Now we get:

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: a == 4
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

That’s great, but if we’re going to diagnose the problem, we probably need to know the value of the variable a, and all the test is telling us is that it’s not 4. So instead, we can change the CHECK statement to the following:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK_EQUAL(4, a);
}

And now the output will be

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: Expected 4 but was 5
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

Much better. Now we get both the error information and the value of the variable under test. Virtually all unit-testing frameworks include different types of CHECK statements to get more information when testing floats, arrays, or other data types. You can even make your own CHECK statement for your own common data types such as colors or lists.

As a bonus, if you’re using an IDE, double-clicking on the test failure message should bring you automatically to the failing test statement.

When To Run

When to run unit tests will depend on what is being tested and how long it takes to do so. In general, the more frequently you run the tests, the better. The sooner you get feedback that something went wrong, the easier it will be to fix. Maybe even before it was checked in and it spread to the rest of the team. On the flip side, realistically, building and running a set of unit tests takes a certain amount of time, so it’s important to find the right balance between feedback frequency and time spent waiting for tests.

At the very least, all tests should run once a day, during the nightly build process in your build server (You have a build server, don’t you? If not, run over and read this column in the August 2008 issue). It doesn’t matter how long they take or how how many different projects you need to run. Just add them to the build script, and hook their output into the build results.

On the other extreme, you can build your tests every time you build the project and execute them as a postbuild step. That way, any time you make a change to a project, all tests will execute and you’ll see if anything went wrong. This is a great approach, but I wouldn’t recommend it if tests add more than a couple of seconds to the incremental build time, otherwise, they’ll be slowing you down more than they help.

For most developers, some approach in the middle will make most sense. For example, take a small, fast subset of tests that are more likely to break, and run those with every build. Whenever any code is checked in to version control, the build server can run those tests plus a few more, slower ones. And finally, at night, you can bring out the big guns and run those really long, really thorough ones that take a few hours to complete. You can separate those tests into different projects, or, if your framework supports them, into different test suites, which allow you to decide which sets of tests to execute at runtime.
Reporting Results

If a unit test fails and nobody notices, is it really an error? Just running the tests isn’t good enough. We have to make sure that someone sees the failure it and fixes the problem.

Most unit-testing frameworks will let you customize how you want the failure errors to be reported. By default they will probably be sent to stdout, but you can easily customize the framework to send them to debug log streams, save to a file, or upload them to a server.

Even more important than the actual error messages is detecting whether there were any failures. After running all the tests, there is usually some way to detect how many tests failed. The program that was running the tests can detect any failures, print an error message, and exit with an error code. That error code will propagate to the build server and trigger a build failure. Hopefully by now alarm sounds are going off across the office and someone is on his way to fix it.

Project Organization

When people start down the unit test path, they often struggle to figure out how to physically lay out the unit tests. In the end, it really doesn’t matter too much as long as it makes sense to you, the final build doesn’t contain any tests, and they’re still easy to build and run.

My personal preference is to keep unit tests separate from the rest of the code. Usually I end up creating one file of tests for every cpp file. So FirstPersonCameraController.h and .cpp have a corresponding TestFirstPersonCameraController.cpp. Since I use this convention regularly throughout all of my code, I have a custom IDE macro to toggle between a file and its corresponding test file. I also put all the tests in a separate subdirectory to keep them as physically separate as possible.

I prefer to break up my code into several static libraries for each major subsystem: graphics, networking, physics, animations, etc. Each of those libraries has a set of unit tests, but instead of compiling them into the library, I create a separate project that creates a simple executable program. That project contains all the unit tests and links against the library itself, and in its main entry point it just calls the function to runs all unit tests and returns the number of failures. This keeps the tests separate from the library, but still very easy to build.

If all your code is organized into libraries, and your game is just a collection of libraries linked together, that’s all you need. Most games and tools, however, have a fair amount of code that you might want to test in the project itself. Since the game is an executable, you can’t easily link against it from a different project like we did before. In this case, I build the unit tests into the game itself, and I optionally call them whenever a particular command-line parameter like -runtests is present. Just make sure to #ifdef out all the tests in the final build.

Multiplatform Testing

Running the tests on the PC where you build the code is very straightforward. But unless you’re only creating games and tools for that platform, you will definitely want to run your tests on different platforms as well. Unit tests are an invaluable tool for catching slight platform inconsistencies caused by different compilers, architecture idiosyncrasies, or varying floating point rules.

Unfortunately, running unit tests on a different platform from your build machine is usually a bit more involved and not nearly as fast as doing it locally. You need to start by compiling the tests for the target platform. This is usually not a problem since you’re already building all your code for that platform, and hopefully your unit testing framework already supports it. Then you need to upload your executable with the tests and any data required to the target platform and run it there. Finally, you need to get the return code back to detect if there were any failures. This is surprisingly the trickiest part of the process with a lot of console development kits. If getting the exit code is not a possibility, you’ll need to get creative by parsing the output channel, or even waiting for a notification on a particular network port.

Some target platforms are more limited than others in both resources and C++ support. One of the features that makes UnitTest++ a good choice for games is that it requires minimal C++ features (no STL) and it can be trimmed down even further (no exceptions or streams).

For example, running unit tests on the PS3 SPUs was extremely useful, but it required stripping the framework down to the minimum amount of features. It was also tricky being able to fit the library code plus all the tests in the small amount of memory available. To get around that, we ended up changing the build rules for the SPUs so each test file created its own SPU executable (or module). We then wrote a simple main SPU program that would load each module separately, run its tests, keep track of all the stats, and finally report them.

Running a set of unit tests on the local machine can be an almost instant process, but running them on a remote machine is usually much slower, and can take up to 10 or 20 seconds just with the overhead of copying them and launching the program remotely. For this reason, you’ll want to run tests on other platforms less frequently.

No Leaking Allowed

Finally, if you’re going to have this all this unit testing code running on a regular basis, you might as well get as much information out of it as possible. I have found it invaluable to keep track of memory leaks around the unit test code.

You’ll have to hook into your own memory manager, or use the platform-specific memory tracking functions. The basic idea is to get a memory status before running the tests, and another one after all the tests execute. If there are any extra memory allocations, that’s probably a leak. In that case, you can report it as a failed build by returning the correct error code.

Watch out for static variables or singletons that allocate memory the first time they’re used. They might be reported as memory leaks even though it wasn’t what you were hoping to catch. In that case, you can explicitly initialize and destroy all singletons, or, even better, not use them at all, and keep your memory leak report clean.

You’re now armed with all you need to know to set up unit tests into your project and build pipeline. Grab a testing framework and get started today.

This article was originally printed in the May 2009 issue of Game Developer.

What becomes significantly more difficult is learning to effectively apply TDD to code with more dependencies. A question that I’m often asked from people trying to use TDD for the first time is “How can you possibly use TDD for high-level game code? It’s impossible!”.

When The Going Gets Tough

At that point, the temptation to give up on using TDD for high-level code becomes very strong. It seems that all the time spent writing mocks and hooking objects together is a waste of time and that you could be writing game code much faster without it. “Maybe it would be best to skip TDD, just for that part”.

Bad idea!

By giving up on TDD for high-level code, you’ll be missing out on the main benefit of TDD: designing better code. I’ve said this many times, but it bears repeating: TDD is not a testing technique. It’s a design technique. You get lots of benefits from applying TDD [2], but one of the main one is much more modular and dependency-free code.

That’s because TDD turns our weaknesses into advantages. We’re humans and we’re lazy. We don’t want to repeat ourselves constantly or write complex code. If we’re writing a test for some code we’ll write in the future, we’re going to create a very simple test. We’re probably going to want to put some object or data on the stack and make a function call and nothing else. We certainly don’t want to initialize the graphics system, create a new world, add some level data, start the physics simulation, and then call the function! So by being lazy, we end up writing very simple code with the least amount of dependencies.

What does that mean for our high-level code? The AI logic seems to need access to every single game system, at least the way we implemented it in the past. Using TDD to implement AI logic doesn’t mean writing tests to write the same code you would have written before. It means writing tests and then writing some code that makes those tests pass and nothing else. For example, we might realize that the AI doesn’t need to access the physics system, cast a ray in the world, find out what entities is interesting, and return that information. Instead, all it might need to do is output some data saying that it wants a ray query done at a later time (which in turn is the key to batching those ray casts and achieving high performance).

You soon realize that the code you write is very different (and better!) to what you would have written before. Once you get over that hump, TDDing high-level code is not about twisting your code with mocks, but becomes simpler code that takes simple input data and creates simple output data. By pushing through and forcing yourself to apply TDD, you made a breakthrough and reaped all the benefits.

That’s the main reason why TDD books push the idea that no code can be written without a test first. Otherwise, inexperienced TDD programmers would be too quick to quit and would miss out on the technique completely.

Not only that, but code that is not created with TDD tends to have the bad habit of spreading. When code was not designed through TDD, it means that other code that uses it will probably be difficult to TDD itself. So it’s a bit like const-correctness: You’re either in our out, and there’s very little room for half measures.

On the other hand, I would argue that using TDD on a math library is a bad idea. It’s essential to write good unit tests for a math library, but probably not to design it through TDD. Are you really going to implement a cross product differently just because you wrote tests before? The emphasis there has to be on correctness and performance, not on creating the interface or implementation through tests.

It’s A Tool To Help Development

A few days ago, I received an email from Caleb Gingles with a great question:

I’ve started a new game project and have been making an effort to strictly follow TDD from the beginning. Which has been a challenge, as much of the coding at the beginning of the project has consisted of operating system requirements and framework stuff. [...] However, there are other situations that seem much harder to test. Like the main loop in a game. According to Kent Beck’s book, everything begins with testing, and no code is written unless a test requires it. But how can a test require the main loop framework in a game? The loop just… is. It exists to allow you to do certain other things, it doesn’t have much purpose in and of itself.

Applying what I just talked about, we could use TDD to create the main loop of a game. It definitely requires thinking about it differently and breaking preconceptions. It’s about writing code to pass the tests, not writing tests to fit the code we want to write.

So we could do something like this:

TEST(MainLoopExecutesCorrectNumberOfTimes)
{
    MainLoopState loopState;
    MainLoopUtils::Execute(loopState, 4);
    CHECK_EQUAL(4, loopState.frameCount);
}

All of a sudden we see a way to test something that before it seemed impossible to test. We can add tests to make sure certain functions are getting called (maybe we add those function pointers to the loopState), that the main loop exits under certain conditions, etc.

For some games with complex GUIs and different game modes, it might make a lot of sense to have a more flexible main loop and develop it completely through TDD. On the other hand, for a simple iPhone game that has a single main loop that never changes, there might be no point at all in using TDD. The loop code is going to end up looking more or less the same as it would without TDD, just more complex and generic. TDD didn’t help any in that case.

How about code that calls into OpenGL, or some other non-TDD-friendly library? Do we have to TDD that? Frankly, I wouldn’t bother. I did give it a good try at one point several years ago and I can say it is possible, but it’s a pain and you gain almost nothing from it. The main benefits is trusting that the library you’re using really does what it’s supposed to do, but it won’t affect much the design of your code. So what you’re really doing is adding tests to that library (which might or might not be of value).

My advice in a situation like that: Call the library from as few places as possible (I didn’t want to say make a thin wrapper because that implies isolation which is not the goal here), and just test that your code is called at the right time with the right data. The actual code that calls the library should be small and simple enough that you should feel OK not having any tests for it.

Even though it seems to go against what I said in the previous section, TDD is not all or nothing. It’s not a religion.

When you’re starting out, I encourage you to push yourself to think how you could apply TDD to situations you don’t think are possible. Once you’re experienced enough, you’ll know when to say enough is enough, and use TDD only when it actually benefits you. At that point you’ll also know how to write code in a way that plays well with TDD, even if the code itself was not developed through TDD.

In the end, TDD is a tool to help you develop better and faster. Don’t ever let it get in the way of that.

[1] That’s actually a horrible place to apply TDD. More on that in a second.

[2] Usable API, simple code, unit tests, documentation, safety net for refactoring among others.

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

Every developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

• All my code is written in C and C++ (except for a bit of ObjC and assembly).
• It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
• I always try to write important code through Test-Driven Development.
• I’m the only programmer (and only designer).
• Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

• I’m showing all the data there is and how big it is. Nothing is hidden.
• I’m making it clear that it’s free of pointers and temporary variables.
• I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions.

None of that would be an issue except that TDD creates lots of unit tests and encourages almost constant incremental builds. In the past I’ve stressed how important it is to keep them fast. But this week that I was able to really feel what a dramatic difference slow unit tests make.

For many months, our unit tests ran under one second. Each library has its own set of unit tests, and so does the game and each tool. So even though we were writing lots of unit tests, we weren’t always running them all at the same time. There was a definite snappiness to coding: write test, compile, fail, write code, compile, pass, move on. Go, go, go.

As the game project got larger, we started accumulating more unit tests and testing times started creeping up into the second range. At the time, I couldn’t quite put my finger on what had changed, but there was a definite change in feel. Things weren’t quite as snappy.

Then, about a couple of weeks ago, we hit the two second mark and things definitely changed.

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 2.13 seconds.

As the tests were running, my mind would wander, thinking about what test I would write next, or whether the code I had just written would be cache-friendly, or whether we’d do Honey’s or Manhattan for lunch. I would be jerked out of my state of flow. It wasn’t just me, Charles felt it too. The difference in productivity was huge.

The Zones

For those of you laughing at those times and dismissing them as insignificant, you need to keep in mind that our build times are very short:

• An incremental build with just a change in a cpp file takes under a second (including link times).
• A full build of *all* our runtime code, including libraries, is around 50 seconds in a dual-core machine [1].

So adding on 2 seconds to the incremental build time is going from “practically instant” to “OK, I’m waiting…”.

Of course, none of this has anything to do with unit tests in particular. It’s total incremental build times (including compiling, linking, and running unit tests) that matter. An incremental build that takes 3 seconds with no unit tests is just as bad as one that is under one second with 3 second unit tests. They both interrupt the flow the same way.

I’ve often thought about how a company size affects communication and “feel”. It’s close to a step function with logarithmic scale: there’s a distinct jump from a company of 3 to a company of 10, to one of 35, to one of 100+.

I think something similar applies to build times [2]

• 0 to 2 seconds. Programmer stays in the flow. Productivity is way high.
• 2 to 8 seconds. Flow is broken. Definite feel of things being a bit painful and sluggish.
• 8 to 30 seconds. Attention wanders to other parts of the code, email, etc. Multitasking kicks in and overall productivity plummets.
• 30+ seconds. Builds seen as batch processing. After each build is started, some other action is started. Maybe even physically get up, get a drink, start a conversation. Full brain reboot between coding tasks.
• 5+ minutes. Programmers bang their heads against the desk, actively consider the merits of different forms of suicide, or at the very least start polishing their resumes.

The Fix

Having been spoiled by sub-second build times, neither one of us wanted to put up with the broken flow of 2+ second builds, so we decided it was time to fix things [3].

The good news is that our slow build times were solely due to running the unit tests. 2.13 seconds to run only about 1100+ tests is really bad, even on a so-so desktop machine like the ones we can afford. That’s almost 2ms per test! Remember that these are unit tests, not functional tests. So I expect to be able to run several thousands of these under one second. Something was way off.

Things had gotten much worse in the last couple of weeks, so it wasn’t a slow, steady creep up. Chances are we had introduced something very slow recently, which meant it was probably easy to fix.

As a first pass, we turned on the option to fail any test that takes over a certain amount of time in UnitTest++:

RunAllTests(reporter, UnitTest::Test::GetTestList(), NULL, 5); // fail any test over 5ms

That brought about tons of failing test, most of them added recently and creating one particular object in their fixtures. Digging a big deeper, that particular class had an array of 25,000 (yes, that many zeros) objects. Those objects were plain-old-data structures, but had a default constructor. Even though the constructor was simply initializing its members to zero, that was causing a significant slow down when so many objects were created. To make things worse, some of the code I had written was iterating over the array resetting the entries or shifting them up or down, repeatedly calling the constructor. Ouch!

A few minutes and lots of code hacking later, and the object didn’t call a single constructor anymore. Our test times went down by almost half. Victory!

We were already at 1 second, which is in the acceptable range, but since we were here, maybe we could improve things a bit more. After all, 1ms per test is still too slow.

UnitTest++ wasn’t reporting failing tests over 1ms very reliably. Different runs would cause very different results. Things were too spread out or other things in the computer were interfering too much.

About a fifth or so of our tests use Havok. Not with mocks, but directly. They each initialize Havok, create a world, and deal with a simple rigid body, constraint, or something of the sort. I suspected all along that the Havok system initialization and shutdown for each test was probably causing quite a slow down, so I changed it to be initialized only once before we ran any tests.

And that did the trick. With that change we were down to 0.16 seconds. Back to snappy, happy land!

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 0.16 seconds.

In release mode things are even better, but we do most development in debug mode, so that’s the really important one:

> bin\SweetPea.exe -unittestSuccess: 1145 tests passed.Test time: 0.06 seconds.

Perhaps sub-second build times are not realistic for all projects, but I think there’s a lot to be gained by keeping compile, link, and test times as short as possible. For tips on how to keep compile times down, check out these articles I wrote a while ago. If your unit tests are slow and you really can’t speed them up anymore, consider splitting them into two sets: one that is important or recent ones that runs with every build, and another, more comprehensive one that runs in the build server.

Even if you cut down from 20 seconds down to 10, you’ll be making a huge shift in how you’re able to work and iterate on your programming.

[1] We’d love to keep detailed metrics with each build: executable size, number of lines of code, number of tests, etc, etc. Unfortunately, when the rubber hits the road, that ends up in the “nice to have category” and we simply have no resources to spare on a two-man startup.

[2] For a whole whooping statistical population of one since I’m only talking about my experience. Still, I suspect that the same applies to many other programmers by scaling the function by some constant.

[3] Perfect use of agile mentality. Fix things when you need them fixed, not before. But make sure you do fix them when you need them. Otherwise it’s not agile, it’s just lazy.

Bug Hunting

Unit tests, by their very nature, are limited to a single class or function at the time. There are all sorts of complex interactions between objects and systems that they simply can’t test. Even if you use mock objects and are extremely careful to test all your object interactions, there will be lots of unexpected cases and bugs that crawl out while running the game under real world conditions. Dr. Chaos is alive and well.

Besides, unit tests just test that the code does what you think it should do. So if the algorithm you have in your head is completely wrong, they’ll just make sure that the implementation is as broken as you imagined. Bugs like that might only surface once the code is put in the context of the whole game and interacts with other systems.

Traditionally, game companies rely on QA to uncover and squash most of the bugs created by complex interactions. You know, the type of bug that gets only triggered when the player enter the cave while there’s a blue moon and he spun around in place a seven times. Not only is this is an expensive process, but it’s not even very good at uncovering all the bugs. Most games have millions and millions of combinations of possibilities and interactions, and it’s completely impractical to try to run through all of them by hand.

Our Hero: LeChimp

At Power of Two Games it’s just two of us. So no QA or even interns to play the game endlessly. But, even in preproduction, we can’t afford to ignore those types of bugs. Instead, we enlisted the help of our hero: LeChimp.

LeChimp is our functional test server. It tirelessly runs the game every couple of hours and makes sure it loads and runs without any problem. Sure, ideally it should run more frequently, but LeChimp doubles up as our build server, and we don’t have another good computer to spare (maybe if y’all bought more t-shirts we could afford to buy another cheapo Dell).

Running the game is a good start. It checks that it’s possible to load every level and that nothing crashes. Frankly, that’s a good percentage of what QA does a lot of the time, and a lot of teams would really benefit from having such a simple test and know as soon as a level stops loading. Still, it we can do much better than that.

Monkey business

LeChimp runs the game for a fixed number of frames, and makes sure the game doesn’t crash or hits any asserts. But running the game without any action going on is not very useful, so it runs it with the -monkey switch, which feeds pseudo-random input to the game, as if a monkey were playing the game.

Actually, the input from -monkey is not random at all. I first made it truly random by generating new inputs through std::rand() every frame, but it looked more like a monkey on crack was playing the game and the characters shook violently back and forth and never managed to do anything interesting. Instead, now the monkey input holds the controller sticks and the buttons for some varying time interval, and it looks more like a monkey without its ADD medicine, which is clearly a step up.

When I first heard of this technique at a GDC tutorial a few years ago, I thought it would be pretty useless. How many bugs could feeding random input to the game really uncover? Shouldn’t we try to be doing something more intelligent to mimic how players interact with the game? But since it was really easy to implement I decided to give it a try. Boy, was I in for a treat! Within a few runs, it uncovered several major bugs that nobody hadn’t seen during their runs of the game. Since then, there isn’t a game I work on that doesn’t get treated with some monkey input love..

It’s so simple to implement that if you haven’t done it already, I really encourage you to go and do it right away. Do it over your next lunch break even. I guarantee you’ll be amazed at what it uncovers (or I’ll refund your money for this article .

Recording for posterity

In addition to just trying to crash the game, LeChimp records all the inputs and the state of the game at every frame. Then, after running the game for a while, it runs it again, feeding it the recorded input, and verifies that the game is in the same state as it was during the recording session (every enemy is in the same place, every prop has the same orientation, every player has the same health, etc). This verifies that the game is fully deterministic, that is, given the same inputs, it always produces the same output.

We’re planning on some cool features that rely on the game being fully deterministic, so this something very important to us. But even if we weren’t planning on doing anything with it, determinism is an extremely useful feature to have when it comes time to track down bugs. Instead of getting lengthy descriptions of a crash from testers (which somehow always seems impossible to reproduce), they can include the playback file that lead to the crash along with the bug report, allowing us to catch it in the debugger in no time. Of course, it’s not like we have actual testers, but we affectionately think of LeChimp as one, and he’s always very careful to save all his playback files with every functional test run.

Right now we’re just recording the inputs to the game: delta time every frame and controller inputs. Just with that, the game runs exactly the same time in and time out (thanks to Havok for being deterministic!). Things get more complicated as soon as multiple threads are involved, since the exact timing of context switches between threads can affect the output. Some tools out there, like ReplayDirector claim to address this, but I haven’t looked into it very much.

Both the input file and the same state file are opened, written to, and closed every frame. That way there is no data loss if the game crashes unexpectedly and you get all the input leading up to that frame.

As far as checking that the world state is the same, it’s totally an ad-hoc process. We simply pick some of the obvious state and save them to a file: player positions, enemy position, props transforms, etc. If we ever see something get out of sync, we add it to the game state that gets saved and compared so it doesn’t happen in the future.

No waiting around

So far we have LeChimp running the following with every functional test:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name

followed by

sweetpea -playback=functional_test -level=level_name

At 60 Hz, running 10,000 frames is almost three minutes. 10,000 is just a number we pulled out of a hat. The longer you let the monkey loose with the game, the better the chance of uncovering something. Multiply that by the number of levels and sandboxes, and the functional test now takes quite a while to complete.

A good chunk of the time of the functional test is spent rendering each frame and waiting for the vertical sync signal. But nobody is actually looking at the output [2], so why bother?

We added a couple more command line switches to make the game run without rendering or waiting on vsync. To make that really useful, we also added the ability to force the frame time to be a fixed timestep. So we can run the game like this:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name -render=no -vsync=no -timestep=0.01566

The game will cruise through the simulation as fast as possible, often cramming all three minutes of gameplay into 10 seconds or so. Perfect for poor monitor-less LeChimp.

Testing, testing

There’s even more to LeChimp that just monkeying around. It also runs several other functional tests checking some high-level functionality:

• Player attacks. The player character attacks enemies using each different type of attack and verifies that each attack is successful. This is particularly useful when there are a few types of attacks (which themselves are unit tested), but there are many combinations that can be created between attacks and targets.

• Level restart. LeChimp loads a level and then restarts it hundreds of times, checking for crashes and memory leaks.

• Torture chamber. A tiny level in which the hero (in god mode) frantically mows down hundreds of enemies that get immediately respawned. This is a perfect stress test for performance and hardcoded limits.

We’re writing these tests as we go along. Whenever there’s a feature that seems complex enough, or that it relies on other systems, or that it seems to break repeatedly, we take a few minutes, write a new functional test, and throw it to LeChimp to run with all the others.

Functional tests like these are about as high level as it gets. They deal with actions such as “move the player to the right”, “spawn an enemy here”, or “perform special attack XXX”, so it would make sense to implement them in the same way you implement game logic (which in our case is still C++, although we’re considering a switch to Lua in some not very distant future).

Long Live LeChimp

LeChimp has been invaluable battling against Dr. Chaos. Several times I made a refactoring or introduced a new feature, all the unit tests passed, I checked it in, and a little while later LeChimp screams at us that something is wrong. Once we see the functional test fail, it’s usually pretty obvious how to fix it: a memory pool is too small, or a combination of events that causes the player to enter some unexpected state. Fixing it is a matter of writing a unit test, fixing the logic, and checking it in, all in a few minutes.

A few times, however, the problem hasn’t been that obvious. The world gets out of sync, but only in release mode. Running it from the debugger often results in yet a different state. Sounds like some annoying memory overwrite, or perhaps some uninitialized memory. Any programmer who’s had to deal with this before probably has shivers running down his back. Fortunately, LeChimp has a secret weapon in its arsenal to deal with that. But that’s another story and shall be told another time.

Until then, happy holidays, everybody!

[1] Don’t underestimate the power of keeping programmers happy! At a previous company I used to work for, a manager rewarded one of the programmers by buying him a DVD set of a TV show he was really into. That was only about $40, but they had a huge effect on the programmer’s morale and productivity. Talk about well-spent money.

[2] LeChimp doesn’t even have a monitor, although that sucks sometimes. RemoteDesktop is pretty cool, but it locks up the DirectX surfaces in some weird way, and then the graphics renderer refuses to initialize correctly. So we’re forced to use… get this… NetMeeting! With fake phone rings and all! Ring, ring, calling LeChimp…

I had been using a modified version of CppUnitLite for quite a while, slowly fixing the parts in need of mending, and adding new functionality as it became needed. Eventually I released most of those changes as CppUnitLite2, and I thought that would be the end of that. It turns out that was just the beginning.

Shortly after that, several people started emailing me with new functionality, fixes, patches, suggestions, etc. At the same time, Charles Nicholson was starting to get quite a bit of response as well for his own testing framework, sometimes with similar fixes and suggestions.

If this was going to grow to be a real project, developing over time, supporting users’ needs, and accepting code contributions, clearly it would make sense to join forces. And that’s exactly what we did. We grabbed the best features of each framework and created what we think it’s the best C++ unit-testing framework out there (for our needs anyway).

We took the results and put them up in Sourceforge under a very unrestrictive license, and that’s how UnitTest++ was born.

Our ultimate goal is to apply test-driven development to game development. All of the existing frameworks fell short in one area or another. Specifically, the driving forces behind the design of UnitTest++ are:

• Portability. As game developers, we need to write tests for a variety of platforms, most of which are not supported by normal software packages (all the game consoles). So the ability to easily port the framework to a new platform was very important.
• Simplicity. The simpler the framework, the easier it is to add new features or adapt it to meet new needs, especially in very limited platforms.
• Development speed. Writing and running tests should be as fast and straightforward as possible. We’re going to be running many tests hundreds of times per day, so running the tests should be fast and the results well integrated with the workflow.

UnitTest++ is a fully featured testing framework, with many of the features you may expect from Xunit-style frameworks such as fixtures and reporting. Additionally, here are some of the specific features that make UnitTest++ unique:

• Minimal work required to create a new test. For TDD development, we write lots and lots of tests, so creating a new test should be as quick and painless as possible. UnitTest++ does not require explicit test registrations, and creating new tests requires minimal work.
• Good assert and crash handling. When automated tests are executed, it’s very important not to hang the process or abort the tests any time the program crashes or an assert fails. UnitTest++ will catch and report exceptions as failed tests and print a lot of information about them. It will translate signals to exceptions as well as be able to catch invalid memory accesses and other problems.
• Minimal footprint and minimal reliance on heavy libraries. When you intend to run tests on a Cell SPU with a measly 256 KB of RAM, you really want to be as lean and mean as possible. UnitTest++ is very lightweight, and it will get even lighter weight once strstream is replaced.
• No dynamic memory allocations done by the framework, which makes it much easier to track memory leaks and generally more attractive for embedded systems.
• Multiplatform support. Right now it supports Win32, Linux, and Mac OS X out of the “box,” and other platforms will probably work without any changes. The source code makes use of platform-specific functions whenever necessary, so it is easy to hook up special timers, allocators, or reporters for specific platforms. The code comes with a makefile as well as with project and solution files for Visual Studio 2003 and 2005.
• Full set of unit tests for itself. UnitTest++ was TDD’d from the ground up, including some of the ugly macros (imagine how much fun it was bending the framework to test itself that way!). It goes without saying, all the tests are included with the source code, so you can go totally wild with them

This was just the first release. UnitTest++ is ready for full production work, but we already have plans beyond this release. This is a peek at some of the features we have in mind for upcoming versions:

• Out-of-the-box support for game console platforms (Xbox 360, PS3 PPU and SPU, and Nintendo Revolution). Of course, this is dependent on us getting permission from the console manufacturers to release some code that works with their SDK.
• Ability to trade some features (such as rich check reporting) for some extra memory. In some platforms, like Cell SPUs, you really need to go barebones.
• Suites to group tests together.
• Different reporters (HTML, GUIs, etc). Don’t worry, all those modules will be optional and well separated, so they won’t affect the simplicity of the framework.
• Per-test timings. Maybe the ability to flag some tests as not exceeding a certain amount of time, or just failing any test that goes over a threshold.
• Built-in memory leak detection.

So that’s it. Download the source code and give it a try. We welcome any comments and suggestions (best channels would be through the project mailing list or comments here). We hope you find it useful and that it makes your TDDing even more productive.

See how doing something so apparently backwards as writing unit tests before any code can help with all those problems.

This paper will be presented at the 2006 Game Developers Conference

Noel Llopis and Sean Houghton
High Moon Studios

Printer-friendly format.
Presentation slides.
Sample code from presentation.

1. Introduction

Part of the appeal of programming is that, as Brooks describes, it allows us to build ornate castles in the air, where our imagination is the only limit. As our target platforms increase in power and memory, our castles become larger and more intricate.

However, it’s the same facility to weave code out of thin air that can become our greatest danger. Code has a tendency to grow beyond initial expectations, quickly surpassing our capacity to fully understand it. Our castles in the air can quickly become snowballs that are too large to be moved and shaped to fit our needs.

We have all experienced how development slows down to a crawl towards the end of a project. We have seen first-hand the difficulty of squashing insidious many-headed bugs. We have wrestled with somebody else’s code, just to give up or fully re-write it in despair. We have sat in frustration, unable to do any work for several hours while the game build is broken. We have seen countless hours of work go up in smoke as code is thrown away and started from scratch for the next project.

Code can get too complex for its own good. Milestone pressures, a fluctuating game industry, growing teams and budgets, and the breakneck pace of hardware change don’t help an already difficult situation.

This is where test-driven development comes in.

2. What is test-driven development?

Let’s start by looking at the traditional programming process. Once you decide to implement a specific piece of functionality, you break the problem down mentally into smaller problems, and you start implementing each of them. You write code, and you compile it to know whether things are going well. After a while, you might have written enough code that you can run the game and try to see if the feature works. Maybe you run through a level, checking that feature and making sure nothing else looks obviously broken, or maybe you even step in the debugger to make sure your code is getting called and doing what it’s supposed do. Once you’re happy with it, you check it into source control and move on to another task.

Test-driven development (TDD) turns the programming process around: As before, you break down a problem mentally into smaller problems (or at least a single smaller problem), but now you write a unit test for that small feature, see it fail (since you still haven’t implemented anything), and then write the code to make that test pass. Then you repeat the process with another, very small piece of functionality that will get you closer to the full feature you’re trying to implement. The cycles are very short, perhaps only a minute or two.

Let’s have a more detailed look at the TDD cycle:

• Write a single unit test for a very small piece of functionality.
• Run it and see it fail (in C++, it’s likely that it won’t even compile).
• Write the simplest amount of code that will make the test compile and pass.
• Refactor the code and/or tests. Run tests and see them pass.

A unit test is a test that verifies a single, small behavior of the system (also called a developer test). Specifically, in this context, each unit test deals with something almost trivially small, which can probably be done in less than a minute. For example, it can test that a health pack doesn’t add more health past the player’s maximum, or it can test that a particular effect disables depth writes. We’ll see an example of a simple test in the next section.

How does TDD help us deal with the complexity of software, and, more specifically, how does it solve some of the problems listed in the introduction? These are the benefits, listed roughly in our order of importance:

Better code design

What is the first thing you do when writing some code with TDD? You are forced to use that code in a test. That simple fact makes it so all code is created with the user of the code in mind, not the implementation details. The resulting code is much easier to work with as a result, and it directly solves the problems it was intended to fix. Also, because the code was first created by testing it in isolation from the rest of the code, you will end up with a much more modular, simpler design.

Safety net

With TDD, just about every bit of code has some associated tests with it. That means you can be merciless about refactoring your code, and you can still be confident that it will work if all the tests continue to run. Even though we find refactoring to be an extremely important part of the development process, the safety net goes way beyond refactoring. You can confidently apply obscure performance optimizations to squeeze that last bit of performance out of the hardware and know that nothing is broken. Similarly, changing functionality or adding new feature towards the end of the development cycle suddenly becomes a lot less risky and scary.

Instant feedback

The unit tests provide you with instant feedback up to several times per minute. If at any point you thought tests should be passing and they aren’t, you know something has gone wrong: not an hour ago, not ten minutes ago, but sometime in the last minute. In the worst case, you can just revert your changes and start over. This is quite subjective, but the constant feedback has a surprising morale-boosting effect. No problem seems too large as long as you’re taking small steps in its direction and getting feedback that you’re in the correct track. For some of us, TDD has rekindled the same joy of programming that we discovered in the 8-bit computers we cut our teeth on.

Documentation

What’s worse than uncommented code? Code with outdated comments. We all know how easy it is for comments to get out of date, yet there is very little we can do about it. The unit tests created through TDD serve as a very effective form of documentation. You can browse through them to see what kind of use a class is intended to have, you can look up a particular test to see what kind of assumptions a function makes about its parameters, or you can even comment out some code that makes no sense and see what tests break to give an idea of what it does. The best thing about unit tests as documentation: they can never get out of date. We have found that in codebases that are developed with TDD, comments have almost disappeared, being used only to explain why something was done in a particular way, or to document what paper an algorithm we implemented came from.

One thing that should be clear and is important to stress is that TDD is not just writing unit tests. TDD is a development methodology, not a testing one. That’s why TDD’s benefits deal with better code design and structure, ease of refactoring, etc, and not with correctness. TDD ensures that your code does whatever you wanted it to do, not that it does it correctly.

3. Implementing TDD

Let’s put TDD in practice by following the normal practice and writing our first test. Let’s assume we already have a game up and running, and our task is to implement health powerups. The very first thing we want to implement is that if the pl
ayer walks over a health powerup, he receives some amount of health. That’s actually even more complicated than what we want for our small testing steps, so let’s start with an even simpler test that requires the least amount of effort: if we have a player and a powerup, and the player isn’t anywhere near the powerup, his amount of health doesn’t change. Simple, right?

Ideally, how would we like to test that? Probably by writing some code like this:

World world;
const initialHealth = 60;
Player player(initialHealth);
world.Add(&player, Transform(AxisY, 0, Vector3(10,0,10)));
HealthPowerup powerup;
world.Add(&powerup, Transform(AxisY, 0, Vector3(-10,0,20)));
world.Update(0.1f);
CHECK_EQUAL(initialHealth, player.GetHealth());

And that’s exactly what the test will look like, only we’ll have to surround it with a macro to take care of all the bookkeeping and to give it a descriptive name. Our full test looks like this:

TEST (PlayersHealthDoesNotIncreaseWhileFarFromHealthPowerup)
{
    World world;
    const initialHealth = 60;
    Player player(initialHealth);
    world.Add(&player, Transform(AxisY, 0, Vector3(10,0,10)));
    HealthPowerup powerup;
    world.Add(&powerup, Transform(AxisY, 0, Vector3(-10,0,20)));
    world.Update(0.1f); CHECK_EQUAL(initialHealth, player.GetHealth());
}

The macros TEST and CHECK_EQUAL are part of a unit testing framework. The framework is intended to make the task for writing and running unit tests as simple as possible. As you can see, it can’t get much easier than that. There are a variety of freely-available unit-test frameworks for C++ and other languages. We recommend UnitTest++, which is a lightweight C++ unit-test framework that supports multiple platforms, is easy to adapt and port, and was created after using TDD in games for several years.

Now we compile this code and we get the following output:

Running unit tests...
1 tests run There were no test failures.
Test time: 0 seconds.

The CHECK_EQUAL macro is the part that performs the actual check, and it takes as parameters the expected value and the actual value. If they are different, it will mark the test as failed, and output a message with as much information as possible. As a bonus, UnitTest++ formats the failed test message so it can be parsed by Visual Studio and it is easy to navigate to the location of the test.

The reason we have macros like CHECK_EQUAL is that unit tests need to be fully automated. There should be no need for visual inspection or reading some text. The test program should know whether any tests failed or not without any ambiguity, that way it can be easily integrated into the build process.

4. How to test

When writing unit tests, there are three main ways of testing your code:

Return value

Make a function call, and check the return value. This is the most direct way of testing, and it works great for functions that do computations and return the computed value. Because this is the easiest and most straightforward way of testing, we should use this approach whenever possible.

For example, a function called GetNearestEnemy() would be a perfect candidate to be tested this way. We can easily set up a world with a couple of enemies, call that function, and verify that it returns the enemy entity we expected.

Beware of testing functions that return boolean results indicating if the function failed or succeeded. You really want to test that the function does things correctly, not just that it reports it did them.

Object state

Make a function call, then check that the state of the object or some part of the system has changed correctly. This tests that state changes directly, so it’s also a very straightforward form of testing.

For example, we could send an event of nearby noise to an AI in “idle” state and check that its state changes to “alert” in response.

Sometimes it might force you to expose some state that would otherwise be private, but if it’s something that you need to test from the user point of view, then making it public is probably not a bad idea anyway. It might lead to larger interfaces than absolutely necessary, but it also sometimes shows that a class really should be split into two, and one of the classes should contain the other one.

Object interaction

This is the trickiest aspect to test. Here we make a function call, and we want to test that the object under test did a sequence of actions with other objects. We don’t really care about state, just that a certain number of function calls happened. The most common testing pattern for this situation is a mock object. A mock object is an object that implements the interface of another object, but its only purpose is to help with the test. For example, a mock object could record what functions were called and what values were passed, or could be set up to return a set of fixed values when one of its functions is called.

Because this is the most complex form of testing, we recommend using it only when the other two forms of testing are not possible. Using mock objects frequently could be an indication that the code relies too much on heavy objects with complex interactions instead of many, loosely-coupled, simpler objects.

A good situation for using a mock object could be testing HUD rendering. We want to verify that the HUD elements are rendered in a specific order, so we create a mock for the rendering canvas which will keep an ordered list of the elements that get passed to it. We pass the mocked rendering canvas to the HUD renderer and make the render call. Then we can examine that the list in our mocked canvas matches our expectations.

5. Best practices

This is a set of best practices we have found to be particularly important for all of our development with TDD.

Run tests frequently

Being able to write tests easily is a requirement for successfully rolling out TDD. Nobody wants to spend extra time writing unit tests when they can be implementing the features that are due for this milestone. But even more importantly, tests should run very frequently, and a failed test should be treated the same way as a failed build.

At High Moon, every library has a separate test project that creates an executable that links with the library and runs all the tests. We have hooked up running the executable itself as the post-build step in Visual Studio (or as the last command in a make file). That means that making any changes to the library or tests and triggering a compile will also run all the tests. Additionally, since the test executable returns a value with the number of failed tests, a failing test will return a non-zero value, which is interpreted by the build chain as a failed build. This makes it so everybody is running unit tests for all code all the time, which greatly improves the build stability.

In addition to running tests locally, the build server also runs all the unit tests at the same time it does code builds. In our case, since the tests are a postbuild step, there was nothing special we had to do in the build server, and a failed test would be reported just as code that didn’t compile correctly.

Test only code under test

This is a key practice for writing good unit tests. The most important reason to minimize the
amount of code involved in a test is to keep things simple. Ideally, when something breaks, you want only a small number of tests failing so you can quickly pinpoint the problem. If some tests involve a large amount of the codebase, they will constantly break for unrelated reasons.

Also, if you are able to use the code under test with a minimal amount of other code or libraries, it means you are creating very modular, self-contained code, which will help with the overall design. Finally, another very good reason to avoid involving extra code is that tests that only work on a small set of code are typically much faster than tests that involve large systems and complex initialization/shutdown sequences.

Continuing with the health powerup example, notice that the test involved a World object, a Player object, and a HealthPowerup object. There was no graphics system initialization, databases, etc. The World object can be a very lightweight container for game entities without any other dependencies. Try setting something like that in your current engine and you might be surprised by how many implicit dependencies you find in different parts of the engine.

A test that involves a lot of code across many different systems is usually referred to as a functional test(also called a customer test). Functional tests are extremely useful, especially if they are fully automated, but they fill a very different role than unit tests.

Keep tests simple

This is related to the previous best practice. You want a test to check one thing and only one thing; that way, when something breaks, it’s immediately clear what went wrong.

A good start is to label each test clearly with a name that describes exactly what the test is supposed to do. For example, a test named PlayerHealth is not very helpful. A much better name would be PlayerHealthGoesUpWhenRunningOverHealthPowerup.

Keeping each unit test to just a handful of lines makes it much easier to understand at a glance. In our case, it is unusual to have unit tests that are more than 15 lines long. Needing to write overly long unit tests is usually a sign that the test is involving too much code and could probably be re-written in a better way.

We also found that limiting each unit test to a single check statement or two resulted in tests that were the easiest to understand. Sometimes that means you’ll have to write some duplicate setup code between two tests, but it’s a small price to pay to keep the tests as simple as possible.

Whenever you have some common code in two or more tests, you can use a fixture. A fixture is a set of common code that is executed before and after each test that uses that fixture. Using fixtures can cut down tremendously on the amount of test code you have to write. Any decent unit test framework should support fixtures, so use them whenever they’re needed.

Keep tests independent

Unit tests should be completely independent of each other. Creating an object in a test and reusing it in a different test is asking for trouble for the same reasons as we discussed earlier: when a test fails, you want to be able to zero in on the failing test and the problem that is causing it to fail. Having tests depend on each other will cause chains of failing tests, making it difficult to track the problem down to the source.

Keep tests very fast

The unit tests we write are compiled and executed as a postbuild step. We have hundreds or thousands of tests per library, so that means they have to run blazingly fast or they’ll get in the way. If unit tests are only dealing with the minimum amount of code, they should be able to run really fast. That means they shouldn’t be talking to the hardware, they shouldn’t be initializing and shutting down expensive systems, and they most definitely should not be doing any file I/O.

All our unit tests are timed (another feature of UnitTest++), and the overall time for the test run is printed after it runs. As a general rule, whenever a set of unit tests goes over two seconds, it means something is wrong and we try to fix it (even if a unit test takes a full ms, which is a huge amount for a unit test, you can still run 1000 of them in a single second).

6. TDD and game development

Applying TDD to game development has its own unique challenges that are not usually discussed in the TDD literature.

Different platforms

Most game developers today need to develop for a variety of platforms: PCs (Windows, Macs, or Linux), game consoles, handhelds, etc. Even though at High Moon we develop console games, we use Windows as our primary development environment because of the good development tools and the fast iteration time. That means we always have a version of our engine and tools that runs under Windows, which makes running unit tests very easy and convenient.

We also wanted to run the unit tests in each platform we develop for, but we had to make a few changes to compensate for the shortcomings of those platforms. The minimum support that we needed was the ability to run an executable through the command line and capture the output and, ideally, the return code. Surprisingly, none of the game console environments we develop for supports that simple operation, so we were forced to write a small set of programs using the system API to do exactly that.

After we wrote those small utility programs, we were able to run unit tests on the consoles just like we did under Windows. The main difference was that running any program on the consoles had a noticeable startup time delay. It’s a matter of just a few seconds, but it was too long to run the unit tests automatically as a postbuild step after every compilation. Besides, we don’t yet have one dev kit for every developer station, so we couldn’t count on always having a target platform available. Because of that, our unit tests on platforms other than Windows are only executed manually, and by the build server. It is not ideal, but the large majority of the problems show up in the Windows tests, so as long as those continue being run all the time, we catch most problems on time.

Graphics, middleware, and other APIs

Probably the biggest barrier that people see to doing unit testing and TDD with games is how to deal with graphics. It’s certainly not the textbook-perfect example you’ll read about in TDD books, but it’s certainly possible.

The first thing to realize is that graphics are just part of a game. It is common sense and a good software engineering practice to keep all the graphics-related code in one library or module, and make the rest of the game independent of the platform graphics API and hardware. Once we had that organization, we were able to test any part of the game or engine without having to worry about graphics.

Ideally, we wanted to develop the graphics renderer library with TDD as well, and there’s no way to avoid dealing with platform-specific graphics calls. We tried three different approaches, from most involved to least involved:

• Catch all graphics function calls. We inserted a layer between the graphics renderer and the graphics API that exactly mimicked the platform API. That way we could make any graphics API calls that we needed without having to worry about the underlying hardware. As a bonus, that layer could report which functions were called and with what parameters. This approach made for extremely thorough tests. The testing layer could be removed in the final build and the functions would call directly into the graphics API so there would be no performance penalties. However, it was quite labor-intensive, especially for the Direct3D API because it uses classes and not just plain functions like OpenGL.
• Check for state. Another approach is to actually work on the graphics hardware and check that things are working correctly by querying the graphics AP
  I state. For example, rendering a certain mesh should set a specific vertex declaration. This approach is much simpler, but there were some things we couldn’t test. For example, we could render a mesh, but we wouldn’t be able to find out how many triangles were sent to the hardware. Also, since OpenGL is so state-based, it was important for most functions to clean up after themselves, so it was often very hard to check for any state. On platforms in which the internals of the graphics API are more open, you might be able to examine more states by looking at the graphics command buffer.
• Isolate graphics calls. When all else fails, this is a useful technique for dealing with calls into external APIs. Isolate an API function call or a group of API function calls into a single function, and test that your function is called at the right time. The function that calls into the graphics API itself won’t be tested, but all it does it make some straight calls. Remember, what we really want to test is that our code behaves correctly, not that the graphics API works as documented.

We started with the first approach, wrapping the API as we were using it, but the amount of extra work we had to do to wrap complex APIs like Direct3D and OpenGL quickly became overwhelming. Maybe if we were working on a commercial graphics rendering middleware it would be worth the effort. For us, after doing that for a few weeks, we realized that the benefits we were deriving from it weren’t worth the time we were spending. It was also discouraging us from using new API functions just because they hadn’t been wrapped before.

In the end, we ended up settling for a combination for the second and third approaches: testing the state whenever possible, and isolating the calls the rest of the time. One of the drawbacks of this approach is that we’re working directly with the hardware, so tests actually do initialize and shutdown the graphics system every time (except for those platforms out there that can only initialize the graphics system once), so they can be more time consuming. On the positive side, because the tests are exercising the graphics API and hardware directly, we catch things that the first approach wouldn’t have caught (for example, sending incorrect parameters to a function).

The same three approaches can be used for any middleware or external API. The important thing to remember is to make sure you’re testing your code, not the API itself. Keeping that in mind helps to write true unit tests and not functional tests for the API.

A good example of this approach is how we wrote our input system with TDD. The input system deals with getting input values from gamepads and other controllers. Even though at first glance it might seem like the whole system is about making system calls to poll the data, there is a lot of common code that is totally platform independent: button mappings, edge detection, filtering, plugging/unplugging controllers, etc.

In our system, we have an interface named GameController with a Sample() function. Each platform implements a platform-specific version of the GameController (for example D3DController) and does the raw sampling of all buttons and axis through platform-specific API calls. That part is fully untested. The rest of the input system works through the GameController interface, but for all the tests we provide a MockGameController() which allows us to control what input values we feed to the tests.

We are hoping that as TDD becomes more common in the games industry, middleware providers will make their APIs more TDD friendly and even ship with their unit tests.

Third-party game engines

If dealing with APIs was not straightforward, working with a full third-party game engine that was not developed with TDD is even more challenging. After all, an API is just a list of classes or functions, and you have a lot of control over how and when they get called. Working with a full game engine, you might end up writing a small module that is fully surrounded by the engine code. If the engine was not developed with TDD, it can be very difficult to use your code in isolation or figure out how to break things up so they can be tested.

At High Moon, we are working on several projects with the Unreal Engine 3, and we’re using TDD on them. Even though the engine is not TDD-friendly at all, we still find more benefit from doing TDD with it. Imagine then how useful TDD can be on a full engine developed with TDD from the start.

The most important thing to do in this situation is to separate the code we write as much as possible from the engine code. Not only does this allow us to unit-test our code in isolation, but we also keep future merges with Epic’s codebase as simple as possible. However, that severely limits the amount of large-scale refactorings we can do to keep things as testable as possible, so it’s a tough trade-off..

One unique aspect of the Unreal Engine is that it makes heavy use of its scripting language, UnrealScript. Because a lot of the high-level game engine is written in UnrealScript, we had little choice but to use it to write most of the game code. The first thing we had to do was to create a unit-testing framework for UnrealScript. The resulting framework is called UnUnit and is freely available for download through UDN (Unreal Developer Network). Several companies working with the Unreal Engine are currently using UnUnit for their game projects.

UnrealScript is a surprisingly good language for unit testing. By default, all functions are virtual, so overriding behaviors and creating mocks is simpler than C++. Also, compilation is very fast, which keeps iteration times low (large, badly organized C++ codebases can take a long time to link). All of that makes TDD with UnrealScript not just possible, but very effective.

Randomness and games

Most games involve a fair amount of randomness: the next footstep sound you play can be any one of a set of sounds, the next particle emitted has a random speed between a minimum and a maximum, etc. As a general rule, you want to remove the randomness from your tests. A few times we caught ourselves starting to write a unit test that called a function many times in a loop and then averaged the result, but that’s totally the wrong way to go. Unit tests are supposed to be simple and fast, so any loops in a test are usually very suspect.

A better approach is to separate the random decision from the code that uses it. For example, instead of just having a PlayFootstep() function that takes care of computing a random footstep and then playing it, we can break it into int ComputeNextFootstep() which is just a random function call, and a PlayFootstep(int index), which we can now test very easily.

Another approach is to take control over the random number generator at the beginning of the test and rig the output so we know what sequence of numbers is going to come up. We can do this either by mocking the random number generator object or by setting some global state on the random number generator, depending on how it is implemented. Then tests can count on a specific sequence of numbers being generated and check the results accordingly.

High-level game scripts

How far is it worth taking TDD? Should TDD be used for every single line of code? How about game-specific script code? The answer depends on your priorities and game.

As a general rule, we find that if any other part of the game is going to depend on the code we are writing, then it’s probably worth doing it with TDD and having a full set of unit tests for it. Otherwise, if it’s a one-shot deal with the highest-level code, then it’s probably fine without TDD. Also, code at that level is often writ
ten by designers in a game-specific scripting language, so TDD might not be a viable option.

An example of some code that we would not use TDD for is trigger code: when the player goes around the corner, wake up two AIs and trigger a different background music.

Functional tests are a great complement to unit tests, so it’s important not to forget about them. Automated functional tests can be extremely useful catching high-level problems, gathering performance and memory utilization data, and removing some of the mechanical testing from QA and letting them concentrate on issues such as gameplay balance and flow.

7. Lessons learned

Design and TDD

One of the most important benefits we get from TDD is the better code design that it creates. For this to happen, it’s important to let the tests guide the code and not the other way around. It can be disconcerting to always be looking only a few minutes into the future and implementing the “simplest thing that could possibly work,” but it really works. The key to working with TDD is to realize that refactoring is an integral part of the development process and should happen regularly every few tests. Good design will come up through those refactorings as needed by the tests we have written and the code we have implemented.

Does this mean you shouldn’t do any design ahead of time? That depends on your situation and experience, and the type of code you’re writing. In general, the less known or more likely to change something is, the less design we do. If you’re working on the 10^th iteration of a well-known sports game franchise, for a known platform, and you know exactly what you’re going to do, up front design can be more beneficial.

In our case, we prefer to discuss very rough concepts of where we expect something to go, what it should do in the future, and maybe some very, very rough organizational structure. In general, we prefer not to even think of classes or draw UML diagrams on whiteboards before starting, because such discussions can then lead implementation too much. We prefer to let the tests guide us, and if at some point we’re going away from what we had in mind at the beginning, we can stop to reconsider if we’re heading in the right direction. Most of the time we are, and we simply hadn’t thought things through enough at the beginning (or thought through them too much and we simply didn’t need that level of flexibility).

TDD and high-level code

One of the questions we had when we jumped into TDD is whether it was going to hold for high-level code. We had seen in practice from previous projects that we can certainly do TDD to create low-level and intermediate-level libraries (math, collision, messaging, etc). But would it really work for high-level code that would build on low-level code?

The answer is an unconditional yes. We have developed a full codebase doing TDD from the start, and we had no difficulty writing high-level code with TDD. Things like character state machines, game flow, or specific game entities were done through TDD without any problems, and greatly benefited from the TDD approach.

Two tips that helped us apply TDD to high-level code:

• Keep tests as true unit tests. When working on high-level code, it can be tempting to let tests degenerate into functional tests that involve the whole game engine. Don’t do that. Even if it takes a few more minutes to make an enemy character testable in isolation, it is well worth it in the long run.
• Keep the engine architecture as flat as possible. This is a general good practice, but it is more so with TDD. Clearly, avoid having your engine as one big, intertwined module. But even if it’s broken down into libraries or modules, keep them as independent of each other as possible instead of as one long list of dependencies. For example, there’s no reason the AI module needs to know anything about graphics or sound, but it will need to know about a world representation and have access to the messaging system.

Tests as a measure of progress

Software developers have been struggling for a long time to find a good measure of progress or work done. Some projects use code line counts to determine progress, others use features completed, while others just look at the number of hours spent at the office. Clearly, none of those approaches are ideal.

We have found that counting the number of unit tests is a really good measure of progress. Especially when tests are developed through TDD, they deal less with corner cases and more with program features. If a library has 500 tests associated with it, we can say that it’s roughly half as complex as a library with 1000 tests.

As an added benefit, people tend to behave based on how they think they are being measured. So if this is made clear, people will be much more likely to be strict about applying TDD and not falling back on writing code without tests. In our groups, we have a test chart, which is updated every day and shows the total count of tests. If tests aren’t going up, or they’re going up more slowly than other times, we know something is wrong and we’re not making much progress.

Build stability

As we expected, having almost full code coverage with unit tests greatly improves the code stability. Broken builds happen much less frequently, and they’re usually caused by a missing file or a different platform not compiling correctly. What was a pleasant surprise is that broken builds are much easier to fix. By following our best practices for unit tests, it becomes really obvious when something breaks, and we can usually check in a fix right away. Builds are rarely broken for more than a few minutes at a time, which can completely change how you organize your code in source control.

Amount of code

It will probably come as no surprise to anybody that writing code with TDD results in more code being written. Sometimes the test code is as large as the code under test itself. Even though this can initially sound like a scary proposition, it really isn’t a problem. The extra code does not take significantly longer to write initially, and it allows us to move a lot faster once we have accumulated more code and we need to refactor or optimize the code in any way.

Just because we have twice as much code it doesn’t mean we have twice the complexity. Quite the contrary. The test code has been written so it’s extremely simple, so its complexity is minimal. Its only job is to check the non-test code, so it’s keeping an eye out for us, helping us, and letting us know when something breaks. Additionally, the TDD approach results in much simpler, decoupled code. Overall, the complexity of the codebase is greatly reduced with TDD.

A good analogy to TDD is scaffolding in a building construction. It’s not something you’re going to deliver to your customer, but it’s an absolute necessity, it needs some time commitment to set it up, and you wouldn’t dream of doing any complex building without it.

Development speed

This is the million-dollar question: Does TDD slow development? We’re not doing TDD because it’s a “good” thing to do, but because we want to ship a better-quality product faster and cheaper than we did before. If TDD doesn’t help with this, then there’s very little point to it (other than keeping programmers happy).

As with other aspects of software development, it is difficult to make an objective study and measure exactly the effects of TDD versus a control group. Things change too much from project to project and team to team. Still, some initial studies have some interesting initial findings (http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf), even if the study was not very rigorous and had a very small sample.

Our experience is that TDD, like any other new development technique, slows development down at the beginning while the team is learning it and becoming familiar with it. Th
is can take as long as a couple of months. Once the team is over the hump, and given the right tools and development environment, the impact of TDD on development speed is minimal.

It is possible to simply write code faster than it is to write the tests first, but as soon as that code needs to be refactored, debugged, used by somebody else, or simply gets more complex, any time savings quickly disappear. The larger the team and the more complex the problem, the more TDD saves time in the long run. It’s very important to not fall for the temptation of skipping TDD for a very short-term gain (aka milestone of the month).

Ideally, TDD and refactoring can flatten out the classical cost-of-change over time curve into something like this. Notice the trade-off between slightly slower short-term speed vs. massive gains later on.

Adopting TDD

We started with TDD by first trying it with a small group on a separate project. In our particular case, not only were we doing TDD, but we were doing all the extreme programming practices (pair programming, continuous integration, collective code ownership, etc). When doing this, it can be extremely useful to have somebody on board with previous TDD experience who can help guide the process, set up the environment, and avoid common early mistakes.

Applying it on a small scale initially was very useful in many different ways. It let the team become more comfortable with TDD and get to the point where they are as productive as they were before. It also makes any bad practices apparent early on and they can be dealt with before rolling it out (making overly complex tests or functional, rather than unit tests).

It also had the effect of creating new evangelists for the new technique. Members of that team were then moved on to other teams, where they became the resident TDD expert.

TDD fits best with other agile development practices. Having an agile mindset fits very well with the idea of finding the design through tests. Pair programming can be extremely helpful, especially at the beginning while rolling out TDD and getting everybody on board. Pair programing also has the added advantage of making programmers less likely to skip writing tests, which can be a common reaction early on.

If you’re interested in applying TDD but you don’t have a commitment from your manager or lead, it is possible to start doing it on the side on your assigned tasks. The most important thing is to make sure your tests run automatically (remember the postbuild trick). Slowly, other people might see how useful the tests are, or how much easier it is to refactor that code, and eventually the time might be right to roll out TDD.

A few people can have a hard time switching over to the TDD mentality, but to take full advantage of TDD, it is best to let the design emerge from the tests and the refactoring rather than trying to plan everything up front. It is very interesting to note though, that most programmers at High Moon quickly accepted TDD, and soon became very enthusiastic about it and started using it in all their code, including their home projects.

There is no doubt that the best situation to roll out TDD is with a fresh new codebase. Not everybody has that luxury, so it is important to learn how to apply TDD even with an existing legacy codebase. Michael Feathers’ book Working Effectively with Legacy Code, explains exactly that situation and gives some very good guidelines on how to go about unit testing with a codebase without existing unit tests. Sometimes it just takes a little bit of refactoring of the existing code and it becomes a lot easier to add new tests.

8. Conclusion

Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and we’re convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development.

9. Resources

This is required reading before you start doing any TDD:

• Beck, Kent, Test Driven Development: By Example. Addison-Wesley, 2002

These other books will also come in handy:

• Fowler, Martin, Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999
• Astels, David, Test Driven Development: A Practical Guide. Prentice Hall, 2003
• Beck, Kent, and Cynthia Andres, Extreme Programming Explained: Embrace Change (2nd Edition), Addison-Wesley, 2004

Very useful mailinglists and web sites:

• TestDriven.com
• TDD mailing list

Resources dealing with TDD and agile game development:

• Noel’s blog, Games from Within
• Clinton Keith’s blog, Agile Game Development

C++ unit-testing frameworks:

• UnitTest++. A C++ unit-testing framework designed with game development in mind.
• CppUnit
• Comparison of C++ unit-test frameworks

CppUnitLite2_1_1.tar.gz

About a year ago, I wrote an article comparing unit test frameworks that went to become one of the most popular in this site. The article concluded with the recommendation of one of the less well known frameworks: CxxTest.

Shortly after writing that article, it came time to roll out test-driven development at work. Interestingly, once I weighted in all the requirements, I ended up not following my own advice. Instead, I decided to go with a customized version of CppUnitLite, which was listed in the article as the second choice. The idea of an extremely simple unit testing framework was too attractive to pass up in an environment where we have to support a large variety of environments and target platforms.

At this point, we have been using CppUnitLite2 for a year at High Moon Studios to do TDD on Windows, Xbox 360, and some PS3 (we even have a stripped-down version that runs on a PS3 SPU). It has been used to unit test libraries of an engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2 was originally released along with my article a year ago. It was a heavily modified version of the original CppUnitLite by Michael Feathers. This version introduces the changes we made at work as different issues came up and we needed more functionality from the framework. In particular, Jim Tilander and Charles Nicholson were responsible for a lot of those changes. By now, I doubt there’s a single line left from the original code. Additionally, CppUnitLite2 itself has a set of tests using itself in an interestingly recursive way.

What’s new

Fixtures created and destroyed around each test.

I can’t believe that the original version of CppUnitLite2 didn’t have this feature. It was the biggest glaring problem with the framework (I even pointed it out in the article). Each fixture used to be created at static initialization time, which, apart from being a really bad idea, caused a lot of objects to be created and live in memory at the same time. Considering that some of those objects would initialize/shutdown some important game systems, we clearly needed to manage their lifetime a bit better.

The test macro was changed to register a temporary test with the TestRegistry, which in turn will create the fixture at execution time. Now fixtures and tests are created just before each test, and destroyed right after, just as you would expect from any sane framework.

For those masochists out there, here’s the ugly macro for a test with fixture in all its glory. Notice how the TestRegistrar registers the dummy test class that we created as part of the macro.

 #define TEST_F(fixture, test_name)                                                      \ struct fixture##test_name : public fixture {                                        \ fixture##test_name(const char * name_) : m_name(name_) {}                       \ void test_name(TestResult& result_);                                            \ const char * m_name;                                                            \ };                                                                                  \ class fixture##test_name##Test : public Test                                        \ {                                                                                   \ public:                                                                         \ fixture##test_name##Test ()                                                 \ : Test (#test_name "Test", __FILE__, __LINE__) {}                       \ protected:                                                                      \ virtual void RunTest (TestResult& result_);                                 \ } fixture##test_name##Instance;                                                     \ TestRegistrar fixture##test_name##_registrar (&fixture##test_name##Instance);       \ void fixture##test_name##Test::RunTest (TestResult& result_) {                      \ fixture##test_name mt(m_name);                                                  \ mt.test_name(result_);                                                          \ }                                                                                   \ void fixture ## test_name::test_name(TestResult& result_)

Improved check statements

Another one of the major weaknesses of CppUnitLite as it stood was the check statements. You needed to have a custom check macro for each data type you cared to check. The generic CHECK_EQUAL macro will now work on any data type as long as it can be compared with operator == and output to a stream.

This version of CppUnitLite2 also includes some special macros to check arrays with one and two dimensions. Again, this works on any data type that supports operator == and operator[]. It came it very handy for checking equality of vectors and matrices in a game engine. For example, the following code checks that two matrices are equal:

Ironically, in some heavily restricted environments (like the PS3 cell processors), we can’t use streams, so in that case we’re forced to go back to the old specialized check macros.

Invariant statements in checks

Another really annoying feature of the previous version of CppUnitLite2 was that the code in a check statement was called multiple times, which could cause strange side effects.

Consider the following code:

The way the check statement was expanded out, both those functions could be evaluated multiple times. If the functions had side effects, they would be executed several times, causing unexpected results (for example, the check could fail but the printed result could be different). The fix wasn’t trivial because we couldn’t save off the value of each statement because they could be of any type and we have no way of finding the type from within the macro.

The check macro now expands out to a templated function, which is able to evaluate both statements only once and work with the saved value to avoid any surprising side effects.

Optional fixture initialization

As we continued hammering on the unit test framework for all our development, one clear pattern appeared that wasn’t handled by the framework: Being able to create fixtures with parameters defined in the test. For example, the fixture might do ten different steps, but we want to set a particular value in one of those steps, and by the time control reaches the test code itself it’s too late.

We started getting around that by creating different fixtures, and then by inhereting from other fixtures and changing some parameters. Both of those solutions were less than ideal (lots of code duplication and added complexity). Things got out of hand when we made the fixtures a templated class on a value, and created them by specifying the value as part of the texture name. Now it was a pain to debug and things were even less clear.

So we finally implemented it natively in CppUnitLite2 as fixtures with initialization parameters. We introduced a new macro, TEST_FP (test with fixture and parameters), which takes any parameters you want to pass to the fixture constuctor.

Now you can declare tests like this:

We should prob
ably simplify that to avoid repeating the fixture name, but that works fine for now.

Include cleanup

The original version of CppUnitLite was pretty casual about what headers it included. Unfortunately, that means that all test code were automatically exposed to a variety of standard headers, whether you wanted it or not. This version is a lot more strict, and the only header that will get pulled in from using it will be iosfwd, which is even a fairly lightweight one.

Memory allocations

We have very strict memory allocation wrappers around our tests, and if any memory leaks, it is reported as a failed test. Since checking for memory allocations is very platform specific, it is left out of the framework (although it could be added pretty easily in the platform-specific sections). In any case, some of the static-initialization time allocations were confusing the memory tracker, so we removed them completely. All it means is that we use an array to register tests instead of a std::vector and we cap the maximum number of tests to a fixed amount. If you need more than the default 5000 tests, go right ahead and add a few zeros.

New license

Previous versions of CppUnitLite were not released with an explicit license. To make things clear, I explicitly added the license text for the MIT license with an added restriction. The MIT license allows you to use the code in any project, commercial or otherwise, without any restrictions. The added restriction to the license prohibits its use in any military applications or projects with any military funding.

Ratings

How does this release of the CppUnitLite2 compare with respect to my initial set of features I used to rate the different unit tests frameworks?

• Minimal amount of work needed to add new tests. Nothing has changed there. Still passes with flying colors.
• Easy to modify and port. That has always been the strong suite of this framework.
• Supports setup/teardown steps (fixtures). Much improved now by creating them correctly around each test.
• Handles exceptions and crashes well. No changes there. Still one of the best I’ve seen.
• Good assert functionality. The check macros have been much improved to match the best frameworks out there.
• Supports different outputs. Yup. Still there.
• Supports suites. It still doesn’t. It just shows that I really haven’t needed this feature yet.

Future work

So, what’s next for CppUnitLite2? It’s really quite usable right now, but I could see a few changes happening in the next year, and maybe a few others will come up and surprise me.

Test registration is based on static initialization of global variables, which is not very reliable across compliers. For example, if you try to put the tests in a static library, MSVC will strip out the registration code leaving you with no tests. To get around that, we could introduce a custom build step in a scripting language that would generate a simple file with explicit registration of tests. This could also be used to collect tests from many different libraries into a single executable.

A couple of times it would have been convenient to have parameterized tests (I think that’s the correct term for it). Basically, to have tests that could run with different data. Something like this:

 TEST_PARAM(MyCustomTest, int a, int b) { CHECK (something); CHECK(something else); CHECK(yadda yadda); }  TEST(MyTest) { CALL_TEST(MyCustomTest, 10, 5); } TEST(MyTest) { CALL_TEST(MyCustomTest, 12, 7); }

The key point is that a test fails in the parameterized test, it should report the error correctly indicating where it came from.

This is actually pretty easy to implement with a couple of macros. We simply haven’t needed it more than a couple of times, so we haven’t bothered yet. But I imagine next time we run up against this we’ll bite the bullet and implement it.

Wrap-up

Charles had so much fun working with this framework, he decided to write his own from scratch. He made it available on his web site, so go check it out.

Thanks to High Moon Studios for being open and letting us release the changes we made and share them back with everybody. If you end up using the framework and you make any changes, drop me an email and I’ll add them for the next release.

This third and last part will cover some tips that should help smooth out the learning curve and make you more productive from the start. We’ll also look at some of the consequences of applying TDD to a project and what kind of things you can expect on the other side of the looking glass.

Tips from the trenches

When I started doing test-driven development, I was going through the motions, following the advice from Kent Beck’s book, and there was no doubt I was getting a lot of benefit in what I was doing. However, I recently looked back at the tests I was writing back then and I was a bit shocked. They’re quite different from what I would write today. In particular, they were not very clean, maintainable tests. These are some tips that I wish I had known when I started out.

Use fixtures extensively

Using fixtures (objects that automatically set and destroy some state for you for every test), is a great way to keep tests small and to the point. All the setup and shutdown code gets refactored away into the fixture and the test can deal just with the things we’re testing. They will also make the tests much easier to refactor later on (which, if you follow TDD, you will be doing extensively).

Make sure your unit test framework allows you to have fixtures and that you can create them as easily as possible.

The rule I follow for using fixtures is the following (very similar to other XP rules about refactoring): the first time I write a test that has some setup and teardown, I just put it in the test itself because it’s the fastest way to get the test up and running. The second time I write a test that requires a similar setup and teardown, I also put it in the test, make the test pass, and then I consider whether to fold it into a fixture or not (depending on how extensive the setup code is). The third time I write a test that needs the same code, I also put it in the test first, but then I make a fixture right away.

I try to never refactor tests into a fixture while I have failing tests, though. First make the tests pass, then move common code into the fixture (which is part of the test refactoring step if you remember the first article).

Keep your tests short and simple

This one is really important. Probably the most important of all the advice in this article. So if you’re only going to take one thing away, make sure it’s this one.

When developing using TDD, you will create as much test code as production code. Sometimes as much as one and a half times as much! That code isn’t set in stone either. Quite the contrary, it’s supposed to be your safety net, and you’ll be changing it and moving it around as you create new code and refactor existing classes.

That means that test code should be very easy to change and refactor itself. It’s like the scaffolding around a building. It’s not enough to make it reach places and be secure. You have to make sure you can take it down and move it around as needed.

I’m a big fan of simple code, but it’s extremely important that your tests should be as simple as possible. The idea is that when a test fails, you want to know right away what failed. If the test itself is 100 lines long and is full of check statements and loops, you’ll have to resort to the debugger to find out what went wrong.

How do we keep the tests as simple as possible?

• Keep tests extremely short. I try to keep my tests under about 10 lines. Preferably less. I want something I can look at and know at a glance what it does. My favorite tests are the ones that are one or two lines long.
• Label your tests correctly. The name of your test should tell you what it does. Don’t call your test TestHealth (what exactly does that do?). Instead, call it DamageLowersHealth.
• Aim to have one CHECK statement per test. The idea here is that you want to be testing only one thing at a time. If there’s something else to test, even if it’s with the same inputs, write a new test. Not everybody buys into this, but it has made a huge difference in the tests I write. When I started out doing TDD I was not shy about making more complex tests with multiple CHECKs. Then, when a test failed, it wasn’t always immediate obvious what had gone wrong. Clearly to do this effectively you need to use fixtures as described in the previous item.

Here’s an example of what I consider a bad unit test:

// BAD UNIT TEST!
TEST (TestArmor)
{
    ArmorComponent armor(100, 10);
    CHECK_EQUALS (100, armor.Amount());
    CHECK_EQUALS (10, armor.RechargeRate());
    armor.ApplyDamage(30);
    CHECK_EQUALS (70, armor.Amount());
    armor.ApplyDamage(80);
    CHECK_EQUALS (0, armor.Amount());
    armor.Update(1.0f);
    CHECK_EQUALS (10, armor.Amount());
    armor.Update(10.0f);
    CHECK_EQUALS (100, armor.Amount());
}

It actually doesn’t look too bad, right? It’s not too long a test, and it doesn’t have any nasty loops. Still, it’s a really bad unit test. Imagine you just made some changes to some other part of the system and the fifth CHECK statement fails. Can you look at it for a second and tell me what went wrong? Probably not. Heck, I don’t even know what the test is doing by the test name! The test is far too complex. Here’s how it should look instead:

class ArmorFixture
{
public:
    ArmorFixture() : armor(100, 10) {}
    ArmorComponent armor;
};  

TEST_F (ArmorFixture, ArmorHasCorrectInitialValue)
{
    CHECK_EQUALS (100, armor.Amount());
} 

TEST_F (ArmorFixture, ArmorHasCorrectRechargeRate)
{
    CHECK_EQUALS (10, armor.RechargeRate());
} 

TEST_F (ArmorFixture, DamageLowersArmorByCorrectAmount)
{
    armor.ApplyDamage(30);
    CHECK_EQUALS (70, armor.Amount());
} 

TEST_F (ArmorFixture, DamageDoesNotGoBelowZero)
{
    armor.ApplyDamage(300);
    CHECK_EQUALS (0, armor.Amount());
} 

TEST_F (ArmorFixture, UpdateRechargesArmorByCorrectAmount)
{
    armor.ApplyDamage(50);
    armor.Update(1.0f);
    CHECK_EQUALS (60, armor.Amount());
} 

TEST_F (ArmorFixture, UpdateDoesNotRechargePastMax)
{
    armor.ApplyDamage(50);
    armor.Update(20.0f);
    CHECK_EQUALS (100, armor.Amount());
}

If a test fails right now, it should be immediately obvious what went wrong. Also notice how the largest test is only three lines long. It’s trivial enough to see what we’re testing at a glance.

Keep your tests fast

We’re going to have lots of unit tests. Lots of them! Thousands upon thousands, so they’d better run fast. Think about it. If your average test takes 100 ms, if we run 5000 of them that means it’s going to take over 8 minutes to run them! An ideal unit test should take less than 1 ms, ideally around 1 micro second or so (so 5000 tests will take less than half a second, which is a bit more acceptable).

What does that mean?

• Forget about doing file operations in your unit tests. Or at least minimize them. You probably have a stream class, so anything that requires serialization should use memory instead. Besides, accessing the disk probably means keeping data files along with the unit tests, which makes them harder to keep up to date. Still, if you have to do it for a few tests, that’s fine. Just don’t make a habit out of it.
• Don’t initialize/shutdown expensive systems in your fixtures. If you absolutely need to have a system that’s expensive to initialize, do it once per test project (and make sure you reset it to a default state after every test).
• Work with one library at a time. Ideally, you should only be working with the code you’re actively changing. Pull in the minimum number of libraries you need to develop those tests and keep your build times down to a minimum to have the fastest possible iteration times. You’ll also only need to constantly run the tests for the library you’re working on. Once you’ve implemented a feature you can build the whole project and run all the unit tests for all the libraries to make sure everything else is still A-OK

If the number of unit tests for a single library gets overwhelming, start thinking of breaking them into suites, so you can run a subset of them while developing. Personally, I haven’t gotten there yet, but I like to keep my libraries small and simple also, so running a few thousand tests still keeps my incremental build times to about a second or less.

Refactoring should be just that: refactoring

Resist the temptation to add new functionality when you’re supposed to be refactoring. It’s very easy. I admit it. It’s also too easy to sneak in a few changes here and there when nobody is looking (yet another reason why pair programming is great). But resist it.

If you decide that you should check that the index is within range, save that for the next test. What are you going to do if a pointer is NULL, return false? Write a test. So what if it’s a trivial test? It’s only going to take you 10 seconds to write it, so why not? That way you keep maximum code coverage and it’s harder for things to slip between the cracks.

Refactoring should be just that: changing the format and structure of the program, but not its functionality.

Bug – Test – Fix

Pop quiz: A producer walks through the door and tells you there’s a bug in the inventory code. Apparently if you add two similar objects, when you remove one from the inventory, the other one disappears. You immediately realize that you were searching based on object name, not on unique ID, and you can fix it in 30 seconds. What do you do?

Let me answer that by asking another question: Is fixing that bug a refactoring of code, or are you changing existing behavior? What does TDD say about changing behavior/adding features? Right. You got it. Write a test first.

Clearly, the current tests you have are not catching that situation, so go ahead and create a test that fails because it shows that situation:

TEST (RemovingOneOfTwoSimilarObjectsRemovesOnlyOne)
{
    Inventory inv;
    GameObject potion1("Health potion");
    GameObject potion2("Health potion");
    inv.Add(potion1);
    inv.Add(potion2);
    inv.Remove(potion1);
    CHECK_EQUAL (1, inv.ItemCount());
}

Once you see that test fail, then you can go and implement the fix. Now the test should be passing and everybody is happy. Check in the code and the test and you’ll make sure that bug never comes back. This is one of the best ways of keeping your codebase in good health. Let’s call it preventative medicine.

Classes and tests

When writing tests, don’t obsess about keeping a one-to-one correlation between classes and test files (which contain many tests, of course). That’s the way tests will start out, but I find that sometimes I have too many tests checking fairly different things, so I end up putting them in different files, even though they’re testing the same class.

Could that be a “smell” that the class needs some refactoring and needs to be split? Perhaps. But sometimes it just feels right to separate the tests into logical groups that way without affecting the class. Don’t worry about it. It’s normal.

Private stuff

Should you test private methods and member variables? Not everybody agrees, but I would say no. Test the public interface. That’s what you care about, after all.

However, I find very useful to have access to non-public library classes. Fortunately, I keep my tests as a subdirectory of the library, so in a way they’re almost part of the library itself. I find that having access to classes that are not exposed outside of the library is very helpful to use as mock objects or to help with other parts of the test.

Introducing TDD

It’s great if you’re psyched about TDD and want to use it in your project. What about the rest of your team, though? How should you introduce TDD in your current project? You’re not going to convert everybody overnight, so plan on doing it incrementally.

First, you can start doing it quietly by yourself. Nobody is stopping you from using TDD to develop the functionality you were going to do. Check in your test code as well, but make it so it’s not built and executed every time your code is modified. Probably a few programmers are going to notice your tests being checked in and will be curious about it. This is your chance to start seeding the idea.

Next, you need to get the lead programmer involved and get him on board. You’ll need to agree that those tests should always pass and checking in code that breaks them is not acceptable. At this point, make sure the tests are executed with every automated build, and even whenever somebody modifies the code under test. It’s important that people see the tests as a good thing and understand what they’re doing as opposed to being things that are getting in the way. If people start commenting out tests to prevent them from failing, you know they’re not having the desired effect.

Finally, assuming that everything was successful and people recognize the tests as being useful, you might want to propose using TDD at the start of a new project, or for some specific tool or library. Make sure everybody is on board with the idea, get a few copies of Kent Beck’s book, and help educate them with your experience.

When all you have is a hammer…

Don’t let the novelty of TDD blind you. Even though I believe that TDD is a great technique that is widely applicable to most of the code we write, there are some cases when I would not recommend TDD. It’s important to recognize that and know when not to use it. After all, TDD is supposed to help us. Whenever it gets in the way, or the benefits we reap don’t outweigh the drawbacks, out with it!

• Experimental code. If you’re just experimenting, putting together a prototype, or doing anything that is going to be thrown away, then there’s probably little benefit to using TDD. This is what XP calls “spikes”: short tasks with the intent of experimenting with something to learn more about how to proceed. There’s very little reason to use TDD in this situation.
• High-level scripting code. Chances are your game has a scripting language. How about telling your designers to write tests for their code before they write the scripts? Yeah, right! Not only would that be an exercise in futility, but it wouldn’t be that useful (unless your scripts extend to a fairly low level). Designer scripts, by definition, are probably changing all the time and no other parts of the code directly depend on them. They should probably be relatively simple, so writing unit tests for them is not all that important.
• Don’t test hardwired data. This goes without saying, but don’t test the specific data values in your tests. If you’re writing a test for a grenade weapon that deals damage within a certain radius, don’t hardwire the specific radius in your tests, because that’s going to be constantly tweaked until the game ships. Check the radius in your test and check against that.
• Shaders. A couple of years ago that wasn’t a big deal because shaders were so limited. Now they’re starting to be quite complicated and general, so maybe starting to think about TDD wouldn’t be a bad idea. If somebody out there is doing TDD with graphics shaders, please drop me an email. I’d love to know more about how you’re doing it and whether it’s useful.
• Multithreaded code. TDD uses unit tests, which means it tests very small bits of functionality in isolation. The current emphasis on multithreaded code makes it so there will be things we can’t easily test through TDD, especially dealing with thread interactions. I haven’t had to deal with this yet, but I’m sure it’s going to come up. For now, I’ll just ignore threading issues when I’m writing unit tests and deal with it later.

GUI tools

Can we apply TDD to GUI tools? And the more important question is, even if we can, should we? Absolutely!

A lot of the bulk of game development goes into tools and program plug-ins. Especially now, for this next generation of consoles coming up, I’m convinced that the asset pipeline is going to play a key role in differentiating games from each other and making the ones with a really solid pipeline stand out from the rest. Not doing TDD on tools is going to exclude a good percentage of all code developed for a game, and one of the most important parts at that.

There’s enough material here for a whole other article ( or even a whole book). You can also refer to the Test First User Interfaces mailing list.

The main idea is to separate the GUI from the logic. Unfortunately a lot of GUI development tools encourage you to write your logic right in the function that handles the button pressing. Resist that temptation. Put all the logic for your tool in a separate library, and hook it up to the GUI with the minimum amount of code. This is a bit like the “humble dialog” technique described by Michael Feathers.

Keeping that separation also makes it a lot easier to provide different interfaces to your code. Maybe you want to also have a command-line only version of the tool, or maybe you want to make it into a Windows service. All that is trivial if you’ve completely separated your GUI from your logic.

Consequences of doing TDD

Amount of code

You should expect the overall amount of code you write to increase, maybe by as much as 100% to 150%. That’s a lot of code. It’s all right, though. That’s no ordinary code. It’s code that is looking out for the other half of your code. It’s not code that you have to worry about architecting and dealing with complex dependencies, or anything. It’s extremely simple, and it’s driven by the features you implement. There’s just quite a bit of it.

Development speed

What does writing all that code do to your development speed? Doesn’t it slow things down? Yes, it will. At first.

Whenever you start learning a new programming language, technique, or tool, it’s normal for your productivity to drop temporarily. You’ve spent years honing your skills on a different technique, so switching takes a while and you need some practice.

In my own experience with doing TDD in a not very TDD-friendly environment and with C++ and a game engine, I will admit that it took one or two months to get to the point where I felt I was as productive as before. That’s a fair amount of time, so I don’t recommend switching to doing TDD towards the end of a project. Wait until the beginning of the next project, or do it if you’re not in the critical path and you think you can afford a small productivity hit.

Even before you reach the point where you’re comfortable enough with TDD to develop at the same speed as before, you’ll already be reaping all of the benefits that TDD provides (safety net, better design, documentation, etc, etc).

Here’s the interesting part, though: I claim that once you’re comfortable with TDD and you’ve applied it to a large amount of code, your development speed will actually be faster than it would be without TDD. So not only you get all the other benefits, but things will go faster and more smoothly. How’s that possible? You have to write all that extra code and that takes time!

Think of it as a snowball (or as a Katamari). Doing development without TDD allows you to get going pretty quickly. You take a little snowball and run with it. But as you roll it on the snow, it keep growing and growing. At first it’s not a big deal, but eventually it’ll start to slow you down. Eventually it’ll become a huge ball that takes several people just to move it a small amount. At that point, you better be ready to ship the game because it’s not going much further.

Doing TDD, on the other hand, is like starting out with a medium size ball (say, a foot or two in diameter), but it’ll never get significantly larger than that. You can happily keep rolling it around, adding features and changing things, and you’ll still be as fast and flexible as you were the first day.

Of course, I have no hard data to back any of this other than my own personal experience. If anybody has seen some studies on the subject, please let me know so I can see if they agree with my observations.

More interfaces

In C++, the only (or at least, the easiest) way to use mock objects is to use virtual functions through an interface class. The more you use that technique, the more virtual functions you’ll end up with, even if you didn’t really need them for the production code itself.

A good example could be a graphics rendering library, which is just a light wrapper around the graphics hardware abstraction. If you know you’re only going to use one type of graphics hardware, you don’t really need an interface class and a lot of virtual functions. But if you want to insert a mock object there (which you probably will, to avoid using a real renderer in your tests), it’ll force you to create unnecessary virtual functions.

Of course, many times, when you need a mock object, you also need to use polymorphism at runtime, so you needed those virtual functions anyway.

How much of a big deal is it? I’d say for the most part it’s not worth sweating it. Those are still fairly coarse functions, so having a virtual call isn’t going to affect your frame rate very much at all. They certainly won’t for higher-level code that is not called as frequently. If the extra virtual function calls are an issue, you can change those tests to check the state of the objects directly instead of using mock objects.

I hope you found this peek behind the looking glass useful. Hopefully it convinced you to give test-driven development an honest try. And at the very least, I hope that it made you think about how you develop software and question your assumptions. But don’t worry, this is not the end: this is a topic I plan on revisiting in future articles. By now you should know your way into the looking glass. Feel free to step through any time you want.

What to test

By now you have a pretty good idea of what TDD is and how to apply it. But what exactly do we test? How do we go about testing it?

Depending on the code in question, I use one of three main approaches to test that the code is doing what it should.

Check output

This is the easy one. You make a function call, and check the result value. Everything you need to know is there at your fingertips.

TEST (TargetingCantLockOnObjectOutsideCone)
{
    TargetingSystem targeting;
    targeting.SetTransform(...);
    Vector3 positionBehind (...);
    CHECK (!targeting.CanLockOn(positionBehind));
}

Check state

In this situation, you’re checking that a particular operation changes some state. You might get a success/failure return code, but that doesn’t tell you enough (you still want to test that separately though). For example, applying a speed powerup to an entity returns true or false depending on whether the entity consumed that powerup, but we need to test that it did the right thing.

TEST (EntityDoesNotConsumesSpeedPowerupByDefault)
{
    GameEntity entity;
    SpeedPowerup powerup;
    CHECK (!entity.Apply(powerup));
}  

TEST (EntityConsumesSpeedPowerup)
{
    GameEntity entity;
    entity.Accepts(SpeedPowerup);
    SpeedPowerup powerup;
    CHECK (entity.Apply(powerup));
}  

TEST (EntityConsumesSpeedPowerup)
{
    GameEntity entity;
    entity.Accepts(SpeedPowerup);
    float oldSpeed = entity.GetSpeed();
    SpeedPowerup powerup; entity.Apply(powerup);
    CHECK_CLOSE (oldSpeed + powerup.GetSpeedBoost(), entity.GetSpeed());
}

If I had to add a third test related to that, I would refactor the tests into a fixture. More on that in a minute.

What if the entity class didn’t have a GetSpeed() method? Ideally you want to test things by using them the way you expect them to be used in the game. I suppose you could test the speed by the slightly roundabout method of letting the entity move for one frame and see the difference, but that’s a bit too convoluted.

In a case like this, I’d say that adding a GetSpeed() method is justified. You want to stay clear of always falling back into this pattern and littering your interfaces with GetXXX() methods though. Remember, testing is supposed to make your code more robust and easier to understand, not get in the way and complicate interfaces.

What if you ever come up against something that seems untestable? Hard-core TDDers will claim that if you can’t test something, you shouldn’t implement it. That’s a sentiment I agree with, but I have encountered a just a handful of things over time that I considered too much of a pain to test and skipped the tests. I suspect the more experience once acquires with TDD, the less common that situation is. But in general, if you can’t test it, then think about how to implement it in a different way. Chances are the new design is going to be better. It’s yet another case of TDD in action affecting the final design.

Check interaction between objects

A lot of the code in a game is more than simple functions that return values or set states. In a complex game engine, objects communicate with each other, sending messages, calling methods, and performing a sequence of actions. Testing that is just as important as testing the individual state in an object.

To test interactions between objects, you use mock objects. Mock objects are objects that look like the regular object to the code, but are nothing more than a pretty face with testing code behind it for us to verify that things worked as expected. Most mock objects I work with are extremely simple, and are limited to increasing a counter when something happens, or storing a copy of some argument passed to a function.

One of the mock objects I end up using quite frequently is one to verify that object lifetimes are managed correctly. For example, imagine we have an entity system that uses components. The entity owns the components, so if the entity is ever destroyed, it should destroy all the components. How would we implement that? Pretty easy: in the destructor we write a loop that…. No, wrong answer. Before we write anything, we need to write a test for it. Something like this:

TEST (EntityDestroysComponents)
{
    GameEntity entity;
    entity.Add(new Component());
    delete entity;
    // ??? How exactly do we test that the component was deleted?
}

So we create an entity, we add a component, and we destroy the entity. How can we verify that the component was destroyed along with it? I suppose we could try to access the component we allocated and see if that memory is valid, but that’s a really ugly and non-portable solution. It would be much cleaner to use a mock object: that simply keeps track of how many instances of that class are currently allocated.

class MockComponentDestruction : class Component
{
public:
    MockComponentDestruction() { ++s_instances; }
    virtual ~MockComponentDestruction() { --s_instances; }  

    static int s_instances;
};  

int MockComponentDestruction::s_instances = 0;

Now the test becomes quite trivial.

TEST (EntityDestroysComponents)
{
    GameEntity entity;
    entity.Add(new MockComponentDestruction());
    CHECK_EQUAL (1, MockComponentDestruction::s_instances);
    delete entity;
    CHECK_EQUAL (0, MockComponentDestruction::s_instances);
}

TDD and Games

All right, how about about using TDD for game development? The great majority of the code in a game is code just like in any other type of application. You have objects that are doing some processing and communicating with other objects. We can test that very easily. So, what’s different about games?

Graphics

There’s a misconception that games are all about graphics. That’s just plain wrong. It’s true that games make heavy use of graphics, and games interact with the user primarily through the use of graphics (along with audio and force feedback). But the bulk of the code in a game is not doing low-level graphics operations. Instead, it’s running AI, figuring out what things are visible this frame, sending objects down the pipeline, moving entities in the world, updating physics simulations, etc.

The best thing you can do is forget about low-level graphics. Wrap it up neatly at a really low level in a little library and push it aside. Then you can concentrate on testing everything that uses graphics instead of getting bogged down with the graphics themselves.

For example, we can easily develop through TDD the code that will determine what’s in view given a camera in our world. We can also use TDD to come up with a solution to correctly sort the meshes sent to the graphics renderer to optimize performance. All those are higher-level operations that are not involved with the low-level graphics API or hardware, so nothing is stopping us from testing them.

But it is even possible to test fairly low-level operations. As an example, say you’re about to send to the hardware an object that is lit with two point lights and a directional light, and you want to make sure the right shader is selected (or the right states are set). In this situation, you’re making a function call (Render()), and you can either check the state of the graphics renderer directly, or you can insert a mock object between your code and the graphics API and detect that the correct functions were called.

To make the tests more effective, try to make sure you can run most of your game code without having to create a graphics device, initialize it, create a window, set a video mode, etc. Not only will that decouple your code from the graphics library (which is a good thing), but it will let you run the tests much more quickly and on a variety of platforms. Otherwise, any machine that runs the tests might be required to have DirectX 9.0c, a fancy graphics card, etc. If there’s really no way around it, you can go ahead and do any graphics initialization at the beginning of the test run for the whole graphics library and shut it down when the tests complete, but ideally you shouldn’t have to do that at all.

What about verifying that the right calls and data are sent to the graphics hardware? That’s up to you. TDD is not a religion or an absolute mandate. It’s a tool that I happen to think can be applied to most situations and give great results. If you’re writing a driver for some hardware you should definitely do it. If you’re writing graphics middleware, you might want to consider that too. Otherwise, it’s really not worth it. If you’ve tested everything up until the call where you just set the right render states and push the data down, you can safely trust the graphics hardware to do the rest.

Middleware

Middleware is a big topic in game development (shameless plug), and chances are good you’re using some form of middleware in your projects. Maybe it’s something as simple as Bink, or something as complex as RenderWare. In any case, it is true that working with an external library (which may not even come with source code) makes things a bit more difficult.

You should treat middleware very much like what we did with graphics. Assume it works correctly, but test everything leading up to the interface with the middleware library.

The key idea is to remember that with TDD, you want to test that the code you’re about to implement is doing the right thing. For example, imagine you’re integrating some physics middleware into your engine. You might start with a simple test like this:

TEST (PhysicsObjectStaysAtRestWithNoForces)
{
    PhysicsObject obj;
    Vector3 oldPosition = obj.GetPosition();
    obj.Update();
    CHECK_EQUAL (oldPosition, obj.GetPosition());
}

Next you might want to check that if you apply a force it moves in the right direction. Then you check for friction, collisions, bouncing, deformation, etc. The fact that you’re using an external middleware library to solve those problems should not matter as far as the tests are concerned.

Now let me get on my soapbox for a second. If you’re a middleware provider, I hope you’re using unit tests (and of course, I’d recommend that they be developed before the code). If you do that, please, please, make sure you also release your unit tests to source code licensees. The unit tests will serve both as up-to-date documentation and as a safety net in case they make any changes. And who needs more safety nets than the people who didn’t write the code in the first place?

Hardware

A lot of games are developed on a PC but run on different platforms (PS2, Xbox, Gamecube, custom arcade hardware, handhelds, etc). How do we test them? As much as possible, I would recommend trying to keep as much of the code as platform-independent as possible. Not only will it help with testing but it’s probably a good business and engineering decision. At that point, you can test it just like any other code, hopefully in the same step as you build your code, without any delays.

Additionally, it’s a good idea to run the unit tests on the target platforms themselves. This is something you could run more infrequently, such as right before you’re about to check in code, and, of course, by the automated build machine itself. That way you’ll catch any platform differences, like subtle changes introduced by different endianness, etc.

Large amounts of data

Of all the things that make game development different, this is probably one of the most unique ones. Games, and especially modern games, often use many gigabytes of data. What does that mean for unit testing and TDD?

Fortunately, not very much. These tests we’re writing are unit tests. They test the functionality inside one small part of a class. They shouldn’t have to deal with any data, and especially not with many gigabytes worth of it. They should test that the code does the right thing, period.

Let’s do some examples

In case you’re not fully convinced yet, let’s run through some real-world examples. Thanks to Ivan-Assen Ivanov for providing the specific examples (taken out of some real-world tasks he had to work on).

Example #1: Computing a passability grid

Compute a passability grid for use by the pathfinder code. The passability values are computed based on terrain features such as slope, road textures, water depth, etc.

Where do we start? Before we start thinking of algorithms, let’s just create a test that checks the obvious. I should be able to create an empty grid with some default values.

TEST (DefaultPassabilityGridIsPassable)
{
    PassabilityGrid grid(1,1);
    CHECK (grid.IsPassable(0,0));
}

Next I might want to try to do something about checking values out of range. It seems like a good idea to make all values outside the grid as non-passable.

TEST (ElementsJustOutsideGridAreNotPassable)
{
    PassabilityGrid grid(1,1);
    CHECK (!grid.IsPassable(-1,0));
    CHECK (!grid.IsPassable(1,0));
    CHECK (!grid.IsPassable(0,-1));
    CHECK (!grid.IsPassable(0,1));
    CHECK (!grid.IsPassable(1,1));
}

Notice the tests aren’t even providing full coverage or anything. I simply sampled five points outside of the grid. If I ever find there’s a bug there, I’ll go back, write a specific test, see it fail, and then fix it. For now, this is good enough.

Next I would check that our tests work with a slightly larger grid since I could have easily hardwired a 1×1 grid in the grid class.

TEST (DefaultPassabilityGridIsPassableOnLargerGrid)
{
    PassabilityGrid grid(2,3);
    CHECK (grid.IsPassable(1,0));
    CHECK (grid.IsPassable(1,2));
}

What next? On a totally flat terrain without any roads, the resulting passability grid is uniform and the nodes are passable. By this time you’ll probably already have a terrain class (which should have hopefully been developed using TDD), so I’ll just assume we have a Terrain class.

TEST (EmptyTerrainCreatesUniformGrid)
{
    Terrain terrain(2,2,Grass);
    PassabilityGrid grid(terrain);
    CHECK (grid.IsPassable(0,0));
    CHECK (grid.IsPassable(1,1));
}

Notice that we just created a new constructor for the PassabilityGrid. It wasn’t part of a grand master plan or a fancy UML design. It simply felt like the right thing to do when using PassabilityGrids this way. That’s the beauty of TDD: the code design follows the use of the code.

I think you see where this is heading. Next I would try creating a terrain with one mountain cell, and seeing that the corresponding grid node was impassable. Then create a road and see that those cells are more easily passable than others.

Example #2: Creating a thin layer around audio library

One particular feature of this layer, which is not simply reordering parameters and calling another function, is multi-sample sounds: when the sound engine receives an order to play “UnitWalk”, it chooses randomly with specific probabilities between UnitWalk1.wav, UnitWalk2.wav and UnitWalk3.wav; this mapping, along with probabilities and some other parameters needed by the sound system, comes from a bunch of XML files.

That’s a pretty big task, so we need to break it down into something smaller just to even start thinking about it. If it’s really a thin layer, it’s possible that some functions might call directly into the audio library. Those I would just do without a test (whenever I needed them). For example, it’s possible that our library might have an Initialize() call that simply calls SoundLibrary::Initialize(). That’s fine. No tests there.

Let’s concentrate on the mapping of sound events to specific sounds with specific probabilities. Where do we start? Reading XML files? Figuring out random numbers? No, let’s take it from the top. Let’s write a really simple test of a trivial case and make it pass. How about this?

TEST (SoundEventPlaysCorrectSoundFile)
{
    GameSoundSystem soundSystem;
    soundSystem.PlayEvent("UnitWalk");
    CHECK_EQUAL ("UnitWalk1.wav", soundSystem.GetLastSoundPlayed());
}

OK, that’s not exactly a pretty-looking test, but it’s a start. It gives us a rough guide of where to go. Forget about XML files and probabilities, or anything. I play an event, and I want to see a particular wav file played out the other end. How do I implement that? I hardcode a “UnitWalk1.wav” in the sound library, of course. Test passed. Yeah, I know, the horror! But it took 20 seconds and we made some progress (all tests are passing). Besides, we already started making decisions about how we want to be using this sound system, so we definitely made some progress.

One thing I don’t like about that previous test is that it relies on the function GetLastSoundPlayed(). That’s not a function that’s necessary to work with the GameSoundSystem, and the only reason we introduced it was to help us test it. It’s OK to do that from time to time, but I’d rather keep my class interfaces as simple and uncluttered as possible. So instead, we’ll go ahead and refactor that test to use a mock object representing the real system sound library that will collect the name of the last sound played.

class SoundSystemMock : class SoundSystem
{
public:
    virtual void PlaySound(const char * sound) { lastSoundPlayed(std::string(sound)); }
    std::string lastSoundPlayed;
};  

TEST (SoundEventPlaysCorrectSoundFile)
{
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound(soundSystem);
    gameSound.PlayEvent("UnitWalk");
    CHECK_EQUAL ("UnitWalk1.wav", soundSystem.lastSoundPlayed);
}

This also has the advantage of allowing us to write all these tests without even having to play real sounds and without even linking with the real sound library.

Next. We probably want to play other sounds than UnitWalk1.wav, so let’s write another test that will fail.

TEST (SoundEventPlaysCorrectSoundFile2)
{
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound(soundSystem);
    gameSound.PlayEvent("UnitFire");
    CHECK_EQUAL ("UnitFire1.wav", soundSystem.lastSoundPlayed);
}

This will totally fail because we know that our sound system is playing “UnitWalk1.wav” no matter what the event is. Time to fix that. For that we probably want to add the concept of mapping. For now, let’s just do a one-to-one mapping.

TEST (SoundEventPlaysCorrectSoundFile2)
{
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound(soundSystem);
    gameSound.MapEvent("UnitFire", "UnitFire1.wav");
    gameSound.PlayEvent("UnitFire");
    CHECK_EQUAL ("UnitFire1.wav", soundSystem.lastSoundPlayed);
}

Everything is passing again. Now let’s add some probabilities to it. This part gets a bit tricker because it involves some randomness. Radomness is one of those tricky things when it comes to testing because you don’t want your results to change from run to run. You also don’t want to get into the situation that you’re running the same test multiple times to verify that you get the correct sound played 20% of the time.

At the same time, randomness plays an important part in game development, so it’s crucial to know how to deal with it correctly. I decided to ask in the testdrivendevelopment mailing list looking for some good insights.The best solution seemed to be to move the randomness factor as an input. In the production code, that input will be filled by the game random number generator. In our tests, we can pass whatever value we want and we know what results we’re going to get.

Here’s the next test that includes some different probabilities of different sounds being played.

TEST (SoundEventWithProbabilities)
{
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound(soundSystem);
    gameSound.MapEvent("UnitFire", "UnitFire1.wav", 0.2, "UnitFire2.wav", 0.8);
    gameSound.PlayEvent("UnitFire", 0.3);
    CHECK_EQUAL ("UnitFire2.wav", soundSystem.lastSoundPlayed);
}

Unfortunately, I don’t want the public interface of GameSoundSystem::PlayEvent() to take an event name and a number between 0 and 1. That’s only for internal consumption and shouldn’t be exposed through the interface. On the other hand, I really need that function to test it (and it cleans things up considerably from an implementation point of view). Here’s a case where I think giving the test access to a private function is fine.

If you feel uncomfortable with that idea, you would have to create a new class that sits between the GameSoundSystem and the SoundSystem itself and takes care of doing the mapping. That class can have the number between 0 and 1 as part of its interface because it won’t be exposed outside of the library. That might actually not be a bad idea since the mapping between events and sounds could be the full-time job for a class.

Just because the first implementation of the mapping of events could have been hardwired (after all, do the simplest thing that could possibly work), I would add another similar test with different values:

TEST (SoundEventWithProbabilities)
{
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound(soundSystem);
    gameSound.MapEvent("UnitFire", "UnitFire1.wav", 0.2, "UnitFire2.wav", 0.8);
    gameSound.PlayEvent("UnitFire", 0.1);
    CHECK_EQUAL ("UnitFire1.wav", soundSystem.lastSoundPlayed);
}

This also gives us the perfect opportunity to refactor the tests into a fixture by putting all common code there and simplifying the tests considerably:

class SoundEventFixture
{
public:
    SoundEventFixture() : gameSound(soundSystem)
    {
        gameSound.MapEvent("UnitFire", "UnitFire1.wav", 0.2, "UnitFire2.wav", 0.8);
    }
    SoundSystemMock soundSystem;
    GameSoundSystem gameSound;
};  

TEST_F (SoundEventFixture, FirstSoundPlays)
{
    gameSound.PlayEvent("UnitFire", 0.1);
    CHECK_EQUAL ("UnitFire1.wav", soundSystem.lastSoundPlayed);
} 

TEST_F (SoundEventFixture, FirstSoundPlays2)
{
    gameSound.PlayEvent("UnitFire", 0.0);
    CHECK_EQUAL ("UnitFire1.wav", soundSystem.lastSoundPlayed);
} 

TEST_F (SoundEventFixture, SecondSoundPlays)
{
    gameSound.PlayEvent("UnitFire", 0.22);
    CHECK_EQUAL ("UnitFire2.wav", soundSystem.lastSoundPlayed);
} 

TEST_F (SoundEventFixture, SecondSoundPlays2)
{
    gameSound.PlayEvent("UnitFire", 1.0);
    CHECK_EQUAL ("UnitFire2.wav", soundSystem.lastSoundPlayed);
}

I would then refactor the tests to come up with a better way to express the mapping between events and sounds than passing huge lists of parameters, check for valid probabilities (can’t add up to anything different than 1.0), etc. We never got around to the XML part, but that’s something totally orthogonal to this. First let’s worry about getting the data to the object, then we can load that data in any way we want (but we’ll test that too when we get to it, of course).

That’s enough for one day. The next (and final!) part of this article will deal with specific tips for doing TDD that I found particularly useful, especially dealing with C++ and games. I’ll also cover what are some of the immediate consequences you can expect from doing TDD, including build times, development speed, etc. In the meanwhile, get your unit-test framework ready and give it a whirl.

Traditional development

Let’s take a moment to think about how you write code without using test-driven development. You decide you need to implement some piece of functionality, you mentally break the problem into smaller problems, and you start working on each of them. While you write code, your only guide for whether you’re heading in the right direction or not is whether your code continues to compile. Once you’ve assembled a minimum number of those smaller subtasks, your program might even be able to run. So you fire up the game or tool and “see” if it works. If it’s not obvious (or if it doesn’t work at all), maybe you break in the debugger and try to step into the code you just wrote to make sure the program is doing what you intended. Once you see it do its thing, you happily commit it to version control and move on to some other task.

Does that more or less describe how you write code? You can replace any specific details in that paragraph, but in general that seems to sum up in my experience how most programmers approach writing code.

What is Test-Driven Development?

The idea behind Test-Driven Development (TDD) is extremely simple: before you write a piece of functionality, you write a test for it. Only once you have that test, which should fail at first, do you actually implement the functionality and see the test pass.

I know it sounds like a backwards approach if you’re not used to it, but it makes a lot more sense than it seems to at first. The thing to keep in mind is that you’re not writing a bunch of tests first and then making them pass. You’re just writing a single test for a very small bit of functionality that you can implement in a minute or two.

TDD development has a very well defined, short cycle, usually only taking a few minutes per cycle:

• Write a single test for a very small piece of functionality.
• Run it and see it fail (or not even compile in C++).
• Write code to make the test compile and pass.
• Run the test and see it pass.
• Refactor code. See test pass.
• Refactor tests. See tests pass.

Comparing it to the non-test-driven development approach, you’re replacing all the mental checking and debugger stepping with code that verifies that your program does exactly what you intended it to do.

There really isn’t much more to test-driven development than that. The devil, as they say, is in the details. You need to learn to effectively do that cycle over and over during your development, how to tackle the right task size, how to organize things so you can test them, how to write tests easily, how to make sure tests can be executed constantly, etc.

A very good starting point is Kent Beck’s book Test Driven Development. He’ll take you through an example and give you an idea of the kind of pacing and techniques you’re likely to use. Don’t be put off by the use of Java as their development language. Java is close enough to C++, and besides, it’s not the language that’s important, it’s the concept.

Benefits of TDD

So, what exactly do you get by doing things apparently backwards? A lot, as it turns out.

Safety net

Developing your code using TDD means that every single piece of functionality you add will have a test coverage (or close to it anyway). If anything ever changes, you’ll know right away. This is extremely important for many reasons.

Refactoring becomes a lot easier, so you can always keep your codebase healthy (I always like to say that the quality of a codebase is directly tied to how easy it is to change—if it ever becomes difficult or “a pain” to change, the quality is already on its way down and it’s just going to get worse if untreated).

Making changes late in the project also becomes a lot easier, effectively flattening out the change vs. cost curve.

This means you can be making significant changes late in the project based on real playtest feedback or finding out what really works and what doesn’t, and still be confident you’re not breaking anything major. This alone can make the difference between a so-so and an outstanding game.

We all know about those dusty corners of codebases. You know, the places with all the cobwebs, and sometimes gnarled oak trees and a troll or two hiding in the shadows. The places no sane programmer dares approach, especially as a milestone looms over the horizon. TDD gives you the courage and the tools to dive in head first into any part of the code, even the scary parts, make any necessary changes, and walk away like a hero.

Usable design

Here’s something that might come as a surprise: Code that you develop through TDD is going to end up looking very different from code you would have written otherwise. Why? After all, the only thing we’re doing differently is writing the tests first. TDD is not a design paradigm or anything like that.

That’s because with TDD, the first thing you’re forced to do is to think about how you want to use your code. You don’t start with a UML diagram, or a detailed class header file, or anything like that. You start by using the feature you’re about to implement. You could call it “extreme dog food eating.” You’re going to be your first user of your code, and you’re going to think about how to use it from the beginning.

What does that mean in practice? You’ll find that it’s very easy to create new objects of the classes you’re designing. Most of the time you’ll be able to create them on the stack and that will be it. It also means you won’t have to have a complicated sequence of registration, creation, binding, yada, yada, yada. You create an object and it does what it’s supposed to do.

Quick test: Think about your current game codebase. Could you write something like the following code?

#include "GameEntity.h"
int main()
{
    GameEntity entity(optional parameters);
    entity.DoSomething();
    return 0;
}

Chances are you can’t. You probably need a world for the entity to live in. And the world needs to pull in physics and graphics, and initialize the video card, and the sound system, and the networking… Then you probably have to go through a complex creation procedure. Or maybe you can’t even create an entity that way and you can only load it from disk along with some resources.

If simplicity is something important (and I clearly think so), then TDD will help you tremendously.

Modularity

This is another side effect of having to eat your own dog food all the time and use the code you’re about to write. You’ll quickly see that testing code that depends on lots of other code is a nuisance that can be avoided most of the time. As a result, the code you write will be extremely modular. It’ll be very easy to use an object in isolation from other classes or systems.

To draw on the earlier example, you will be able to create a game entity without a world, and certainly without expensive operations like initializing the graphics system and pulling in the rest of the codebase.

Modularity schodularity. Why should you care? There are the usual reasons: refactoring, reusing the code, adding new features later on, etc. But there are also much more practical ones. A highly modular codebase will compile much more quickly than one that has grown organically (without resorting to the likes of Incredibuild). It’ll also allow you to break it up into smaller, separate libraries which can be tested and linked separately (which means really fast iteration times instead of always having a 2-minute link step if you’re dealing with the full codebase).

Documentation

What’s wrong with comments in code? I used to be a big one for having a header on every class and every function, generating cute help files with Doxygen and all that, but I eventually gave up. When you’re developing at a real-world pace, those pretty comments quickly fall by the wayside. Then you’re left either with obvious comments (assign a to b!), outdated ones, or, even worse, incorrect ones.

The tests you write while doing TDD are the ideal low-level documentation for your code. For every feature, you have at least one example on how to use it correctly and how it’s expected to be used. And the best part is that it can’t ever get out of date.

Comments in code still have their place, but it should be very rare. You should still add comments explaining why you did something a certain way, or including the URL of the web page where you got certain algorithms or code snippets. Other than that, I believe documentation should come in the form of good class and function names and a comprehensive set of TDD-generated unit tests.

WYGIWYM (What You Get Is What You Meant)

When you’re doing TDD, even though you’re writing unit tests, you’re not verifying that the algorithm you’re implementing is correct. That’s not the goal. What TDD gives you is the confidence that the code you just wrote does what you wanted it to do. If you meant to do the wrong thing, you’ll only be testing that you did the wrong thing. TDD isn’t going to help you in that respect (other than make you realize really early on that you’re doing the wrong thing, since you’re using your own code before you even write it). A bit more on this later on.

Instant feedback

When you write any code doing TDD, you get instant feedback. You write a test, build, run the tests, and you see them fail. Then you write the code, and you see the tests pass. You refactor a few things, and see the tests pass still. Instant feedback is part of the TDD cycle, and since the cycle is so short, you’ll get feedback on how things are going every few minutes, or even several times per minute.

Apart from keeping you honest and showing that things are progressing in the right direction, this instant feedback can have a very curious effect on the mood of the programmer. I don’t know exactly how to describe it, but I hadn’t had as much fun programming in a long, long time. Somehow, TDD brought back the excitement and satisfaction I felt writing Basic programs on an Amstrad CPC twenty years ago.

I know that’s a very subjective “benefit” of TDD, but it’s one that can have a profound effect on the team and the overall productivity. Don’t underestimate the power of a happy programmer!

TDD in practice

Enough theory. I hope I have convinced you of the potential benefits and you’re willing to give it a try. Let’s get into the nitty-gritty details. How exactly do we go about doing TDD? I’m assuming you’re using C++ since that’s the most-used language for game development today.

Get a unit-testing framework.

You need a unit-testing framework. The goal is to make writing a test as painless as possible. The easier it is to write a test, the less people will resist writing new tests. Since unit tests are at the core of TDD, the more streamlined the process the better.

I recently looked at some of the most popular C++ unit-testing frameworks. If you’re just starting out, don’t worry about the details. Use whatever is most convenient. I’m currently using a modified version of CppUnitLite (which I’m hoping to put up here in the next few days) just because it’s so small, easy to use, and easy to modify and port.

With my current unit-testing framework, adding a new test is as easy as writing this:

TEST (MyFirstTest)
{
    CHECK_EQUAL (3, GetSomeValue());
}

Put tests into a separate executable.

I recommend splitting your code into many separate libraries, each of them dealing with one specific subject: sound, networking, AI, scene management, physics, bitmaps, file IO, input, etc, etc. That’s just a good practice whether you’re doing TDD or not. Extra bonus points for managing dependencies between libraries correctly and avoiding cyclical dependencies.

For each library, create a new executable that contains the tests for that library. My preferred directory structure looks something like this, but feel free to organize it in any way you like

I definitely don’t want my tests in the same file as the class they’re testing. I try to keep the class as simple as possible, and that would just add clutter. Some people prefer to put the test files in the same directory as the files for the library itself. I like to keep them separate but nearby, especially because there’s no one-to-one correspondence between library files and test files.

Build and run tests often.

Make sure to make the test executable dependent on the library it’s testing (and linking with). As the final step of the build process, run the test itself. That way it’ll be impossible to build a library without building the tests and running them. Since the tests should be very fast (taking less than a few seconds), this shouldn’t get in the way of fast iteration.

Most unit-testing frameworks return by default the number of failed tests they encountered. Since calling that executable was part of the build process, if it returns anything other than zero, the build framework considers the build failed, which is exactly what you want. If you’re using an IDE such as Visual Studio or KDevelop, you’ll also get a nicely formatted error message that you can select and it’ll go to the failing test. Tests have become an integral part of the build, and a failing test is treated just like a syntax error.

If you’re using make files, just run the executable as your last step of the build rule:

test:
    $(objects) ${COMPILER} -o test $(objects) ${LIBFLAGS}
    ./test

With Scons, just use AddPostAction:

test = env.Program('test', list)
env.AddPostAction(test, './test')

In Visual Studio, call $(TargetPath) as your PostBuild rule.

Make sure the test executables are built and run in your automated build machine as well. Do both debug and release, and if you can also run them in your target platforms (Xbox, PS2, or whatever), so much the better.

Step size.

Before letting you loose with the TDD tools in hand, it’s important to talk about step size. Before you start a TDD cycle, you need to decide how much functionality you should tackle in that one cycle. It’s extremely important that you start by taking baby steps. Don’t worry about the fact that it looks silly or pointless. The point at first is to get comfortable with the cycle and to get the hang of it.

A full iteration of the TDD cycle should take just a few minutes (and sometimes less than a minute). If it takes significantly longer than that to get your tests running again (>15 minutes), it probably means that you tried taking too big a step. If you find yourself in that situation with no obvious way to make the tests pass, you might even want to consider undoing what you did and starting again with an even simpler task.

The concept of step size is very important. Choosing the right step size is probably the hardest part of TDD, and it only comes with experience and tripping a few times. At the beginning, you should start with trivial steps and stick with it. Once you’re totally comfortable with it, you can take steps a bit larger. But as soon as you find yourself with unexpected failing tests or taking too long before getting passing tests again, you should back off and take baby steps again for a while.

We’ll see some specific examples of TDD in action in the second part of this article, which will show how you can take tiny steps.

Resources

• Test Driven Development: By Example by Kent Beck. The first book you should read.
• Test Driven Development: A Practical Guide by Dave Astels. If you are dying to read more.
• TestDriven.com. Good portal to test-driven topics.
• testdrivendevelopment mailing list.

Now you should have a good idea of what TDD is and what’s involved in actually doing it. But TDD is like riding a bicycle. You can read all you want about it, but eventually, you’ll have to try it for yourself if you want to learn it. I hope this is enough information for you to go out and take it out for a quick spin.

The second part of this article will deal with how to applying TDD to game development (with specific examples) and will cover tips from the trenches and what you can expect when applying TDD to your projects.

Update (Apr 2010): It’s been quite a few years since I originally did this comparison. Since then, Charles Nicholson and I created Unit Test++, a C/C++ unit-testing framework that addresses most of my requirements and wish-list items. It’s designed to be a light-weight, high-performance testing framework, particularly aimed at games and odd platforms with limited functionality. It’s definitely my framework of choice and I haven’t looked at new ones in several years because it fits my needs so well. I definitely encourage you to check it out.

———-

One of the topics I’ve been meaning to get to for quite a while is the applicability of test-driven development in games. Every time the topic comes up in conversations or mailing lists, everybody is always very curious about it and they immediately want to know more. I will get to that soon. I promise!

In the meanwhile I’m now in the situation that I need to choose a unit-testing framework to roll out for my team at work. So, before I get to talk about how to use test-driven development in games, or the value of unit testing, or anything like that, we dive deep into a detailed comparison of existing C++ unit-testing frameworks. Hang on tight. It’s going to be a long and bumpy ride with a plot twist at the end.

If you just want to read about a specific framework, you can go directly there:

• CppUnit
• Boost.Test
• CppUnitLite
• NanoCppUnit
• Unit++
• CxxTest

Overview

How do we choose a unit-testing framework? It depends on what we’re going to do with it and how we’re going to use it. If I used Java for most of my work, the choice would be easy since JUnit seems to be the framework of choice for those working with Java. I don’t hear them arguing over frameworks or proposing new ones very frequently, so it must be pretty good.

Unfortunately that’s not the case with C++. We have our XUnit family member, CppUnit, but we’re clearly not happy with that. We have more unit-testing frameworks than you can shake a stick at. And a lot of teams end up writing their own from scratch. Why is that? Is C++ so inadequate for unit testing that we have trouble fitting the XUnit approach in the language? Not like it’s a bad thing, mind you. Diversity is good. Otherwise I would be stuck writing this under Windows and you would be stuck reading it with Internet Explorer. In any case, I’m clearly not the first one who’s asked this question. This page tries to answer the question, and comes up with some very plausible answers: differences in compilers, platforms, and programming styles. C++ is not exactly a clean, fully supported language, with one coding standard.

A good way to start is to create a list of features that are important given the type of work I expect to be doing. In particular, I want to be doing test-driven development (TDD), which means I’m going to be constantly writing and running many small tests. It’s going to be used for game development, so I’d like to run the tests in a variety of different platforms (PC, Xbox, PS2, next-generation consoles, etc). It should also fit my own personal TDD style (many tests, heavy use of fixtures, etc).

The following list summarizes the features I would like in a unit-testing framework in order of importance. I’ll evaluate each framework on the basis of these features. Thanks to Tom Plunket for providing a slightly different view on the topic that helped me to re-evaluate the relative importance of the different features.

1. Minimal amount of work needed to add new tests. I’m going to be doing this all the time, so I don’t want to do a lot of typing, and I especially don’t want to do any duplicated typing. The shorter and easier it is to write, the easier it’ll be to refactor, which is crucial when you’re doing TDD.
2. Easy to modify and port. It should have no dependencies with non-standard libraries, and it shouldn’t rely on “exotic” C++ features if possible (RTTI, exception handling, etc). Some of the compilers we have to use for console development are not exactly cutting edge. To verify this one, I created a set of unit tests using each library under Linux with g++. Since most of the tests are written with Windows and Visual Studio in mind, it’s not a bad initial test.
3. Supports setup/teardown steps (fixtures). I’ve adopted the style recommended by David Astels in his book Test Driven Development: A Practical Guide about using only one assertion per test. It really makes tests a lot easier to understand and maintain, but it requires heavy use of fixtures. A framework without them is ruled out immediately. Bonus points for frameworks that let me declare objects used in the fixture on the stack (and still get created right before the test) as opposed to having to allocate them dynamically.
4. Handles exceptions and crashes well. We don’t want the tests to stop just because some code that was executed accessed some invalid memory location or had a division by zero. The unit-testing framework should report the exception and as much information about it as possible. It should also be possible to run it again and have the debugger break at the place where the exception was triggered.
5. Good assert functionality. Failing assert statements should print the content of the variables that were compared. It should also provide a good set of assert statements for doing “almost equality” (absolutely necessary for floats), less than, more than, etc. Bonus points for providing ways to check whether exceptions were or were not thrown.
6. Supports different outputs. By default, I’d like to have a format that can be understood and parsed by IDEs like Visual Studio or KDevelop, so it’s easy to navigate to any test failures as if they were syntax errors. But I’d also like to have ways to display different outputs (more detailed ones, shorter ones, parsing-friendly ones, etc).
7. Supports suites. It’s kind of funny that this is so low in my priority list when it’s usually listed as a prominent feature in most frameworks. Frankly, I’ve had very little need for this in the past. It’s nice, yes, but I end up having many libraries, each of them with its own set of tests, so I hardly ever need this. Still, it certainly would be nice to have around in case it starts getting slow to run the unit tests at some point.

Bonus: Timing support. Both for total running time of tests, and for individual ones. I like to keep an eye on my running times. Not for performance reasons, but to prevent them from getting out of hand. I prefer to keep running time to under 3-4 seconds (it’s the only way to be able to run them very frequently). Ideally, I’d also like to see a warning printed if any single test goes over a certain amount of time.

Easy of installation was not considered a priority; after all, I only have to go through that once—it’s creating new tests that I’m going to be doing all day long. Non-commercial-friendly licenses (like GPL or LGPL) are also not much of an issue because the unit test framework is not something we’re going to link to the executable we ship, so they don’t really impose any restrictions on the final product.

Incidentally, during my research for this article, I found that other people have compiled lists of what they wish for in C++ unit-testing frameworks. It’s interesting to contrast that article with this one and make a note of the differences and similarities between what we’d like to see in a unit test framework.

Ideal Framework

Before I start going over each of the major (and a few minor) C++ unit-testing frameworks, I decided I would apply the philosophy behind test-driven development to this analysis and start by thinking what I would like to have. So I decided to write the set of sample tests in some ideal unit-testing framework without regard for language constrains or anything. In the ideal world, this is what I would like my unit tests to be like.

The simplest possible test should be trivial to create. Just one line to declare the test and then the test body itself:

TEST (SimplestTest)
{
    float someNum = 2.00001f;
    ASSERT_CLOSE (someNum, 2.0f);
}

A test with fixtures is going to be a bit more complicated, but it should still be really easy to set up:

SETUP (FixtureTestSuite)
{
    float someNum = 2.0f;
    std::string str = "Hello";
    MyClass someObject("somename");
    someObject.doSomethng();
}

TEARDOWN (FixtureTestSuite)
{
    someObject.doSomethingElse();
}

TEST (FixtureTestSuite, Test1)
{
    ASSERT_CLOSE (someNum, 2.0f); someNum = 0.0f;
}

TEST (FixtureTestSuite, Test2)
{
    ASSERT_CLOSE (someNum, 2.0f);
}

TEST (FixtureTestSuite, Test3)
{
    ASSERT_EQUAL(str, "Hello");
}

The first thing to point out about this set of tests is that there is a minimum amount of code spent in anything other than the tests themselves. The simplest possible test takes a couple of lines and needs no support other than a main file that runs all the tests. Setting up a fixture with setup/teardown calls should also be totally trivial. I don’t want to inherit from any classes, override any functions, or anything. Just write the setup step and move on.

Look at the setup function again. The variables that are going to be used in the tests are not dynamically created. Instead, they appear to be declared on the stack and used directly there. Additionally, I should point out that those objects should only be created right before each test, and not before all tests start. How exactly are the tests going to use them? I don’t know, but that’s what I would like to use. That’s why this is an ideal framework.

Now let’s contrast it to six real unit-testing frameworks that have to worry about actually compiling and running. For each of the frameworks I look at the list of wanted features and I try to implement the two tests I implemented with this ideal framework. Here is the source code for all the examples.

CppUnit is probably the most widely used unit-testing framework in C++, so it’s going to be a good reference to compare other unit tests against. I had used CppUnit three or four of years ago and my impressions back then were less than favorable. I remember the code being a mess laced with MFC, the examples all tangled up with the framework, and the silly GUI bar tightly coupled with the software. I even ended up creating a patch to provide console-only output and removed MFC dependencies. So this time I approached it with a bit of apprehension to say the least.

I have to admit that CppUnit has come a long way since then. I was expecting the worst, but this time I found it much easier to use and configure. It’s still not perfect, but it’s much, much better than it used to. The documentation is pretty decent, but you’ll have to end up digging deep into the module descriptions to even find out that some functionality is available.

1. Minimal amount of work needed to add new tests. This is one of the major downfalls of CppUnit, and, ironically, it’s the highest-rated feature I was looking for. CppUnit requires quite a bit of work for the simplest possible test.

// Simplest possible test with CppUnit
#include <cppunit/extensions/HelperMacros.h>
class SimplestCase : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE( SimplestCase );
    CPPUNIT_TEST( MyTest );
    CPPUNIT_TEST_SUITE_END();
protected:
    void MyTest();
};

CPPUNIT_TEST_SUITE_REGISTRATION( SimplestCase );  

void SimplestCase::MyTest()
{
    float fnum = 2.00001f;
    CPPUNIT_ASSERT_DOUBLES_EQUAL( fnum, 2.0f, 0.0005 );
}

2. Easy to modify and port. It gets mixed marks on this one. On one hand, it runs under Windows and Linux, and the functionality is reasonably well modularized (results, runners, outputs, etc). On the other hand, CppUnit still requires RTTI, the STL, and (I think) exception handling. It’s not the end of the world to require that, but it could be problematic if you want to link against libraries that have no RTTI enabled, or if you don’t want to pull in the STL.
3. Supports fixtures. Yes. If you want the objects to be created before each test, they need to be dynamically allocated in the setup() function though, so no bonus there.

#include <cppunit/extensions/HelperMacros.h>
#include "MyTestClass.h"
class FixtureTest : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE( FixtureTest );
    CPPUNIT_TEST( Test1 );
    CPPUNIT_TEST( Test2 );
    CPPUNIT_TEST( Test3 );
    CPPUNIT_TEST_SUITE_END();
protected:
    float someValue;
    std::string str;
    MyTestClass myObject;
public:
    void setUp();
protected:
    void Test1();
    void Test2();
    void Test3();
};

CPPUNIT_TEST_SUITE_REGISTRATION( FixtureTest );  

void FixtureTest::setUp()
{
    someValue = 2.0;
    str = "Hello";
}  

void FixtureTest::Test1()
{
    CPPUNIT_ASSERT_DOUBLES_EQUAL( someValue, 2.0f, 0.005f );
    someValue = 0;
    //System exceptions cause CppUnit to stop dead on its tracks
    //myObject.UseBadPointer();
    // A regular exception works nicely though myObject.ThrowException();
}

void FixtureTest::Test2()
{
    CPPUNIT_ASSERT_DOUBLES_EQUAL( someValue, 2.0f, 0.005f );
    CPPUNIT_ASSERT_EQUAL (str, std::string("Hello"));
}

void FixtureTest::Test3()
{
    // This also causes it to stop completely
    //myObject.DivideByZero();
    // Unfortunately, it looks like the framework creates 3 instances of MyTestClass
    // right at the beginning instead of doing it on demand for each test. We would
    // have to do it dynamically in the setup/teardown steps ourselves.
    CPPUNIT_ASSERT_EQUAL (1, myObject.s_currentInstances);
    CPPUNIT_ASSERT_EQUAL (3, myObject.s_instancesCreated);
    CPPUNIT_ASSERT_EQUAL (1, myObject.s_maxSimultaneousInstances);
}

4. Handles exceptions and crashes well. Yes. It uses the concept of “protectors” which are wrappers around tests. The default one attempts to catch all exceptions (and identify some of them). You can write your own custom protectors and push them on the stack to combine them with the ones already there. It didn’t catch system exceptions under Linux, but it would have been trivial to add with a new protector. I don’t think it had a way to easily turn off exception handling and let the debugger break where the exception happened though (no define or command-line parameter).
5. Good assert functionality. Pretty decent. It has the minimum set of of assert statements, including one for comparing floating-point numbers. It’s missing asserts for less than, greater than, etc. The contents of the variables compared are printed to a stream if the assert fails, giving you as much information as possible about the failed test.
6. Supports different outputs. Yes. Has very-well defined functionality for “outputters” (which display the results of the tests), as well as “listeners” (which get notified while the tests are happening). It comes with an IDE-friendly output that is perfect for integrating with Visual Studio. Also supports GUI progress bars and the like.
7. Supports suites. Yes.

Overall, CppUnit is frustrating because it’s almost exactly what I want, except for my most wanted feature. I really can’t believe that it takes so much typing (and duplicated typing at that) to add new tests. Other than that, the main complaint is the need for RTTI or exceptions, and the relative complexity of the source code, which could make it challenging to port to different platforms.

Update: I’ve revised my comments and ratings of the Boost.Test framework in light of the comments from Gennadiy Rozental pointing out how easy it is to add fixtures in boost.

I’m a big fan of Boost, but I have to admit that it wasn’t until about a year ago that I even learned that Boost was providing a unit testing library. Clearly, I had to check it out.

The first surprise is that Boost.Test isn’t exclusively a unit-testing framework. It also pretends to be a bunch of other things related to testing. Nothing terribly wrong with that, but to me is the first sign of a “smell.” The other surprise is that it wasn’t really based on the XUnit family. Hmmm… In that case, it had better provide some outstanding functionality.

The documentation was top notch. Some of the best I saw for any testing framework. The concepts were clearly explained, and had lots of simple examples to demonstrate different features. Interestingly, from the docs I saw that Boost.Test was designed to support some things that I would consider bad practices such as dependencies between tests, or long tests.

1. Minimal amount of work needed to add new tests. Almost. Boost.Test requires really minimal work to add new tests. It’s very much like the ideal testing framework described earlier. Unfortunately, adding tests that are part of a suite requires more typing and explicit registration of each test.

#include <boost/test/auto_unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>  

BOOST_AUTO_UNIT_TEST (MyFirstTest)
{
    float fnum = 2.00001f;
    BOOST_CHECK_CLOSE(fnum, 2.f, 1e-3);
}

2. Easy to modify and port. It gets mixed marks on this one, for the same reasons as CppUnit. Being part of the Boost libraries, portability is something that they take very seriously. It worked flawlessly under Linux (better than most frameworks). But I question how easy it is to actually get inside the source code and start making modifications. It also happens to pull into quite a few supporting headers from other Boost libraries, so it’s not exactly small and self-contained.
3. Supports fixtures. Boost.Test eschews the setup/teardown structure of NUnit tests in favor of plain C++ constructors/destructors. At first this threw me off for a loop. After years of being used to setup/teardown, and a fairly complex suite setup, I didn’t see the obvious ways of using fixtures with composition.Now that I’ve tried it this way I’ve come to like it almost better than setup/teardown fixtures. One of the great advantages of this approach is that you don’t need to create fixture objects dynamically, and instead you can put the whole fixture on the stack.On the downside, it’s annoying to have to refer to the variables in the fixture through the object name. It would be great if they could somehow magically appear in the same scope as the test case itself. Also, it would have been a bit cleaner if the fixture could have been setup on the stack by the BOOST_AUTO_UNIT_TEST macro instead of having to explicitely put it on the stack for every test case.

#include <boost/test/auto_unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>
#include "MyTestClass.h"  

struct MyFixture
{
    MyFixture()
    {
        someValue = 2.0;
        str = "Hello";
    }
    float someValue;
    std::string str;
    MyTestClass myObject;
};

BOOST_AUTO_UNIT_TEST (TestCase1)
{
    MyFixture f;
    BOOST_CHECK_CLOSE (f.someValue, 2.0f, 0.005f);
    f.someValue = 13;
}  

BOOST_AUTO_UNIT_TEST (TestCase2)
{
    MyFixture f;
    BOOST_CHECK_EQUAL (f.str, std::string("Hello"));
    BOOST_CHECK_CLOSE (f.someValue, 2.0f, 0.005f);
    // Boost deals with this OK and reports the problem
    //f.myObject.UseBadPointer();
    // Same with this
    //myObject.DivideByZero();
}  

BOOST_AUTO_UNIT_TEST (TestCase3)
{
    MyFixture f;
    BOOST_CHECK_EQUAL (1, f.myObject.s_currentInstances);
    BOOST_CHECK_EQUAL (3, f.myObject.s_instancesCreated);
    BOOST_CHECK_EQUAL (1, f.myObject.s_maxSimultaneousInstances);
}

4. Handles exceptions and crashes well. This is one of the aspects where Boost.Test is head and shoulders above all the competition. Not only does it handle exceptions correctly, but it prints some information about them, it catches Linux system exceptions, and it even has a command-line argument that disables exception handling, which allows you to catch the problem in your debugger on a second run. I really couldn’t ask for much more.
5. Good assert functionality. Yes. Has assert statements for just about any operation you want (equality, closeness, less than, greater than, bitwise equal, etc). It even has support for checking whether exceptions were thrown. The assert statements correctly print out the contents of the variables being checked. Top marks on this one.
6. Supports different outputs. Probably, but it’s not exactly trivial to change. At least the default output is IDE friendly. I suspect I would need to dig deeper into the unit_test_log_formatter, but I certainly didn’t see a variety of preset output types that I could just plug in.
7. Supports suites. Yes, but with a big catch. Unless I’m missing something (which is very possible at this point–if so make sure to let me know), creating a suite requires a bunch of fairly verbose statements and also requires modifying the test runner itself in main. Have a look at the example below. Couldn’t that have been simplified to the extreme? It’s not a big deal as this is my least-wanted requirement, but I wish I could label all the test cases in one file as part of a suite with a simple macro at the beginning of the file. Another minor shortcoming is the lack of setup/teardown steps for whole suites, which could come in really handy (especially if suite creation were streamlined).

#include <boost/test/unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>
using boost::unit_test::test_suite;   

struct MyTest
{
    void TestCase1()
    {
        float fnum = 2.00001f;
        BOOST_CHECK_CLOSE(fnum, 2.f, 1e-3);
    }

    void TestCase2()
    {}
};

test_suite * GetSuite1()
{
    test_suite * suite  = BOOST_TEST_SUITE("my_test_suite");
    boost::shared_ptr instance( new MyTest() );
    suite->add (BOOST_CLASS_TEST_CASE( &MyTest::TestCase1, instance ));
    suite->add (BOOST_CLASS_TEST_CASE( &MyTest::TestCase2, instance ));
    return suite;
}

#include <boost/test/auto_unit_test.hpp>
using boost::unit_test::test_suite; 

extern test_suite * GetSuite1();  

boost::unit_test::test_suite* init_unit_test_suite( int /* argc */, char* /* argv */ [] )
{
    test_suite * test = BOOST_TEST_SUITE("Master test suite");
    test->add( boost::unit_test::ut_detail::auto_unit_test_suite() );
    test->add(GetSuite1());
    return test;
}

Boost.Test is a library with a huge amount of potential. It has great support for exception handling and advanced assert statements. It also has other fairly unique functionality such as support for checking for infinite loops, and different levels of logging. On the other hand, it’s very verbose to add new tests that are part of a suite, and it might be a bit heavy weight for game console environments.

CppUnitLite has a funny story behind it. Michael Feathers, the original author of CppUnit, got fed up with the complexity of CppUnit and how it didn’t fit everyone’s needs, so we wrote the ultra-light weight framework CppUnitLite. It is as light on the features as it is on complexity and size, but his philosophy was to let people customize it to deal with whatever they need.

Indeed, CppUnitLite is only a handful of files and it probably adds up to about 200 lines of very clear, easy to understand and modify code. To be fair, in this comparison I actually used a version of CppUnitLite I modified a couple of years ago (download it along with all the sample code) to add some features I needed (fixtures, exception handling, different outputs). I figured it was definitely in the spirit that CppUnitLite was intended, and if nothing else, it can show what can be accomplished by just a few minutes of work with the source code.

On the other hand, CppUnitLite doesn’t have any documentation to speak of. Heck, it doesn’t even have a web site of its own, which I’m sure is not helping the adoption rate by other developers.

1. Minimal amount of work needed to add new tests. Absolutely! Of all the unit-test frameworks, this is the one that comes closest to the ideal. On the other hand, it could be the fact that I’ve used CppUnitLite the most and I’m biased. In any way, it really fits my idea of minimum amount of work required to set up a simple test or even one with a fixture (although that could be made even better).

#include "lib/TestHarness.h"  

TEST (Whatever,MyTest)
{
    float fnum = 2.00001f;
    CHECK_DOUBLES_EQUAL (fnum, 2.0f);
}

2. Easy to modify and port. Definitely. Again, it gets best of the class award in this category. No other unit-test framework comes close to being this simple, easy to modify and port, and at the same time having reasonably well separated functionality. The original version of CppUnitLite even had a special lightweight string class to avoid dependencies on STL. In my modified version I changed it to use std::string since that’s what I use in most of my projects, but the change took under one minute to do. Also, using it under Linux was absolutely trivial, even though I had only used it under Windows before.

3. Supports fixtures. This is where the original CppUnitLite starts running into trouble. It’s so lightweight that it doesn’t have room for many features. This was an absolute must for me, so I went ahead and added it. I’m sure it could be improved to make it so adding a fixture requires even less typing, but it’s functional as it stands. Unfortunately, it suffers from the problem that objects need to be created dynamically if we want them to be created right before each test. To be fair though, every single unit-test framework in this evaluation has that requirement. Oh well.

#include "lib/TestHarness.h"
#include "MyTestClass.h"   

class MyFixtureSetup : public TestSetup
{
public:
    void setup()
    {
        someValue = 2.0;
        str = "Hello";
    }
    void teardown()
    {}
protected:
    float someValue;
    std::string str;
    MyTestClass myObject;
};   

TESTWITHSETUP (MyFixture,Test1)
{
    CHECK_DOUBLES_EQUAL (someValue, 2.0f);
    someValue = 0;
    // CppUnitLite doesn't handle system exceptions very well either
    //myObject.UseBadPointer();
    // A regular exception works nicely though myObject.ThrowException();
}  

TESTWITHSETUP (MyFixture,Test2)
{
    CHECK_DOUBLES_EQUAL (someValue, 2.0f);
    CHECK_STRINGS_EQUAL (str, std::string("Hello"));
}   

TESTWITHSETUP (MyFixture,Test3)
{
    // Unfortunately, it looks like the framework creates 3 instances of MyTestClass
    // right at the beginning instead of doing it on demand for each test. We would
    // have to do it dynamically in the setup/teardown steps ourselves.
    CHECK_LONGS_EQUAL (1, myObject.s_currentInstances);
    CHECK_LONGS_EQUAL (3, myObject.s_instancesCreated);
    CHECK_LONGS_EQUAL (1, myObject.s_maxSimultaneousInstances);
}

4. Handles exceptions and crashes well. The original CppUnitLite didn’t handle them at all. I added minor support for this (just an optional try/catch). To run the tests without exception support it requires recompiling the tests with a special define turned on, so it’s not at slick as the command-line argument that Boost.Test features.

5. Good assert functionality. Here is where CppUnitLite really shows its age. The assert macros are definitely the worst of the lot. They don’t use a stream to print out the contents of their variables, so we need custom macros for each object type you want to use. It comes with support for doubles, longs, and strings, but anything else you need to add by hand. Also, it doesn’t have any checks for anything other than equality (or closeness in the case of floating-point numbers).

6. Supports different outputs. Again, the original only had one type of output. But it was very well isolated and it was trivial to add more.

7. Supports suites. Probably the only framework that doesn’t support suites. I never really needed them, but they would probably be very easy to add on a per-file basis.

CppUnitLite is as barebones as it gets, but with a few modifications it hits the mark in all the important categories. If it had better support for assert statements, it would come very close to my ideal framework. Still, it’s a worthy candidate for the final crown.

I had never heard of NanoCppUnit until Phlip brought it up. From reading the feature list, it really appeared to be everything that I wanted CppUnitLite to be, except that it was better and ready to work out of the box.

The first point against NanoCppUnit is the awful “packaging” of the framework. If you thought that CppUnitLite was bad (not having a web page of its own), well, at least you could download it as a zip file. For NanoCppUnit you actually have to copy and paste the five files that make up the framework from a web page. I’m not kidding. That makes for some “lovely” formatting issues I might add. The documentation found in the web page wasn’t exactly very useful either.

In any case, I continued my quest to get a simple test program up and running with NanoCppUnit. Out of the box (or out of the web page, rather) it’s clearly aimed only at Windows platforms. I thought it would be trivial to fix, but changing it required more time than I thought at first (I personally gave up when I started getting errors buried three macros deep into some assert statement). Unlike CppUnitLite, the source code is not very well structured at all, full of ugly macros everywhere, making it not trivial to add new features like new output types. Unless I’m totally mistaken, it even looks like it has sample code inside the test framework itself. Eventually I had to give up on running it under Linux, so my comments here are just best guesses by looking at the source code.

1. Minimal amount of work needed to add new tests. I think so. I’m not sure it’s possible to create a standalone test that is part of a global suite, but at least creating a suite doesn’t require manual registration of every test. This is (probably) the simplest possible test with NanoCppUnit.

struct MySuite:  TestCase { };  

TEST_(MySuite, MyTest)
{
    float fnum = 2.00001f;
    CPPUNIT_ASSERT_DOUBLES_EQUAL(fnum, 2.0f, 0.0001);
}

2. Easy to modify and port. Not really. Windows dependencies run deeper than it seems on the surface. The code is small, but it’s messy enough that it’s a pain to work with. I’m sure it can be ported with a bit of effort though since it’s so small.
3. Supports fixtures. Yes. Setup and teardown calls very similar to the modified version of CppUnitLite.
4. Handles exceptions and crashes well. No idea since I wasn’t able to run it. I see some try/catch statements in the code, but no way to turn them on or off. Probably no better than CppUnitLite.
5. Supports different outputs. Not really. Everything is hardwired to use a stream that sends its contents to OutputDebugString() in Windows. I think the default output text is formatted to match the Visual Studio error format.
6. Good assert functionality. Yes. Good range of assert statements, including floating point closeness, greater than, less than, etc.
7. Supports suites. Yes. I don’t know what’s involved in just running a single suite though. Not a big deal either way.

One of NanoCppUnit’s unique features is regular expression support as part of its assert tests. That’s very unusual, but I can see how it could come in handy. A few times in the past, I’ve had to check that a certain line of code has some particular format, so I had to sscanf it, and then check on some of the contents. A regular expression check would have done the job nicely.

Unfortunately, NanoCppUnit doesn’t really live up to the standards of other frameworks. Right now it feels too much as a work in progress, with too much missing functionality and not clearly structured code.

The further along we get in this evaluation, the less Xunit-like the frameworks become. Unit++’s unique feature is that it pretends to be more C++-like than CppUnit. Wait a second, did I hear that right? More C++ like? Is that supposed to be a good thing? Looking back at my ideal test framework, it really isn’t very much like C++ at all. Once I started thinking about that topic I realized that there really is no reason at all why the tests framework itself needs to be in C++. The tests you write need to be in the language of the code being tested, but all the wrapper code doesn’t. That’s a point that the next, and final, testing framework will drive home.

So, what does it mean to be more C++ like? No macros for a start. You create suites of tests by creating classes that derive from suite. That’s the same thing we were doing in most other frameworks, really, but it was just happening behind the scenes. It really doesn’t help me any to know that that is what I’m doing, and I would certainly not call it a “feature.” As a result, tests are more verbose than they could be.

The documentation is simply middle-of-the-road. It’s there, but it’s not particularly detailed and it doesn’t come loaded with examples.

1. Minimal amount of work needed to add new tests. I’m afraid it gets failing marks for this. It requires manual registration of tests, and every test needs to be part of a suite. This makes adding new tests tedious and error prone (by writing a new test and forgetting to register it). I don’t know about you, but with all the C++ cruft, I look at the code below and it’s not immediately obvious what it does until I’ve scanned it a couple of times. The signal to noise ratio is pretty poor.

#define __UNITPP #include "unit++.h"
using namespace unitpp;
namespace
{
    class Test : public suite
    {
        void test1()
        {
            float fnum = 2.00001f;
            assert_eq("Checking floats", fnum, 2.0f);
        }
    public:
        Test() : suite("MySuite")
        {
            add("test1", testcase(this, "Simplest test", &Test::test1));
            suite::main().add("demo", this);
        }
    }; 

    Test* theTest = new Test();
}

2. Easy to modify and port. So-so. It needs the STL and it pulls in some stuff like iostreams (which I remember having distinct problems with when I was working with STLPort). On the other hand the source code is relatively small and self-contained so it’s certainly doable to port and modify if you’re willing to put in some time.
3. Supports fixtures. Another framework that I just can’t see how to do fixtures with. Like Boost.Test, it seems to think that using the constructor and destructor for each class is all you need. A quick search for fixture or setup or teardown in the documentation doesn’t reveal anything. I don’t know if I’m totally missing something or if other people just write very different tests from me. I suppose I could create a new class for every fixture I want, put the setup in the constructor and the teardown in the destructor and inherit from it for every test case (and somehow figure out how to create an instance of that class and use it for each test run). It’s probably possible, but it’s not exactly trivial, is it? Again, the lack of fixtures puts this framework out of the running.
4. Handles exceptions and crashes well. Average. It manages to catch regular exceptions without crashing, but that’s about it. No system exceptions in Linux. No way to turn it off for debugging.
5. Supports different outputs. I couldn’t figure out how to do it from the documentation. There is probably a way to do it since it even supports GUI functionality , but it’s not obvious (and there are no examples). Besides, by this point, having failed points 1 and 3, I wasn’t really motivated to spend a while learning the framework. Incidentally, this is one of the few frameworks whose default text output is not formatted correctly for IDEs like KDevelop.
6. Good assert functionality. It scrapes by with the minimum in this department. It provides equality and condition checks, but that’s it. It doesn’t even provide a float version of assert to check for “close enough.” At least it prints the contents of the variables to a stream correctly.
7. Supports suites. Yes, like most of them.

Overall, Unit++ is not really a candidate. Perhaps it’s because it’s not intended for the type of testing I intend to use it for, but it doesn’t offer anything new over other frameworks and it has a lot of drawbacks of its own. The lack of fixtures is simply unforgivable.

After looking into a framework that tried to be different from XUnit (Unit++), I wasn’t particularly looking forward to evaluating possibly the wackiest one of them all, CxxTest. I had never heard of it until a few days ago, but I knew that it required using Perl along the way to generate some C++ code. My spider senses were tingling.

Boy was I wrong!! Within minutes of using CxxTest and reading through its great documentation (the best by far), I was completely convinced this was the way to go. This came as a complete surprise to me since I was ready to leave somewhat dissatisfied and pronounce a victor between CppUnit and CppUnitLite.

Let’s start from the beginning. What’s with the use of Perl and why is it different from CppUnit? Erez Volk, the author of CxxTest, had the unique insight that just because we’re testing a C++ program, we don’t need to rely on C++ for everything. Other languages, such as Java, are better suited to what we want to do in a unit-testing framework because they have good introspection (reflection) capabilities. C++ is quite lacking in that category, so we’re forced to use kludges like manual registration of tests, ugly macros, etc. CxxTest gets around that by parsing our simple tests and generating a C++ test runner that calls directly into our tests. The result is simply brilliant. We get all the flexibility we need without the need for any ugly macros, exotic libraries, or fancy language features. As a matter of fact, CxxTest’s requirements are as plain vanilla as you can get (other than being able to run Perl).

The code-generation step is also trivial to integrate into the regular build system. The wonderful documentation gives explicit step-by-step instructions on how to integrate it with make files, Visual Studio projects files, or Cons. Once you have it set up, you won’t even remember there’s anything out of the ordinary going on.

Let’s see how it stacks up against the competition.

1. Minimal amount of work needed to add new tests. Very good. It’s almost as simple as the best of them. If I could nit-pick, I would have wished for an even simpler way to create tests without the need to declare the class explicitly. Since we’re doing processing with a Perl script, there’s no reason we couldn’t have taken it a step beyond that and used a syntax even closer to my ideal test framework.

class SimplestTestSuite : public CxxTest::TestSuite
{
    public: void testMyTest()
    {
        float fnum = 2.00001f;
        TS_ASSERT_DELTA (fnum, 2.0f, 0.0001f);
    }
};

2. Easy to modify and port. CxxUnit requires the simplest set of language features (no RTTI, no exception handling, no template functions, etc). It also doesn’t require any external libraries. It is also distributed simply as a set of header files, so there’s no need to compile into a separate library or anything like that. Functionality is pretty well broken down and separated in the original source code, so making modifications should be fairly straightforward.
3. Supports fixtures. CxxUnit gets the “top of its class” label in this category. Not only does it support setup/teardown steps on a per-test level, but it also supports them at the suite and at the world (global) level. Creating fixtures is pretty straightforward and just requires inheriting from a class and creating as many functions as you want starting with the letters “test.” To be really picky, I would have loved it if they had taken it a step further and, apart from simplifying the code a bit more, also inserted the setup and teardown code around the code for each test. That would have allowed us to work with those objects directly on the stack and their lifetime would have been managed correctly around each test. Oh well. Can’t have everything.

#include "MyTestClass.h"  

class FixtureSuite : public CxxTest::TestSuite
{
public:
    void setUp()
    {
        someValue = 2.0;
        str = "Hello";
    } 

    void tearDown() {}  

    void test1()
    {
        TS_ASSERT_DELTA (someValue, 2.0f, 0.0001f);
        someValue = 13.0f;
        // A regular exception works nicely though myObject.ThrowException();
    } 

    void test2()
    {
        TS_ASSERT_DELTA (someValue, 2.0f, 0.0001f);
        TS_ASSERT_EQUALS (str, std::string("Hello"));
    } 

    void test3()
    {
        //myObject.UseBadPointer();
        TS_ASSERT_EQUALS (1, myObject.s_currentInstances);
        TS_ASSERT_EQUALS (3, myObject.s_instancesCreated);
        TS_ASSERT_EQUALS (1, myObject.s_maxSimultaneousInstances);
    }  

    float someValue;
    std::string str;
    MyTestClass myObject;
};

4. Handles exceptions and crashes well. Great support. It catches all exceptions and prints information about them formatted like any other error (no system exceptions under Linux though). You can easily re-run the tests with a command-line argument to the Perl script to avoid catching exceptions and catch them in the debugger instead. It also gives you a custom version of every assert macro that lets you catch the exceptions yourself in case you ever need to do that.
5. Supports different outputs. Different outputs are supported by passing a parameter indicating which type of output you want to the Perl processing step. The default one (error-printer) was formatted correctly for IDE parsing, and you can use several others (including GUIs for those of you addicted to progress bars, a yes/no report, or a stdio one). Adding new output formatting sounds very straightforward and it’s even covered in the documentation.
6. Good assert functionality. Again, it gets “top of its class” for this one. It has a whole suite of very comprehensive assert functions, including ones for exception handling, checking predicates, and arbitrary relations. It even has a way to print out warnings which can be used to differentiate between two parts of the code calling the same test, or to print reminder “TODO” messages to yourself.
7. Supports suites. Yes. All tests are part of a suite.

Another feature supported by CxxUnit that I haven’t had time to look into is some support for mock objects. Anybody doing TDD knows the value of mock objects when it comes to testing the interactions between a set of objects. Apparently CxxUnit allows you to override global functions with specific mock functions (it gives an example of overriding fopen()). I don’t think it helps any with regular classes; for those you’re on your own.

So, what’s not to like in CxxTest? Not much, really. Other than wishing that the test syntax were a bit tighter, the only thing to watch out for is what happens with large projects. If you follow the examples in the documentation, it will create a single runner for all the tests you give it. This can be problematic if you’re going to be having thousands of tests, and then making one small change in one of them causes a full recompilation of all your code.

Update: After talking with Erez and re-checking the documentation, I realized this is already fully supported in CxxUnit. By default, when you generate a test runner, it adds a main function and some global variables, so linking with other similar runners gives all sorts of problems. However, it turns out you can generate a test runner with the –part argument, and it will leave out the main function and any other globals. You can then link together all the runners and have a single executable. I wonder if it would be worth going as far as creating a runner for every suite, or if it would be best to cluster suites together. Worth investigating at some point whenever I get enough tests to make a difference.

Conclusion

After going through all six C++ unit-testing frameworks, four stand out as reasonable candidates: CppUnit, Boost.Test, a modified CppUnitLite, and CxxTest.

Of the four, CxxTest is my new personal favorite. It fits very closely the requirements of my ideal framework by leveraging the power of an external scripting language. It’s very usable straight out of the “box” and it provides some nifty advanced features and great assert functionality. It does require the use of a scripting language as part of the build process, so those unconfortable with that requirement, might want to look at one of the other three frameworks.

CppUnit is a solid, complete framework. It has come a long, long way in the last few years. The major drawbacks are the relative verbosity for adding new tests and fixtures, as well as the reliance on STL and some advanced language issues.

If what you need is absolute simplicity, you can do no wrong starting with CppUnitLite (or a modified version), and tweaking it to fit your needs. It’s a well-structured, ultra-light framework with no external dependencies, so modifying it is extremely easy. Its main drawback is the lack of features and the primitive assert functionality.

If you’re going to be working mostly on the PC, you don’t expect to have to modify the framework itself, and you don’t mind pulling in some additional Boost libraries, Boost.Test could be an excellent choice.

Should you roll your own unit-test framework? I know that Kent Beck recommends it in his book Test-Driven Development: By Example, and it might be a great learning experience, but I just can’t recommend it. Just as it’s probably good to write a linked-list and a stack data structure a few times but I wouldn’t recommend actually doing that in production code instead of using the ones provided in the STL, I strongly recommend starting with one of the three unit-testing frameworks mentioned above. If you really feel the need to roll your own, grab CppUnitLite and get hacking.

Whichever one you choose, you can really do no wrong with one of those three frameworks. The most important thing is that you are writing unit tests, or, even better, doing test-driven development. To paraphrase Michael Feathers, code without unit tests is legacy code, and you don’t want to be writing legacy code, do you?

unit_test_frameworks.tar.gz