Stepping Through the Looking Glass: Test-Driven Game Development (Part 1)

If you’re a typical game developer, you probably don’t write any tests for the code you create. So how would you feel about not just writing tests, but creating them before the code they are testing? What if I told you those tests don’t even verify that the code you write is correct? “It’s madness,” you might say; “it’s all backwards!” Not really. It all makes sense in its own way. Follow me through the looking glass and I’ll show you the wonderful upside-down world of test-driven development and how we can apply it to games.

Traditional development

Let’s take a moment to think about how you write code without using test-driven development. You decide you need to implement some piece of functionality, you mentally break the problem into smaller problems, and you start working on each of them. While you write code, your only guide for whether you’re heading in the right direction or not is whether your code continues to compile. Once you’ve assembled a minimum number of those smaller subtasks, your program might even be able to run. So you fire up the game or tool and “see” if it works. If it’s not obvious (or if it doesn’t work at all), maybe you break in the debugger and try to step into the code you just wrote to make sure the program is doing what you intended. Once you see it do its thing, you happily commit it to version control and move on to some other task.

Does that more or less describe how you write code? You can replace any specific details in that paragraph, but in general that seems to sum up in my experience how most programmers approach writing code.

What is Test-Driven Development?

The idea behind Test-Driven Development (TDD) is extremely simple: before you write a piece of functionality, you write a test for it. Only once you have that test, which should fail at first, do you actually implement the functionality and see the test pass.

I know it sounds like a backwards approach if you’re not used to it, but it makes a lot more sense than it seems to at first. The thing to keep in mind is that you’re not writing a bunch of tests first and then making them pass. You’re just writing a single test for a very small bit of functionality that you can implement in a minute or two.

TDD development has a very well defined, short cycle, usually only taking a few minutes per cycle:

• Write a single test for a very small piece of functionality.
• Run it and see it fail (or not even compile in C++).
• Write code to make the test compile and pass.
• Run the test and see it pass.
• Refactor code. See test pass.
• Refactor tests. See tests pass.

Comparing it to the non-test-driven development approach, you’re replacing all the mental checking and debugger stepping with code that verifies that your program does exactly what you intended it to do.

There really isn’t much more to test-driven development than that. The devil, as they say, is in the details. You need to learn to effectively do that cycle over and over during your development, how to tackle the right task size, how to organize things so you can test them, how to write tests easily, how to make sure tests can be executed constantly, etc.

A very good starting point is Kent Beck’s book Test Driven Development. He’ll take you through an example and give you an idea of the kind of pacing and techniques you’re likely to use. Don’t be put off by the use of Java as their development language. Java is close enough to C++, and besides, it’s not the language that’s important, it’s the concept.

Benefits of TDD

So, what exactly do you get by doing things apparently backwards? A lot, as it turns out.

Safety net

Developing your code using TDD means that every single piece of functionality you add will have a test coverage (or close to it anyway). If anything ever changes, you’ll know right away. This is extremely important for many reasons.

Refactoring becomes a lot easier, so you can always keep your codebase healthy (I always like to say that the quality of a codebase is directly tied to how easy it is to change—if it ever becomes difficult or “a pain” to change, the quality is already on its way down and it’s just going to get worse if untreated).

Making changes late in the project also becomes a lot easier, effectively flattening out the change vs. cost curve.

This means you can be making significant changes late in the project based on real playtest feedback or finding out what really works and what doesn’t, and still be confident you’re not breaking anything major. This alone can make the difference between a so-so and an outstanding game.

We all know about those dusty corners of codebases. You know, the places with all the cobwebs, and sometimes gnarled oak trees and a troll or two hiding in the shadows. The places no sane programmer dares approach, especially as a milestone looms over the horizon. TDD gives you the courage and the tools to dive in head first into any part of the code, even the scary parts, make any necessary changes, and walk away like a hero.

Usable design

Here’s something that might come as a surprise: Code that you develop through TDD is going to end up looking very different from code you would have written otherwise. Why? After all, the only thing we’re doing differently is writing the tests first. TDD is not a design paradigm or anything like that.

That’s because with TDD, the first thing you’re forced to do is to think about how you want to use your code. You don’t start with a UML diagram, or a detailed class header file, or anything like that. You start by using the feature you’re about to implement. You could call it “extreme dog food eating.” You’re going to be your first user of your code, and you’re going to think about how to use it from the beginning.

What does that mean in practice? You’ll find that it’s very easy to create new objects of the classes you’re designing. Most of the time you’ll be able to create them on the stack and that will be it. It also means you won’t have to have a complicated sequence of registration, creation, binding, yada, yada, yada. You create an object and it does what it’s supposed to do.

Quick test: Think about your current game codebase. Could you write something like the following code?

#include "GameEntity.h"
int main()
{
    GameEntity entity(optional parameters);
    entity.DoSomething();
    return 0;
}

Chances are you can’t. You probably need a world for the entity to live in. And the world needs to pull in physics and graphics, and initialize the video card, and the sound system, and the networking… Then you probably have to go through a complex creation procedure. Or maybe you can’t even create an entity that way and you can only load it from disk along with some resources.

If simplicity is something important (and I clearly think so), then TDD will help you tremendously.

Modularity

This is another side effect of having to eat your own dog food all the time and use the code you’re about to write. You’ll quickly see that testing code that depends on lots of other code is a nuisance that can be avoided most of the time. As a result, the code you write will be extremely modular. It’ll be very easy to use an object in isolation from other classes or systems.

To draw on the earlier example, you will be able to create a game entity without a world, and certainly without expensive operations like initializing the graphics system and pulling in the rest of the codebase.

Modularity schodularity. Why should you care? There are the usual reasons: refactoring, reusing the code, adding new features later on, etc. But there are also much more practical ones. A highly modular codebase will compile much more quickly than one that has grown organically (without resorting to the likes of Incredibuild). It’ll also allow you to break it up into smaller, separate libraries which can be tested and linked separately (which means really fast iteration times instead of always having a 2-minute link step if you’re dealing with the full codebase).

Documentation

What’s wrong with comments in code? I used to be a big one for having a header on every class and every function, generating cute help files with Doxygen and all that, but I eventually gave up. When you’re developing at a real-world pace, those pretty comments quickly fall by the wayside. Then you’re left either with obvious comments (assign a to b!), outdated ones, or, even worse, incorrect ones.

The tests you write while doing TDD are the ideal low-level documentation for your code. For every feature, you have at least one example on how to use it correctly and how it’s expected to be used. And the best part is that it can’t ever get out of date.

Comments in code still have their place, but it should be very rare. You should still add comments explaining why you did something a certain way, or including the URL of the web page where you got certain algorithms or code snippets. Other than that, I believe documentation should come in the form of good class and function names and a comprehensive set of TDD-generated unit tests.

WYGIWYM (What You Get Is What You Meant)

When you’re doing TDD, even though you’re writing unit tests, you’re not verifying that the algorithm you’re implementing is correct. That’s not the goal. What TDD gives you is the confidence that the code you just wrote does what you wanted it to do. If you meant to do the wrong thing, you’ll only be testing that you did the wrong thing. TDD isn’t going to help you in that respect (other than make you realize really early on that you’re doing the wrong thing, since you’re using your own code before you even write it). A bit more on this later on.

Instant feedback

When you write any code doing TDD, you get instant feedback. You write a test, build, run the tests, and you see them fail. Then you write the code, and you see the tests pass. You refactor a few things, and see the tests pass still. Instant feedback is part of the TDD cycle, and since the cycle is so short, you’ll get feedback on how things are going every few minutes, or even several times per minute.

Apart from keeping you honest and showing that things are progressing in the right direction, this instant feedback can have a very curious effect on the mood of the programmer. I don’t know exactly how to describe it, but I hadn’t had as much fun programming in a long, long time. Somehow, TDD brought back the excitement and satisfaction I felt writing Basic programs on an Amstrad CPC twenty years ago.

I know that’s a very subjective “benefit” of TDD, but it’s one that can have a profound effect on the team and the overall productivity. Don’t underestimate the power of a happy programmer!

TDD in practice

Enough theory. I hope I have convinced you of the potential benefits and you’re willing to give it a try. Let’s get into the nitty-gritty details. How exactly do we go about doing TDD? I’m assuming you’re using C++ since that’s the most-used language for game development today.

Get a unit-testing framework.

You need a unit-testing framework. The goal is to make writing a test as painless as possible. The easier it is to write a test, the less people will resist writing new tests. Since unit tests are at the core of TDD, the more streamlined the process the better.

I recently looked at some of the most popular C++ unit-testing frameworks. If you’re just starting out, don’t worry about the details. Use whatever is most convenient. I’m currently using a modified version of CppUnitLite (which I’m hoping to put up here in the next few days) just because it’s so small, easy to use, and easy to modify and port.

With my current unit-testing framework, adding a new test is as easy as writing this:

TEST (MyFirstTest)
{
    CHECK_EQUAL (3, GetSomeValue());
}

Put tests into a separate executable.

I recommend splitting your code into many separate libraries, each of them dealing with one specific subject: sound, networking, AI, scene management, physics, bitmaps, file IO, input, etc, etc. That’s just a good practice whether you’re doing TDD or not. Extra bonus points for managing dependencies between libraries correctly and avoiding cyclical dependencies.

For each library, create a new executable that contains the tests for that library. My preferred directory structure looks something like this, but feel free to organize it in any way you like

I definitely don’t want my tests in the same file as the class they’re testing. I try to keep the class as simple as possible, and that would just add clutter. Some people prefer to put the test files in the same directory as the files for the library itself. I like to keep them separate but nearby, especially because there’s no one-to-one correspondence between library files and test files.

Build and run tests often.

Make sure to make the test executable dependent on the library it’s testing (and linking with). As the final step of the build process, run the test itself. That way it’ll be impossible to build a library without building the tests and running them. Since the tests should be very fast (taking less than a few seconds), this shouldn’t get in the way of fast iteration.

Most unit-testing frameworks return by default the number of failed tests they encountered. Since calling that executable was part of the build process, if it returns anything other than zero, the build framework considers the build failed, which is exactly what you want. If you’re using an IDE such as Visual Studio or KDevelop, you’ll also get a nicely formatted error message that you can select and it’ll go to the failing test. Tests have become an integral part of the build, and a failing test is treated just like a syntax error.

If you’re using make files, just run the executable as your last step of the build rule:

test: 
    $(objects) ${COMPILER} -o test $(objects) ${LIBFLAGS} 
    ./test

With Scons, just use AddPostAction:

test = env.Program('test', list) 
env.AddPostAction(test, './test')

In Visual Studio, call $(TargetPath) as your PostBuild rule.

Make sure the test executables are built and run in your automated build machine as well. Do both debug and release, and if you can also run them in your target platforms (Xbox, PS2, or whatever), so much the better.

Step size.

Before letting you loose with the TDD tools in hand, it’s important to talk about step size. Before you start a TDD cycle, you need to decide how much functionality you should tackle in that one cycle. It’s extremely important that you start by taking baby steps. Don’t worry about the fact that it looks silly or pointless. The point at first is to get comfortable with the cycle and to get the hang of it.

A full iteration of the TDD cycle should take just a few minutes (and sometimes less than a minute). If it takes significantly longer than that to get your tests running again (>15 minutes), it probably means that you tried taking too big a step. If you find yourself in that situation with no obvious way to make the tests pass, you might even want to consider undoing what you did and starting again with an even simpler task.

The concept of step size is very important. Choosing the right step size is probably the hardest part of TDD, and it only comes with experience and tripping a few times. At the beginning, you should start with trivial steps and stick with it. Once you’re totally comfortable with it, you can take steps a bit larger. But as soon as you find yourself with unexpected failing tests or taking too long before getting passing tests again, you should back off and take baby steps again for a while.

We’ll see some specific examples of TDD in action in the second part of this article, which will show how you can take tiny steps.

Resources

• Test Driven Development: By Example by Kent Beck. The first book you should read.
• Test Driven Development: A Practical Guide by Dave Astels. If you are dying to read more.
• TestDriven.com. Good portal to test-driven topics.
• testdrivendevelopment mailing list.

Now you should have a good idea of what TDD is and what’s involved in actually doing it. But TDD is like riding a bicycle. You can read all you want about it, but eventually, you’ll have to try it for yourself if you want to learn it. I hope this is enough information for you to go out and take it out for a quick spin.

The second part of this article will deal with how to applying TDD to game development (with specific examples) and will cover tips from the trenches and what you can expect when applying TDD to your projects.