Mock Objects: Friends Or Foes?

In a previous article we covered all the details necessary to start using unit testing on a real-world project. That was enough knowledge to get started and tackle just about any codebase. Eventually you might have found yourself doing a lot of typing, writing redundant tests, or having a frustrating time interfacing with some libraries and still trying to write unit tests. Mock objects are the final piece in your toolkit that allow you to test like a pro in just about any codebase.

Testing Code

The purpose of writing unit tests is to verify the code does what it’s supposed to do. How exactly do we go about checking that? It depends on what the code under test does. There are three main things we can test for when writing unit tests:

  • Return values. This is the easiest thing to test. We call a function and verify that the return value is what we expect. It can be a simple boolean, or maybe it’s a number resulting from a complex calculation. Either way, it’s simple and easy to test. it doesn’t get any better than this.
  • Modified data. Some functions will modify data as a result of being called (for example, filling out a vertex buffer with particle data). Testing this data can be straightforward as long as the outputs are clearly defined. If the function changes data in some global location, then it can be more complicated to test it or even find all the possible places that can be changed. Whenever possible, pass the address of the data to be modified as an input parameter to the functions. That will make them easier to understand and test.
  • Object interaction. This is the hardest effect to test. Sometimes calling a function doesn’t return anything or modify any external data directly, and it instead interacts with other objects. We want to test that the interaction happened in the order we expected and with the parameters we expected.

Testing the first two cases is relatively simple, and there’s nothing you need to do beyond what a basic unit testing-framework provides. Call the function and verify values with a CHECK statement. Done. However, testing that an object “talks” with other objects in the correct way is much trickier. That’s what we’ll concentrate on for the rest of the article.

As a side note, when we talk about object interaction, it simply refers to parts of the code calling functions or sending messages to other parts of the code. It doesn’t necessarily imply real objects. Everything we cover here applies as well to plain functions calling other functions.

Before we go any further, let’s look at a simple example of object interaction. We have a game entity factory and we want to test that the function CreateGameEntity() finds the entity template in the dictionary and calls CreateMesh() once per each mesh.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    EntityDictionary dict;
    MeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
    // How do we test it called the correct functions?
}

We can write a test like the one above, but after we call the function CreateGameEntity(), how do we test the right functions were called in response? We can try testing for their results. For example, we could check that the returned entity has the correct number of meshes, but that relies on the mesh factory working correctly, which we’ve probably tested elsewhere, so we’re testing things multiple times. It also means that it needs to physically create some meshes, which can be time consuming or just need more resources than we want for a unit test. Remember that these are unit tests, so we really want to minimize the amount of code that is under test at any one time. Here we only want to test that the entity factory does the right thing, not that the dictionary or the mesh factory work.

Introducing Mocks

To test interactions between objects, we need something that sits between those objects and intercepts all the function calls we care about. At the same time, we want to make sure that the code under test doesn’t need to be changed just to be able to write tests, so this new object needs to look just like the objects the code expects to communicate with.

A mock object is an object that presents the same interface as some other object in the system, but whose only goal is to attach to the code under test and record function calls. This mock object can then be inspected by the test code to verify all the communication happened correctly.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

dict.meshCount = 3;

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);

    CHECK_EQUAL(1, dict.getEntityInfoCallCount);
    CHECK_EQUAL(gameEntityUid, dict.lastEntityUidPassed);
    CHECK_EQUAL(3, meshFactory.createMeshCallCount);
}

This code shows how a mock object helps us test our game entity factory. Notice how there are no real MeshFactory or EntityDictionary objects. Those have been removed from the test completely and replaced with mock versions. Because those mock objects implement the same interface as the objects they’re standing for, the GameEntityFactory doesn’t know that it’s being tested and goes about business as usual.

Here are the mock objects themselves:

struct MockEntityDictionary : public IEntityDictionary
{
    MockEntityDictionary() 
        : meshCount(0)
        , lastEntityUidPassed(0)
        , getEntityInfoCallCount(0)
    {}

    void GetEntityInfo(EntityInfo& info, int uid)
    {
        lastEntityUidPassed = uid;
        info.meshCount = meshCount;
        ++getEntityInfoCallCount;
    }

    int meshCount;
    int lastEntityUidPassed;
    int getEntityInfoCallCount;
};


struct MockMeshFactory : public IMeshFactory
{
    MockMeshFactory() : createMeshCallCount(0)
    {}

    Mesh* CreateMesh()
    {
        ++createMeshCallCount;
        return NULL;
    }
};

Notice that they do no real work; they’re just there for bookkeeping purposes. They count how many times functions are called, some parameters, and return whatever values you fed them ahead of time. The fact that we’re setting the meshCount in the dictionary to 3 is how we can then test that the mesh factory is called the correct number of times.

When developers talk about mock objects, they’ll often differentiate between mocks and fakes. Mocks are objects that stand in for a real object, and they are used to verify the interaction between objects. Fakes also stand in for real objects, but they’re there to remove dependencies or speed up tests. For example, you could have a fake object that stands in for the file system and provides data directly from memory, allowing tests to run very quickly and not depend on a particular file layout. All the techniques presented in this article apply both to mocks and fakes, it’s just how you use them that sets them apart from each other.

Mocking Frameworks

mock.jpgThe basics of mocking objects are as simple as what we’ve seen. Armed with that knowledge, you can go ahead and test all the object interactions in your code. However, I bet that you’re going to get tired quickly from all that typing every time you create a new mock. The bigger and more complex the object is, the more tedious the operation becomes. That’s where a mocking framework comes in.

A mocking framework lets you create mock objects in a more automated way, with less typing. Different frameworks use different syntax, but at the core they all have two parts to them:
A semi-automatic way of creating a mock object from an existing class or interface.
A way to set up the mock expectations. Expectations are the results you expect to happen as a result of the test: functions called in that object, the order of those calls, or the parameters passed to them.

Once the mock object has been created and its expectations set, you perform the rest of the unit test as usual. If the mock object didn’t receive the correct calls the way you specified in the expectations, the unit test is marked as failed. Otherwise the test passes and everything is good.

GoogleMock

GoogleMock is the free C++ mocking framework provided by Google. It takes a very straightforward implementation approach and offers a set of macros to easily create mocks for your classes, and set up expectations. Because you need to create mocks by hand, there’s still a fair amount of typing involved to create each mock, although they provide a Python script that can generate mocks automatically from from C++ classes. It still relies on your classes inheriting from a virtual interface to hook up the mock object to your code.

This code shows the game entity factory test written with GoogleMock. Keep in mind that in addition to the test code, you still need to create the mock object through the macros provided in the framework.

TEST(CreateGameEntityCallsCreateMeshForEachMesh) 
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    EXPECT_CALL(dict, GetEntityInfo())
        .Times(1)
        .WillOnce(Return(EntityInfo(3));

    EXPECT_CALL(meshFactory, CreateMesh())
        .Times(3);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
}

MockItNow

This open-source C++ mocking framework written by Rory Driscoll takes a totally different approach from GoogleMock. Instead of requiring that all your mockable classes inherit from a virtual interface, it uses compiler support to insert some code before each call. This code can then call the mock and return to the test directly, without ever calling the real object.

From a technical point of view, it’s a very slick method of hooking up the mocks, but the main advantage of this approach is that it doesn’t force a virtual interface on classes that don’t need it. It also minimizes typing compared to GoogleMock. The only downside is that it’s very platform-specific implementation, and the version available only supports Intel x86 processors, although it can be re-implemented for PowerPC architectures.

Problems With Mocks

There is no doubt that mocks are a very useful tool. They allow us to test object interactions in our unit tests without involving lots of different classes. In particular, mock frameworks make using mocks even simpler, saving typing and reducing the time we have to spend writing tests. What’s not to like about them?

The first problem with mocks is that they can add extra unnecessary complexity to the code, just for the sake of testing. In particular, I’m referring to the need to have a virtual interface that objects are are going to be mocked inherit from. This is a requirement if you’re writing mocks by hand or using GoogleMock (not so much with MockItNow), and the result is more complicated code: You need to instantiate the correct type, but then you pass around references to the interface type in your code. It’s just ugly and I really resent that using mocks is the only reason those interfaces are there. Obviously, if you need the interface and you’re adding a mock to it afterwards, then there’s no extra complexity added.

If the complexity and ugliness argument doesn’t sway you, try this one: Every unnecessary interface is going to result in an extra indirection through a vtable with the corresponding performance hit. Do you really want to fill up your game code with interfaces just for the sake of testing? Probably not.

But in my mind, there’s another, bigger disadvantage to using mock frameworks. One of the main benefits of unit tests is that they encourages a modular design, with small, independent objects, that can be easily used individually. In other words, unit tests tend to push design away from object interactions and more towards returning values directly or modifying external data.

A mocking framework can make creating mocks so easy, to the point that it doesn’t discourage programmers from creating a mock object any time they think of one. And when you have a good mocking framework, every object looks like a good mock candidate. At that point, your code design is going to start looking more like a tangled web of objects communicating in complex ways, rather than simple functions without any dependencies. You might have saved some typing time, but at what price!

When to Use Mock Frameworks

That doesn’t mean that you shouldn’t use a mocking framework though. A good mocking framework can be a lifesaving tool. Just be very, very careful how you use it.

The case when using a mocking framework is most justified when dealing with existing code that was not written in unit testing in mind. Code that is tangled together, and impossible to use in isolation. Sometimes that’s third-party libraries, and sometimes it’s even (yes, we can admit it) code that we wrote in the past, maybe under a tight deadline, or maybe before we cared much about unit tests. In any case, trying to write unit tests that interface with code not intended to be tested can be extremely challenging. So much so, that a lot of people give up on unit tests completely because they don’t see a way of writing unit tests without a lot of extra effort. A mocking framework can really help in that situation to isolate the new code you’re writing, from the legacy code that was not intended for testing.

Another situation when using a mocking framework is a big win is to use as training wheels to get started with unit tests in your codebase. There’s no need to wait until you start a brand new project with a brand new codebase (how often does that happen anyway?). Instead, you can start testing today and using a good mock framework to help isolate your new code from the existing one. Once you get the ball rolling and write new, testable code, you’ll probably find you don’t need it as much.

Apart from that, my recommendation is to keep your favorite mocking framework ready in your toolbox, but only take it out when you absolutely need it. Otherwise, it’s a bit like using a jackhammer to put a picture nail on the wall. Just because you can do it, it doesn’t mean it’s a good idea.

Keep in mind that these recommendations are aimed at using mock objects in C and C++. If you’re using other languages, especially more dynamic or modern ones, using mock objects is even simpler and without many of the drawbacks. In a lot of other languages, such as Lua, C#, or Python, your code doesn’t have to be modified in any way to insert a mock object. In that case you’re not introducing any extra complexity or performance penalties by using mocks, and none of the earlier objections apply. The only drawback left in that case is the tendency to create complex designs that are heavily interconnected, instead of simple, standalone pieces of code. Use caution and your best judgement and you’ll make the best use of mocks.


This article was originally printed in the June 2009 issue of Game Developer.

Nitty Gritty Unit Testing

It’s one thing to see someone drive a car and have a theoretical understanding of what the pedals do and how to change gears. It’s is a completely different thing to be able to drive a car safely on the street. There are some activities that require many small details and some hands-on experience to be able to execute them successfully.

The good news is that unit testing is a lot simpler than driving a standard shift, but there are a lot of small details you need to get right to do it successfully. Even after reading about unit testing and being convinced of its benefits, programmers are often not sure how to get started. This month’s column is not going to try to convince you of the many benefits of unit testing (I hope you are already convinced), but rather, describe some very concrete tips to help you get started right away.

Goals of Unit Testing

There are many different reasons to write unit tests:

  • Correctness testing. Checking that the code behaves as designed.
  • Boundary testing. Checking that the code behaves correctly in odd or boundary situations.
  • Regression testing. Checking that the behavior of the code doesn’t change unintentionally over time.
  • Performance testing. Checking that the program meets certain minimum performance or memory constraints.
  • Platform testing. Checking that the code behaves the same across multiple platforms.
  • Design. Tests provide a way to advance the code design and architecture. This is usually referred as Test-Driven Development (TDD).
  • Full game or tool testing. Technically this is a functional test, not a unit test anymore because it involves the whole program instead of a small subset of the code, but a lot of the same techniques apply.

Some developers use unit tests only for one of the reasons listed above, while others use many kinds of tests for a variety of different reasons. It’s important to recognize that because there are so many different uses for unit tests, no single solution is going to fit everybody. The ideal setup for some of those situations is going to be slightly different than for others. The basics are the same for all of them though.

When working with unit tests, these are our main goals:

  • Spend as little time as possible writing a new test.
  • Be notified of failing tests, and see at a glance which ones failed and why.
  • Trust our tests. Have them be consistent from run to run and robust in the face of bad code.

Testing Framework

lack_of_tests.jpgMost of us have created one-off programs in the past to test some particularly complicated code. It’s usually a quick command line program that runs through a bunch of cases and asserts after each one that the results were correct. That’s the most bare-bones way of creating unit tests.

Unfortunately, it’s also a pain and it misses on most of the unit-testing goals described in the previous section: creating a new program just for that is a pain, we have to go out of our way to run the tests, and it usually gets out of date faster than the latest Internet meme. That is in part why a lot of programmers have an initial aversion to writing unit tests.

If you’re considering writing even a small unit test, you should use a unit-testing framework. A unit-testing framework removes all the busy work from writing unit tests and lets you spend your time on the logic of what to test. This doesn’t mean that the framework writes the tests for you. Be very wary of any tool that claims to do that! No, a unit-testing framework is simply a small library that provides all the glue for running unit tests and reporting the results. Sorry, you still need to use your brain and do (some of) the typing.

A quick search will reveal plenty of unit-testing frameworks to choose from for your language of choice, and most of them are free and open source so you can rely on them and modify them to suit your needs. For C/C++ and game development, I strongly recommend starting with UnitTest++. Charles Nicholson and I wrote that framework a few years ago specifically with games and consoles in mind. Many game teams have adopted it for their games and tools, and it has been used on lots of different game platforms including current and last generation consoles, Windows, Linux, and Mac PCs, and even on the iPhone. In most situations, it should be a straight drop-in to your project and you’re up and running.

If you end up using a different testing framework, or even if you roll your own, the techniques described here still apply, even if the syntax is slightly different.

Hello Tests

Writing your first test is easy as pie:

#include <UnitTest++.h>

TEST(MyFirstTest)
{
    int a = 4;
    CHECK(a == 4);
}

To run it you need to add the following line to your executable somewhere. We’ll talk more about the physical organization of tests in a moment.

int failedCount = UnitTest::RunAllTests();

Done! Easy, wasn’t it? When you compile and run the program you should see the following output:

Success: 1 test passed.
Test time: 0.00 seconds.

Let’s add a failing test:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK(a == 4);
}

Now we get:

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: a == 4
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

That’s great, but if we’re going to diagnose the problem, we probably need to know the value of the variable a, and all the test is telling us is that it’s not 4. So instead, we can change the CHECK statement to the following:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK_EQUAL(4, a);
}

And now the output will be

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: Expected 4 but was 5
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

Much better. Now we get both the error information and the value of the variable under test. Virtually all unit-testing frameworks include different types of CHECK statements to get more information when testing floats, arrays, or other data types. You can even make your own CHECK statement for your own common data types such as colors or lists.

As a bonus, if you’re using an IDE, double-clicking on the test failure message should bring you automatically to the failing test statement.

When To Run

When to run unit tests will depend on what is being tested and how long it takes to do so. In general, the more frequently you run the tests, the better. The sooner you get feedback that something went wrong, the easier it will be to fix. Maybe even before it was checked in and it spread to the rest of the team. On the flip side, realistically, building and running a set of unit tests takes a certain amount of time, so it’s important to find the right balance between feedback frequency and time spent waiting for tests.

At the very least, all tests should run once a day, during the nightly build process in your build server (You have a build server, don’t you? If not, run over and read this column in the August 2008 issue). It doesn’t matter how long they take or how how many different projects you need to run. Just add them to the build script, and hook their output into the build results.

On the other extreme, you can build your tests every time you build the project and execute them as a postbuild step. That way, any time you make a change to a project, all tests will execute and you’ll see if anything went wrong. This is a great approach, but I wouldn’t recommend it if tests add more than a couple of seconds to the incremental build time, otherwise, they’ll be slowing you down more than they help.

For most developers, some approach in the middle will make most sense. For example, take a small, fast subset of tests that are more likely to break, and run those with every build. Whenever any code is checked in to version control, the build server can run those tests plus a few more, slower ones. And finally, at night, you can bring out the big guns and run those really long, really thorough ones that take a few hours to complete. You can separate those tests into different projects, or, if your framework supports them, into different test suites, which allow you to decide which sets of tests to execute at runtime.
Reporting Results

If a unit test fails and nobody notices, is it really an error? Just running the tests isn’t good enough. We have to make sure that someone sees the failure it and fixes the problem.

Most unit-testing frameworks will let you customize how you want the failure errors to be reported. By default they will probably be sent to stdout, but you can easily customize the framework to send them to debug log streams, save to a file, or upload them to a server.

Even more important than the actual error messages is detecting whether there were any failures. After running all the tests, there is usually some way to detect how many tests failed. The program that was running the tests can detect any failures, print an error message, and exit with an error code. That error code will propagate to the build server and trigger a build failure. Hopefully by now alarm sounds are going off across the office and someone is on his way to fix it.

Project Organization

When people start down the unit test path, they often struggle to figure out how to physically lay out the unit tests. In the end, it really doesn’t matter too much as long as it makes sense to you, the final build doesn’t contain any tests, and they’re still easy to build and run.

My personal preference is to keep unit tests separate from the rest of the code. Usually I end up creating one file of tests for every cpp file. So FirstPersonCameraController.h and .cpp have a corresponding TestFirstPersonCameraController.cpp. Since I use this convention regularly throughout all of my code, I have a custom IDE macro to toggle between a file and its corresponding test file. I also put all the tests in a separate subdirectory to keep them as physically separate as possible.

I prefer to break up my code into several static libraries for each major subsystem: graphics, networking, physics, animations, etc. Each of those libraries has a set of unit tests, but instead of compiling them into the library, I create a separate project that creates a simple executable program. That project contains all the unit tests and links against the library itself, and in its main entry point it just calls the function to runs all unit tests and returns the number of failures. This keeps the tests separate from the library, but still very easy to build.

If all your code is organized into libraries, and your game is just a collection of libraries linked together, that’s all you need. Most games and tools, however, have a fair amount of code that you might want to test in the project itself. Since the game is an executable, you can’t easily link against it from a different project like we did before. In this case, I build the unit tests into the game itself, and I optionally call them whenever a particular command-line parameter like -runtests is present. Just make sure to #ifdef out all the tests in the final build.

Multiplatform Testing

Running the tests on the PC where you build the code is very straightforward. But unless you’re only creating games and tools for that platform, you will definitely want to run your tests on different platforms as well. Unit tests are an invaluable tool for catching slight platform inconsistencies caused by different compilers, architecture idiosyncrasies, or varying floating point rules.

Unfortunately, running unit tests on a different platform from your build machine is usually a bit more involved and not nearly as fast as doing it locally. You need to start by compiling the tests for the target platform. This is usually not a problem since you’re already building all your code for that platform, and hopefully your unit testing framework already supports it. Then you need to upload your executable with the tests and any data required to the target platform and run it there. Finally, you need to get the return code back to detect if there were any failures. This is surprisingly the trickiest part of the process with a lot of console development kits. If getting the exit code is not a possibility, you’ll need to get creative by parsing the output channel, or even waiting for a notification on a particular network port.

Some target platforms are more limited than others in both resources and C++ support. One of the features that makes UnitTest++ a good choice for games is that it requires minimal C++ features (no STL) and it can be trimmed down even further (no exceptions or streams).

For example, running unit tests on the PS3 SPUs was extremely useful, but it required stripping the framework down to the minimum amount of features. It was also tricky being able to fit the library code plus all the tests in the small amount of memory available. To get around that, we ended up changing the build rules for the SPUs so each test file created its own SPU executable (or module). We then wrote a simple main SPU program that would load each module separately, run its tests, keep track of all the stats, and finally report them.

Running a set of unit tests on the local machine can be an almost instant process, but running them on a remote machine is usually much slower, and can take up to 10 or 20 seconds just with the overhead of copying them and launching the program remotely. For this reason, you’ll want to run tests on other platforms less frequently.

No Leaking Allowed

Finally, if you’re going to have this all this unit testing code running on a regular basis, you might as well get as much information out of it as possible. I have found it invaluable to keep track of memory leaks around the unit test code.

You’ll have to hook into your own memory manager, or use the platform-specific memory tracking functions. The basic idea is to get a memory status before running the tests, and another one after all the tests execute. If there are any extra memory allocations, that’s probably a leak. In that case, you can report it as a failed build by returning the correct error code.

Watch out for static variables or singletons that allocate memory the first time they’re used. They might be reported as memory leaks even though it wasn’t what you were hoping to catch. In that case, you can explicitly initialize and destroy all singletons, or, even better, not use them at all, and keep your memory leak report clean.


You’re now armed with all you need to know to set up unit tests into your project and build pipeline. Grab a testing framework and get started today.


This article was originally printed in the May 2009 issue of Game Developer.

When Is It OK Not To TDD?

The basic principles of Test-Driven Development (TDD) are very simple and easy to understand. Every programmer quickly grasps those and is able to apply them to simple cases and low level libraries (math libraries seem to be everybody’s favorite TDD proving ground [1]).

What becomes significantly more difficult is learning to effectively apply TDD to code with more dependencies. A question that I’m often asked from people trying to use TDD for the first time is “How can you possibly use TDD for high-level game code? It’s impossible!”.

When The Going Gets Tough

At that point, the temptation to give up on using TDD for high-level code becomes very strong. It seems that all the time spent writing mocks and hooking objects together is a waste of time and that you could be writing game code much faster without it. “Maybe it would be best to skip TDD, just for that part”.

Bad idea!

tdd_cycle.jpgBy giving up on TDD for high-level code, you’ll be missing out on the main benefit of TDD: designing better code. I’ve said this many times, but it bears repeating: TDD is not a testing technique. It’s a design technique. You get lots of benefits from applying TDD [2], but one of the main one is much more modular and dependency-free code.

That’s because TDD turns our weaknesses into advantages. We’re humans and we’re lazy. We don’t want to repeat ourselves constantly or write complex code. If we’re writing a test for some code we’ll write in the future, we’re going to create a very simple test. We’re probably going to want to put some object or data on the stack and make a function call and nothing else. We certainly don’t want to initialize the graphics system, create a new world, add some level data, start the physics simulation, and then call the function! So by being lazy, we end up writing very simple code with the least amount of dependencies.

What does that mean for our high-level code? The AI logic seems to need access to every single game system, at least the way we implemented it in the past. Using TDD to implement AI logic doesn’t mean writing tests to write the same code you would have written before. It means writing tests and then writing some code that makes those tests pass and nothing else. For example, we might realize that the AI doesn’t need to access the physics system, cast a ray in the world, find out what entities is interesting, and return that information. Instead, all it might need to do is output some data saying that it wants a ray query done at a later time (which in turn is the key to batching those ray casts and achieving high performance).

You soon realize that the code you write is very different (and better!) to what you would have written before. Once you get over that hump, TDDing high-level code is not about twisting your code with mocks, but becomes simpler code that takes simple input data and creates simple output data. By pushing through and forcing yourself to apply TDD, you made a breakthrough and reaped all the benefits.

That’s the main reason why TDD books push the idea that no code can be written without a test first. Otherwise, inexperienced TDD programmers would be too quick to quit and would miss out on the technique completely.

Not only that, but code that is not created with TDD tends to have the bad habit of spreading. When code was not designed through TDD, it means that other code that uses it will probably be difficult to TDD itself. So it’s a bit like const-correctness: You’re either in our out, and there’s very little room for half measures.

On the other hand, I would argue that using TDD on a math library is a bad idea. It’s essential to write good unit tests for a math library, but probably not to design it through TDD. Are you really going to implement a cross product differently just because you wrote tests before? The emphasis there has to be on correctness and performance, not on creating the interface or implementation through tests.

It’s A Tool To Help Development

A few days ago, I received an email from Caleb Gingles with a great question:

I’ve started a new game project and have been making an effort to strictly follow TDD from the beginning. Which has been a challenge, as much of the coding at the beginning of the project has consisted of operating system requirements and framework stuff. [...] However, there are other situations that seem much harder to test. Like the main loop in a game. According to Kent Beck’s book, everything begins with testing, and no code is written unless a test requires it. But how can a test require the main loop framework in a game? The loop just… is. It exists to allow you to do certain other things, it doesn’t have much purpose in and of itself.

Applying what I just talked about, we could use TDD to create the main loop of a game. It definitely requires thinking about it differently and breaking preconceptions. It’s about writing code to pass the tests, not writing tests to fit the code we want to write.

So we could do something like this:

TEST(MainLoopExecutesCorrectNumberOfTimes)
{
    MainLoopState loopState;
    MainLoopUtils::Execute(loopState, 4);
    CHECK_EQUAL(4, loopState.frameCount);
}

All of a sudden we see a way to test something that before it seemed impossible to test. We can add tests to make sure certain functions are getting called (maybe we add those function pointers to the loopState), that the main loop exits under certain conditions, etc.

For some games with complex GUIs and different game modes, it might make a lot of sense to have a more flexible main loop and develop it completely through TDD. On the other hand, for a simple iPhone game that has a single main loop that never changes, there might be no point at all in using TDD. The loop code is going to end up looking more or less the same as it would without TDD, just more complex and generic. TDD didn’t help any in that case.

How about code that calls into OpenGL, or some other non-TDD-friendly library? Do we have to TDD that? Frankly, I wouldn’t bother. I did give it a good try at one point several years ago and I can say it is possible, but it’s a pain and you gain almost nothing from it. The main benefits is trusting that the library you’re using really does what it’s supposed to do, but it won’t affect much the design of your code. So what you’re really doing is adding tests to that library (which might or might not be of value).

My advice in a situation like that: Call the library from as few places as possible (I didn’t want to say make a thin wrapper because that implies isolation which is not the goal here), and just test that your code is called at the right time with the right data. The actual code that calls the library should be small and simple enough that you should feel OK not having any tests for it.


Even though it seems to go against what I said in the previous section, TDD is not all or nothing. It’s not a religion.

When you’re starting out, I encourage you to push yourself to think how you could apply TDD to situations you don’t think are possible. Once you’re experienced enough, you’ll know when to say enough is enough, and use TDD only when it actually benefits you. At that point you’ll also know how to write code in a way that plays well with TDD, even if the code itself was not developed through TDD.

In the end, TDD is a tool to help you develop better and faster. Don’t ever let it get in the way of that.


[1] That’s actually a horrible place to apply TDD. More on that in a second.

[2] Usable API, simple code, unit tests, documentation, safety net for refactoring among others.

The Always-Evolving Coding Style

This is my first entry into #iDevBlogADay. It all started very innocently with a suggestion from Miguel, but the ball got rolling pretty quickly. The idea is to have one independent iPhone game developer write a blog entry each day of the week. At first we thought we would be hard-pressed to get 7 developers, but it’s starting to seem we might have multiples per day!

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

 

matrix.jpgEvery developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

  • All my code is written in C and C++ (except for a bit of ObjC and assembly).
  • It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
  • I always try to write important code through Test-Driven Development.
  • I’m the only programmer (and only designer).
  • Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

  • I’m showing all the data there is and how big it is. Nothing is hidden.
  • I’m making it clear that it’s free of pointers and temporary variables.
  • I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

 

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

What’s Your Pain Threshold?

Mine is two seconds.

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there :-) ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions. Continue reading