When Is It OK Not To TDD? | Games from Within

The basic principles of Test-Driven Development (TDD) are very simple and easy to understand. Every programmer quickly grasps those and is able to apply them to simple cases and low level libraries (math libraries seem to be everybody’s favorite TDD proving ground [1]).

What becomes significantly more difficult is learning to effectively apply TDD to code with more dependencies. A question that I’m often asked from people trying to use TDD for the first time is “How can you possibly use TDD for high-level game code? It’s impossible!”.

When The Going Gets Tough

At that point, the temptation to give up on using TDD for high-level code becomes very strong. It seems that all the time spent writing mocks and hooking objects together is a waste of time and that you could be writing game code much faster without it. “Maybe it would be best to skip TDD, just for that part”.

Bad idea!

By giving up on TDD for high-level code, you’ll be missing out on the main benefit of TDD: designing better code. I’ve said this many times, but it bears repeating: TDD is not a testing technique. It’s a design technique. You get lots of benefits from applying TDD [2], but one of the main one is much more modular and dependency-free code.

That’s because TDD turns our weaknesses into advantages. We’re humans and we’re lazy. We don’t want to repeat ourselves constantly or write complex code. If we’re writing a test for some code we’ll write in the future, we’re going to create a very simple test. We’re probably going to want to put some object or data on the stack and make a function call and nothing else. We certainly don’t want to initialize the graphics system, create a new world, add some level data, start the physics simulation, and then call the function! So by being lazy, we end up writing very simple code with the least amount of dependencies.

What does that mean for our high-level code? The AI logic seems to need access to every single game system, at least the way we implemented it in the past. Using TDD to implement AI logic doesn’t mean writing tests to write the same code you would have written before. It means writing tests and then writing some code that makes those tests pass and nothing else. For example, we might realize that the AI doesn’t need to access the physics system, cast a ray in the world, find out what entities is interesting, and return that information. Instead, all it might need to do is output some data saying that it wants a ray query done at a later time (which in turn is the key to batching those ray casts and achieving high performance).

You soon realize that the code you write is very different (and better!) to what you would have written before. Once you get over that hump, TDDing high-level code is not about twisting your code with mocks, but becomes simpler code that takes simple input data and creates simple output data. By pushing through and forcing yourself to apply TDD, you made a breakthrough and reaped all the benefits.

That’s the main reason why TDD books push the idea that no code can be written without a test first. Otherwise, inexperienced TDD programmers would be too quick to quit and would miss out on the technique completely.

Not only that, but code that is not created with TDD tends to have the bad habit of spreading. When code was not designed through TDD, it means that other code that uses it will probably be difficult to TDD itself. So it’s a bit like const-correctness: You’re either in our out, and there’s very little room for half measures.

On the other hand, I would argue that using TDD on a math library is a bad idea. It’s essential to write good unit tests for a math library, but probably not to design it through TDD. Are you really going to implement a cross product differently just because you wrote tests before? The emphasis there has to be on correctness and performance, not on creating the interface or implementation through tests.

It’s A Tool To Help Development

A few days ago, I received an email from Caleb Gingles with a great question:

I’ve started a new game project and have been making an effort to strictly follow TDD from the beginning. Which has been a challenge, as much of the coding at the beginning of the project has consisted of operating system requirements and framework stuff. [...] However, there are other situations that seem much harder to test. Like the main loop in a game. According to Kent Beck’s book, everything begins with testing, and no code is written unless a test requires it. But how can a test require the main loop framework in a game? The loop just… is. It exists to allow you to do certain other things, it doesn’t have much purpose in and of itself.

Applying what I just talked about, we could use TDD to create the main loop of a game. It definitely requires thinking about it differently and breaking preconceptions. It’s about writing code to pass the tests, not writing tests to fit the code we want to write.

So we could do something like this:

TEST(MainLoopExecutesCorrectNumberOfTimes)
{
    MainLoopState loopState;
    MainLoopUtils::Execute(loopState, 4);
    CHECK_EQUAL(4, loopState.frameCount);
}

All of a sudden we see a way to test something that before it seemed impossible to test. We can add tests to make sure certain functions are getting called (maybe we add those function pointers to the loopState), that the main loop exits under certain conditions, etc.

For some games with complex GUIs and different game modes, it might make a lot of sense to have a more flexible main loop and develop it completely through TDD. On the other hand, for a simple iPhone game that has a single main loop that never changes, there might be no point at all in using TDD. The loop code is going to end up looking more or less the same as it would without TDD, just more complex and generic. TDD didn’t help any in that case.

How about code that calls into OpenGL, or some other non-TDD-friendly library? Do we have to TDD that? Frankly, I wouldn’t bother. I did give it a good try at one point several years ago and I can say it is possible, but it’s a pain and you gain almost nothing from it. The main benefits is trusting that the library you’re using really does what it’s supposed to do, but it won’t affect much the design of your code. So what you’re really doing is adding tests to that library (which might or might not be of value).

My advice in a situation like that: Call the library from as few places as possible (I didn’t want to say make a thin wrapper because that implies isolation which is not the goal here), and just test that your code is called at the right time with the right data. The actual code that calls the library should be small and simple enough that you should feel OK not having any tests for it.

Even though it seems to go against what I said in the previous section, TDD is not all or nothing. It’s not a religion.

When you’re starting out, I encourage you to push yourself to think how you could apply TDD to situations you don’t think are possible. Once you’re experienced enough, you’ll know when to say enough is enough, and use TDD only when it actually benefits you. At that point you’ll also know how to write code in a way that plays well with TDD, even if the code itself was not developed through TDD.

In the end, TDD is a tool to help you develop better and faster. Don’t ever let it get in the way of that.

[1] That’s actually a horrible place to apply TDD. More on that in a second.

[2] Usable API, simple code, unit tests, documentation, safety net for refactoring among others.

http://www.appsizematters.com Bob Koon

Another great post!

What I like to do is start with a tracer bullet to try to get close to a general end-to-end flow (or get as close to it as possible) and then TDD the main modules. As you say, using TDD for the whole shebang might not be worth the effort.

@Bob_at_BH
Steve Breen

Solid post. Tools come in all shapes and sizes.
Knowing when and where to swing foo at bar is what makes good devs good.
http://www.sunetos.com Doug Sjoquist

The fact that TDD changes how I approach my code really resonates with me. I love how much cleaner and maintainable my code is when I approach a project like this. There is a very clear difference between my projects that follow this pattern and those that don’t.

Also, I’m starting a fresh project for my game and have taken a data-centric, TDD approach with it. I am only using the multiple prototypes developed over the last few months as a source of ideas, but not a source of code — I’m already feeling much better about what I’ve accomplished the last week than I have the last two months. (I’m using GHUnit for the tests, seems to be working pretty well so far.)

Thanks for the articles, reprints and new!

Doug
http://tilander.org/aurora Jim Tilander

Noel,

It’s funny, I’ve completely shifted away from TDD (I still sometimes write unit tests, but that’s something else entirely as you said). Instead I try to write as simple, non dependent code as possible. You touched it a little, having explicit dependencies makes the code hard to TDD, I think this is the only real virtue of TDD. I feel though that you can do this without bringing in “TD” in the “D” it’s just slightly harder and requires more discipline — the TDD tend to keep you honest and not do stupid things like:

http://xkcd.com/292/

If you do this without the “TD” then you can also make the external interface to the user *as simple as possible*, not exposing anything extra. I think this is the thing I dislike about TDD, it keeps me having to expose internals to the user, instead of writing closed modules — I find myself writing more and more stuff like this:

namespace foobar {

void init(); // module startup (do once)
void term(); // module quit

void update(uint64_t ticks); // process data
void notify(uint32_t message); // enqueue data

}

Which internally has more components and simple stuff, but externally is super simple to use (comments for illustration only).

I think you can either use TDD, or have pair programming (and your buddy keeps you honest) or just be a super obsessive compulsive programmer

/j
http://www.gamesfromwithin.com Noel

Jim, You’re right that you don’t need TDD to write code in the TDD style. I really think you need to have done TDD (or have learned in an evironment that encourages that kind of architecture) to be able to do it naturally without the unit tests. And of course, you’re missing out on all the other benefits of TDD if you skip the tests.

One of my next posts is going to be “Confessions of a TDD-holic” and how I have been lass than perfect about doing TDD when I should (!!) ;-b
http://tilander.org/aurora Jim Tilander

I think you sufficiently hammered home TDD to me while we were coding I just think it’s a tool to better code, just like pair programming is, but you can get to the goal of better code many different way and you don’t always need to use the car you used on the way once you’ve arrived…
http://defragdev.com/blog/ Mark

For me, the main thing about TDD is that it forces me to immediately program against my code’s interface/API/whatever. This nearly always results in an improvement of some sort, be it a less clunky API, quickly eliminating would-be bugs or — best of all — getting rid of surprises. TDD is the principle of least surprise’s best friend, because they go hand in hand.

If I ‘think’ like I’m TDDing but don’t actually write the tests, I usually have loosely coupled classes and similar looking code, but it’s not quite the same quality really and I’ll usually run into a few more problems down the road. Plus I like having loads of regression tests to lean on in future.
Caleb Gingles

Noel,

After reading this over multiple times, I feel like I almost get it, but I think I’m still missing something.

For example, the OpenGL case. You don’t test OpenGL, you test your use of it. So let’s say you wrap up your OpenGL calls into a set of functions that perform specific tasks. Let’s say you have a function to create and upload a vertex buffer. And, as you say, it is a simple function that does nothing more than interface with an external library, so you just write it, without writing tests for it beforehand.

But now you’re ready to use this function to create the vertex buffer. This is “real” code now, so you have to write a test first. But what would be the test? All you’re going to do is call the function, pass data to it, and let the external library do it’s thing. Do you just try to mock it and record that it got called? If so, the test isn’t really guaranteeing that you passed valid data, or that the function did anything at all for that matter. All I know is I called the function. Is that useful?

It doesn’t feel like I’m testing something meaningful if the tests are so finely-grained that I’m essentially mirroring each function call with a test that confirms that I made the function call. What am I missing here?
http://www.gamesfromwithin.com Noel

Hi Caleb,

Yes, you test that the function was called. You have the right idea that the funciton shouldn’t be a one-to-one wrapper of OpenGL, but something that gives you a bit more functionality (sends a mesh and some graphics states to be rendered for example).

If now you need to write the function that renders a character in game, you can test that the correct vertex buffer is being sent, that it has all the skinning information, that the correct states are being set, the correct bone matrices are being loaded, etc. All very useful stuff (if it’s not useful, then you don’t need to write that code

Sometimes I have my render functions return a buffer with commands and data. Testing the rendering functions is very easy because I just have to look at that output buffer. Then, I can write a function (with or without tests) that converts those commands into OpenGL calls (or real push buffer commands or whatever).

If you really want to, you can also involve OpenGL in the unit tests. For example, if you want to TDD the function that converts commands into OpenGL calls, you can usually query OpenGL back for some of the state. It’s not ideal (it requires OpenGL to run all unit tests now), but it’s an option and it can come in handy sometimes.