Last year I wrote about the basics of Data-Oriented Design (see the September 2009 issue of Game Developer). In the time since that article, Data-Oriented Design has gained a lot of traction in game development and many of teams are thinking in terms of data for some of the more performance-critical systems.

As a quick recap, the main goal of Data-Oriented Design is achieving high-performance on modern hardware platforms. Specifically, that means making good use of memory accesses, multiple cores, and removing any unnecessary code. A side effect of Data-Oriented Design is that code becomes more modular and easier to test.

Data-Oriented Design concentrates on the input data available and the output data that needs to be generated. Code is not something to focus on (like traditional Computer Science), but is something that is written to transform the input data into the output data in an efficient way. In modern hardware, that often means applying the same code to large, contiguous blocks of homogeneous memory.

Applying Data-Oriented Design

It’s pretty easy to apply these ideas to a self-contained system that already works over mostly-homogeneous data. Most particle systems in games are probably designed that way because one of their main goals is to be very efficient and handle a large number of particles at high framerates. Sound processing is another system that is naturally implemented thinking about data first and foremost.

So, what’s stopping us from applying it to all the performance-sensitive systems in a game code base? Mostly just the way we think about the code. We need to be ready to really look at the data and be willing to split up the code into different phases. Let’s take a high-level example and see how the code would have to be restructured when optimizing for data access.

Listing 1 shows pseudocode for what could be a typical update function for a generic game AI. To make things worse, that function might even be virtual and different types of entities might implement it in different ways. Let’s ignore that for now and concentrate on what it does. In particular, the pseudocode highlights that, as part of the entity update, it does many conditional ray casting queries, and also updates some state based on the results of those queries. In other words, we’re confronted with the typical tree-traversal code structure so common in Object-Oriented Programming.

void AIEntity::Update(float dt)
{
    DoSomeProcessing();
    if (someCondition && Raycast(world))
       DoSomething();
    if (someOtherCondition && BunchOfRayCasts(world))
       DoSomethingElse();
    UpdateSomeOtherStuff();
}

Listing 1

Ray casts against the world are a very common operation for game entities. That’s how they “see” what’s around them and that’s what allows them to react correctly to their surroundings. Unfortunately, ray casting is a very heavy weight operation, and it involves potentially accessing many different areas in memory: a spatial data structure, other entity representations, polygons in a collision mesh, etc.

Additionally, the entity update function would be very hard to paralellize on multiple cores. It’s unclear how much data is read or written in that function, and some of that data (like the world data structure) might be particularly hard and expensive to protect from updates from multiple threads.

If we re-organize things a bit, we can significantly improve performance and paralellization.

Break Up And Batch

Without seeing any of the details of what’s going on inside the entity update, we can see the raycasts sticking out like a sore thumb in the middle. A raycast operation is fairly independent of anything else related to the entity, it’s heavyweight, and there could be many of them, so it’s a perfect candidate to break up into a separate step.

Listing 2 shows how the broken up entity update code would look like. The update is now split in two different passes: The first pass does some of the updating that can be done independently of any ray casts, and decides which, if any, raycasts need to be performed sometime this frame.

void AIEntity::InitialUpdate(float dt, RayCastQueries& queries)
{
    DoSomeProcessing();
    if (someCondition)
        AddRayCastQuery(queries);
    if (someOtherCondition)
        AddBunchOfRayCasts(queries);
}

void AIEntity::FinalUpdate(const RayCastResults& results)
{
    UpdateSomeOtherStuff(results);
}

Listing 2

The game code in charge of updating the game processes all AI entities in batches (Listing 3). So instead of calling InitialUpdate(), solve ray casts, and FinalUpdate() for each entity, it iterates over all the AI entities calling InitialUpdate() and adds all the raycast query requests to the output data. Once it has collected all the raycast queries, it can process them all at once and store their results. Finally, it does one last pass and calls FinalUpdate() with the raycast results on each entity.

RayCastQueries queries;
for (int i=0; i<entityCount; ++i)
    entities[i].InitialUpdate(dt, queries);

// Other update that might need raycasts

RayCastResults results;
for (int i=0; i<queries.count; ++i)
    PerformRayCast(queries[i], results);

for (int i=0; i<entityCount; ++i)
    entities[i].FinalUpdate(results);

Listing 3

By removing the raycast calls from within the entity update function, we’ve shortened the call tree significantly. The functions are more self-contained, easier to understand, and probably much more efficient because of better cache utilization. You can also see how it would be a lot easier to parallelize things now by sending all raycasts to one core while another core is busy updating something unrelated (or maybe by spreading all raycasts across multiple cores, depending on your level of granularity).

Note that after calling InitialUpdate() on all entities, we could do some processing on other game objects that might also need raycast queries and collect them all. That way, we can batch all the raycasts at compute them all at once. For years, we’ve been drilled by graphics hardware manufacturers how we should batch our render calls and avoid drawing individual polygons. This is the same way: By batching all raycasts in a single call, we have the potential to achieve much higher performance.

Splitting Things Up

Have we really gained much by reorganizing the code this way? We’re doing two full passes over the AI entities, so wouldn’t that be worse from a memory point of view? Ultimately you need to measure it and compare the two. In modern hardware platforms, I would expect performance to be better because, even though we’re traversing through the entities twice, we’re using the code cache much better and we’re accessing them sequentially (which allows us to pre-fetch the next one too).

If this is the only change we make to the entity update, and the rest of the code is the usual deep tree traversal code, we might not have gained much because we’re still blowing the cache limits with every update. We might need to apply the same design principles to the rest of the update function to start seeing performance improvements. But at the very least, even with this small change, we have made it easier to parallelize.

One thing we’ve gained now is the ability to modify our data to fit the way we’re using it, and that’s the key to big performance gains. For example, after seeing how the entity is updated in two separate passes, you might notice that only some of the data that was stored in the entity object is touched from the first update, and the second pass accesses more specific data.

At that point we can split up the entity class into two different sets of data. One of the most difficult things at this point is naming these sets data in some meaningful way. They’re not representing real objects or real-world concepts anymore, but different aspects of a concept, broken down purely by how the data is processed. So what before was an AIEntity, can now become a EntityInfo (containing things like position, orientation, and some high-level data) and AIState (with the current goals, orders, paths to follow, enemies targeted, etc).

The overall update function now deals with EntityInfo structures in the first pass, and AIState structures in the second pass, making it much more cache friendly and efficient.

Realistically, both the first and second passes will have to access some common data (for example the entity current state: fleeing, engaged, exploring, idle, etc). If it’s only a small amount of data, the best solution might be to simply duplicate that data on both structures (going against all “common wisdom” in Computer Science). If the common data is larger or is read-write, it might make more sense to give it separate data structure of its own.

At this point, a different kind of complexity is introduced: Keeping track of all the relationships from the different structures. This can be particularly challenging while debugging because some of the data belonging to the same logical entity isn’t stored in the same structure and it’s harder to explore in a debugger. Even so, making good use of indices and handles makes this problem much more manageable (see Managing Data Relationships in the September 2008 issue of Game Developer).

Conditional Execution

So far things are pretty simple because we’re assuming that every AI entity needs both updates and some ray casts. That’s not very realistic because entities are probably very bursty: sometimes they need a lot of ray casts, and sometimes they’re idle or following orders and don’t need any for a while. We can deal with this situation by adding a conditional execution to the second update function.

The easiest way to conditionally execute the update would be to add an extra output parameter to the FirstUpdate() function indicating whether the entity needs a second update or not. The same information could be derived form the calling code depending on whether there were any raycasts queries added. Then, in the second pass, we only update those entities that appear in the list of entities requiring a second update.

The biggest drawback of this approach is that the second update went from traversing memory linearly to skipping over entities, potentially affecting cache performance. So what we thought was going to be a performance optimization ended up making things slower. Unless we’re gaining a significant performance improvement, it’s often better to simply do the work for all entities whether they need it or not. However, if on average less than 10 or 20 percent of the entities need a ray cast, then it might be worth avoiding doing the second update on all the other entities and paying the conditional execution penalty.

If the number of entities to be updated in the second pass is fairly small, another approach would be to copy all necessary data from the first pass into a new temporary buffer. The second pass can then process that data sequentially without any performance penalties and it would completely offset the performance hit of copying the data.

Finally, another alternative, especially if the conditional execution remains fairly similar from frame to frame, is to relocate entities that need raycasting together. That way the copying is minimal (swapping an entity to a new location in the array whenever it needs a raycast), and we still get the benefit of the sequential second update. For this to work all your entities need to be fully relocatable, which means working with handles or some other indirection, or updating all the references to the entities that swapped places.

Different Modes

What if the entity can be in several, totally different modes of execution? Even if it’s the same type of entity, traversing through them linearly calling the update function could end up using completely different code for each of them, so it will result in poor code cache performance.

There are several approached we can take in a situation like that:

• If the different execution modes also are tied to different parts of the entity data, we could treat them as if they were completely different entities and break each of their data components apart. That way, we can iterate through each type separately and get all the performance benefits.
• If the data is mostly the same, and it’s just the code that changes, we could keep all the entities in the same memory block, but rearrange them so that entities in the same mode are next to each other. Again, if you can relocate your data, this is very easy and efficient (it only requires swapping a few entities whenever the state changes).
• Leave it alone! Ultimately, Data-Oriented Design is about thinking about the data and how it affects your program. It doesn’t mean you always have to optimize every aspect of it, especially if the gains aren’t significant enough to warrant the added complexity.

The Future

Is thinking about a program in terms of data and doing these kind of optimizations a good use of our time? Is this all going to go away in the near future as hardware improves? As far as we can tell right now, the answer is a definite no. Efficient memory access with a single CPU is a very complicated problem, and matters get much worse as we add more cores. Also, the amount of transistors in CPUs (which is a rough measure of power) continues to increase much faster than memory access time. That tells us that, barring new technological breakthroughs, we’re going to be dealing with this problem for a long time. This is a problem we need to deal with right now and build our technology around it.

There are some things I’d like to see in the future to make Data-Oriented Design easier. We can all dream up of a new language that will magically allow for great memory access and easy paralellization, but replacing C/C++ and all existing libraries is always going to be a really hard sell. Historically, the best advances in game technology have been incremental, not throwing away existing languages, tools, and libraries (that’s why we’re still stuck with C++ today).

Here are two things that could be done right now and work with our existing codebases. I know a lot of developers are working on similar systems in their projects, but it would be great to have a common implementation released publicly so we can all build on top of them.

Language

Even though a functional language might be ideal, either created from scratch or reusing an existing one, we could temporarily extend C to fit our needs. I would like to see a set of C extensions where functions have clearly defined inputs and outputs, and code inside a function is not allowed to access any global state or call any code outside that function (other than local helper functions defined in the same scope). This could be done as a preprocessor or a modified C compiler, so it remains very compatible with existing libraries and code.

void FirstEntityUpdate(input Entities* entities, input int entityCount, output RayCastQueries* queries, output int queryCount);

Dependencies between functions would be expressed by tying the outputs of some functions to the input of other functions. This could be done in code or through the use of GUI tools that help developers manage data relationships visually. That way we can construct a dependency diagram of all the functions involved in every frame.

Scheduler

Once we have the dependencies for every function, we can create a directed acyclic graph (DAG) from it, which would give us a global view of how data is processed every frame. At that point, instead of running functions manually, we can leave that job in the hands of a scheduler.

The scheduler has full information about all the functions as well as the number of available cores (and information from the previous frame execution if we want to use that as well). It can determine the critical path through the DAG and optimize the scheduling of the tasks so the critical path is always being worked on. If temporary memory buffers are a limitation for our platform, the scheduler can take that into account and trade some performance time for a reduced memory footprint.

Just like the language, the scheduler would be a very generic component, and could be made public. Developers could use it as a starting point, build on top of it, and add their own rules for their specific games and platforms.

Even if we’re not ready to create those reusable components, every developer involved in creating high-performance games should be thinking about data in their games right now. Data is only going to get more important in the future as the next generation of consoles and computers rolls in.

This article was originally printed in the September 2010 issue of Game Developer.

We’ve all had things nagging us in the back of our minds. They’re nothing we have to worry about this very instant, just something we need to do sometime in the future. Maybe that’s changing those worn tires in the car, or making an appointment with the dentist about that tooth that has been bugging you on and off.

Dynamic memory allocation is something that falls in that category for most programmers. We all know we can’t just go on allocating memory willy-nilly whenever we need it, yet we put off dealing with it until the end of the project. By that time, deadlines are piling on the pressure and it’s usually too late to make significant changes. With a little bit of forethought and pre-planning, we can avoid those problems and be confident our game is not going to run out of memory in the most inopportune moment.

On-Demand Dynamic Memory Allocation

The easiest way to get started with memory management is to allocate memory dynamically whenever you need it. This is the approach many software engineering books consider as ideal and it’s often encouraged in Computer Science classes.

It’s certainly an easy approach to use. Need a new animation instance when the player is landing on a ledge? Allocate it. Need a new sound when we reach the goal? Just allocate another one!

On-demand dynamic memory allocation can help to keep memory usage to a minimum, because, you only allocate the memory that you need and no more. In practice it’s not quite as neat and tidy because there can be a surprisingly large amount of overhead per allocation, which adds up if programmers become really allocation-happy.

It’s also a good way to shoot yourself in the foot.

Games don’t live inside a Computer Science textbook, so we have to deal with real world limitations, which make this approach cumbersome, clunky, and potentially disastrous. What can go wrong with on-demand dynamic memory allocation? Almost everything!

Limited Memory

Games, or any software for that matter, run on machines with limited amounts of memory. As long as you know what that limit is, and you keep extremely careful track of your memory usage, you can remain under the limit. However, since the game is allocating memory any time it needs it, there will most likely come a time when the game tries to allocate a new block but there is no memory left. What can you do then? Nothing easy I’m afraid. You can try to free an older memory block and allocate the new one there, or you can try to make your game tolerant to running out of memory. Both those solutions are very complex and difficult to implement correctly.

Even setting memory budgets and sticking to them can be very difficult. How can a designer know that a given particle system isn’t going to run out of memory? Are these AI units going to create too many pathfinding objects and crash the game? Hard to say until we run the game in all possible combinations. And even then, how do you know it isn’t going to crash five minutes later? Or ten? It’s almost impossible to know for certain.

If you insist in using this approach, at the very least, you should tag all memory allocations, so you have an idea of how memory is being used. You can either tag each allocation based on what system initiated it (physics, textures, animation, sound, AI, etc) or even on the filename where it originated, which has the advantage that it can be automated and should still give you a good picture of the overall memory usage.

Memory Fragmentation

Even if you take lots of pain not to go over your the available memory, you might still run into trouble because of memory fragmentation. You might have enough memory for a new allocation, but in the form of many small memory blocks instead of a large contiguous one. Unless you provide your own memory allocation mechanism, fragmentation is something that is very hard to track on your own, so you can’t even be ready for it until the allocation fails.

Virtual Memory

Virtual memory could solve all those problems. In theory, if you run out of real memory, the operating system swaps out some older, unused pages to disk and makes room for the new memory you requested. In practice, it’s just a bad caching scheme because it can be triggered at the worst possible moment, and it doesn’t know about what data it’s swapping out or how your game uses it.

Games, unlike most other software, have a “soft realtime” requirement: The game needs to keep updating at an acceptable interactive rate, which is somewhere around 15 or more frames per second. That means that gamers are going to make a trip to the store to return your game if it pauses for a couple of seconds every few minutes to “make some room” for new memory. So relying on virtual memory isn’t a particularly attractive solution.

Additionally, lots of games run in platforms with fixed amounts of RAM and no virtual memory. So when memory runs out, things won’t get slow and chuggy, they’ll crash hard. When the memory is gone, it’s really gone.

Performance Problems

There are some performance issues that are relatively easy to track down and fix. Usually ones that occur every frame and are happening in a single spot: some expensive operation, a O(n3) algorithm, etc. Then there are performance problems introduced by dynamic memory allocations, which can be really hard to track down.

Standard malloc returns memory pretty quickly, and usually doesn’t ever register on the the profiler. Every so often though, whenever the game has been running for a while and memory is pretty fragmented, it can spike up and cause a significant delay for just a frame. Trying to track down those spikes has caused more than one programmer to age prematurely. You can avoid some of those problems by using your own memory manager, but don’t attempt to write a generic one yourself from scratch. Instead start with some of the ones listed in the references.

Malloc spikes are not the only source of performance problems. Allocating many small blocks of memory can lead to bad cache coherence when the game code access them sequentially. This problem usually manifests itself as a general slowdown that can’t be narrowed down in the profiler. With today’s hardware of slow memory systems and deep caches, good memory access patterns are more important than ever.

Keeping Track Of Memory

Another source of problems with dynamic memory allocation are bugs in the logic that keeps track of the allocated memory blocks. If we forget to free some of them, our program will have memory leaks and has the potential to run out of memory.

The flip side of memory leaks are invalid memory access. If we free a memory block and later we access it as if it were allocated, we’ll either get a memory access exception, or we’ll manage to corrupt our own game.

Some techniques, such as reference counting and garbage collection can help keep track of memory allocations, but introduce their own complexities and overhead.

Introducing Pre-allocation

On the opposite corner of the boxing ring is the purely pre-allocated game. It excels at everything that the dynamically-allocated game is weak at, but it has a few weaknesses of its own. All in all, it’s probably a much safer approach for most games though.

The idea behind a pre-allocation memory strategy is to allocate everything once and never have to do any dynamic allocations in the middle of the game. Usually you grab as big a block of memory as you can, and then you carve it out to suit your game’s needs.

Some advantages are very clear: no performance penalties, knowing exactly how your memory is used, never running out of memory, and no memory fragmentation to worry about. There are some other more subtle advantages, such as being able to put data in contiguous areas of memory to get best cache coherency, or having blazingly-fast load times by loading a baked image of a level directly into memory.

The main drawback of pre-allocation is that is more complex to implement than the dynamic allocation approach and it takes some planning ahead.

Know Your Data

For preallocation to work, you need to know ahead of time how much of every type of data you will need in the game. That can be a daunting proposition, especially to those used to a more dynamic approach. However, with a good data baking system (see last month’s Inner Product column), you can get a global view of each level and figure out how big things need to be.

There is one important design philosophy that needs to be adopted for preallocation to work: Everything in the game has to be bounded. That shouldn’t feel too restrictive; after all, the memory in your target platform is bounded, as well as every single resource. That means that everything that can create new objects, including high-level game constructs, should operate on a fixed number of them. This might seem like an implementation detail, but it often bubbles up to what’s exposed to game designers. A common example is an enemy spawner. Instead of designing a spawner with an emission rate, it should have a fixed number of enemies it can spawn (and potentially reuse them after they’re dead).

Potentially Wasted Space

If you allocate enough data for the worst case in your game, that can lead to a lot of unused data most of the time. That’s only an issue if that unused data is preventing you from adding more content to the game. We might initially balk at the idea of having 2000 preallocated enemies when we’re only going to see 10 of them at once. But when you realize that each of those enemies is only taking 256 bytes and the total overhead is 500 KB, which can be easily accommodated in most modern platforms today.

Preallocation doesn’t have to be as draconian as it sounds though. You could relax this approach and commit to having each level preallocated and never having dynamic memory allocations while the game is running. That still allows you to dynamically allocate the memory needed for each level and keep wasted space to a minimum. Or you can take it even further and preallocate the contents of memory blocks that are streamed in memory. That way each block can be divided in the best way for that content and wasted space is kept to a minimum.

Reuse, RecycleM

If you don’t want to preallocate every single object you’ll ever use, then you can create a smaller set, and reuse them as needed. This can be a bit tricky though. First of all, it needs to be very much specific to the type of object that is reused. So particles are easy to reuse (just drop the oldest one, or the ones not in view), but might be harder with enemy units or active projectiles. It’s going to take some game knowledge of those objects to decide which ones to reuse and how to do it.

It also means that systems need to be prepared to either fail an allocation (if your current set of objects is full and you don’t want to reuse an existing one), or they need to cope with an object disappearing from one frame to another. That’s a relatively easy problem to solve by using handles or other weak references instead of direct pointers.

Then there’s the issue that reusing an object isn’t as simple as constructing a new one. You really need to make sure that when you reuse it, there’s nothing left from the object it replaced. This is easy when your objects are just plain data in a table, but can be more complicated when they’re complex C++ classes tied together with pointers. In any case, you can’t apply the Resource Acquisition Is Initialization (RAII) pattern, but it doesn’t seem to be a pattern very well suited for games, and it’s a small price to pay for the simplicity that preallocation provides.

Specialized Heaps

Truth be told, a pure pre-allocated approach can be hard to pull off, especially with highly dynamic environments or games with user-created content. Specialized heaps is a combination of dynamic memory allocation and pre-allocation that takes the best of both worlds.

The idea behind specialized heaps is that the heaps themselves are pre-allocated, but they allow some form of specialized dynamic allocation within them. That way you avoid the problems of running out of memory, or memory fragmentation globally, but you still can perform some sort of dynamic allocation when needed.

One type of specialized heaps is based on the object type. If you can guarantee that all objects allocated in that heap are going to be of the same size, or at least a multiple of a certain size, memory management becomes much easier and less error prone, and removes a lot of the complexity of a general memory manager.

My favorite approach for games is to create specialized heaps based on the lifetime of the objects allocated in them. These heaps use sequential allocators, always allocating memory from the beginning of a memory block. When the lifetime of the objects is up, the heap is reset and allocations can start from the beginning again. The use of a simple sequential allocator bypasses all the insidious problems of general memory management: fragmentation, compaction, leaks, etc. See the code in http://gdmag.com/resources/code.htm for an implementation of a SequentialAllocator class.

The heap types most often used in games are:

• Level heap. Here you allocate all the assets and data for the level at load time. When the level is unloaded, all objects are destroyed at once. If your game makes heavy use of streaming, this can be a streaming block instead of a full level.
• Frame heap. Any temporary objects that only need to last a frame or less get allocated here, and destroyed at the end of the frame.
• Stack heap. This one is a bit different from the others. Like the other heaps, it uses a sequential allocator and objects are allocated from the beginning, but instead of destroying all objects at once, it only destroys objects up to the marker that is popped fro the stack.

What About Tools?

You can take everything I’ve written here, and (almost) completely ignore it for tools. I fall in the camp of the programmers who consider the runtime as a totally separate beast from the tools. That means that the runtime can be lean and mean and minimalistic, but I can relax and use whatever technique makes me more productive when writing tools. That means you can allocate memory any time you want, you can use complex libraries like STL and Boost, etc. Most tools are going to run on a beefy PC and a few extra allocations here and there won’t make any difference.

Be careful with performance-sensitive tools though. Tools that build assets or compute complex lighting calculations might be a bottleneck in the build process. In that case, performance becomes crucial again and you might want to be a bit more careful about memory layout and cache coherency.

On the other hand, if the tool you’re writing is not performance sensitive, you should ask yourself if it really needs to be written in C++. Maybe C# or Python are better languages if all you’re doing is transforming XML files or verifying that a file format is correct. Trading performance for ease of development is almost always a win with development tools.

Next time you reach out for a malloc inside your main loop, think about how it can fail. Maybe you can pre-allocate that memory and stop worrying about what’s going to happen the day you prepare the release candidate.

This article was originally printed in the February 2009 issue of Game Developer.

No Flash? Try the QuickTime video version.

Let me get one thing out of the way to stop all the trigger-happy, const-bashing, would-be-commenters: const doesn’t make any guarantees that values don’t change.

You can change a const variable by casting the constness away, or referencing it through a pointer, but you really had to go out of your way to do that. If it helped with that, const would solve (or improve) the memory aliasing problem like Hitler pointed out. It doesn’t, so const is pretty weak as far as promises go. It just says “I, the programmer, promise not to change this value on purpose (unless I’m truly desperate)”. Still, even a promise like that goes a long way helping with readability and maintenance.

With that out of the way, what exactly do I mean by using const everywhere?

Const non-value function parameters

Any reference or pointer function parameters that are pointing to data that will not be modified by the function should be declared as const. If you’re going to use const just for one thing, this is the one to use. It’s invaluable glancing at a function signature and seeing which parameters are inputs and which ones are outputs.

void Detach(PhysicsObject& physObj, int attachmentIndex, const HandleManager& mgr);

Marking those parameters as const also serves as a warning sign in case a programmer in the future tries to modify one of them. Imagine the disaster if the calling code assumes data never changes, but the function suddenly starts modifying that data! const won’t prevent that from happening, but will remind the programmer that he’s changing the “contract” and needs to revisit all calling code and check assumptions.

Const local variables

This is a very important use of const and one of the ones hardly anyone follows. If I declare a local (stack) variable and its value never changes after initialization, I always declare it const. That way, whenever I see that variable used later in the code, I know that its value hasn’t changed.

const Vec2 newPos = AttachmentUtils::ApplySnap(physObj, unsnappedPos);
const Vec2 deltaPos = newPos - physObj.center;
physObj.center = newPos;

This is one of the reasons why I did a 180 on the ternary C operator (?). I used to hate it and find it cryptic and unreadable, but now I find it compact and elegant and it fulfills my const fetish very well.

Imagine you have a function that is going to work in one of two objects and you need to compute the index to the object to work on. You could do it this way:

int index;
if (some condition)
	index = 0;
else
	index = 2;

DoSomethingWithIndex();

Not only does that take several lines not to do much, but index isn’t const (argh!). So every time I see index anywhere later on in that function, I’m going to have to spend the extra mental power to make sure nothing has changed (and, with my current coding style, I would assume it has changed).

Instead, we can simply do this:

const int index = (some condition) ? 0 : 2;
DoSomethingWithIndex();

Ahhhh… So much better!

Const member variables

This one doesn’t really apply to me anymore because I don’t use classes and member variables. But if you do, I strongly encourage you do mark every possible member function as const whenever you can.

The only downside is that sometimes you’ll have some internal bit of data that is really not changing the “logical” state of an object, but it’s still modifying a variable (usually some caching or logging data). In that case, you’ll have to resort to the mutable keyword.

Const value function parameters

Apparently I’m not a total Const Nazi because this is one possible use of const that I choose to skip (even though I tried it for a while because of Charles).

Marking a value function parameter as const doesn’t make any difference from the calling code point of view, but it serves the same purpose as marking local stack variables as const in the implementation of the function. You’re just saying “I’m not going to modify that parameter in this function” so it makes the code easier to understand.

I’m actually all for this, but the only reason I’m not doing it is because C/C++ makes it a pain. Marking parameters as const in the function declaration adds extra verbosity and doesn’t help the person browsing the functions at all. You could actually put the const only in the function definition and it will work, but at that point the declaration and the definition are different, so you can’t copy and paste them or use other automated tools or scripts.

The concept of const is one of the things I miss the most when programming other languages like C#. I don’t understand why they didn’t add it to the language. On something like Python or Perl I can understand because they’re supposed to be so free form, but C#? (Edit: How about that? Apparently C# has const. It was either added in the last few years or I completely missed it before). It also really bugs me that Objective C or the Apple API doesn’t make any use of const.

Frankly, if it were up to me, I would change the C/C++ language to make every variable const by default and adding the nonconst or changeable (or take over mutable) keyword for the ones you want to modify. It would make life much more pleasant.

But then again, that’s why the call me the Const Nazi.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

Every developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

• All my code is written in C and C++ (except for a bit of ObjC and assembly).
• It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
• I always try to write important code through Test-Driven Development.
• I’m the only programmer (and only designer).
• Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

• I’m showing all the data there is and how big it is. Nothing is hidden.
• I’m making it clear that it’s free of pointers and temporary variables.
• I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

If you haven’t seen the presentation, go download it right now. It’s really well put together and he has some great detailed examples on how caches are affected with traditional object-oriented design vs. a more data-oriented approach.

[1] Thanks to Christer Ericson for pointing that out.

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

hinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is Thre Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

Figure 1a. Call sequence with an object-oriented approach

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

Figure 1b. Call sequence with a data-oriented approach

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

Thinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is There Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

This article was originally printed in the September 2009 issue of Game Developer.

Here’s a Korean translation of this article by Hakkyu Kim.

Size

Charles and I were priding ourselves in keeping things small and minimal. But truth be told, it’s not like we were keeping track of how many lines of code we had written. Were things as small as we hoped they were?

The most convenient way of counting lines of code that I know is CLOC. It’s an extremely easy to use open source program which counts the lines of code in a codebase, gives very detailed information, strips out whitespace, breaks things down by language, and does just about everything you’d want from a program like that.

Running it on the latest version of our code (not including any 3rd party libraries) produces this:

    1621 text files.    1579 unique files.    3721 files ignored.
-------------------------------------------------------------------------------
Language                    files          blank        comment          code
-------------------------------------------------------------------------------
C++                            485          13577            303          46181
C#                             324           4935            712          22966
C/C++ Header                   407           4153             95          11975
MSBuild scripts                 18              0            126          1490
-------------------------------------------------------------------------------
SUM:                          1234          22665           1236          82612

Almost 60K lines of C++ code seemed very high. At first I thought it was because CLOC was counting files twice: once in their regular location and once in the .svn directory, but apparently it’s already removing all duplicates, so that wasn’t it.

Almost more scary than the amount of C++ code (which is all our runtime and some of our tools) is the amount of C# code. For a language that claims to be of significantly higher level than C++, that’s quite a mouthful of code!

Another surprising count in there is the number of lines with comments. Since we make heavy use of TDD, I really didn’t expect more than a couple dozen lines of code in the whole codebase. Still, I’m kind of proud that we have less than one line of code per file on average :-)

Here’s a more detailed breakdown, with the line count just for our runtime (engine and game):

1089 text files.    1053 unique files.    2338 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            441          11997            245          40943
C/C++ Header                   385           3964             90          11405
-------------------------------------------------------------------------------
SUM:                           826          15961            335          52348

and for our tools:

532 text files.     531 unique files.    1383 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                             324           4935            712          22966
C++                             44           1580             58           5238
MSBuild scripts                 18              0            126           1490
C/C++ Header                    23            199              5            591
-------------------------------------------------------------------------------
SUM:                           409           6714            901          30285

Tests

Then I realized that a good chunk of those were tests. So excluding all directories matching *Tests* gets the following result:

1206 text files.    1187 unique files.    4199 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            283           6464            150          22464
C#                             213           2636            534          12402
C/C++ Header                   380           3824             94          10782
MSBuild scripts                 12              0             84            978
-------------------------------------------------------------------------------
SUM:                           888          12924            862          46626

A bit more than half the C++ code consisted of tests. That’s pretty consistent with my experience with TDD. C# seems to follow a similar percentage as well.

As for the exact number of tests, running a grep for TEST shows all the C++ tests:

C:\pow2>grep -r TEST SweetPea Engine Tools | grep -v svn | wc   2163    3620  221953

And doing the same thing with [Test] brings up all C# tests:

C:\pow2>grep -r \[Test\] SweetPea Engine Tools | grep -v svn | wc   735    1470  52717

That means that our average C++ test is about 11.5 lines long, and C# tests 14.4. Frankly, that sounds rather high. We make heavy use of fixtures whenever possible and each test usually only checks for a single condition (even if it involves a couple check statements). I suppose that number is higher than expected because it probably includes all the lines from #include statements and all the fixtures as part of the average.

* If we only count cpp files, that goes down to 49%

I was curious about that last part of checking a single thing per test, so I ran a grep for the number of CHECK statements in our code:

C:\pow2>grep -r CHECK SweetPea Engine Tools | grep -v svn | wc   3886   15079  399598

That’s 1.8 CHECK statements per TEST, which is about right. Even though we’re checking for a single condition, we’ll often check a couple things about it (i.e. the camera stopped and it reached its final destination).

Build Times

So, given that amount of code, how long does it take to build it? Clearly it depends on your hardware. Since we’re not exactly rolling in money, we don’t have particularly powerful machines. Here at home, I’m using a modest Core 2 Duo E4300 (overclocked to 2.6 GHz) with fast memory and a relatively fast SATA hard drive, so that’s what I used for all my timings.

A full build of our game, plus all the libraries, all the tests, and running all the tests takes exactly 1 minute and 10 seconds. That’s pretty good for two reasons:

• When we work with the game we don’t build and run the unit tests for the engine. We have a separate solution for that. A full build of just the engine, the game, and the game unit tests only takes 43 seconds.
• The game itself is a fairly large project and devenv doesn’t know how to paralellize that build, so it’s only using half the available CPU power for about half the build time.

An incremental build after changing a single cpp file takes slightly over a second (including half a second of unit test execution).

As you can imagine, working with that codebase is a dream come true. Snappy, responsive. Nothing is hard enough that can’t be changed.

Unfortunately that’s where the fairy tale ends. The tools are another story altogether. Our C# tools, with all their unit tests, build in a mere 18 seconds, and the C++ tools in 1 minute and 10 seconds. That’s not too bad, except that it’s a surprisingly large amount of time for the C++ tools since there aren’t that many of them.

Here’s the kicker, doing another build without changing a single thing take 38 seconds. Whoa! We’re doing some C++/CLI trickery and apparently dependency checking is totally broken in VS2005 (either that, or we just don’t know how to set it up right).

Keeping things fast

What’s the secret of a lighting-fast build? Clearly, keeping the code size down is crucial. If your codebase is 2 million lines of code, builds are going to be painful no matter what. But they can be a little less painful with some gentle care.

One of the main build-time killers that we’re avoiding is the use of STL or Boost. Those libraries pull in everything and the kitchen sink, and their heavy use of templates make build and link time go through the roof. No thanks.

Our template use is pretty minimal. We have a couple containers (which I love and I’ll write about it one of these days) and that’s about it.

We’re pretty anal when it comes to keeping physical dependencies to a minimum. We forward declare aggressively, and we only include the headers that are necessary for each cpp file (PC Lint is “kind” of enough to remind us every time we have unnecessary #includes). We’re not using external include guards or #pragma once.

Precompiled headers are either not used, or kept to a minimum. I think the only project that uses them is the game and only for Havok headers. We don’t even have windows.h in a precompiled header (which would be a really bad idea because you’d be putting all the junk in windows.h available to your whole program).

Finally, we are using incremental links whenever possible. I remember a few versions of Visual Studio ago they were pretty broken, but they’re not giving us any problems. The only caveat is that if you modify a static library your program is linking with, it will force a full link. So they’re really only good for modifying the executable itself.

We’re not using any distributed builds. First of all, we don’t have enough computers to make it worthwhile. And second, I had horrible experiences with distributed builds in the past. They would help with a badly structured codebase, at the cost of longer incremental builds and mysterious spurious bad builds. Besides, once they’re in place, they tend to encourage even further disregard for keeping dependencies to a minimum.

How about you?

So, that’s it for the Power of Two codebase. How about you? Want to share your size, build times, or any other data?

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there :-) ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions.

None of that would be an issue except that TDD creates lots of unit tests and encourages almost constant incremental builds. In the past I’ve stressed how important it is to keep them fast. But this week that I was able to really feel what a dramatic difference slow unit tests make.

For many months, our unit tests ran under one second. Each library has its own set of unit tests, and so does the game and each tool. So even though we were writing lots of unit tests, we weren’t always running them all at the same time. There was a definite snappiness to coding: write test, compile, fail, write code, compile, pass, move on. Go, go, go.

As the game project got larger, we started accumulating more unit tests and testing times started creeping up into the second range. At the time, I couldn’t quite put my finger on what had changed, but there was a definite change in feel. Things weren’t quite as snappy.

Then, about a couple of weeks ago, we hit the two second mark and things definitely changed.

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 2.13 seconds.

As the tests were running, my mind would wander, thinking about what test I would write next, or whether the code I had just written would be cache-friendly, or whether we’d do Honey’s or Manhattan for lunch. I would be jerked out of my state of flow. It wasn’t just me, Charles felt it too. The difference in productivity was huge.

The Zones

For those of you laughing at those times and dismissing them as insignificant, you need to keep in mind that our build times are very short:

• An incremental build with just a change in a cpp file takes under a second (including link times).
• A full build of *all* our runtime code, including libraries, is around 50 seconds in a dual-core machine [1].

So adding on 2 seconds to the incremental build time is going from “practically instant” to “OK, I’m waiting…”.

Of course, none of this has anything to do with unit tests in particular. It’s total incremental build times (including compiling, linking, and running unit tests) that matter. An incremental build that takes 3 seconds with no unit tests is just as bad as one that is under one second with 3 second unit tests. They both interrupt the flow the same way.

I’ve often thought about how a company size affects communication and “feel”. It’s close to a step function with logarithmic scale: there’s a distinct jump from a company of 3 to a company of 10, to one of 35, to one of 100+.

I think something similar applies to build times [2]

• 0 to 2 seconds. Programmer stays in the flow. Productivity is way high.
• 2 to 8 seconds. Flow is broken. Definite feel of things being a bit painful and sluggish.
• 8 to 30 seconds. Attention wanders to other parts of the code, email, etc. Multitasking kicks in and overall productivity plummets.
• 30+ seconds. Builds seen as batch processing. After each build is started, some other action is started. Maybe even physically get up, get a drink, start a conversation. Full brain reboot between coding tasks.
• 5+ minutes. Programmers bang their heads against the desk, actively consider the merits of different forms of suicide, or at the very least start polishing their resumes.

The Fix

Having been spoiled by sub-second build times, neither one of us wanted to put up with the broken flow of 2+ second builds, so we decided it was time to fix things [3].

The good news is that our slow build times were solely due to running the unit tests. 2.13 seconds to run only about 1100+ tests is really bad, even on a so-so desktop machine like the ones we can afford. That’s almost 2ms per test! Remember that these are unit tests, not functional tests. So I expect to be able to run several thousands of these under one second. Something was way off.

Things had gotten much worse in the last couple of weeks, so it wasn’t a slow, steady creep up. Chances are we had introduced something very slow recently, which meant it was probably easy to fix.

As a first pass, we turned on the option to fail any test that takes over a certain amount of time in UnitTest++:

RunAllTests(reporter, UnitTest::Test::GetTestList(), NULL, 5); // fail any test over 5ms

That brought about tons of failing test, most of them added recently and creating one particular object in their fixtures. Digging a big deeper, that particular class had an array of 25,000 (yes, that many zeros) objects. Those objects were plain-old-data structures, but had a default constructor. Even though the constructor was simply initializing its members to zero, that was causing a significant slow down when so many objects were created. To make things worse, some of the code I had written was iterating over the array resetting the entries or shifting them up or down, repeatedly calling the constructor. Ouch!

A few minutes and lots of code hacking later, and the object didn’t call a single constructor anymore. Our test times went down by almost half. Victory!

We were already at 1 second, which is in the acceptable range, but since we were here, maybe we could improve things a bit more. After all, 1ms per test is still too slow.

UnitTest++ wasn’t reporting failing tests over 1ms very reliably. Different runs would cause very different results. Things were too spread out or other things in the computer were interfering too much.

About a fifth or so of our tests use Havok. Not with mocks, but directly. They each initialize Havok, create a world, and deal with a simple rigid body, constraint, or something of the sort. I suspected all along that the Havok system initialization and shutdown for each test was probably causing quite a slow down, so I changed it to be initialized only once before we ran any tests.

And that did the trick. With that change we were down to 0.16 seconds. Back to snappy, happy land!

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 0.16 seconds.

In release mode things are even better, but we do most development in debug mode, so that’s the really important one:

> bin\SweetPea.exe -unittestSuccess: 1145 tests passed.Test time: 0.06 seconds.

Perhaps sub-second build times are not realistic for all projects, but I think there’s a lot to be gained by keeping compile, link, and test times as short as possible. For tips on how to keep compile times down, check out these articles I wrote a while ago. If your unit tests are slow and you really can’t speed them up anymore, consider splitting them into two sets: one that is important or recent ones that runs with every build, and another, more comprehensive one that runs in the build server.

Even if you cut down from 20 seconds down to 10, you’ll be making a huge shift in how you’re able to work and iterate on your programming.

[1] We’d love to keep detailed metrics with each build: executable size, number of lines of code, number of tests, etc, etc. Unfortunately, when the rubber hits the road, that ends up in the “nice to have category” and we simply have no resources to spare on a two-man startup.

[2] For a whole whooping statistical population of one since I’m only talking about my experience. Still, I suspect that the same applies to many other programmers by scaling the function by some constant.

[3] Perfect use of agile mentality. Fix things when you need them fixed, not before. But make sure you do fix them when you need them. Otherwise it’s not agile, it’s just lazy.

C++ enums are great in practice, but they’re a bit lacking once you really start flexing them. Don’t get me wrong, they’re definitely a step up from #defines, but you can’t help but wish for more. Type safety is only so-so (enums get promoted to ints silently), you can’t forward declare them so you’re forced to have extra includes, you can’t split up a declaration of enums across multiple files (the compiler would have to do more work, but it would be awesome to be able to create uids for different classes in their own header files instead of a global one). But the most grating of design decisions is their scoping rules.
Take for example a set of enums describing the game flow:

enum GameFlowType
{
    Run,
    Exit,
    Restart,
    Restore,
};

Now functions can return a GameFlowType to indicate that the main loop should do. So far so good.

Except that, look at how it’s used:

GameFlowType DoMainLoop(bool render)
{
    //...
    if (someCondition)
        return Exit;
    return Run;
}

Ouch! We’ve managed to pollute the global namespace with totally generic keywords such as Run and Exit. What if I want to have an enum that controls what action an AI is going to take:

enum AIAction
{
    Enter,
    Exit,
    Stop,
    Walk,
    Run,
};

That won’t even compile because the symbols conflict with the GameFlowType ones. Even worse, it will compile just fine if they don’t happen to be included in the same compilation unit. So you might be fine all along until you write some AI code that has control over the game flow. Oops!
OK, so you can prefix all your enums like this:

enum GameFlowType
{
    GameFlowTypeRun,
    GameFlowTypeExit,
    GameFlowTypeRestart,
    GameFlowTypeRestore,
};

and

enum AIAction{
    AIActionEnter,
    AIActionExit,
    AIActionStop,
    AIActionWalk,
    AIActionRun,
};

That will technically fix the problem, but it’s a bit ugly, and we’re still polluting the global namespace (what if I wanted to have a class called AIActionEnter?).

Another potential solution is to score the enum inside the class that is using them. So we could have:

class AIAction{
public:
    enum AIActionType
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };
};

Much cleaner, do doubt, but also with its share of problems: First of all, it forces you to associate a set of enums with a class, which is not something you always want to do. But the biggest problem is that as soon as you have more than one set of enums per class, they all get promoted to the same scope:

class AIAction
{
public:
    enum AIActionBehavior
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };

    enum AIActionGroup
    {
        None,
        Self,
        Team,
        All,
    };
};

When we see one of those enums in code referred to as AIAction::Self, we really have no idea that it’s referring to that group the action is applied to, as opposed to AIAction::Enter, which is the type of action.
Our solution? Move all enums into a descriptive namespace.

namespace GameFlowType
{
    enum Enum
    {
        Invalid,
        Run,
        Exit,
        Restart,
        Restore,
    };
}

When you have a variable of that enum type, it’s listed as GameFlowType::Enum type, which clearly indicates what it does. And its values are referred to as GameFLowType::Exit. You get the ideal scoping, you’re not tied to any particular class, things are explicit but not verbose, you’re no polluting the global namespace, and it’s doesn’t affect the runtime or compilation times.
Sometimes the best solutions are the simplest ones.

I had been using a modified version of CppUnitLite for quite a while, slowly fixing the parts in need of mending, and adding new functionality as it became needed. Eventually I released most of those changes as CppUnitLite2, and I thought that would be the end of that. It turns out that was just the beginning.

Shortly after that, several people started emailing me with new functionality, fixes, patches, suggestions, etc. At the same time, Charles Nicholson was starting to get quite a bit of response as well for his own testing framework, sometimes with similar fixes and suggestions.

If this was going to grow to be a real project, developing over time, supporting users’ needs, and accepting code contributions, clearly it would make sense to join forces. And that’s exactly what we did. We grabbed the best features of each framework and created what we think it’s the best C++ unit-testing framework out there (for our needs anyway).

We took the results and put them up in Sourceforge under a very unrestrictive license, and that’s how UnitTest++ was born.

Our ultimate goal is to apply test-driven development to game development. All of the existing frameworks fell short in one area or another. Specifically, the driving forces behind the design of UnitTest++ are:

• Portability. As game developers, we need to write tests for a variety of platforms, most of which are not supported by normal software packages (all the game consoles). So the ability to easily port the framework to a new platform was very important.
• Simplicity. The simpler the framework, the easier it is to add new features or adapt it to meet new needs, especially in very limited platforms.
• Development speed. Writing and running tests should be as fast and straightforward as possible. We’re going to be running many tests hundreds of times per day, so running the tests should be fast and the results well integrated with the workflow.

UnitTest++ is a fully featured testing framework, with many of the features you may expect from Xunit-style frameworks such as fixtures and reporting. Additionally, here are some of the specific features that make UnitTest++ unique:

• Minimal work required to create a new test. For TDD development, we write lots and lots of tests, so creating a new test should be as quick and painless as possible. UnitTest++ does not require explicit test registrations, and creating new tests requires minimal work.
• Good assert and crash handling. When automated tests are executed, it’s very important not to hang the process or abort the tests any time the program crashes or an assert fails. UnitTest++ will catch and report exceptions as failed tests and print a lot of information about them. It will translate signals to exceptions as well as be able to catch invalid memory accesses and other problems.
• Minimal footprint and minimal reliance on heavy libraries. When you intend to run tests on a Cell SPU with a measly 256 KB of RAM, you really want to be as lean and mean as possible. UnitTest++ is very lightweight, and it will get even lighter weight once strstream is replaced.
• No dynamic memory allocations done by the framework, which makes it much easier to track memory leaks and generally more attractive for embedded systems.
• Multiplatform support. Right now it supports Win32, Linux, and Mac OS X out of the “box,” and other platforms will probably work without any changes. The source code makes use of platform-specific functions whenever necessary, so it is easy to hook up special timers, allocators, or reporters for specific platforms. The code comes with a makefile as well as with project and solution files for Visual Studio 2003 and 2005.
• Full set of unit tests for itself. UnitTest++ was TDD’d from the ground up, including some of the ugly macros (imagine how much fun it was bending the framework to test itself that way!). It goes without saying, all the tests are included with the source code, so you can go totally wild with them :-)

This was just the first release. UnitTest++ is ready for full production work, but we already have plans beyond this release. This is a peek at some of the features we have in mind for upcoming versions:

• Out-of-the-box support for game console platforms (Xbox 360, PS3 PPU and SPU, and Nintendo Revolution). Of course, this is dependent on us getting permission from the console manufacturers to release some code that works with their SDK.
• Ability to trade some features (such as rich check reporting) for some extra memory. In some platforms, like Cell SPUs, you really need to go barebones.
• Suites to group tests together.
• Different reporters (HTML, GUIs, etc). Don’t worry, all those modules will be optional and well separated, so they won’t affect the simplicity of the framework.
• Per-test timings. Maybe the ability to flag some tests as not exceeding a certain amount of time, or just failing any test that goes over a threshold.
• Built-in memory leak detection.

So that’s it. Download the source code and give it a try. We welcome any comments and suggestions (best channels would be through the project mailing list or comments here). We hope you find it useful and that it makes your TDDing even more productive.

CppUnitLite2_1_1.tar.gz

About a year ago, I wrote an article comparing unit test frameworks that went to become one of the most popular in this site. The article concluded with the recommendation of one of the less well known frameworks: CxxTest.

Shortly after writing that article, it came time to roll out test-driven development at work. Interestingly, once I weighted in all the requirements, I ended up not following my own advice. Instead, I decided to go with a customized version of CppUnitLite, which was listed in the article as the second choice. The idea of an extremely simple unit testing framework was too attractive to pass up in an environment where we have to support a large variety of environments and target platforms.

At this point, we have been using CppUnitLite2 for a year at High Moon Studios to do TDD on Windows, Xbox 360, and some PS3 (we even have a stripped-down version that runs on a PS3 SPU). It has been used to unit test libraries of an engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2 was originally released along with my article a year ago. It was a heavily modified version of the original CppUnitLite by Michael Feathers. This version introduces the changes we made at work as different issues came up and we needed more functionality from the framework. In particular, Jim Tilander and Charles Nicholson were responsible for a lot of those changes. By now, I doubt there’s a single line left from the original code. Additionally, CppUnitLite2 itself has a set of tests using itself in an interestingly recursive way.

What’s new

Fixtures created and destroyed around each test.

I can’t believe that the original version of CppUnitLite2 didn’t have this feature. It was the biggest glaring problem with the framework (I even pointed it out in the article). Each fixture used to be created at static initialization time, which, apart from being a really bad idea, caused a lot of objects to be created and live in memory at the same time. Considering that some of those objects would initialize/shutdown some important game systems, we clearly needed to manage their lifetime a bit better.

The test macro was changed to register a temporary test with the TestRegistry, which in turn will create the fixture at execution time. Now fixtures and tests are created just before each test, and destroyed right after, just as you would expect from any sane framework.

For those masochists out there, here’s the ugly macro for a test with fixture in all its glory. Notice how the TestRegistrar registers the dummy test class that we created as part of the macro.

 #define TEST_F(fixture, test_name)                                                      \ struct fixture##test_name : public fixture {                                        \ fixture##test_name(const char * name_) : m_name(name_) {}                       \ void test_name(TestResult& result_);                                            \ const char * m_name;                                                            \ };                                                                                  \ class fixture##test_name##Test : public Test                                        \ {                                                                                   \ public:                                                                         \ fixture##test_name##Test ()                                                 \ : Test (#test_name "Test", __FILE__, __LINE__) {}                       \ protected:                                                                      \ virtual void RunTest (TestResult& result_);                                 \ } fixture##test_name##Instance;                                                     \ TestRegistrar fixture##test_name##_registrar (&fixture##test_name##Instance);       \ void fixture##test_name##Test::RunTest (TestResult& result_) {                      \ fixture##test_name mt(m_name);                                                  \ mt.test_name(result_);                                                          \ }                                                                                   \ void fixture ## test_name::test_name(TestResult& result_)

Improved check statements

Another one of the major weaknesses of CppUnitLite as it stood was the check statements. You needed to have a custom check macro for each data type you cared to check. The generic CHECK_EQUAL macro will now work on any data type as long as it can be compared with operator == and output to a stream.

This version of CppUnitLite2 also includes some special macros to check arrays with one and two dimensions. Again, this works on any data type that supports operator == and operator[]. It came it very handy for checking equality of vectors and matrices in a game engine. For example, the following code checks that two matrices are equal:

CHECK_ARRAY2D_CLOSE (m1, m2, 4, 3, kEpsilon);

Ironically, in some heavily restricted environments (like the PS3 cell processors), we can’t use streams, so in that case we’re forced to go back to the old specialized check macros.

Invariant statements in checks

Another really annoying feature of the previous version of CppUnitLite2 was that the code in a check statement was called multiple times, which could cause strange side effects.

Consider the following code:

CHECK_EQUAL (FunctionWithSideEffects1(), FunctionWithSideEffects1());

The way the check statement was expanded out, both those functions could be evaluated multiple times. If the functions had side effects, they would be executed several times, causing unexpected results (for example, the check could fail but the printed result could be different). The fix wasn’t trivial because we couldn’t save off the value of each statement because they could be of any type and we have no way of finding the type from within the macro.

The check macro now expands out to a templated function, which is able to evaluate both statements only once and work with the saved value to avoid any surprising side effects.

Optional fixture initialization

As we continued hammering on the unit test framework for all our development, one clear pattern appeared that wasn’t handled by the framework: Being able to create fixtures with parameters defined in the test. For example, the fixture might do ten different steps, but we want to set a particular value in one of those steps, and by the time control reaches the test code itself it’s too late.

We started getting around that by creating different fixtures, and then by inhereting from other fixtures and changing some parameters. Both of those solutions were less than ideal (lots of code duplication and added complexity). Things got out of hand when we made the fixtures a templated class on a value, and created them by specifying the value as part of the texture name. Now it was a pain to debug and things were even less clear.

So we finally implemented it natively in CppUnitLite2 as fixtures with initialization parameters. We introduced a new macro, TEST_FP (test with fixture and parameters), which takes any parameters you want to pass to the fixture constuctor.

Now you can declare tests like this:

TEST_FP (VectorFixture, VectorFixture(10), MyTest) { }

We should prob ably simplify that to avoid repeating the fixture name, but that works fine for now.

Include cleanup

The original version of CppUnitLite was pretty casual about what headers it included. Unfortunately, that means that all test code were automatically exposed to a variety of standard headers, whether you wanted it or not. This version is a lot more strict, and the only header that will get pulled in from using it will be iosfwd, which is even a fairly lightweight one.

Memory allocations

We have very strict memory allocation wrappers around our tests, and if any memory leaks, it is reported as a failed test. Since checking for memory allocations is very platform specific, it is left out of the framework (although it could be added pretty easily in the platform-specific sections). In any case, some of the static-initialization time allocations were confusing the memory tracker, so we removed them completely. All it means is that we use an array to register tests instead of a std::vector and we cap the maximum number of tests to a fixed amount. If you need more than the default 5000 tests, go right ahead and add a few zeros.

New license

Previous versions of CppUnitLite were not released with an explicit license. To make things clear, I explicitly added the license text for the MIT license with an added restriction. The MIT license allows you to use the code in any project, commercial or otherwise, without any restrictions. The added restriction to the license prohibits its use in any military applications or projects with any military funding.

Ratings

How does this release of the CppUnitLite2 compare with respect to my initial set of features I used to rate the different unit tests frameworks?

• Minimal amount of work needed to add new tests. Nothing has changed there. Still passes with flying colors.
• Easy to modify and port. That has always been the strong suite of this framework.
• Supports setup/teardown steps (fixtures). Much improved now by creating them correctly around each test.
• Handles exceptions and crashes well. No changes there. Still one of the best I’ve seen.
• Good assert functionality. The check macros have been much improved to match the best frameworks out there.
• Supports different outputs. Yup. Still there.
• Supports suites. It still doesn’t. It just shows that I really haven’t needed this feature yet.

Future work

So, what’s next for CppUnitLite2? It’s really quite usable right now, but I could see a few changes happening in the next year, and maybe a few others will come up and surprise me.

Test registration is based on static initialization of global variables, which is not very reliable across compliers. For example, if you try to put the tests in a static library, MSVC will strip out the registration code leaving you with no tests. To get around that, we could introduce a custom build step in a scripting language that would generate a simple file with explicit registration of tests. This could also be used to collect tests from many different libraries into a single executable.

A couple of times it would have been convenient to have parameterized tests (I think that’s the correct term for it). Basically, to have tests that could run with different data. Something like this:

 TEST_PARAM(MyCustomTest, int a, int b) { CHECK (something); CHECK(something else); CHECK(yadda yadda); }  TEST(MyTest) { CALL_TEST(MyCustomTest, 10, 5); } TEST(MyTest) { CALL_TEST(MyCustomTest, 12, 7); }

The key point is that a test fails in the parameterized test, it should report the error correctly indicating where it came from.

This is actually pretty easy to implement with a couple of macros. We simply haven’t needed it more than a couple of times, so we haven’t bothered yet. But I imagine next time we run up against this we’ll bite the bullet and implement it.

Wrap-up

Charles had so much fun working with this framework, he decided to write his own from scratch. He made it available on his web site, so go check it out.

Thanks to High Moon Studios for being open and letting us release the changes we made and share them back with everybody. If you end up using the framework and you make any changes, drop me an email and I’ll add them for the next release.

C++ refactoring is hard. There’s no doubt about that. Just parsing C++ is hard, as I quickly found out when I wrote some scripts a couple of years ago to analyze the structure of a C++ program. Even with the help of Doxygen pre-processing, my analysis was full of holes and exceptions. Writing a robust program to correctly interpret and modify the structure of a C++ program is a nightmare come true.

Frankly, I had pretty much given up all hope of every having good automated refactoring tools for C++. My best hope was that new language designers keep refactoring in mind as a primary goal of their languages, so at least we’ll have those tools in whatever language we’re programming with in ten years.

So along comes SlickEdit version 10 with great promises of C++ refactoring. Surprisingly enough, that’s just one small bullet point completely hidden by a bunch of bullet-point items of no great interest or uniqueness (smart line selections? keyboard emulation? what year is this, 1992?). As a matter of fact, refactoring isn’t even mentioned in SlickEdit’s “cool features” web page! That’s not an auspicious beginning.

Fortunately, things get better. The installation for the Linux demo version of SlickEdit v10 worked without a hitch. The refactoring actions feature prominently in the Tools menu and the right-click context menu. I loaded a small project I was working on (just a handful of classes-nothing big) and started applying them one at the time.

Rename

This is the big one. If I can only have one refactoring, I’ll take this one. Being able to rename variables, functions, and classes takes care of 75% of the refactorings I do. A tool that could reliably do that would be a major productivity boost (reliably being the key word there).

How does SlickEdit’s rename work? Member variables were a dream to rename. But frankly, I can usually do that with “replace in file” and doing one edit in the header file, so that’s not very challenging. Still, good start.

Trying to rename member functions was not quite as successful. Sometimes it would work like a charm, and rename the function in the cpp file, the header file, and everywhere else it was called from. Sometimes however, it would look at my code and proclaim that there was nothing to rename. Nothing? I had just right clicked on top of the function to rename! Some other times it would rename the function in a couple of classes and totally miss where it was being used.

After some experimentation, it turns out that “Quick Rename” (which supposedly is not nearly as robust) worked time and time again without a single problem. That’s good I guess, but it left me with a very uneasy feeling.

Thumbs up for the user interface for refactoring too. As soon as you enter your refactoring and your parameters, you are presented with a list of the files that will be modified and a diff of each of them, so you can cancel the operation if something went horribly wrong. It certainly makes you feel a lot more in control.

Extract method

This is a fairly common refactoring. I actually thought it would be fairly complicated, but in every test I did, SlickEdit handled it admirably. The tricky part is to figure out what arguments the new function needs to take, and every time it figured it out correctly. You even get an option to name the new function and tweak the arguments it takes before applying the refactoring. Again, great user interface.

My only quibble with this refactoring is that the new methods are all made public. Uh? Would it have been so difficult to give us an option? If anything, I have just moved code from a member function to another member function in that class. By default it should be private, not public. Oh well, I just have to move one line. But it’s a bit annoying to have to tweak the header file by hand.

Modify parameter list

Another big one. Sometimes it’s quite painful to add a new parameter to a function that is called from dozens or hundreds of tests. So much so that it really discourages us from doing the “right” thing and is tempting to create a new function instead. Sometimes, after the second time you need to add a new parameter, we just create a struct with the parameters that the function takes and save some pain in the future. Not a particularly elegant solution, so this refactoring tool would come in very handy.

Unfortunately this one completely failed for me. Most of the time, trying to add a new parameter to a function would just come back and tell me there’s nothing to do. Same thing with rearranging the order of parameters. At best, once I got it to actually add the new parameter, it only changed the function and the header file, not the calling code. Doh!

Misc inheritance chain refactorings

SlickEdit has a set of refactorings intended to do operations in the inheritance chain: push down, pull up, extract super class, etc. I don’t work with big inheritance chains these days, so they’re not very useful to me and I didn’t test them at all. Actually, now that I think about it, the only inheritance I use are interfaces and mock objects. Then again, I suspect this would be extremely handy for manipulating legacy code (like some currently popular game engines).

Extract class

This seems like a really large refactoring. Extracting a whole class out of a set of code? Rename didn’t work consistently, and modify parameter list didn’t work at all. I certainly wouldn’t want to try something of this magnitude. Let’s get the other ones working before we jump on this one.

Convert local to field

This one seems marginally useful, and it’s not something I do in a regular basis. All it does is to prepend an m_ to the variable and add it to the header file. What if you don’t prefix your member variables with m_? Is there a way to turn that off? Also, the new member variable wasn’t added to the constructor(s) initialization list.

Move method

At first I thought that it was a totally pointless refactoring since it’s describing as moving a method from one class to another (I can’t remember when was the last time I did that). But it turns out it can be quite useful to move a non-member function into a class. That’s something I do sometimes when a helper function in an anonymous namespace starts needing more and more state from the object that uses it, and eventually it becomes clear that it wants to be a member function.

Move method will make it a static member function, and then you can follow that up with Convert static method to instance method and voila.

Encapsulate field

I’m sure some people will rave about this one, but I’m not too thrilled about it. It creates set/get methods of a member variable. Frankly, I don’t like set/get methods. I think that, most of the time, they’re a smell of bad design. I’m not sure I want to make it any easier for people to create them, so I would be happier without this refactoring tool at all. Besides, what’s with making them inline by default? At least give us an option to put them in the cpp file.

Replace literal with declared constant

This one takes a literal (magic number/string) and replaces with with a constant or a define. Now we’re really reaching the bottom of the barrel. Is it really faster to navigate a three-level menu to select this than to do it by hand? It’s good that you get an option to create the new constant as const, static const, or an evil #define, but it doesn’t place it in an anonymous namespace, even if one already exists in that compilation unit.

Create standard methods

This one will add a constructor, destructor, copy constructor, and assignment operator to the class. At first I thought it wasn’t a big deal. I already have macros that do that for new classes. Then I realized you can do this with an existing class, and it will take care of assigning all member variables. That’s pretty cool, but of limited practical use. I haven’t had to go back to many existing classes and add assignment operators. Still, I could see how it could be useful.

All in all, SlickEdit managed to impress me because I wasn’t really expecting functional C++ refactorings. On the other hand, it is clearly far from being robust enough to be relied upon for production work. Maybe the next version? For me, they just need to fix the rename and I’m almost there.

One cool aspect of SlickEdit is its powerful scripting language, where it has access to all the commands you can do from the menu. That means that I can write a script that, as part of what it does, kicks off some refactoring step. That’s extremely useful to adapt the refactorings to my own environment. One thing I didn’t see is whether the script language has access to the logical structure of the C++ program or whether it’s just limited to calling the refactoring function. It would be great to have more control and be able to create custom refactorings.

SlickEdit is a bit puzzling because it parses all the header files during each refactoring. And by all, I mean all. Not just the ones in your projects, but all the standard include files that you might have included. I don’t really understand why it does that. I’m clearly not going to change standard header files, so I think that limiting a refactoring only to files that are presently loaded in the project would be good. With my toy projects with a handful of classes, the refactorings were very fast (a second or two), but I fear how it might scale to a large project. Next time they have an improved version I’ll have to load it up with a large codebase and see how it fares.

It seems as if refactoring tools for C++ are just around the corner, but still tantalizingly out of reach. At this point I’m not ready to switch to SlickEdit based on the strength of its refactoring tools alone, although I’m still going to take a closer look at it. Right now I would prefer a standalone command-line refactoring tool that could be easily integrated in any environment. Anybody knows of any other decent C++ refactoring tools?

Yes, I do find assert useful. Yes, I’m doing full-out test-driven development. Yes, I’m a hardcore C++ programmer.

Before we even get started, let me recap what I see as the main reasons to use assert.

• Detect a program error or bug right away. The sooner the better. There’s nothing more frustrating than trying to track down bugs that occurred several function calls away, especially if they’ve unwound the stack and started a new series of calls. Even worse are errors that propagate and don’t become apparent until several frames later.Example: Creating a new character from a spwan point during gameplay should always work, so asserting on the pointer returned by the factory is a good idea.
• Document a decision, limitation, or assumption of particular functions. Nothing like calling a function and getting an assert right away to realize that’s not the way it was intended to be used.Example: A function that takes a pointer that can never be NULL and we don’t want to take a reference (maybe because the object is going to store and keep ownership of the pointer).
• Catch bad situations that will soon lead to trouble.Example: Asserting that the matrix passed to a rotate function is orthonormal. Everything will work fine for a little while if it isn’t, but soon things will degenerate if errors keep accumulating.

One key point about the use of assert is that it’s a message from the programmers to the programmers and nobody else. The idea is that whatever caused that assert is a top priority and should be fixed right away. An end user should never see an assert message because a) they don’t care and b) it means their program is toast.

This distinction between programming errors and “user” errors is particularly important in game development. User errors are anything that a user of the tools or game could cause just by using it. In our case that includes the content creators (designers and artists). The last thing we want to do is stop everything just because they typed the wrong file, or tried to load an old version of a model. Those errors need to be caught and dealt with. They are expected.

Programming errors are the ones that can only be caused by something done by a programmer in the code. As an example, in game development we constantly have functions that look like this: void Something::Update(float deltaTimeInSec). Unless you have a simulation system that can go both forwards and backwards, trying to call that function with negative seconds is nonsensical. An assert there tells the programmer that something either went horribly wrong, or they’re trying to use Update in a way it wasn’t intended. Clearly, no user of the program will ever be able to call Update with a negative time value just by using the program. So in this case, I do want to assert and have that fixed right away.

With that out of the way, let’s look at Len’s five major complaints about assert. Let’s take them one at the time.

1. The check will go away in a release build.

From my point of view, an assert is a way to tell a programmer (probably even myself in the future) that something has gone completely wrong. If an assert ever goes off, it’s a bug in the code and a programmer needs to fix it.

So what if asserts go away in release builds? We’re writing “shrink-wrapped” software, so that’s a good thing because there will be no programmers around with their handy debugger to fix the problem. If a user ever manages to come up with a way to cause the code to get in a bad state, then all we can do is pray. Especially in the case of console games, trying to capture traces and exception reports is not going to help anybody.

2. The assert is covering poor design.

I think this is really the crux of the discussion. Is assert really just a crutch for “poor design”? If you define good design as creating bullet-proof code that will work under any circumstance then it’s probably right. But you know, my job is not to write bullet-proof code. My job is to write whatever needs to be written to satisfy our customer stories every two weeks, and eventually ship a game with it.

This is very much in the agile development mindset. I’m not going to try to bomb-proof a piece of code for all foreseeable uses in the future. I’ll make sure it does what it needs to do now and leave it at that. Under this view, asserts are post-it notes saying “This code doesn’t even handle this situation. You either screwed up, or you better update the code to deal with it.”

Let’s go back to the Update() function. What are our options to make it work without assert?

• Implement Update() to go backwards in time as well as forwards. That will easily more than double development effort for something that will have no effect in the final product (unless you’re implementing a mechanic like Sands of Time, although I suspect that not even that game had a negative-time update).
• Make it physically impossible to pass negative numbers into the Update function. We could create a delta time class that cannot be manipulated in any other way than to have positive values. While this is clearly possible, I feel it’s a tradeoff between simplicity and “bullet-proofness.” I’ll take simplicity any day of the week for most situations.
• Have the update function check for negative values and return a failing value. That would be fine except that now everywhere we call Update we need to check for return values. Did you notice that Update(), the way it was declared, returned no values at all? That’s a function that is supposed to work and never fail. What if an Update() function fails? What do we do next? Quietly move on to the next one, or try to fix the system in some way? I’d rather slap the programmer with an assert and have him fix it right away than quietly continue to work while things are not working as expected.

So I see it more as a tradeoff between simplicity, time, and robustness. There’s no single correct answer, and it will depend on your particular situation. If you’re writing flight-control software, to hell with simplicity and time constraints, and go for 100% robustness and throw a few redundant systems for good measure in case of cosmic rays interfering with the hardware. In my case, turnaround time and simplicity trump absolute robustness.

Also, I admit this might not work in every environment. It works for us because we’re the only users of our own code. When something isn’t good enough, we make it good enough. That’s very different from writing a generic library for thousands of users to apply in their production code.

3. The assert makes it very hard to write a unit test for the failure condition.

I don’t understand where this comes from. Maybe with a crappy unit-test framework it’s a problem, but the unit-test framework I use deals with assertions just like any other exception, and fails the current test case. If your unit test framework barfs when your code tries to access an invalid pointer, you really need to look into getting a better one (I keep meaning to put up my version one of these days).

You should be able to hook up assert to anything you want (by using a custom assert or any other way you want). So the test runner first hooks up a special assert handler that throws a particular exception. Not only does this let us fail a test if an exception is triggered, but it even allows us to write tests to check that asserts are thrown. For example, I could write something like this:

 TEST (UpdateWithNegativeTimeAsserts) { Whatever stuff; CHECK_ASSERT(stuff.Update(-0.1f)); }

4. If the object is running without a user interface, then the assert’s dialog box may cause issues.

What dialog box? Maybe I’ve been doing game development too long, but every single project I’ve worked on has had a custom assert macro that gives us a lot more control than the standard one. Seriously, are people calling the standard assert() everywhere? No wonder they have issues.

Game Programming Gems 1 had a great introduction to using custom asserts by Steve Rabin with lots of great ideas (here’s an implementation based on his article). At the very least, you should be able to have full control of what you want the assert to do. A lot of the time that will simply be _asm int 3, but many other times you want to throw an exception, write out a log, send out an email, update a database, or even put up a dialog box to allow the user to bypass it. As a side note, I actually hate the option of continuing after an assert and I would never allow it in my own implementation. Assert means dead. Kaput. Done.

A custom assert takes five minutes to set up and will pay back in gold-pressed latinum in no time.

5. The assert is really just an “exploding comment.”

Yes, that’s completely true. It’s more like a programmer yelling in your ear “YOU JUST SCREWED UP!” But then again, that’s one of the things that make unit tests so great. They’re like comments that complain every time you change the behavior of the system in a way that violates what the tests are checking. Those are the best kind of “comments.”

That takes care of Len’s objections. Clearly, I don’t see any of them being an issue other than the second one, which is a matter of tradeoffs. Having said all that, I do use asserts less than I did before I used TDD. Unit tests take care of some of the things I would have checked with asserts in the past (did this loop really iterate over all the members that needed to be updated?), but unit tests still can’t protect against misuse of functions or simply unexpected situations.

So no, I’m not willing to give up on assert any time soon. It might be an ugly, rusty, oversized tool, but it still has a very definite spot in my toolbox.

Fast-forward a few months and one aborted attempt at implementing that build system, and there are now more unanswered questions than answers. It turns out that Jam and its successors were far from the perfect solution I had envisioned, so I was back to square one. I took this opportunity to look into all the build systems I had left out of the first article, plus all the ones that other people asked me about since then, to come up with a much more comprehensive evaluation.

The Contenders

Jam

Since Jam is the culprit that caused this second go-around, I figured I would start with it. What’s wrong with Jam? Isn’t it the fastest kid on the block? It still is the fastest, but when it comes to supporting new platforms or adding new settings to builds, it’s not the most straightforward of all systems.

Jam is a bit too complex for my tastes. Even after sitting for a while with the Jam tutorial (which is a horrible introduction, by the way), reading the skimpy documentation many times, and playing around with some sample Jam files, it was still very difficult to create non-trivial Jam files. At that point, my best allies were the Jambase and Jamrules files themselves. When it came time to start adding support for features such as precompiled headers, different build configurations, or Xbox 360 support, things got a lot harder very quickly.

Jam is a bit of a puzzle. Out of the “box” it claims to be able to have platform-independent Jam files, and that’s the source of the problem, because it makes it a lot harder to support new platforms. In my case, since I’m targeting a very specific set of platforms, I would much rather specify each build rule myself and have a simple, clean set, instead of a monster set of rules that applies to everything.

At the same time, Jam doesn’t support some really basic features, such as spaces in filenames. Not like I advocate having spaces in filenames, mind you, but that’s an unfortunate reality of the setup of many Windows computers. Yes, I know I can patch it, but this is where the next, and biggest, problem of Jam comes in.

Jam is dead.

Yes, Jam is completely dead. The Jamming mailing list has less traffic than a side road in Siberia during the winter. My attempts at asking there for help went completely unanswered (except for one kind soul who emailed me privately, apparently too embarrassed to post to a dead mailing list).

The fact that the latest official version of Jam dates back to 2002 should have been a good tipoff, but somehow I didn’t catch on to that at first. The nail in the coffin was when I asked where the latest set of patches, or the latest unofficial version of Jam, was located and nobody, *nobody*, answered.

So where is everybody who developed and used Jam? Oh, they must have gone to one of Jam’s successors, like BoostBuild. That’s good, right? Right?…

BoostBuild v2

BoostBuild v2 is promising on paper. It builds on top of Jam to add several much-needed features such as multiple configurations, correct filename parsing, etc. Or at least so it seems.

As soon as I start digging into it it’s very clear that BoostBuild is much more complex than Jam. It’s not like Jam was a walk in the park, but BoostBuild is close to unacceptable. I’ll take a Makefile any day of the week. Still, I would be willing to overlook that, bury myself deep in the documentation, and learn the system inside out if BoostBuild lived up to expectations: Jam speed with more features.

Unfortunately it doesn’t. It doesn’t just fall short either, it misses by several miles and destroys a friendly nearby town in the process. BoostBuild was the slowest of all the build systems I tested. Slower even than Scons! Talk about taking something small and fast and turning it into a slow, lumbering beast.

On the up side, I won’t have to spend months learning the overly complex system. I had to find a silver lining somewhere, right?

Is BoostBuild v1 any better? It could be. But frankly, after my experience with v2 I wasn’t in the mood to go check it out. Besides, it sounds like they’re going to phase it out. If I want a fast, dead build system, I’ll choose plain Jam.

Microsoft Visual Studio 2005

I had heard (from the comments in the previous articles and other posts on the internet) that Visual Studio 2005 had an improved build system. Even though it seems that C++ is getting the shaft and not being rolled in the MSBuild system for this release, apparently it was still improved somewhat. Since a beta version of Visual Studio 2005 fell in my lap at the GameFest conference earlier this year, I decided to give it a try.

Incremental build times were definitely improved, although they are still far from Jam times. The one-second delay for each project is reduced to a bit less than half a second. Better for sure, but I don’t see why it can’t be faster. Still, that’s getting in the range of being more usable.

One interesting addition is that, by default, Visual Studio 2005 will use multiple threads to compile C++ code. That will be a very welcome addition for full builds. Now I really need a quad-processor machine for our build server.

VCBuild

MSBuild, the Ant clone that Microsoft has been hailing as the next great thing to come, doesn’t yet support C++ builds. It’s a matter of priorities, I suppose. In the meanwhile, MSBuild will use the standalone VCBuild tool to compile Visual Studio solutions.

VCBuild promises to be a lightweight, command-line version of devenv.com, ideally suited for build servers. Indeed, VCBuild had slightly better build times than MSVC2005, which is a huge improvement over 2003. That actually puts VCBuild as the fastest build system after Jam and Make.

At this point, I was interested enough to think about starting to use VCBuild for all of our command-line builds (at the server and the quick verification build before submitting code). Unfortunately, I immediately ran into a couple of problems. The first one is clearly listed in the Readme, so I knew it was coming: VCBuild will not add a dependent project to the link list like MSVC does. This is actually a good thing. I always hated that dependencies implied linking in Visual Studio. Unfortunately, since that’s the default behavior, we are relying on it :-( It’s easy to change, but it’s just one small obstacle in the way.

The more serious problem was that VCBuild didn’t seem to parse all the project settings correctly (warning levels for example), so it created very different results than a build with devenv.com.

Finally, I was disappointed to see that VCBuild does not support multiple processors. I had started thinking of VCBuild as a standalone MSVC2005 project compiler, but that’s clearly not the case.

Fast Solution Build MSVC plugin

This one isn’t really a standalone build system like the others covered in this article. However, it directly addresses the problem of speeding up Visual Studio builds, so it was worth including it in. In the past, I had very mixed experiences with it, but after hearing some really good things about it from some co-workers, I decided to give it a whirl.

As promised, Fast Solution Build does speed up incremental Visual Studio builds to speeds similar to Visual C++ 6.0. That is to say, doing an incremental build with no changes on a large solution takes about a second, which is really how it should be. This time around, I didn’t have any problems with strangely broken builds or anything, so it seems pretty stable.

Unfortunately, Fast Solution Build is not without its set of problems. The first few problems are relatively minor. It doesn’t seem to play well at all with C#. If I ever make the mistake of attempting a fast build on a C# project, Visual Studio will start a build and never end. The only way to get out of that state is to kill the process and restart Visual Studio. I guess I just need to train my fingers not to press F7 in C# projects.

The other problem is more of a user problem than anything related to the plugin. In spite of its name, Fast Solution Build, it does not build a solution, but it builds the selected project. That alone caused more problems than anything because most people are used to doing solution builds, not project builds. It’s especially problematic if you want to have several unrelated projects build simultaneously (like in the case of unit tests projects and a game executable in addition to the libraries they depend on). More than once, we ended up checking in code that seemed to build fine, but broke when the server built the full solution.

The lack of a command-line interface also kills Fast Solution Build as a long-term solution. It might be OK while you’re working with the IDE, but you can’t use it for any command-line builds such as in the build server or for a quick sanity check we have before committing any code.

However, the biggest problem of all with Fast Solution Build is that it’s just a plugin to Visual Studio, so you’re left with many of the problems in Visual Studio itself, such as being hard to set up large solutions with lots of dependencies, having the visual representation mixed in with the real dependencies, suspect dependency checking (try deleting a file and then doing a build), or being tied to a single platform.

All in all, if you want a quick fix to your build times, Fast Solution Build is a good temporary band-aid provided you don’t work with C# much and you don’t have very complex solutions.

Scons

As I was wrapping up the new measurements, I noticed the announcement in for the new version Scons (0.96.91 pre-release). Since the last version was many moons ago, and since there was talk about improving Scons’ performance, I decided to give it a try.

The new version is indeed faster, but only by a tiny amount. Scons still lags far, far behind the curve in terms of performance. Unless the developers manage to squeeze performance improvements of several orders of magnitude, it’ll remain unusable as a build system for fast iterations.

Ant

Ant is one of the build systems I consciously left out of my first roundup. I had used it a bit in a build server, but I had the impression it was mostly intended as a “build sequencer,” with high-level actions like archiving files or emailing, and intended for full, nightly builds, rather for fast C++ incremental builds. It turns out that even though Ant was developed primarily with Java development in mind, it does have C++ support through some of the extra contribution tasks, with full dependency checking.

Another fact that put me off a bit from trying Ant was its level of verbosity. Writing XML files is anything but fun, and I much prefer the terseness of a Make file than the verbosity of an Ant file. Still, if a tool does the job well, I’ll put up with a lot.

Ant’s dependency checking was not bad, but not great either. Incremental build times with Ant were a bit worse then MSVS2005, but not by much. Frankly, that’s faster than I expected with my initial preconceptions.

Full build times on Windows were blazingly fast. So much so, that I was convinced I had configured something wrong and I was only doing half the build. It turns out that everything was fine, but Ant will send all cpp files to be compiled to the compiler at once, whereas other build systems will process them one at a time. Under Linux with gcc, this doesn’t make much of a difference, but in Windows using the MSVC compiler, it makes a huge difference. Other build systems should take note of that and take batching much more seriously.

Nant

Nant is Ant’s evil twin written with the .Net framework instead of Java. Frankly, I don’t really see the point of Nant when Ant is already a mature project and runs in many more platforms. To make matters worse, the Nant developers decided to make their syntax different from Ant. Was that really necessary? If at least it were a straight Ant clone, it would be easier to migrate from one to the other. Personally, I would have preferred to see all that energy put into improving Ant instead of creating a .Net clone.

Having said that, one of the early surprises of Nant was its native support for C++ compilation. In Ant it was a custom task, here it works out of the box.

Nant also has a good set of extra contributed tasks, including support for Perforce operations (although for a lot of the operations I wanted to do, I ended up having to fall back to command-line scripts because the tasks didn’t have enough parameters or support for exactly what I wanted to do).

Another potentially interesting feature of Nant is support for Visual Studio solution files. This could be very useful because it would move the dependency checking out of Visual Studio and into Nant. Unfortunately, I tried running it on the solutions at work and it failed miserably. It seems that lots of compiler options are not parsed correctly, so it was impossible to build our existing solutions. I also suspect that C++ isn’t their main target, so C++ project support is probably lagging behind C# and Visual Basic. Although I didn’t try it, I imagine that Nant would choke at trying to build an Xbox 360 solution file.

Just for kicks, I tried running it on Linux with Mono, but apparently it only works with the Microsoft cl.exe compiler, not with gcc.

Build times were almost identical to those of Ant. Nant does very good batching as well, so full builds were blazingly fast, but incremental builds were only middle-of-the-road.

Rake/Rant

I don’t do any Ruby development, so I had never heard of Rake until Ivan-Assen Ivanov told me about it. In for a penny, in for a pound, so I went ahead and ran the tests on Rake as well.

Rake is a build system written in Ruby (and I assume mostly intended for Ruby users as well). Like Scons, it brings the full power and expressiveness of a scripting language (Ruby in this case) to the build script. It looks pretty simple and clean. The Ruby syntax is a bit strange at first sight, but it’s easy to get used to (especially with lots of examples).

Unfortunately, Rake doesn’t have C++ dependency checking built in. I could have probably cobbled something together with makedepend, but that would take more Ruby-foo than I’m comfortable with.

Of course, all of this being open source, somebody, somewhere has had the same itch to scratch, and Rant was born. Apart from the cool name (after all, who doesn’t like to write Rantfiles?), Rant looks very similar to Rake. Don’t be fooled by their parallels to Make and [N]Ant. It’s much more Make meets Scons in Rubyland than a derivative of Ant.

One feature of Rant that I really liked was the good support for error messages. If the Rantfile had a problem in it, it would correctly identify the problem and point me to it. After working with Ant/Nant, that was a welcome change.

The build times were a bit of a mixed bag. Full incremental builds were pretty good, definitely in the top third of the build systems I looked at. Unfortunately, building a single library without any changes was extremely slow. It seems that Rant had to do a lot of work before it could even look at an individual library. If it weren’t because of that, I would be a lot more excited about it.

Make and MSVC2003

Nothing new in this front. For details, just read the first article. Everything there still applies.

The Method

I used the same test case and hardware as the last time. As a quick recap, the source code was made up of 50 static libraries with 100 classes each (2 files per class), 15 includes per cpp file to other files in that library, and 5 includes to files in other libraries.

The Python script generates a tree with a full code base for every build system along with all the necessary build files. Whenever the build files for two platforms are different, two separate trees are created. Thanks to Jim Tilander to writing the Python scripts to generate the BoostBuild files.

The computer was a P4 2.8GHz CPU with 1GB of RAM and a 7200 rpm hard drive. The Linux distro was Mandrake Linux 10.2 with the 2.6 kernel and Windows XP (without any antivirus running–which makes a huge difference!).

generate_libs.zip

The Results

Linux with gcc 3.4.1

Windows XP with MSVC 2003 compiler (except for the MSVC2005 run).

Update: Since several people asked, I timed Scons with MD5 checksum checking turned off in the hopes of getting a better time. I followed the advice described here. Just like last time, turning off MD5 checksum made it go slower if that makes any sense.

Using the option to indicate unchanged dependencies made it go a bit faster, but that’s not an option we can realistically use during development, since that would be changing constantly with the structure of our program. Still, the fact that without doing any dependency checking it’s still taking 47 seconds. What is it doing??

With multiprocessor machines being so common these days, a good build system needs to support parallel execution of tasks in multiple processors. Surprisingly, not all of the build systems compared here support it. Of the usable build systems, only Make, Jam, and MSVC2005 actually support parallel builds.

Conclusion

Jam still remains the best-performing build system of all by quite a lot. Unfortunately, the total lack of support and community around it make it very difficult to recommend. There has been some talk recently in the sweng-gamedev mailing list to do some work for build systems. Maybe we could take over the continued development of Jam. If that happened, I wouldn’t hesitate to recommend it. Until then, unless you’re a Jam expert or are willing to dig through user-created patches in the Perforce Public Depot, Jam is not a realistic option.

MSVS2005 has improved a lot from 2003, and along with VCBuild, it’s one of the best performing build systems under Windows for incremental links. Personally, I hate to be locked in to proprietary tools and single platforms, so I would opt for the next best systems, which are Ant/Nant. Pick your flavor of choice, they’re about the same.

MSBuild can be promising, especially once Microsoft rolls out their XNA Studio extensions specifically designed for asset building. It’ll only run in Windows, but at least they claimed that they’re making it general enough you can use for building any type of assets, not just their own formats. That’s definitely one to watch for the future.

If you’re comfortable with Make files, it still remains as a great option for very fast builds. You’ll probably want to wrap Make in a more comprehensive build system (such as Ant) for your build server, packaging builds, etc. But Make can be your everyday workhorse during development.

Whatever you do, stay far, far away from BoostBuild and Scons. Their performance is simply so horrible there’s no excuse to use them unless build times are of no importance to you at all.

I remain hopeful that we might be able to resurrect Jam. If you’re interested in contributing, contact me through email, comments on this page, or participate in the sweng-gamedev discussion directly

generate_libs.zip

I define an iteration cycle as the time elapsed between making a trivial change and being able to see the results of that change. In particular I’m concentrating on large-scale (around one million lines of code) C++ projects, which is representative of modern PC and console games today.

Of course, there’s more to fast iteration than just the speed of the build system. How quickly you’re able to get in the game and see the results is a big factor (another reason why I wouldn’t want to live without unit tests). The physical dependencies of your program are going to affect how quickly your code builds. And if you’re always working with a full game executable that takes forever to link, your iteration times are going to be shot (yet another reason to use unit tests!).

But let’s put that aside and concentrate on the build system itself.

Fast iteration is more than just about time and speed. It’s also about how you feel about the code and what you dare do with it. When things are slow and painful, you’re going to be a lot less likely to try new things, or fix one last thing to clean up the code, or refactor something out of a header file into its own module. Over time, this is going to accumulate into cruft, hacks, and unmaintainable code. It’s also about not breaking up the flow, the mental state you’re in while you’re writing software. Interruptions of more than just a few seconds are much more detrimental than their time value alone.

When you add test-driven development to the mix, fast iteration becomes even more crucial. With test-driven development, you end up doing micro-cycles of modify-compile-test, sometimes several times per minute. Unless you have very fast build times, you’re dead in the water.

Before we go any further, let’s crunch a few numbers. It’s not so much to show specific improvements, but to have as a reference point going forwards. Think about the project you’re currently working on. How long does it take to build when you modify a single cpp file (or even no files at all)? I’ve seen projects that took over two minutes to build, and anywhere between 30 seconds to one minute is fairly typical. Let’s say 30 seconds for this example. How often do you need to do a build? It’s not very fast, so maybe once every 5 minutes, 8 hours a day. That adds up to a staggering 48 minutes per day, or 10% of your full work day!

Now reduce the build time to two seconds instead. And to make things more interesting, let’s do a build every two minutes. That adds up to be 8 minutes per day, and, most importantly, they feel like almost instant builds, so they don’t bring you out of the flow state. That’s what our goal should be for a build system.

The Goals

As I started looking into different build systems (there are a lot of them out there!), I noticed that they have very different sets of goals, and a lot of them are fairly irrelevant to my particular needs.

As a game developer, I work with a varied, but limited number of platforms (most of which are not supported out of the box by build tools anyway). I’m also not planning on releasing the source code any time soon to let players compile the game in any platform, so I have no need to have a build system that automatically detects all the correct settings and does the right thing for any possible platform.

Some other build systems had useless features built in, such as access to version control or the ability to send emails. I consider those type of tasks to be totally beyond what the core build system should do, and I prefer to have those features in a wrapper system that does build/test/deployment of builds by calling the build system itself.

So what exactly do I want in a build system?

• Super-fast incremental builds (around two seconds). This is the key to fast iteration and I want this at almost any cost.
• Customizable. I’m going to be using unusual compilers and environments. I want to be able to easily set my own rules and actions and not be tied to any particular platform or compiler.
• Correctness. I want the build system to build the minimal amount of files and still do the right thing under most normal circumstances.
• Multiprocessor support. I also care about the speed of a full build, and with multiprocessor machines finally becoming popular, using multiple processors is a great way to speed up build times.
• Scalable. I want all the related source code to be tied to the same build system. I don’t want to create a separate build file for every minor tool so they build at a reasonable speed. I’d like to simply build any target and have the minimal amount of files rebuilt.

The Contenders

Visual Studio .NET

Most game developers doing Windows or Xbox development will be familiar with this build system. I’ve been stuck with it for many years, and while things were not great in Visual Studio 6.0 and earlier versions, it became decidedly unusable when it turned into “.NET”. I don’t know what happened, but I suspect that in their effort to cram all those languages under a single IDE they crippled the C++ build system even more.

My main gripe is how slow Visual Studio .NET is when doing an incremental build on a solution with many projects. It’s roughly a second per project. If you have 50 projects, there goes a full minute for nothing. That’s simply unacceptable, so I’ve always had to work around it by creating many solutions with the minimum amount of projects necessary, or keeping the dependencies in my head and forcing builds by hand. Alternatively you could throw all the code in a couple of projects instead of breaking it up into many different ones, but that’s like jumping off a cliff to avoid being stung by a bee.

Because Visual Studio mixes the build system with the visual representation of the files, large solutions are not only slow to build, but are a positive pain to work with, making browsing the source code extremely difficult.

The problems with Visual Studio don’t end there. The solution and project files are a pain to generate, they’re full of magical GUIDs and references to the registry, and they’re extremely verbose. The .NET framework offers an API to create those files, but the fact remains that they’re much more complex to create than any of the other build systems.

If you don’t generate the project files by hand, you soon enter configuration hell. Anybody who’s had to make sweeping changes to lots of projects with multiple configurations through the IDE will know what a painful process I’m talking about.

Some of my other complaints are not being able to easily set the build order for different projects (it appears to be determined by the order in which they appear in the solution file), or the fact that setting a dependency between two projects forces an implicit linking.

All in all, Visual Studio seems very well suited to small, toy projects. Projects with just a couple of libraries and a few thousand lines. Anything bigger than that and it becomes an exercise in frustration.

On the bright side, it is possible to use “makefile projects” in Visual Studio, which completely bypasses Visual Studio’s own build system and simply calls an external command to build the project. Visual Studio also offers a command-line line interface, so at least it is possible to do builds from the command line without launching the IDE. Finally, there are some third-party plug-ins that can speed up the dependency checking, which can help with some of the problems (unfortunately, FastSolutionBuild doesn’t seem to have a command-line interface, which renders it useless for my needs).

Other third-party add-ons like Incredibuild claim to speed up full builds, but they do so at the expense of incremental build speed, which I consider much more important.

Make

Make is the granddaddy of all the build systems. With a distinguished history of over 20 years, it has certainly proved its worth in the real world by building hundreds of thousands of projects over the years.

But make is far from perfect; otherwise there would be no need for other build systems. However, things are not as bad as people make them out to be, especially with modern versions of make (GNU make for example). The lack of portability is not an issue for us because we can just trivially write a new makefile, or a variant, for every platform we support.

The claims about make not being scalable are more serious. The article “Recursive Make Considered Harmful” certainly did much harm to make’s reputation. Specifically, that paper claims that it’s very hard to get the order of recursion right because using recursive make has no global project view. I think that’s only true if you’re dealing with self-modifying source code files or autogeneration of source code. If that’s not the case, I can’t see how the order of recursion can matter at all as long as the dependencies are met.

As for the claim that make is slow, let’s put that off until we compare it to the other build systems.

Personally, I really like make. It’s small, clean, and elegant. It does one thing and it does it very well. It’s easy to extend and modify. At first I was surprised that make didn’t do implicit dependency checking of C files (building a C file when an included header file changes), but it fits perfectly with the simplicity of make. It’s just a dependency graph with actions. If you don’t tell it about a rule, it won’t know about it. Fortunately, we can use tools like makedepend to generate the C dependencies with extreme ease.

One of my only gripes with main is the silly tab syntax. The fact that action commands have to be preceded by a tab character is awkward and out of place today, but it’s a small quirk I’m willing to adapt to. Fortunately gnu make’s error messages are very clear and it even asks “Did you forget to put a tab before the command?”

Jam

Jam was born an an improved make. It tried to keep all the good things about make and fix all the problems. And to a large extent, it succeeded.

Like make, Jam is small and easily portable. It deals better with inter-project dependencies by avoiding recursive Jam invocations (while still allowing individual sections to be built separately). The Jam language, even though it’s still fairly restrictive, is more expressive than make and it’s easier to write complex functionality.

One of the main differences from make is that Jam actually provides a fair amount of base functionality in its Jambase file. Jam out of the box knows about some of the most popular development environments and languages (including implicit dependency checking for C/C++ files), so it simplifies the build files for the simple cases. In the other cases, you can add your own rules and actions very easily.

I find it funny that while Jam fixes make’s weird tab requirement, it adds its own “space semicolon” weird command terminator (although I know some programmers who think that “space semicolon” is the one and true way). Either way, it really doesn’t matter since it’s such a small thing.

Jam also has spawned some forks that add extra functionality but are fully backwards compatible: FTJam and BoostJam.

Scons

Scons has been hailed as the next step in the evolution of build systems. It is supposed to be a much improved make-like system, not only written in Python, but using Python as the language used to define the build itself. Python is a fully general, object-oriented language, so it’s extremely expressive. It also has the advantage of being a well-established language with a great set of documentation, debuggers, and tools, which can make creating and debugging complex build scripts easier.

Scons also claims to be extremely accurate when it comes to determining what files need to be built. It doesn’t rely in the time stamp for a file, but it uses the MD5 signature instead (a type of checksum approach). Another very intriguing feature I didn’t get around to testing is the network cache of built object files.

Walking into this test I was a bit afraid of what I might find. I had read some reports of several people having problems with Scons performance on large data sets. However, the latest version (0.96.90), released just a couple of months ago, is supposed to have some performance improvements.

The Method

As a test, I decided to run each of the different build systems on the same codebase. Instead of using some real-world codebase, with its own set of quirks and problems (and the difficulty of easily building it with the different systems), I wrote a script to generate a simple C++ codebase. The code structure is based on what I expect to see in my own projects with many different projects. The physical dependencies in the generated codebase are extremely well contained, and header files never include other header files. Real code bases would have more complicated dependencies and would make the tendencies we see here even more exaggerated.

The specific parameters I used for this test were:

• 50 static libraries
• 100 classes (2 files per class, .h and .cpp) per library
• 15 includes from that library in each class
• 5 includes from other libraries in each class

This is by all accounts still a small or at most medium-sized codebase. A full game engine and tools can easily become much larger than this.

Thinking back, I really should have done the test with at least 100 libraries, not 50, because all my libraries have an extra associated project for unit tests. No big deal. I don’t think it would have changed the results very much. The important thing was to get enough code to make measurements noticeable (if we just build 10 files every build system is going to be really snappy).

For each of the build systems, I measured three operations:

• Full rebuild. Compiling the full source code for the first time. I didn’t expect this time to change much at all from build system to build system or even across platforms. I was quite wrong!
• Incremental build: Doing another build without any changes. This is the really interesting measurement that will tell us a lot about potential for fast iteration.
• Incremental build on a single library: Doing a build of a library without any changes.

I did the measurements in both Linux (2.6 kernel) and Microsoft Windows XP for different systems. Clearly some build systems only run in one platform (Visual Studio). But I decided to run some of the other build systems under Windows as well to provide a more fair comparison.

The specific hardware I ran these tests in is not as important since all we’re comparing are their relative merits. But for the curious it’s a P4 2.8 GHz CPU with hyperthreading, 2GB of fast RAM, and a 7200 rpm EIDE hard drive. The most important part is that I had enough memory to prevent thrashing.

GNU make, Jam, and Scons all support parallel builds. While it won’t speed up incremental builds any, this can reduce the time for full builds dramatically. Since this test was done in a single-CPU machine (and the primary measure was incremental builds), I restricted all the builds to use a single process.

The Results

We can make lots of very interesting observations from this table.

First of all, it confirms what I had seen all along, that Visual Studio is horrible for incremental builds with many projects. My off-the-cuff estimate of one second per project ended up being extremely accurate (54 seconds for 50 projects). That’s simply not acceptable for me.

As I feared, Scons, failed the fast iteration test as well. It actually ended up being slower than Visual Studio in all accounts, even for individual library rebuilds. It might do the “right” thing under all conditions, but frankly, that’s not a price I’m willing to pay to get absolutely correct results. I really don’t think I encounter any situation in everyday work in which Scons would do the right thing and make or Jam wouldn’t.

At this point, I was afraid that I just wasn’t going to be able to get the type of iteration I wanted out of file-based, compiled languages. Fortunately that’s not the case. Both make and Jam do a great job and fall in the range of what I consider acceptable (around a couple of seconds).

There are two interesting observations to be made about full build times from the chart above. First of all, Scons with g++ under Linux is twice as slow as Jam or make for a full rebuild. I find that extremely surprising. Although I guess that’s the extra minute of dependency checking plus some extra overhead of its own. I tried some of the suggestions to get faster Scons builds (by trading off accuracy for speed), but they just improved incremental build times by a couple of seconds. Clearly, Scons needs to do some catching up before it can play with the big boys.

The other one is comparing g++/Linux with VC++/Windows XP. Jam is over twice as slow with VC++/Windows XP than it is with g++ under Linux. Is it Windows XP or is it Visual C++? I don’t know. It would be interesting to try the experiment with g++ or some other compiler under Windows and see if the times are reduced at all. I suspect the Windows file system might have something to do with that.

Conclusion

This little experiment cleared up a lot of doubts for me. I’m ready to ditch Visual Studio as a build system and replace it completely with Jam or make. Make is a simpler but Jam probably edges it out because it’s a bit nicer, it doesn’t have any recursive problems, and the default functionality is pretty handy. It’s hard to go wrong with either one.

Since most programmers still expect to work from within the Visual Studio IDE, you can easily create a “makefile” project type and hook it up to the build system of your choice.

One interesting idea that came up during this research in the Scons mailing list is that of a background process that monitors which files change and updates dependency graphs on the fly. So whenever you initiate a build, all the work has already been done and the build can start right away. A variation on this idea that has been brought up in some TDD mailing lists is that of the build system not just computing dependencies in the background, but actually attempting to compile the code and run the unit tests in the background. If any of the tests fail, they can even be highlighted in the source code editor. Sort of like an on-the-fly, smart code checker on steroids.

Of course, we could also choose a language that has much smaller build times. I haven’t worked on a large-scale C# project yet, but the small tools I’ve created have impressed me with how fast the iteration can be. The same can be said for scripting languages such as Python or Lua. Unfortunately, we’re stuck with C++ for the foreseeable future in game development, so we better learn to deal with it the best we can.

For now, I’ll be happy to stick with Jam and two-second incremental builds. Let’s start jamming!

generate_libs.py

The same can be said for large-scale C++ development. Most of the books and articles out there deal with class hierarchies, object-oriented design, and mind-bending template tricks. However, when it comes down to it, a solid physical structure and good code layout will go a long way towards making all programmers productive. When the milestones near and the pressure piles on, a badly structured C++ codebase is likely to be as fatal as the cold, Russian winter was to Napoleon’s army.

I’ve already written before about how important the physical layout of a codebase can be ([1] and [2]), so I’m not going to go over it again. This is going to be about how to use one of the many tools at our disposal to make working with a large codebase more bearable: pre-compiled headers.

Benefits and drawbacks

The only benefit of pre-compiled headers is build speed. Nothing more, nothing less. But we’re not talking about a measly 10-15% build speed improvement. Pre-compiled headers can easily improve build times by an order of magnitude or more depending on your code structure. Clearly, it’s a technique worth exploring.

The major benefit is when doing full rebuilds, but it can also help when building individual files. So it will help with the bane of the C++ programmer: iteration time (it won’t do anything to help with link times, though).

What’s not to like about pre-compiled headers? Several things, it turns out:

• Using a single set of pre-compiled headers exposes more symbols than necessary for many modules. That can lead to increased physical and logical dependencies.
• Modifying a header that is part of the pre-compiled headers set will trigger a full rebuild.
• Pre-compiled headers are not supported in every environment (although these days most compilers seem to support them to some extent).
• Bloated pre-compiled headers can slow things down. This drawback is only from hearsay and minor anecdotal personal evidence. It sort of makes sense (with the pre-compiled header file getting huge), but I would like to see some hard data. Has anybody measured this?

What’s the best way to deal with the drawbacks in order to be able to take full advantage of the super-speedy compile times?

How do pre-compiled headers work?

A C++ compiler operates on one compilation unit (cpp file) at the time. For each file, it applies the pre-preprocessor (which takes care of doing all the includes and “baking” them into the cpp file itself), and then it compiles the module itself. Move on to the next cpp file, rinse and repeat. Clearly, if several files include the same set of expensive header files (large and/or including many other header files in turn), the compiler will be doing a lot of duplicated effort.

The simplest way to think of pre-compiled headers is as a cache for header files. The compiler can analyze a set of headers once, compile them, and then have the results ready for any module that needs them.

I won’t get into the specifics on how to set up pre-compiled headers; that’s very environment-specific. You might want to start with Bruce Dawson’s great article on pre-compiled headers on Visual C++, or the GNU documentation on pre-compiled headers with gcc.

In very general terms, you need to specify one file (here we’ll call it PreCompiled.h) as containing the pre-compiled headers. Anything included in that file will then become part of the pre-compiled headers cache. Then, you just need to make sure to include PreCompiled.h in each module you want to use the pre-compiled headers from. One word of warning: Visual C++ ignores all lines in the cpp file before #include “PreCompiled.h”, so it’s a good habit to make sure that’s the first line of code in each file.

Organizing your code

Unless you’re dealing with very small code bases, you should really split up your code into multiple libraries. Not only will you get the most benefit from pre-compiled headers that way, but you’ll reap a variety of other benefits. Then set up each library with one set of pre-compiled headers.

The big question now is, what should go in the pre-compiled headers? You can put anything you want, really. But by observing a few simple guidelines, you can maximize your build times, which is what this is all about.

• Add “expensive” includes to the pre-compiled header file. “Expensive” headers are the ones that cause a cascade of other includes. Including these every time you compile a file can be quite time consuming. Some of the usual suspects are windows.h, STL headers, single includes for whole APIs, etc.
• Add headers that are included from many different files, even if they’re not very expensive. A header file that is included from 50 different files is almost as bad as a header file that is included once but causes 50 includes of its own. Actually, a simple file that is included in many cpp files is preferable to an expensive one included only once because recompiling one of those many cpp files is going to be fast even if the file is not in the pre-compiled headers, so iteration is still relatively fast.
• Don’t put any headers from the library itself in the pre-compiled headers. The only exception to this is if you have a header in your library that happens to be included everywhere (which is probably a sign that something is wrong anyway). Otherwise, every time you modify a header that was included in the pre-compiled headers file, you’ll cause a full rebuild.

So clearly, the best candidates to add to the pre-compiled header list are expensive includes that happen many times. Those are the ones that are going to get us the big speedup in build times.

At the same time, we don’t want to blindly add every header used by our library. That will make regenerating the pre-compiled headers slightly slower, but, most importantly, will cause all the symbols in those headers to be available to the whole library, which is something undesirable if we’re trying to keep dependencies to a minimum. There’s also the potential issue of pre-compiled header bloat, but I need to confirm that.

Since those guidelines can be applied automatically, I decided to write a script to report the headers that would most benefit from being in the pre-compiled headers for a particular project. Instead of trying to parse the C++ code and find the chains of includes (which is not that hard but requires dealing with include paths, define statements, and almost implementing the full C pre-processor), the script uses the output of building the project with the option to show all includes (-H with gcc, /showincludes in Visual C++). The includes are conveniently formatted with indentation corresponding to the level at which they were included, so it’s very easy for the script to determine which includes are expensive.

The script also accepts a string to ignore all includes which include that string in their path. That way you can easily eliminate includes from the library itself from being recommended to be part of the pre-compiled headers. Unfortunately, gcc outputs the relative path that was used to get to the include, which makes it harder to filter out specific libraries and might result in the same header being included in different ways. To prevent that, you might want to process the output first and automatically change all the paths to be absolute instead of relative (the script can’t do it because there are no guarantees you’re running it on the same platform as the build, let alone in the same directory of the same machine).

Right now the script can parse the output created from gcc and Visual C++, but it should be really easy to extend to any other compiler just by changing the regular expressions it uses to parse the include outputs.

As an example, I ran it the script on the output of building one of the libraries in our game engine at High Moon. These are the results:

 Counting includes in includes.txt
 Cpp files:  38 (Header file, score, times included, includes caused by the header)
 ('file1.h', 11590, 190, 60)
 ('file2.h', 532, 532, 0)
 ('file3.h', 401, 401, 0)
 ('file4.h', 384, 4, 95)
 ('file5.h', 330, 33, 9)
 ('file6.h', 319, 11, 28)
 ('file7.h', 228, 228, 0)
 ('file8.h', 155, 155, 0)
 ('file9.h', 151, 151, 0)
 ('file10.h', 105, 35, 2)
 ('file11.h', 105, 105, 0)
 ('file12.h', 101, 101, 0)

Each include file is reported with a score, the number of times it is included in the program, and the number of other includes it causes. The score is the most important value about each file, which is simply the product of the number of times a file is included and the number of includes that file causes (plus one to take itself into account). So the higher the score, the more expensive an include is.

Interestingly, at the top of the list we have an extremely expensive include that should absolutely be added to the pre-compiled header list. That alone should make a significant positive impact in build times. The rest of the top files listed could be added, although their impact would be much lower.

As another example, I ran the script on the results of building my current mp3 player, Amarok.

 Counting includes in /usr/local/src/amarok-1.2.3/amarok/src/out.txt
 Cpp files:  764 (Header file, score, times included, includes caused by the header)
 ('/usr/lib/qt3//include/qglobal.h', 46620, 630, 73)
 ('/usr/lib/qt3//include/qmap.h', 3740, 110, 33)
 ('/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../../include/c++/3.4.1/vector', 3182, 37, 85)
 ('/usr/lib/qt3//include/qstring.h', 2900, 116, 24)
 ('/usr/include/kconfigskeleton.h', 2280, 30, 75)
 ('/usr/include/klineedit.h', 1955, 17, 114)
 ('/usr/lib/qt3//include/qobject.h', 1605, 107, 14)
 ('/usr/include/kapplication.h', 850, 34, 24)
 ('/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/include/stddef.h', 808, 808, 0)
 ('/usr/lib/qt3//include/qwinexport.h', 784, 784, 0)
 ('/usr/lib/qt3//include/qptrlist.h', 560, 112, 4)
 ('/usr/include/sys/types.h', 546, 42, 12)
 ('/usr/local/include/taglib/taglib.h', 432, 6, 71)
 ('/usr/lib/qt3//include/qnamespace.h', 420, 84, 4)
 ('/usr/include/kaction.h', 396, 22, 17)
 ('/usr/include/kguiitem.h', 216, 27, 7)
 ('/usr/include/kdirlister.h', 189, 9, 20)
 ('/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/include/limits.h', 184, 184, 0)
 ('/usr/include/kfiledialog.h', 182, 7, 25)
 ('/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../../include/c++/3.4.1/fstream', 176, 4, 43)
 ('/usr/include/klistview.h', 161, 23, 6)
 ('/usr/include/kactioncollection.h', 156, 13, 11)
 ('/usr/include/kdiroperator.h', 144, 3, 47)
 ('/usr/include/time.h', 143, 143, 0)
 ('/usr/include/kpopupmenu.h', 143, 13, 10)
 ('/usr/include/bits/wordsize.h', 142, 142, 0)
 ('/usr/lib/qt3//include/qdir.h', 140, 28, 4)
 ('/usr/lib/qt3//include/private/qucomextra_p.h', 129, 43, 2)
 ('/usr/lib/qt3//include/qmetaobject.h', 129, 43, 2)
 ('/usr/lib/gcc/i586-mandrake-linux-gnu/3.4.1/../../../../include/c++/3.4.1/iostream', 126, 3, 41)

This is a larger project than the previous example (764 cpp files), so the potential gains in build time are also much higher. The top 10 headers are all included in many files as well as quite expensive, so they would all be great candidates to add to a pre-compiled header list. Considering that it took about 10 minutes to build on my 3GHz machine, I’d say it would definitely benefit from some judicious use of pre-compiled headers.

Multiplatform development

A lot of C++ compilers support pre-compiled headers nowadays (gcc, Visual C++, and Codewarrior for sure). Unfortunately, there are still compilers and platforms out there without pre-compiled header support that we need to deal with.

Usually, the code will build the same in a platform without pre-compiled header support, but it will be much, much slower because not only do we not have the caching effect of pre-compiled headers, but we’re also including more headers in every compilation unit as part of the PreCompiled.h file. So if we’re not careful, all the build speedups we gained using pre-compiled headers are going to come back and slow down builds in platforms without pre-compiled header support by a huge factor.

The best way to deal with this is to be able to turn on the pre-compiled headers on and off. For example, your pre-compiled header file might look like this:

 #ifdef USE_PRECOMPILED_HEADERS
 #include <string>
 #include <sstream>
 #include <iostream>
 #endif

For platforms without pre-compiled header support, just don’t define USE_PRECOMPILED_HEADERS. Clearly, for this to work every cpp file needs to include all the header files it needs to, independently of whether they’ve been included in PreCompiled.h or not. This is not a bad habit to get in anyway, because it makes dependencies more explicit and it doesn’t slow the build down any (because all those header files are already in the pre-compiled headers, so they’re free).

Having the ability to turn it on and off also allows us to do builds sometimes without relying on pre-compiled headers. This might be useful if we want to verify that files can compile on their own or if we want to generate lists of includes to feed to the script described earlier to find good candidates for adding to the pre-compiled header list.

Maintenance

Pre-compiled headers are mostly fire and forget. You set them up once with the big offenders and leave them be. However, the more a library changes, the more it might benefit from an update of the pre-compiled headers. Whenever a library feels like it’s building slowly, you should see if there are any obvious headers that should be added to the pre-compiled header list.

That’s one reason I like to display the build times for each library (in my case, I like to include unit test times as well). It’s too easy to let a library build more and more slowly over time without ever realizing it. It’s like the proverbial frogs taking a hot bath. But if you have hard data, you can see that the library is now taking 10 seconds to build, but last week it was only taking 6 seconds (yes, it doesn’t seem like much, but when you have 50+ libraries it quickly adds up!). For bonus points, I want my automated build system to keep historical data of build times and display a plot over time, which will make zeroing in build slowdowns much easier.

If you’re interested in maintaining compatibility with other platforms, I’d recommend doing a build with pre-compiled header support turned off every so often (once a night or even once a week), and make sure everything works fine. While you’re at it, take a pass at the code with PCLint, and you’ll be horrified at how many dangerous things you missed.

There really are almost no excuses not to use pre-compiled headers in a large C++ project. If you’re not using them, you’re doing yourself a disservice. And if you’re already using them, you might be able to tweak them a bit and squeeze an extra 10-20% speedup out of your builds. Whatever you do, make sure to keep those build times low to have as fast an edit-build-run cycle as possible.

list_precomp.py

Chances are that if you’re reading this, you already know what Xoreax’s Incredibuild is. If not, here’s the quick summary: Incredibuild is a distributed build system for Microsoft Visual C++ that recruits other machines in the network to parallelize your build; it claims build time reductions of up to 90%. There is no doubt that Incredibuild will speed up most full builds of C++ projects using Microsoft Visual C++ to varying degrees. I’m not going to argue that. But is using Incredibuild in your project really a good idea?

Let’s start with the positives. Incredibuild does work as advertised and can produce significant speed ups from distributing your build. It is well polished and it has clearly been tested in real-world application development. It has good load balancing, you won’t notice your machine being used as a slave, it has the concept of backup coordinators, it minimizes network usage, etc, etc. It does have a few technical glitches, but I’ll get to those later. All in all, it’s pretty robust and well-thought-out product, and there are clearly lots of people happy with it (interestingly, the list includes many game developers).

So, what’s not to like? When I started using Incredibuild last year, I was initially floored by the results. A project that would take 45 minutes to compile, would only take about 5-8 minutes with Incredibuild. Wow! That’s what I call a “significant” speedup!

However, the more I used it, the more it started getting in the way and the more it made me think about all the areas where it was lacking. Be warned that this doesn’t pretend to be a rigorous review with objective data to back up what I’m saying. These are the issues I’ve encountered using Incredibuild without trying to become an expert at it. It’s very likely that are ways around some of the things I’m going to complain about. If so, feel free to let me know in the comments below. Maybe I’ll change my mind and I’ll become an Incredibuild fan after all.

Iteration time

Slow incremental builds. I don’t mean incremental linking (that comes next), I mean non-full builds, like the case where I change a single line in a cpp file and want to generate a new executable. I found that using Incredibuild for such builds is significantly slower than the regular build in MSVC (incidentally, this FAQ claims exactly the opposite). I like to compile my code after every single change. Actually, I like to compile it and run all the tests after every single change. I’ll often even do it while I’m thinking. It’s a (good) habit you get in after you’ve been doing test-driven development for a while. Anything that slows down that cycle is a major impediment. So what if I do incremental builds with the regular MSVC build and just use Incredibuild for full rebuilds? Read on…

Problems mixing and matching object files. Mixing and matching object files created by Incredibuild and MSVC seems to work… sometimes. I know in their FAQ they indicate a few changes you need to do to the debug information settings to prevent some errors. I’m talking about bigger issues. As in, the executable generated is total garbage and crashes right away. Usually doing a full rebuild with Incredibuild fixes it (sit and wait 5-8 minutes). When this is caused just by changing a single cpp file, it gets old very quickly and you start questioning the benefits you’re getting in the first place.

I found that I have similar problems matching object files with incremental linking turned on (this time in the form of errors during the build). The only way I’ve managed to more or less reliably mix both types of object files is by turning off incremental linking completely. Of course, link times are often the bottleneck of the change-build-test cycle, being as high as a full minute or more when you’re dealing with the whole game. Incremental linking, as long as you’re only modifying the executable project, is really a god-send and can reduce a 1+ minute link time to just a couple of seconds. Again, turning it off to be able to use Incredibuild seems a step backwards.

No edit and continue. I admit it, I’ve never been a fan of edit and continue. But some people swear by it, and it’s yet another feature you can’t have turned on with Incredibuild. Interestingly, their FAQ fails to mention the words “incremental linking” or “edit and continue” anywhere. (Actually, they had it under “known issues”).

Technical issues

Tied to Microsoft Visual Studio. Are you developing for other platforms than Windows or Xbox? Are you using other compilers than Visual C++? Then you’re out of luck. You’ll have to come up with a totally different system to build your code for those platforms. You can’t use the same build back end because Incredibuild is totally tied to MSVC. I don’t know exactly why, I just hope they had a very good technical reason and it wasn’t to get prettier integration with colorful progress graphics. At least it is possible to run Incredibuild from the command line (still using MSVC of course), which is essential for build machines. Update: msew just pointed out that version 2.20 has beta support for the Inteal compiler. Good to hear that.

Uses Microsoft Visual Studio’s build system. If I could rip one thing from Visual Studio, it would be the completely broken build system. I don’ t know what they were thinking when they built it, and I have a very hard time believing anybody at Microsoft uses it for complex projects. It seems a great quick way to get toy projects off the ground, but it doesn’t scale well at all to large projects (very much like those GUI wizards, but people seem to put up with them anyway). In a real build system I want to have multiple targets (and combinations of those targets), I want to be able to enforce build order, I want to be able to choose not to link a project just because I make it dependent on another project, I want to be able to have over a hundred different projects in an environment and not bog down because of it, I want to be able to view and change build settings easily across projects. In other words, I want primarily something like make, Jam, or Scons. Fortunately, MSVC allows you to have makefile projects, but Incredibuild doesn’t support them. This is particularly important to me because I would like to use a common back end for all the different platforms, and I’d love to have a single build file that I can use to build any project or platform combination.

Failing to build correctly. Every so often, a build with Incredibuild will completely fail because some dependency check failed along the way (this seems to happen particularly when changing header files outside of the project itself, like middleware or other external APIs). To be fair, this seems to happen just as frequently in MSVC with regular builds using pre-compiled headers, so you probably can’t blame Incredibuild for it. It certainly doesn’t fix the problem though.

Builds are done differently than local ones. I haven’t investigated this one thoroughly, but Incredibuild uses slightly different rules than regular, local builds. As a result, it is possible to set up a build with custom rules that builds correctly in Incredibuild, but fails as a regular build. If you’re trying to mix and match the two, that can’t be good news. I think the source of the problem is that Incredibuild runs custom build rules locally as the first step of the build, and then distributes the rest. Regular local builds will call the custom build rule when they get to the file in question, so you can see how there’s a slight potential for problems there (this, incidentally, is not idle nit picking, but happened in a real project).

Performance

Network issues can affect performance significantly. For a while, Incredibuild kept having all sorts of problems communicating across the network and was causing builds to either take forever or never complete (some slave machine would just never return a result or timeout). I don’t know if that was caused by glitches in our network, too many people using it, or bugs in Incredibuild, but it was highly annoying. I suppose you’re bound to have that happen in any distributed system, but I would expect them to always be able to fall back to local compilation and not take any longer than a non-distributed build.

How much faster than a regular full build? This is the big question. I have seen with my own eyes that using Incredibuild on a project without physical insulation and carefully managed dependencies, and without precompiled headers, can make the build be many, many times faster. But how does it compare to a project that has been carefully managed, with well-thought-out precompiled headers? As an experiment, I retrofitted a large codebase to use precompiled headers. The awful physical dependencies were still there, and I’m sure the precompiled headers were less than ideal. Still, build times went down from 45 minutes to something like 10 minutes. Incredibuild took 5-8 minutes in that same codebase. Throw a faster machine with multiple cores (the future of PCs), and a decent build system that can use multiple threads (not MSVC, go figure), and I think you can get similar build times without any of the drawbacks.

Architectural issues

Here is one argument that cuts both ways.

Programmers will often try to avoid making changes that cause full rebuilds (to avoid spending 30+ minutes waiting for a result), but they end up doing it by hacking things in the wrong place: duplicating code, declaring externs by hand on the wrong file, etc. Ideally, you shouldn’t have many files that can cause a full rebuild (if you do, you have a big problem), but if you need to make a change there, you really should do it. Incredibuild can ease the pain of a full rebuild and prevent the hack, leading to a cleaner codebase and architecture.

On the other hand, having Incredibuild around can cause developers to throw caution to the wind and think they don’t have to worry about physical dependencies anymore. As a result, they end up with the blob antipattern. Ironically, after a few months of that, they’ll probably end up with really slow build times again (because almost every file causes a full rebuild).

Is the time spent in managing and minimizing physical dependencies worth it? Absolutely! Low physical dependencies also means low logical dependencies. Even if build times were instant, you would still be well advised to pay very close attention to dependencies and minimize them (not just for reuse or maintenance, but even during the development of the game, since it makes it much easier to refactor or change any part of your program).

Alternatives

What are the alternatives to Incredibuild then? Are we stuck with really slow builds?

• Manage your physical dependencies. Be as careful as you can about them, and you’ll be way on your way towards minimizing build times. Use the pimpl idiom in any header that is used extensively throughout the codebase.
• Use precompiled headers. Stay away from that option of “automatic precompiled headers” in MSVC because it does more harm than good. Use Bruce Dawson’s paper as a starting point about precompiled headers. Also, precompiled headers are supported across the board for most popular C++ compilers: MSVC, gcc, Codewarrior, etc, so you can get a lot of benefits from a single approach.
• Get a fast dual-core machine and use multiple threads to build. You can’t do that with MSVC, but man, Jam and Scons can certainly do it. I stress the fast CPU part. In the past, I ran some tests, and, assuming you have enough memory, compilation time was not limited by disk space but by CPU speed. Once you start using multiple threads it’ll put more stress on the disk so it’ll probably be a good idea to get fast hard drives as well.
• distcc. I admit I’ve only tested distcc in toy sample projects, but it’s exactly what I would like on paper: a fairly generic distributed build system. It comes with gcc support out of the “box,” but it can be extended to support other compilers. It uses a very clean way of distributing the builds, by running the preprocessor locally and sending the output file to be compiled into an object file. I really don’t know how it scales for a real-world project, but I’d like to learn more about it.
• Scons’ network cache. Scons can transparently cache object files on the network, so if you ever try to build a file that is already in the cache, you’ll get it instantly. This can be a reasonable solution to avoiding full rebuilds after one person (or the build machine) has done it once.

Conclusion

If you’re stuck with some legacy code that takes forever to compile, don’t hesitate to get Incredibuild. It’ll do wonders for your project and will cut down your build times by a huge amount. If you’re developing on a new code base, or your build times are short already, I would recommend against using it. For the project I’m starting now, we’re starting without Incredibuild (even though we already have the licenses, so it’s not a money issue) and we’re carefully managing physical dependencies and using precompiled headers. If at some point it grows very large and we see benefits from Incredibuild, we might switch, but until then, we’re a lot better off without it.

Kyle found that using internal include guards was significantly slower than using #pragma once with the C++ compiler in Microsoft Visual Studio 2003 and 2005 beta. That just flies in the face of the measurements I had done before and what other people had reported. How was it possible?

The first thing that struck me when looking at Kyle’s results is that he’s using an extremely pathological case: 200 header files, each of them including all 200 headers. Still, the point is to measure header include performance, so maybe that was fine. More on that later.

The first thing I did was to write a quick script to generate a set of header files and main file given some parameters. I took the chance to write in Python, which displaced Perl as my favorite scripting language a few months ago. The script will generate a certain number of header files, using either no guards, internal guards, external guards, or pragma statements. In order to be able to compile no guards (and to keep things a bit more realistic), I actually had each header file include all the header files with numbers higher than itself. Feel free to download the script and play with it if you want to run the measurements yourself.

Since I’m running Linux at home, I first ran the experiment with gcc 3.4.1. First I tested no guards, but I had to keep it to a set of 20 header files to avoid taking forever with an astronomical number of includes (they really add up combinatorially!). As expected, even a 20x20 half-matrix of includes too a while to compile: 2+ minutes. The same set of headers with internal include guards was just a blip at 0.15s. So far, so good. That’s what I expected.

Now I cranked things up to 200 headers like Kyle had done. Here comes the first surprise of the day: gcc blows up trying to parse that many includes. I get a “#include nested too deeply” error. Digging through the gcc documentation I wasn’t able to find any way to increase that depth. That made me realize that having include chains of 200+ headers is completely unrealistic. I knew that it was artificially large, but it’s probably so by an order of magnitude at least.

Still, just for the test, I decided to go ahead. It turns out it was choking very near the end, so doing a run with 195 headers worked just fine. The results were what I would have expected. A bit better actually, and certainly much better than the results Kyle saw in Visual Studio: all the runs (internal guards, external guards, and pragma) took about the same time (0.07 seconds), and each of them included exactly 195 headers. No more, no less. That’s exactly how I would expect the compiler to behave.

gcc 3.4.1 (Linux with 2.6 kernel). 195 headers.

• Internal guards: 0.07s
• External guards: 0.09s
• Pragma directive: 0.07s

At work, I’m not as lucky, and I have to use Windows and Microsoft Visual Studio, so I ran the second set of tests there. The results confirmed what Kyle reported: The internal guards were very slow (14+ seconds), the external guards where blazingly fast (0.37 seconds), and the pragma directive was somewhere in between (9.6 seconds).

Microsoft Visual Studio 2003. 195 headers.

• Internal guards: 14.7s
• External guards: 0.37s
• Pragma directive: 9.57s

I was pretty amazed to see that. It seems like a major flaw in the Visual C++ compiler, doesn’t it?

To round out the tests, since I had the Metrowerks PS2 compiler handy, I decided to run the same set of tests with it. It turns out that it completely blew up, complaining of includes nested too deeply with the project with 195 headers. To my surprise, I had to lower the number of includes to 30 in order to be able to compile it at all. Anything over 30 would cause it to blow up.

That made me think again about the worst cases I would see in a typical project, and I realized that having over 30 includes deep at once is probably very, very rare, even if you’re using STL or Boost, which make heavy use of header files.

Just for the sake of completeness, I decided to run the tests with just 30 includes and see if I could discern any patterns from the results. It turns out that Metrowerks is pretty slow overall, but at least the differences between the three approaches are minimal.

Metrowerks PS2 compiler v3.0. 30 headers.

• Internal guards: 0.60s
• External guards: 0.46s
• Pragma directive: 0.49s

I ran the same set of tests with Visual Studio and it showed the external includes being the clear winner, but internal and pragmas being almost the same. I wonder if Visual C++ uses some dynamic algorithm that avoids having a fixed hard limit on include depth at the cost of runtime performance. I’ll take a fixed depth and flawless behavior like gcc any day personally.

Microsoft Visual Studio 2003. 30 headers.

• Internal guards: 0.48s
• External guards: 0.14s
• Pragma directive: 0.41s

gcc was so fast with such a tiny project that I couldn’t reliably measure it. But if the 195 includes was taking 0.07 seconds, you can guess how fast it churned through just 30 header files.

Conclusion

It does seem that Microsoft Visual Studio has some major problems with includes in pathological situations. On the other hand, it also seems that those situations are completely unrealistic and will never happen in a real code base. For a more realistic situation (30x30 half matrix), internal guards and pragmas are the same, and external guards have somewhat of an edge.

gcc behaved like a real champion by being super fast and efficient no matter what technique you threw at it. Way to go! The Microsoft compiler certainly could stand some improvement in that area.

Metrowerks was the slowest of the bunch, but it dealt with all the different techniques just fine for a relatively small set of tests.

Overall, there should be no real difference between using internal guards and pragma directives (which happily confirms some of the measurements we had done in real code bases). So stick with internal guards, which are standard and work on any compiler, but if adding a #pragma once directive surrounded by conditionals (because it’s not standard and it’s not supported in all the compilers) in addition to the internal guards would make you sleep better, go for it. External guards might have an edge with Visual Studio, but the pain and potential trouble of using external guards clearly outweighs any speed benefits gained by using them.

generate_includes.py

-———

One of the topics I’ve been meaning to get to for quite a while is the applicability of test-driven development in games. Every time the topic comes up in conversations or mailing lists, everybody is always very curious about it and they immediately want to know more. I will get to that soon. I promise!

In the meanwhile I’m now in the situation that I need to choose a unit-testing framework to roll out for my team at work. So, before I get to talk about how to use test-driven development in games, or the value of unit testing, or anything like that, we dive deep into a detailed comparison of existing C++ unit-testing frameworks. Hang on tight. It’s going to be a long and bumpy ride with a plot twist at the end.

If you just want to read about a specific framework, you can go directly there:

• CppUnit
• Boost.Test
• CppUnitLite
• NanoCppUnit
• Unit++
• CxxTest

Overview

How do we choose a unit-testing framework? It depends on what we’re going to do with it and how we’re going to use it. If I used Java for most of my work, the choice would be easy since JUnit seems to be the framework of choice for those working with Java. I don’t hear them arguing over frameworks or proposing new ones very frequently, so it must be pretty good.

Unfortunately that’s not the case with C++. We have our XUnit family member, CppUnit, but we’re clearly not happy with that. We have more unit-testing frameworks than you can shake a stick at. And a lot of teams end up writing their own from scratch. Why is that? Is C++ so inadequate for unit testing that we have trouble fitting the XUnit approach in the language? Not like it’s a bad thing, mind you. Diversity is good. Otherwise I would be stuck writing this under Windows and you would be stuck reading it with Internet Explorer. In any case, I’m clearly not the first one who’s asked this question. This page tries to answer the question, and comes up with some very plausible answers: differences in compilers, platforms, and programming styles. C++ is not exactly a clean, fully supported language, with one coding standard.

A good way to start is to create a list of features that are important given the type of work I expect to be doing. In particular, I want to be doing test-driven development (TDD), which means I’m going to be constantly writing and running many small tests. It’s going to be used for game development, so I’d like to run the tests in a variety of different platforms (PC, Xbox, PS2, next-generation consoles, etc). It should also fit my own personal TDD style (many tests, heavy use of fixtures, etc).

The following list summarizes the features I would like in a unit-testing framework in order of importance. I’ll evaluate each framework on the basis of these features. Thanks to Tom Plunket for providing a slightly different view on the topic that helped me to re-evaluate the relative importance of the different features.

1. Minimal amount of work needed to add new tests. I’m going to be doing this all the time, so I don’t want to do a lot of typing, and I especially don’t want to do any duplicated typing. The shorter and easier it is to write, the easier it’ll be to refactor, which is crucial when you’re doing TDD.
2. Easy to modify and port. It should have no dependencies with non-standard libraries, and it shouldn’t rely on “exotic” C++ features if possible (RTTI, exception handling, etc). Some of the compilers we have to use for console development are not exactly cutting edge. To verify this one, I created a set of unit tests using each library under Linux with g++. Since most of the tests are written with Windows and Visual Studio in mind, it’s not a bad initial test.
3. Supports setup/teardown steps (fixtures). I’ve adopted the style recommended by David Astels in his book Test Driven Development: A Practical Guide about using only one assertion per test. It really makes tests a lot easier to understand and maintain, but it requires heavy use of fixtures. A framework without them is ruled out immediately. Bonus points for frameworks that let me declare objects used in the fixture on the stack (and still get created right before the test) as opposed to having to allocate them dynamically.
4. Handles exceptions and crashes well. We don’t want the tests to stop just because some code that was executed accessed some invalid memory location or had a division by zero. The unit-testing framework should report the exception and as much information about it as possible. It should also be possible to run it again and have the debugger break at the place where the exception was triggered.
5. Good assert functionality. Failing assert statements should print the content of the variables that were compared. It should also provide a good set of assert statements for doing “almost equality” (absolutely necessary for floats), less than, more than, etc. Bonus points for providing ways to check whether exceptions were or were not thrown.
6. Supports different outputs. By default, I’d like to have a format that can be understood and parsed by IDEs like Visual Studio or KDevelop, so it’s easy to navigate to any test failures as if they were syntax errors. But I’d also like to have ways to display different outputs (more detailed ones, shorter ones, parsing-friendly ones, etc).
7. Supports suites. It’s kind of funny that this is so low in my priority list when it’s usually listed as a prominent feature in most frameworks. Frankly, I’ve had very little need for this in the past. It’s nice, yes, but I end up having many libraries, each of them with its own set of tests, so I hardly ever need this. Still, it certainly would be nice to have around in case it starts getting slow to run the unit tests at some point.

Bonus: Timing support. Both for total running time of tests, and for individual ones. I like to keep an eye on my running times. Not for performance reasons, but to prevent them from getting out of hand. I prefer to keep running time to under 3-4 seconds (it’s the only way to be able to run them very frequently). Ideally, I’d also like to see a warning printed if any single test goes over a certain amount of time.

Easy of installation was not considered a priority; after all, I only have to go through that once—it’s creating new tests that I’m going to be doing all day long. Non-commercial-friendly licenses (like GPL or LGPL) are also not much of an issue because the unit test framework is not something we’re going to link to the executable we ship, so they don’t really impose any restrictions on the final product.

Incidentally, during my research for this article, I found that other people have compiled lists of what they wish for in C++ unit-testing frameworks. It’s interesting to contrast that article with this one and make a note of the differences and similarities between what we’d like to see in a unit test framework.

Ideal Framework

Before I start going over each of the major (and a few minor) C++ unit-testing frameworks, I decided I would apply the philosophy behind test-driven development to this analysis and start by thinking what I would like to have. So I decided to write the set of sample tests in some ideal unit-testing framework without regard for language constrains or anything. In the ideal world, this is what I would like my unit tests to be like.

The simplest possible test should be trivial to create. Just one line to declare the test and then the test body itself:

TEST (SimplestTest)
{
    float someNum = 2.00001f;
    ASSERT_CLOSE (someNum, 2.0f);
}

A test with fixtures is going to be a bit more complicated, but it should still be really easy to set up:

SETUP (FixtureTestSuite)
{
    float someNum = 2.0f;
    std::string str = "Hello";
    MyClass someObject("somename");
    someObject.doSomethng();
}

TEARDOWN (FixtureTestSuite)
{
    someObject.doSomethingElse();
}

TEST (FixtureTestSuite, Test1)
{
    ASSERT_CLOSE (someNum, 2.0f); someNum = 0.0f;
}

TEST (FixtureTestSuite, Test2)
{
    ASSERT_CLOSE (someNum, 2.0f);
}

TEST (FixtureTestSuite, Test3)
{
    ASSERT_EQUAL(str, "Hello");
}

The first thing to point out about this set of tests is that there is a minimum amount of code spent in anything other than the tests themselves. The simplest possible test takes a couple of lines and needs no support other than a main file that runs all the tests. Setting up a fixture with setup/teardown calls should also be totally trivial. I don’t want to inherit from any classes, override any functions, or anything. Just write the setup step and move on.

Look at the setup function again. The variables that are going to be used in the tests are not dynamically created. Instead, they appear to be declared on the stack and used directly there. Additionally, I should point out that those objects should only be created right before each test, and not before all tests start. How exactly are the tests going to use them? I don’t know, but that’s what I would like to use. That’s why this is an ideal framework.

Now let’s contrast it to six real unit-testing frameworks that have to worry about actually compiling and running. For each of the frameworks I look at the list of wanted features and I try to implement the two tests I implemented with this ideal framework. Here is the source code for all the examples.

CppUnit is probably the most widely used unit-testing framework in C++, so it’s going to be a good reference to compare other unit tests against. I had used CppUnit three or four of years ago and my impressions back then were less than favorable. I remember the code being a mess laced with MFC, the examples all tangled up with the framework, and the silly GUI bar tightly coupled with the software. I even ended up creating a patch to provide console-only output and removed MFC dependencies. So this time I approached it with a bit of apprehension to say the least.

I have to admit that CppUnit has come a long way since then. I was expecting the worst, but this time I found it much easier to use and configure. It’s still not perfect, but it’s much, much better than it used to. The documentation is pretty decent, but you’ll have to end up digging deep into the module descriptions to even find out that some functionality is available.

1. Minimal amount of work needed to add new tests. This is one of the major downfalls of CppUnit, and, ironically, it’s the highest-rated feature I was looking for. CppUnit requires quite a bit of work for the simplest possible test.

// Simplest possible test with CppUnit
#include <cppunit/extensions/HelperMacros.h>
class SimplestCase : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE( SimplestCase );
    CPPUNIT_TEST( MyTest );
    CPPUNIT_TEST_SUITE_END();
protected:
    void MyTest();
};

CPPUNIT_TEST_SUITE_REGISTRATION( SimplestCase );  

void SimplestCase::MyTest()
{
    float fnum = 2.00001f;
    CPPUNIT_ASSERT_DOUBLES_EQUAL( fnum, 2.0f, 0.0005 );
}

3. Easy to modify and port. It gets mixed marks on this one. On one hand, it runs under Windows and Linux, and the functionality is reasonably well modularized (results, runners, outputs, etc). On the other hand, CppUnit still requires RTTI, the STL, and (I think) exception handling. It’s not the end of the world to require that, but it could be problematic if you want to link against libraries that have no RTTI enabled, or if you don’t want to pull in the STL.
4. Supports fixtures. Yes. If you want the objects to be created before each test, they need to be dynamically allocated in the setup() function though, so no bonus there.

#include <cppunit/extensions/HelperMacros.h>
#include "MyTestClass.h"
class FixtureTest : public CPPUNIT_NS::TestFixture
{
    CPPUNIT_TEST_SUITE( FixtureTest );
    CPPUNIT_TEST( Test1 );
    CPPUNIT_TEST( Test2 );
    CPPUNIT_TEST( Test3 );
    CPPUNIT_TEST_SUITE_END();
protected:
    float someValue;
    std::string str;
    MyTestClass myObject;
public:
    void setUp();
protected:
    void Test1();
    void Test2();
    void Test3();
};

CPPUNIT_TEST_SUITE_REGISTRATION( FixtureTest );  

void FixtureTest::setUp()
{
    someValue = 2.0;
    str = "Hello";
}  

void FixtureTest::Test1()
{
    CPPUNIT_ASSERT_DOUBLES_EQUAL( someValue, 2.0f, 0.005f );
    someValue = 0;
    //System exceptions cause CppUnit to stop dead on its tracks
    //myObject.UseBadPointer();
    // A regular exception works nicely though myObject.ThrowException();
}

void FixtureTest::Test2()
{
    CPPUNIT_ASSERT_DOUBLES_EQUAL( someValue, 2.0f, 0.005f );
    CPPUNIT_ASSERT_EQUAL (str, std::string("Hello"));
}

void FixtureTest::Test3()
{
    // This also causes it to stop completely
    //myObject.DivideByZero();
    // Unfortunately, it looks like the framework creates 3 instances of MyTestClass
    // right at the beginning instead of doing it on demand for each test. We would
    // have to do it dynamically in the setup/teardown steps ourselves.
    CPPUNIT_ASSERT_EQUAL (1, myObject.s_currentInstances);
    CPPUNIT_ASSERT_EQUAL (3, myObject.s_instancesCreated);
    CPPUNIT_ASSERT_EQUAL (1, myObject.s_maxSimultaneousInstances);
}

6. Handles exceptions and crashes well. Yes. It uses the concept of “protectors” which are wrappers around tests. The default one attempts to catch all exceptions (and identify some of them). You can write your own custom protectors and push them on the stack to combine them with the ones already there. It didn’t catch system exceptions under Linux, but it would have been trivial to add with a new protector. I don’t think it had a way to easily turn off exception handling and let the debugger break where the exception happened though (no define or command-line parameter).
7. Good assert functionality. Pretty decent. It has the minimum set of of assert statements, including one for comparing floating-point numbers. It’s missing asserts for less than, greater than, etc. The contents of the variables compared are printed to a stream if the assert fails, giving you as much information as possible about the failed test.
8. Supports different outputs. Yes. Has very-well defined functionality for “outputters” (which display the results of the tests), as well as “listeners” (which get notified while the tests are happening). It comes with an IDE-friendly output that is perfect for integrating with Visual Studio. Also supports GUI progress bars and the like.
9. Supports suites. Yes.

Overall, CppUnit is frustrating because it’s almost exactly what I want, except for my most wanted feature. I really can’t believe that it takes so much typing (and duplicated typing at that) to add new tests. Other than that, the main complaint is the need for RTTI or exceptions, and the relative complexity of the source code, which could make it challenging to port to different platforms.

Update: I’ve revised my comments and ratings of the Boost.Test framework in light of the comments from Gennadiy Rozental pointing out how easy it is to add fixtures in boost.

I’m a big fan of Boost, but I have to admit that it wasn’t until about a year ago that I even learned that Boost was providing a unit testing library. Clearly, I had to check it out.

The first surprise is that Boost.Test isn’t exclusively a unit-testing framework. It also pretends to be a bunch of other things related to testing. Nothing terribly wrong with that, but to me is the first sign of a “smell.” The other surprise is that it wasn’t really based on the XUnit family. Hmmm… In that case, it had better provide some outstanding functionality.

The documentation was top notch. Some of the best I saw for any testing framework. The concepts were clearly explained, and had lots of simple examples to demonstrate different features. Interestingly, from the docs I saw that Boost.Test was designed to support some things that I would consider bad practices such as dependencies between tests, or long tests.

1. Minimal amount of work needed to add new tests. Almost. Boost.Test requires really minimal work to add new tests. It’s very much like the ideal testing framework described earlier. Unfortunately, adding tests that are part of a suite requires more typing and explicit registration of each test.

#include <boost/test/auto_unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>  

BOOST_AUTO_UNIT_TEST (MyFirstTest)
{
    float fnum = 2.00001f;
    BOOST_CHECK_CLOSE(fnum, 2.f, 1e-3);
}

3. Easy to modify and port. It gets mixed marks on this one, for the same reasons as CppUnit. Being part of the Boost libraries, portability is something that they take very seriously. It worked flawlessly under Linux (better than most frameworks). But I question how easy it is to actually get inside the source code and start making modifications. It also happens to pull into quite a few supporting headers from other Boost libraries, so it’s not exactly small and self-contained.
4. Supports fixtures. Boost.Test eschews the setup/teardown structure of NUnit tests in favor of plain C++ constructors/destructors. At first this threw me off for a loop. After years of being used to setup/teardown, and a fairly complex suite setup, I didn’t see the obvious ways of using fixtures with composition.Now that I’ve tried it this way I’ve come to like it almost better than setup/teardown fixtures. One of the great advantages of this approach is that you don’t need to create fixture objects dynamically, and instead you can put the whole fixture on the stack.On the downside, it’s annoying to have to refer to the variables in the fixture through the object name. It would be great if they could somehow magically appear in the same scope as the test case itself. Also, it would have been a bit cleaner if the fixture could have been setup on the stack by the BOOST_AUTO_UNIT_TEST macro instead of having to explicitely put it on the stack for every test case.

#include <boost/test/auto_unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>
#include "MyTestClass.h"  

struct MyFixture
{
    MyFixture()
    {
        someValue = 2.0;
        str = "Hello";
    }
    float someValue;
    std::string str;
    MyTestClass myObject;
};

BOOST_AUTO_UNIT_TEST (TestCase1)
{
    MyFixture f;
    BOOST_CHECK_CLOSE (f.someValue, 2.0f, 0.005f);
    f.someValue = 13;
}  

BOOST_AUTO_UNIT_TEST (TestCase2)
{
    MyFixture f;
    BOOST_CHECK_EQUAL (f.str, std::string("Hello"));
    BOOST_CHECK_CLOSE (f.someValue, 2.0f, 0.005f);
    // Boost deals with this OK and reports the problem
    //f.myObject.UseBadPointer();
    // Same with this
    //myObject.DivideByZero();
}  

BOOST_AUTO_UNIT_TEST (TestCase3)
{
    MyFixture f;
    BOOST_CHECK_EQUAL (1, f.myObject.s_currentInstances);
    BOOST_CHECK_EQUAL (3, f.myObject.s_instancesCreated);
    BOOST_CHECK_EQUAL (1, f.myObject.s_maxSimultaneousInstances);
}

6. Handles exceptions and crashes well. This is one of the aspects where Boost.Test is head and shoulders above all the competition. Not only does it handle exceptions correctly, but it prints some information about them, it catches Linux system exceptions, and it even has a command-line argument that disables exception handling, which allows you to catch the problem in your debugger on a second run. I really couldn’t ask for much more.
7. Good assert functionality. Yes. Has assert statements for just about any operation you want (equality, closeness, less than, greater than, bitwise equal, etc). It even has support for checking whether exceptions were thrown. The assert statements correctly print out the contents of the variables being checked. Top marks on this one.
8. Supports different outputs. Probably, but it’s not exactly trivial to change. At least the default output is IDE friendly. I suspect I would need to dig deeper into the unit_test_log_formatter, but I certainly didn’t see a variety of preset output types that I could just plug in.
9. Supports suites. Yes, but with a big catch. Unless I’m missing something (which is very possible at this point–if so make sure to let me know), creating a suite requires a bunch of fairly verbose statements and also requires modifying the test runner itself in main. Have a look at the example below. Couldn’t that have been simplified to the extreme? It’s not a big deal as this is my least-wanted requirement, but I wish I could label all the test cases in one file as part of a suite with a simple macro at the beginning of the file. Another minor shortcoming is the lack of setup/teardown steps for whole suites, which could come in really handy (especially if suite creation were streamlined).

#include <boost/test/unit_test.hpp>
#include <boost/test/floating_point_comparison.hpp>
using boost::unit_test::test_suite;   

struct MyTest
{
    void TestCase1()
    {
        float fnum = 2.00001f;
        BOOST_CHECK_CLOSE(fnum, 2.f, 1e-3);
    }

    void TestCase2()
    {}
};

test_suite * GetSuite1()
{
    test_suite * suite  = BOOST_TEST_SUITE("my_test_suite");
    boost::shared_ptr instance( new MyTest() );
    suite->add (BOOST_CLASS_TEST_CASE( &MyTest::TestCase1, instance ));
    suite->add (BOOST_CLASS_TEST_CASE( &MyTest::TestCase2, instance ));
    return suite;
}

#include <boost/test/auto_unit_test.hpp>
using boost::unit_test::test_suite; 

extern test_suite * GetSuite1();  

boost::unit_test::test_suite* init_unit_test_suite( int /* argc */, char* /* argv */ [] )
{
    test_suite * test = BOOST_TEST_SUITE("Master test suite");
    test->add( boost::unit_test::ut_detail::auto_unit_test_suite() );
    test->add(GetSuite1());
    return test;
}

Boost.Test is a library with a huge amount of potential. It has great support for exception handling and advanced assert statements. It also has other fairly unique functionality such as support for checking for infinite loops, and different levels of logging. On the other hand, it’s very verbose to add new tests that are part of a suite, and it might be a bit heavy weight for game console environments.

CppUnitLite has a funny story behind it. Michael Feathers, the original author of CppUnit, got fed up with the complexity of CppUnit and how it didn’t fit everyone’s needs, so we wrote the ultra-light weight framework CppUnitLite. It is as light on the features as it is on complexity and size, but his philosophy was to let people customize it to deal with whatever they need.

Indeed, CppUnitLite is only a handful of files and it probably adds up to about 200 lines of very clear, easy to understand and modify code. To be fair, in this comparison I actually used a version of CppUnitLite I modified a couple of years ago (download it along with all the sample code) to add some features I needed (fixtures, exception handling, different outputs). I figured it was definitely in the spirit that CppUnitLite was intended, and if nothing else, it can show what can be accomplished by just a few minutes of work with the source code.

On the other hand, CppUnitLite doesn’t have any documentation to speak of. Heck, it doesn’t even have a web site of its own, which I’m sure is not helping the adoption rate by other developers.

1. Minimal amount of work needed to add new tests. Absolutely! Of all the unit-test frameworks, this is the one that comes closest to the ideal. On the other hand, it could be the fact that I’ve used CppUnitLite the most and I’m biased. In any way, it really fits my idea of minimum amount of work required to set up a simple test or even one with a fixture (although that could be made even better).

#include "lib/TestHarness.h"  

TEST (Whatever,MyTest)
{
    float fnum = 2.00001f;
    CHECK_DOUBLES_EQUAL (fnum, 2.0f);
}

3. Easy to modify and port. Definitely. Again, it gets best of the class award in this category. No other unit-test framework comes close to being this simple, easy to modify and port, and at the same time having reasonably well separated functionality. The original version of CppUnitLite even had a special lightweight string class to avoid dependencies on STL. In my modified version I changed it to use std::string since that’s what I use in most of my projects, but the change took under one minute to do. Also, using it under Linux was absolutely trivial, even though I had only used it under Windows before.

4. Supports fixtures. This is where the original CppUnitLite starts running into trouble. It’s so lightweight that it doesn’t have room for many features. This was an absolute must for me, so I went ahead and added it. I’m sure it could be improved to make it so adding a fixture requires even less typing, but it’s functional as it stands. Unfortunately, it suffers from the problem that objects need to be created dynamically if we want them to be created right before each test. To be fair though, every single unit-test framework in this evaluation has that requirement. Oh well.

#include "lib/TestHarness.h"
#include "MyTestClass.h"   

class MyFixtureSetup : public TestSetup
{
public:
    void setup()
    {
        someValue = 2.0;
        str = "Hello";
    }
    void teardown()
    {}
protected:
    float someValue;
    std::string str;
    MyTestClass myObject;
};   

TESTWITHSETUP (MyFixture,Test1)
{
    CHECK_DOUBLES_EQUAL (someValue, 2.0f);
    someValue = 0;
    // CppUnitLite doesn't handle system exceptions very well either
    //myObject.UseBadPointer();
    // A regular exception works nicely though myObject.ThrowException();
}  

TESTWITHSETUP (MyFixture,Test2)
{
    CHECK_DOUBLES_EQUAL (someValue, 2.0f);
    CHECK_STRINGS_EQUAL (str, std::string("Hello"));
}   

TESTWITHSETUP (MyFixture,Test3)
{
    // Unfortunately, it looks like the framework creates 3 instances of MyTestClass
    // right at the beginning instead of doing it on demand for each test. We would
    // have to do it dynamically in the setup/teardown steps ourselves.
    CHECK_LONGS_EQUAL (1, myObject.s_currentInstances);
    CHECK_LONGS_EQUAL (3, myObject.s_instancesCreated);
    CHECK_LONGS_EQUAL (1, myObject.s_maxSimultaneousInstances);
}

6. Handles exceptions and crashes well. The original CppUnitLite didn’t handle them at all. I added minor support for this (just an optional try/catch). To run the tests without exception support it requires recompiling the tests with a special define turned on, so it’s not at slick as the command-line argument that Boost.Test features.

7. Good assert functionality. Here is where CppUnitLite really shows its age. The assert macros are definitely the worst of the lot. They don’t use a stream to print out the contents of their variables, so we need custom macros for each object type you want to use. It comes with support for doubles, longs, and strings, but anything else you need to add by hand. Also, it doesn’t have any checks for anything other than equality (or closeness in the case of floating-point numbers).

8. Supports different outputs. Again, the original only had one type of output. But it was very well isolated and it was trivial to add more.

9. Supports suites. Probably the only framework that doesn’t support suites. I never really needed them, but they would probably be very easy to add on a per-file basis.

CppUnitLite is as barebones as it gets, but with a few modifications it hits the mark in all the important categories. If it had better support for assert statements, it would come very close to my ideal framework. Still, it’s a worthy candidate for the final crown.

I had never heard of NanoCppUnit until Phlip brought it up. From reading the feature list, it really appeared to be everything that I wanted CppUnitLite to be, except that it was better and ready to work out of the box.

The first point against NanoCppUnit is the awful “packaging” of the framework. If you thought that CppUnitLite was bad (not having a web page of its own), well, at least you could download it as a zip file. For NanoCppUnit you actually have to copy and paste the five files that make up the framework from a web page. I’m not kidding. That makes for some “lovely” formatting issues I might add. The documentation found in the web page wasn’t exactly very useful either.

In any case, I continued my quest to get a simple test program up and running with NanoCppUnit. Out of the box (or out of the web page, rather) it’s clearly aimed only at Windows platforms. I thought it would be trivial to fix, but changing it required more time than I thought at first (I personally gave up when I started getting errors buried three macros deep into some assert statement). Unlike CppUnitLite, the source code is not very well structured at all, full of ugly macros everywhere, making it not trivial to add new features like new output types. Unless I’m totally mistaken, it even looks like it has sample code inside the test framework itself. Eventually I had to give up on running it under Linux, so my comments here are just best guesses by looking at the source code.

1. Minimal amount of work needed to add new tests. I think so. I’m not sure it’s possible to create a standalone test that is part of a global suite, but at least creating a suite doesn’t require manual registration of every test. This is (probably) the simplest possible test with NanoCppUnit.

struct MySuite:  TestCase { };  

TEST_(MySuite, MyTest)
{
    float fnum = 2.00001f;
    CPPUNIT_ASSERT_DOUBLES_EQUAL(fnum, 2.0f, 0.0001);
}

3. Easy to modify and port. Not really. Windows dependencies run deeper than it seems on the surface. The code is small, but it’s messy enough that it’s a pain to work with. I’m sure it can be ported with a bit of effort though since it’s so small.
4. Supports fixtures. Yes. Setup and teardown calls very similar to the modified version of CppUnitLite.
5. Handles exceptions and crashes well. No idea since I wasn’t able to run it. I see some try/catch statements in the code, but no way to turn them on or off. Probably no better than CppUnitLite.
6. Supports different outputs. Not really. Everything is hardwired to use a stream that sends its contents to OutputDebugString() in Windows. I think the default output text is formatted to match the Visual Studio error format.
7. Good assert functionality. Yes. Good range of assert statements, including floating point closeness, greater than, less than, etc.
8. Supports suites. Yes. I don’t know what’s involved in just running a single suite though. Not a big deal either way.

One of NanoCppUnit’s unique features is regular expression support as part of its assert tests. That’s very unusual, but I can see how it could come in handy. A few times in the past, I’ve had to check that a certain line of code has some particular format, so I had to sscanf it, and then check on some of the contents. A regular expression check would have done the job nicely.

Unfortunately, NanoCppUnit doesn’t really live up to the standards of other frameworks. Right now it feels too much as a work in progress, with too much missing functionality and not clearly structured code.

The further along we get in this evaluation, the less Xunit-like the frameworks become. Unit++’s unique feature is that it pretends to be more C++-like than CppUnit. Wait a second, did I hear that right? More C++ like? Is that supposed to be a good thing? Looking back at my ideal test framework, it really isn’t very much like C++ at all. Once I started thinking about that topic I realized that there really is no reason at all why the tests framework itself needs to be in C++. The tests you write need to be in the language of the code being tested, but all the wrapper code doesn’t. That’s a point that the next, and final, testing framework will drive home.

So, what does it mean to be more C++ like? No macros for a start. You create suites of tests by creating classes that derive from suite. That’s the same thing we were doing in most other frameworks, really, but it was just happening behind the scenes. It really doesn’t help me any to know that that is what I’m doing, and I would certainly not call it a “feature.” As a result, tests are more verbose than they could be.

The documentation is simply middle-of-the-road. It’s there, but it’s not particularly detailed and it doesn’t come loaded with examples.

1. Minimal amount of work needed to add new tests. I’m afraid it gets failing marks for this. It requires manual registration of tests, and every test needs to be part of a suite. This makes adding new tests tedious and error prone (by writing a new test and forgetting to register it). I don’t know about you, but with all the C++ cruft, I look at the code below and it’s not immediately obvious what it does until I’ve scanned it a couple of times. The signal to noise ratio is pretty poor.

#define __UNITPP #include "unit++.h"
using namespace unitpp;
namespace
{
    class Test : public suite
    {
        void test1()
        {
            float fnum = 2.00001f;
            assert_eq("Checking floats", fnum, 2.0f);
        }
    public:
        Test() : suite("MySuite")
        {
            add("test1", testcase(this, "Simplest test", &Test::test1));
            suite::main().add("demo", this);
        }
    }; 

    Test* theTest = new Test();
}

3. Easy to modify and port. So-so. It needs the STL and it pulls in some stuff like iostreams (which I remember having distinct problems with when I was working with STLPort). On the other hand the source code is relatively small and self-contained so it’s certainly doable to port and modify if you’re willing to put in some time.
4. Supports fixtures. Another framework that I just can’t see how to do fixtures with. Like Boost.Test, it seems to think that using the constructor and destructor for each class is all you need. A quick search for fixture or setup or teardown in the documentation doesn’t reveal anything. I don’t know if I’m totally missing something or if other people just write very different tests from me. I suppose I could create a new class for every fixture I want, put the setup in the constructor and the teardown in the destructor and inherit from it for every test case (and somehow figure out how to create an instance of that class and use it for each test run). It’s probably possible, but it’s not exactly trivial, is it? Again, the lack of fixtures puts this framework out of the running.
5. Handles exceptions and crashes well. Average. It manages to catch regular exceptions without crashing, but that’s about it. No system exceptions in Linux. No way to turn it off for debugging.
6. Supports different outputs. I couldn’t figure out how to do it from the documentation. There is probably a way to do it since it even supports GUI functionality , but it’s not obvious (and there are no examples). Besides, by this point, having failed points 1 and 3, I wasn’t really motivated to spend a while learning the framework. Incidentally, this is one of the few frameworks whose default text output is not formatted correctly for IDEs like KDevelop.
7. Good assert functionality. It scrapes by with the minimum in this department. It provides equality and condition checks, but that’s it. It doesn’t even provide a float version of assert to check for “close enough.” At least it prints the contents of the variables to a stream correctly.
8. Supports suites. Yes, like most of them.

Overall, Unit++ is not really a candidate. Perhaps it’s because it’s not intended for the type of testing I intend to use it for, but it doesn’t offer anything new over other frameworks and it has a lot of drawbacks of its own. The lack of fixtures is simply unforgivable.

After looking into a framework that tried to be different from XUnit (Unit++), I wasn’t particularly looking forward to evaluating possibly the wackiest one of them all, CxxTest. I had never heard of it until a few days ago, but I knew that it required using Perl along the way to generate some C++ code. My spider senses were tingling.

Boy was I wrong!! Within minutes of using CxxTest and reading through its great documentation (the best by far), I was completely convinced this was the way to go. This came as a complete surprise to me since I was ready to leave somewhat dissatisfied and pronounce a victor between CppUnit and CppUnitLite.

Let’s start from the beginning. What’s with the use of Perl and why is it different from CppUnit? Erez Volk, the author of CxxTest, had the unique insight that just because we’re testing a C++ program, we don’t need to rely on C++ for everything. Other languages, such as Java, are better suited to what we want to do in a unit-testing framework because they have good introspection (reflection) capabilities. C++ is quite lacking in that category, so we’re forced to use kludges like manual registration of tests, ugly macros, etc. CxxTest gets around that by parsing our simple tests and generating a C++ test runner that calls directly into our tests. The result is simply brilliant. We get all the flexibility we need without the need for any ugly macros, exotic libraries, or fancy language features. As a matter of fact, CxxTest’s requirements are as plain vanilla as you can get (other than being able to run Perl).

The code-generation step is also trivial to integrate into the regular build system. The wonderful documentation gives explicit step-by-step instructions on how to integrate it with make files, Visual Studio projects files, or Cons. Once you have it set up, you won’t even remember there’s anything out of the ordinary going on.

Let’s see how it stacks up against the competition.

1. Minimal amount of work needed to add new tests. Very good. It’s almost as simple as the best of them. If I could nit-pick, I would have wished for an even simpler way to create tests without the need to declare the class explicitly. Since we’re doing processing with a Perl script, there’s no reason we couldn’t have taken it a step beyond that and used a syntax even closer to my ideal test framework.

class SimplestTestSuite : public CxxTest::TestSuite
{
    public: void testMyTest()
    {
        float fnum = 2.00001f;
        TS_ASSERT_DELTA (fnum, 2.0f, 0.0001f);
    }
};

3. Easy to modify and port. CxxUnit requires the simplest set of language features (no RTTI, no exception handling, no template functions, etc). It also doesn’t require any external libraries. It is also distributed simply as a set of header files, so there’s no need to compile into a separate library or anything like that. Functionality is pretty well broken down and separated in the original source code, so making modifications should be fairly straightforward.
4. Supports fixtures. CxxUnit gets the “top of its class” label in this category. Not only does it support setup/teardown steps on a per-test level, but it also supports them at the suite and at the world (global) level. Creating fixtures is pretty straightforward and just requires inheriting from a class and creating as many functions as you want starting with the letters “test.” To be really picky, I would have loved it if they had taken it a step further and, apart from simplifying the code a bit more, also inserted the setup and teardown code around the code for each test. That would have allowed us to work with those objects directly on the stack and their lifetime would have been managed correctly around each test. Oh well. Can’t have everything.

#include "MyTestClass.h"  

class FixtureSuite : public CxxTest::TestSuite
{
public:
    void setUp()
    {
        someValue = 2.0;
        str = "Hello";
    } 

    void tearDown() {}  

    void test1()
    {
        TS_ASSERT_DELTA (someValue, 2.0f, 0.0001f);
        someValue = 13.0f;
        // A regular exception works nicely though myObject.ThrowException();
    } 

    void test2()
    {
        TS_ASSERT_DELTA (someValue, 2.0f, 0.0001f);
        TS_ASSERT_EQUALS (str, std::string("Hello"));
    } 

    void test3()
    {
        //myObject.UseBadPointer();
        TS_ASSERT_EQUALS (1, myObject.s_currentInstances);
        TS_ASSERT_EQUALS (3, myObject.s_instancesCreated);
        TS_ASSERT_EQUALS (1, myObject.s_maxSimultaneousInstances);
    }  

    float someValue;
    std::string str;
    MyTestClass myObject;
};

6. Handles exceptions and crashes well. Great support. It catches all exceptions and prints information about them formatted like any other error (no system exceptions under Linux though). You can easily re-run the tests with a command-line argument to the Perl script to avoid catching exceptions and catch them in the debugger instead. It also gives you a custom version of every assert macro that lets you catch the exceptions yourself in case you ever need to do that.
7. Supports different outputs. Different outputs are supported by passing a parameter indicating which type of output you want to the Perl processing step. The default one (error-printer) was formatted correctly for IDE parsing, and you can use several others (including GUIs for those of you addicted to progress bars, a yes/no report, or a stdio one). Adding new output formatting sounds very straightforward and it’s even covered in the documentation.
8. Good assert functionality. Again, it gets “top of its class” for this one. It has a whole suite of very comprehensive assert functions, including ones for exception handling, checking predicates, and arbitrary relations. It even has a way to print out warnings which can be used to differentiate between two parts of the code calling the same test, or to print reminder “TODO” messages to yourself.
9. Supports suites. Yes. All tests are part of a suite.

Another feature supported by CxxUnit that I haven’t had time to look into is some support for mock objects. Anybody doing TDD knows the value of mock objects when it comes to testing the interactions between a set of objects. Apparently CxxUnit allows you to override global functions with specific mock functions (it gives an example of overriding fopen()). I don’t think it helps any with regular classes; for those you’re on your own.

So, what’s not to like in CxxTest? Not much, really. Other than wishing that the test syntax were a bit tighter, the only thing to watch out for is what happens with large projects. If you follow the examples in the documentation, it will create a single runner for all the tests you give it. This can be problematic if you’re going to be having thousands of tests, and then making one small change in one of them causes a full recompilation of all your code.

Update: After talking with Erez and re-checking the documentation, I realized this is already fully supported in CxxUnit. By default, when you generate a test runner, it adds a main function and some global variables, so linking with other similar runners gives all sorts of problems. However, it turns out you can generate a test runner with the –part argument, and it will leave out the main function and any other globals. You can then link together all the runners and have a single executable. I wonder if it would be worth going as far as creating a runner for every suite, or if it would be best to cluster suites together. Worth investigating at some point whenever I get enough tests to make a difference.

Conclusion

After going through all six C++ unit-testing frameworks, four stand out as reasonable candidates: CppUnit, Boost.Test, a modified CppUnitLite, and CxxTest.

Of the four, CxxTest is my new personal favorite. It fits very closely the requirements of my ideal framework by leveraging the power of an external scripting language. It’s very usable straight out of the “box” and it provides some nifty advanced features and great assert functionality. It does require the use of a scripting language as part of the build process, so those unconfortable with that requirement, might want to look at one of the other three frameworks.

CppUnit is a solid, complete framework. It has come a long, long way in the last few years. The major drawbacks are the relative verbosity for adding new tests and fixtures, as well as the reliance on STL and some advanced language issues.

If what you need is absolute simplicity, you can do no wrong starting with CppUnitLite (or a modified version), and tweaking it to fit your needs. It’s a well-structured, ultra-light framework with no external dependencies, so modifying it is extremely easy. Its main drawback is the lack of features and the primitive assert functionality.

If you’re going to be working mostly on the PC, you don’t expect to have to modify the framework itself, and you don’t mind pulling in some additional Boost libraries, Boost.Test could be an excellent choice.

Should you roll your own unit-test framework? I know that Kent Beck recommends it in his book Test-Driven Development: By Example, and it might be a great learning experience, but I just can’t recommend it. Just as it’s probably good to write a linked-list and a stack data structure a few times but I wouldn’t recommend actually doing that in production code instead of using the ones provided in the STL, I strongly recommend starting with one of the three unit-testing frameworks mentioned above. If you really feel the need to roll your own, grab CppUnitLite and get hacking.

Whichever one you choose, you can really do no wrong with one of those three frameworks. The most important thing is that you are writing unit tests, or, even better, doing test-driven development. To paraphrase Michael Feathers, code without unit tests is legacy code, and you don’t want to be writing legacy code, do you?

unit_test_frameworks.tar.gz

This review was first published in the January 2002 issue of Game Developer Magazine. Printer-friendly format.

OK, welcome back. Here is the good news: Effective STL is to STL what Effective C++ is to C++. It’s equally great and it’s written in the same light, conversational Scott Meyers style that makes reading it a pleasure. It assumes that you have basic knowledge of STL (and C++) and builds on that to show you how to use it effectively and avoid common pitfalls.

Those of you unfamiliar with STL must be wondering what it is and how is it useful for game development. STL stands for Standard Template Library: A library of generic data structures and algorithms that you can use across many compilers/platforms, including most of the current consoles. The library is implemented with templates so the resulting code will be quite efficient, possibly even more than your hand-coded structures and algorithms. Additionally, it has been implemented, tested, debugged and optimized by thousands of people, so it’s code you can usually rely on. STL also empowers you by putting very powerful constructs at your fingertips. Maybe before you would have thrown a bunch of objects in an array and searched through them in linear time (does that sound familiar?), but now you can just as easily put them in a hash table and have constant-time access to them.

Effective STL starts out with a fairly thorough discussion about containers (vector, list, map, etc) and iterators. It immediately goes beyond the typical description of the containers and their O(n) performance characteristics. Instead it deals with many real-world questions: Is the memory for the elements allocated contiguously? Are the iterators invalidated when the elements change? What the most efficient way of removing elements for a specific container? Those are not issues you’ll see addressed in most STL books.

It then moves on to algorithms and functors. Just like with the containers, instead of listing all the available algorithms, it points out common mistakes and how to deal with them. For example, it will discuss the different ways of sorting elements, or how std::remove really works (and why it doesn’t really remove anything).

The final chapters present more general, but very useful, information on the STL that will save you a few headaches along the way: When to use STL algorithms and when to use your own, style guidelines, or even how to deal with Microsoft’s Visual C++ broken STL implementation and template support.

But STL is not perfect. Reading the book will give you some ideas of where STL is lacking, but it won’t spell them out for you. Sometimes you’ll need to read between the lines and think about how things will apply to game development. For example, memory allocation can be an issue, especially if you’re developing for a console, so you’ll probably want to end up writing your own allocators. The book has a brief couple of items discussing what you can and can’t do with allocators, but not enough to write your own. You’ll need to look elsewhere for that.

Another issue often brought up when dealing with STL is the difficulty debugging STL code, from cryptic multi-line error messages to the difficulty viewing the elements of a container in the debugger. The book will help you a bit by showing you how to “parse” the intimidating error messages and directs you to some STL resources on the Internet.

Effective STL works very well both as a book to read cover to cover and as a reference later on. It only uses source code where it has to; it won’t bore you with pages and pages of pointless code. As a matter of fact, it won’t bore you at all since it packs a lot of information in a mere 250 pages. Overall, you simply must read this book before you decide to use (or not to use) STL in your next game or tools. If you’re already using STL, then it should already be on your bookshelf.

As I write this there are still a few cardboard boxes strewn across our home as we finish unpacking everything. My wife and I just finished a surprisingly painless cross-country move to sunny San Diego. The town house we’re moving into is definitely larger than our previous condo, but it wasn’t until we had to pack everything, move it, and then unpack it that we realized how much stuff we had accumulated over the years. Yes, there was a reason for everything we had, but it was still a lot of stuff. Then we came to the realization that we prefer a more open, simpler environment with fewer pieces of furniture, fewer paintings covering the walls, and less clutter in general.

That is an interesting insight because it applies to many different things, including programming: There is a cost associated with having a complex, cluttered environment, whether it be the furniture in your house, your class hierarchy, or your build process. In particular it will create resistance to change, which can be in the form of a cross-country move, or a simple class refactoring.

Goals

The ultimate reason to write code is to have the computer carry out a sequence of instructions. In our case, that sequence of instructions is a game or a tool to help create the game. It is tempting to think of the computer as the “audience” for our program, but it is not a very good audience. It is way too forgiving, and as long as things compile, it’ll be happy with anything you throw at it. We can do better than that.

We can still write code that is correct, but with people in mind as the primary audience. The usual reason for choosing people as your audience is that software development is a team effort, and it’s very important that other people be able to understand and modify your code. True, but there’s more to it than that. Even if you were the only person writing code, you should still write for people, not computers. Assuming it’s more than a toy project, you will have to refactor your code as you learn new things about it. Probably not once, not twice, but many times over the course of the project’s lifetime. You will have to keep it up to date, fix bugs, add new features, etc. I am convinced that refactoring (or lack thereof) is one of the major influences on code quality, and, as a consequence, product quality, so anything to help with that is extremely important in my book.

Lately, I’ve been taking it a step further and thinking of my audience not just other programmers, but other people who are not programmers. If you manage to write some code that a non-programmer can more or less understand, then you know you’re there. The code should be simple and self-explanatory enough that should be a breeze to refactor in the future.

But wait, as if ease of refactoring wasn’t enough, there are more reasons for writing really simple code. Edsger W. Dijkstra in his lecture “The Humble Programmer” advocated the avoidance of clever tricks and urged programmers to be fully aware of the limitations of the human mind when writing programs.

“The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.”

Dijkstra’s reason for keeping clever tricks away is not so much for ease of refactoring, but the quality of the program itself. Programmers are all too eager to tackle something too ambitious at once, without breaking it down into smaller, more understandable sub-problems, and, as a result, create a disastrous program that crashes often and doesn’t even implement all the required features.

Today, this sounds like a surprisingly modern attitude. After all, it’s right up the alley of agile development. But amazingly, this lecture took place in 1972. Talk about timeless advice (OK, OK, in the really short scale of computer history, not in the grand scheme of things).

Brian Kernighan had also something to contribute to the idea of writing simple code, but coming from a very different angle:

“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

While I totally agree with the sentiment, I feel it is not as important as the other two reasons for aiming towards code simplicity. Frankly, if you use test-driven development and take steps of the correct size, you should hardly ever find yourself knee-deep in the debugger trying to make sense of your program. However, a decade or two ago that made a lot more sense, especially given that debuggers were less sophisticated then as well. Still, anything that encourages programmers to keep things simple is a step in the right direction. So if that convinces some people, so be it.

Attaining Simplicity

Great. Simple is good. How do we write “simple” code? That’s not something they teach in school and it’s not something that is pushed for in the workplace, so it’s probably not something you’re used to thinking about. The rest of the article will be a series of suggestions and techniques on how to make code as simple as possible. Specific examples are given in C++, but the general techniques should apply to most object-oriented languages.

Avoid unnecessary language tricks.

Listen to Dijkstra and go easy on “clever code.” If you feel the need to write a comment to explain to other people what that cute line of code you just wrote does, you should very strongly consider re-writing it so a comment is not necessary. I’m not saying to avoid taking advantage of language features, just the ones that can be done in another, simpler way. For example, don’t use a template when a non-templated implementation is perfectly fine. Don’t rely on a clever arrangement of variables so they’re initialized in the correct order even though the language standard says it should work.

Avoid or postpone performance optimizations that obscure the meaning of the code for as long as possible.

You’re all familiar with the Donald Knuth quote: “Premature optimization is the root of all evil.” Programs have a way of twisting into evil, misshapen fiends when they’re optimized, and they usually become much harder to refactor afterwards. Specifically in this case, I suggest avoiding (or delaying at least) optimizations that affect how simple or readable a section of code is. There are many ways to skin a cat, and there are also many ways to optimize a program. I find that there is often a way to optimize things that doesn’t cause a major loss in code simplicity. But if that’s not possible, at least defer that optimization as long as possible. Maybe you won’t need it, and maybe the code will change before then and you’ll save yourself a lot of rework.

Functions should do one thing only.

The secret to a good function is that it should do one thing and one thing only. Functions that follow that rule are usually quite small (at most 10-15 lines in C++) and their whole meaning can be easily understood at a glance. Because the function has only one purpose, refactoring becomes very easy: No need to worry about how changes in that function will affect other parts of the function, no pesky local variables reused in many parts of the function, etc.

Maybe this is a bit extreme, but ideally, functions should be so simple that I should be able to look at a function, delete it and re-write it from scratch without any difficulty.

I follow two guidelines to keep functions as simple as possible:

• If a function goes over 10 to 15 lines of code, I look at it really hard and try to split it. Chances are it’s doing more than one thing.
• If I ever feel the need to write an explanatory comment about what a few lines of code do, I move those lines into a function of their own and give it a name that explains what it does (usually the same as the what the comment would have been). By following this rule, the only comments in my code are the ones that explain *why* the code does things the way it does, and deal with higher-level such as the purpose of a class or a library.

Classes should do one thing only.

This should come as no big surprise. If we want our functions to be small, simple, and just one thing, we also want the same qualities from our classes. When classes are left unchecked, they can easily grow into unmanageable blobs of many thousands of lines. The largest class I had the misfortune of encountering had a staggering 10,000 lines in it, and the problem was compounded by inheriting from another class with over 8,000 lines! As you can imagine, having to do any work near that class was quite a punishment.

Personally, I feel that C++ classes should ideally be no more than about 200-300 lines (remember, with virtually no other comments). Anything that goes beyond that, and I start thinking hard about what exactly the class does. If a class reaches 1000 lines, it’s time to get the axe out.

One technique I’ve been following to keep classes as simple as possible is to use the controversial suggestion from Scott Meyers on how to decide on the scope of a function. The quick summary is that functions should be pushed out of a class as much as possible: if a function can be implemented as a non-member function, then do it; if it can be a class static, make it so; otherwise, leave it as a member function.

One immediate consequence of following that advice is that a class is stripped of anything that is not essential. Helper functions (both public and private to the class) are pushed out, and the meaning of the class is made more clear. The second consequence is that refactoring is a lot easier. By using non-member functions in anonymous namespaces whenever possible, we avoid the need to edit any header files at all, which results in faster compile and iteration times. Also, since those functions are completely independent of the class, they’re trivial to move around, promote to higher levels, etc. while member functions require some massaging to move around.

I’m suggesting that we keep functions small, but also that we keep classes small. We’re not going to end up with any less code than we would have before, but we’re going to end up with many more classes scattered all over our program. We have pushed the complexity from the function and class level up to the program organization and architecture level. The next article in this series will deal with how to manage this newfound complexity, but here’s a preview: Use hierarchies.

Timings, logging, and error handling

Ideally, I’d like my code to be really small, simple, and to the point so its meaning is always clear and obvious. Unfortunately, in the real world, that’s is not always possible. We can start with a very clean 5-line function, but then we add timings around it to get an idea of its performance, and logging statements to know what is going on, and error handling code to deal with the unexpected, and before we know it, we have a monster function whose meaning is lost in a mass of details.

What can we do about it? We could argue that timings should be done with non-intrusive methods such as using profilers, but that’s not always possible or desirable. We often want very specific timings that only affect a particular section of the code (load times), or we want to continuously monitor performance of certain sections while the game is running. For example, we probably want to get an idea of how much time we’re spending each frame on each of the major tasks: entity updates, AI calculations, physics, animation, rendering, sound, etc. We need to wrap each of those areas in timing statements. The more detailed we want those timings to be, the more junk we’ll have to add to the code. We can try making that as clean as possible by using macros that only require one line, but it’s still junk that obscures the meaning.

Logging doesn’t even have a non-intrusive counterpart unless you want to automatically add prologue and epilogue code to every function (which would generate way too many messages and would affect performance too much). So we’re stuck writing messages by hand to the log system.

Error handling at least as has the possibility of using exceptions, which add a lot less clutter than returning and checking error codes. Unfortunately, we have to deal with several real-world issues which often make exception-handling impractical. The first one is performance. As much as I hate bringing this up, exception handling can easily have a significant performance impact even in modern hardware. This is something that will hopefully go away in the near future as PC hardware and consoles become more powerful. The second problem is compiler support. Even today, some console manufacturers are admitting that exception-handling support in their compilers is not usable and we shouldn’t rely on it. So much for that idea. Finally, and probably most importantly, is complexity and programmer knowledge. If you have read Exceptional C++ you how complicated it can be to write exception-safe code, and not every C++ programmer out there today is going to feel comfortable using exceptions. All of this is a vicious circle, because until programmers demand robust and efficient exception handling, compiler and system creators aren’t going to provide them, which means a lot of programmers are not going to be exposed to them, etc, etc.

So, what great solutions do I have to this problem? None, I’m afraid. This is where I’d like to hear from some of you. How do you deal with timing, logging, and error handling in a way to minimize the clutter and leave your code as clear as possible? Email me and I’ll edit this article or make a new one with any great solutions I receive.

Striking a balance

In the end, even if we have the best goals in mind about simplicity, we will be forced to make compromises. Error handling is one case. Another case is often performance. Sometimes, having many scattered functions can add up to a noticeable performance hit in some inner loop of a performance-critical section. In that case simplicity needs to give way to practicality and do whatever is necessary. Just make really sure that having many functions is the cause of the performance problems before you make any changes.

It is also important not to confuse code simplicity with algorithmic simplicity. Writing simple code doesn’t mean that you have to use a silly bubble sort to order a list. Always choose the most appropriate algorithm for the task at hand, just implement it with the simplest possible code. Implementing it in a very simple way will also make possible to easily change it down the line with a more efficient algorithm.

The next article will deal with how to manage all the complexity we’ve pushed up into the program structure. Until then, keep your functions short, your classes clean, and remember The Humble Programmer:

“…We shall do a much better programming job, provided that we approach the task with a full appreciation of its tremendous difficulty, provided that we stick to modest and elegant programming languages, provided that we respect the intrinsic limitations of the human mind and approach the task as Very Humble Programmers. "

You know one of the reasons I refuse to live near most cities in the US? Traffic. I’m not talking about the “this road is crowded” type of traffic that lets you zip along at a good speed, or even about the “I can’t get out of my lane because this is so packed” type of traffic that moves slowly along. No, I’ve come to accept and deal with that. It’s the “we might as well get out of the car and enjoy the sunshine because we aren’t moving” type of traffic that I can’t stand. Amazingly enough, it seems to happen around most major cities in the US during rush hour, and sometimes this “rush hour” stretches from 7AM until 8PM. It’s a bad sign when car manufacturers have advertisements telling you how much more comfortable you’ll be in their car when you’re stuck in traffic. Under those conditions it can easily take over an hour just to cover a distance of 3 or 4 miles.

Other than pointing out that you’re much better off walking, cycling, or using public transportation, what’s the point of all this and how does it relate to the physical structure of a program? There are some things that just don’t scale well. They appear to work perfectly fine for a small number of units, but as soon as a certain threshold is reached, things seem to bog down and eventually collapse under their own weight. Just adding more lanes doesn’t appear to solve the problem either, judging by the number of clogged-up 5-lane highways everywhere. Sometimes, you need to take a step back and deal with the problem in a different way. Either that, or buy a nice music system and enjoy your time in traffic.

We might not have much of a say over how traffic should be dealt with where we live, but we certainly have a lot of choices when it comes down to structuring our C++ source code. For small projects, we can blissfully code away without paying any attention to physical structure and we won’t be any worse off for it. However, as a project grows, it reaches a critical point, and compilation times start getting slower and slower, to the point where tiny changes could make you wish you were stuck in traffic instead of staring powerlessly at your monitor. Adding a faster CPU, more memory, or a better hard drive can help make things faster, but is usually not a good long-term solution.

Build Types

We are usually concerned with the time for two types of builds:

• Full builds. In this case we care about the time it takes to build the whole project from scratch, starting from a totally clean build. This situation comes about when we just want to use the result of the build of a project we’re not actively modifying. For example, an automated build machine will most likely be doing full builds of the game, so the turnaround time before a build is ready will depend on the full build time. Another example might be if you need to link your code with a library for which you have the source code.
• Minimal builds. Once we have done a full build on a project, we then make a very small change to its source code and build it again. That’s the time for a minimal build. This is what you really care about when you’re actively working on a project, making modifications and compiling constantly. Ideally, building the project after a small change should require very little time This allows for very fast turnaround time for debugging, or even to get feedback from the compiler on silly syntax errors we just typed.

Improving the physical structure of a program often reduces the time of both types of builds. Unfortunately things don’t always work out so neatly and there are times where some changes will make one type of build faster and the other slower. Understanding what affects each compilation time allows us to optimize our compilation strategy and strike a balance that fits our needs.

Clearly, the time for both types of builds depends on the number of files and the complexity of those files. Both types of builds are also affected by the number of files each file depends on (the number of #include statements in each file). However, as we’ll see in a moment, in the case of a full build there is the chance of caching the includes of some files and reusing them for other files.

There is something very different about minimal builds. Their build time is usually dominated by the number of files that depend on the modified files. In the worst situation, every file will depend on the file that changed and a full build will be triggered. In the ideal case, only the file with the changes itself will be compiled and no other files will have been affected. In one case the build could take less than a second, and in the other it could easily take multiple hours.

The rest of this article will look at different techniques to reduce build times and how they affect each of those two build types.

Counting Includes

It is easy to underestimate how quickly include statements can compound. If file A includes file B, and file B includes file C and D, every time someone includes file A they’re including three other files for the ride. Add a few more levels of inclusion with header files including many other header files, and you have a recipe for disaster (or for really long build times at least).

As an experiment, I added one more feature to the script I wrote last week. The script analyzes a set of source code files and determines how many times each file is included by other files in a recursive way. So, in our trivial example above, file C will be reported as being included twice (once by B directly, and once by A indirectly). I then decided to test it on the source code for a high-level game library (I’m not going to be any more specific since it wasn’t particularly good code and it had a pretty hideous physical structure). I wouldn’t be surprised if it’s not very different from the level of complexity of a lot of game code out there. As a point of reference, the library was composed of 300 cpp files and 312 header files.

Before I ran the script, I tried to guess how many times the most included file in the whole library was included by other files. My guess was around 600 times, just because I knew that the physical structure of that code wasn’t pretty. I figured maybe almost half the files included that one header file, and a few others included it indirectly. Boy was I wrong! Here are the shocking results:

That means that during the course of a full build for those 300 cpp files, one header file could be included over 10,000 times! No wonder this particular library seemed to take a long time to compile. Notice that the other top files quickly drop to being included around 800 times each (which is still even higher than my initial estimate).

As a comparison, I tried running that same script on another, much smaller library, but also one with a much better physical structure and many fewer dependencies between files. This second library was only made up of 33 cpp files and 39 header files. The most included file was only included a total of 23 times (with the second one being included less than 10 times). So, having the number of classes grow by a factor of 10, caused the number of includes to grow by a factor of 1000. Clearly not a very scalable situation.

Things aren’t quite that bad though. Header files typically have a set of include guards in them, to prevent the compiler from adding duplicate symbols if it encounters the same header file multiple times during the compilation of one cpp file. This is what include guards look like:

// SomeFile.h

#ifndef SOMEFILE_H_ #define SOMEFILE_H_

// Normal code goes here, even other #include statements if necessary

#endif

With every header file having include guards around it, I turned on the /showincludes switch in Visual C++ and performed a full build. The total number of includes during the course of building the 300 classes in the library was an astounding 15,264. Better than the worst-case-scenario we calculated earlier, but still tremendously high.

Apparently some C++ compilers try to optimize this situation by automatically caching header files and avoiding hitting the disk to reload them over and over. Unfortunately, there is very little hard data about that, and you’re always at the mercy of your current compiler writer. Was that true for Visual Studio .NET 2003?

Redundant Guards

To test if I could speed up the compilation any, I added redundant include guards to the whole library. Redundant include guards are like the regular include guards, but they are placed around the actual #include statement. I first saw them mentioned in the book Large Scale C++ Software Design by John Lakos (written in 1996), but popular wisdom claims that they are unnecessary with modern compilers. Well, time to test that.

This is what redundant include guards look like:

// Somefile.cpp

#ifndef SOMEFILE_H_ #include “SomeFile.h” #endif #ifndef SOMEOTHERFILE_H_ #include “SomeOtherFile.h” #endif

//…

I wrote a quick script to add redundant guards to all the source code and did a full build again. The number of includes reported by the compiler went down to 10,568 (from over 15,000). That means that there were about 5,000 redundant includes in a full build. However, the overall build time didn’t change at all.

Result: Zero. Apparently Visual Studio .NET 2003 (and probably most of the major compilers) does a pretty good job caching those includes by itself.

Recommendation: Stay away from redundant include guards. I never liked having the including files know about the internal define, and if the guard ever changes it can easily break things. Besides, the code looks a lot messier and unreadable. It might have been worth it if we could define #include to expand to a redundant guard automatically, but I don’t think that’s possible with the standard C preprocessor.

#pragma once

Just in case, I decided to test another strategy and see if I obtained similar results. Instead of using redundant include guards, I added the #pragma once preprocessor directive to all header files. Visual C++ will treat files with that directive differently and it’ll make sure that those files are only included once per compilation unit. In other words, it accomplishes the same thing as the external guards, just in a non-portable way. Here’s another really simple script to add #pragma once to all the header files.

Result: No difference. Just as with redundant include guards, it seems that the compiler was smart enough already to optimize that case.

Recommendation: Don’t bother with it. It’s a non-standard construct that doesn’t get any apparent benefit. If you still feel compelled to use it, at least wrap it up with #ifdef checks for the correct version of Visual Studio.

Precompiled Headers

During a full build, every cpp file is treated as a separate compilation unit. For each of those files, all the necessary includes are pulled in, parsed, and compiled. If you look at all the includes during a full build, you’re bound to find a lot of common headers that get included over and over for every compilation unit. Those are usually headers for other libraries that the code relies on, such as STL, boost, or even platform-specific headers like windows.h or DirectX headers. They are usually also particularly expensive headers to include because they tend to include many other header files in turn.

From our findings in the previous two sections, it is clear that some compilers cache the headers encountered for each compilation unit. However, they don’t do anything about duplicated headers found across multiple cpp files, and that’s where precompiled headers come in.

When using precompiled headers, we can flag a set of headers as being part of the precompiled set. The compiler will then process them all at once and save those results. Every compilation unit will then automatically include all the headers that were part of the precompiled set at the very beginning, but at a much lower cost than parsing them from scratch every time.

The catch is that if any of the contents of the precompiled headers changes, a full rebuild is necessary to compile the program again. This means that we should only add headers that are included very often throughout our project but that don’t change frequently. Perfect candidates are the ones we mentioned earlier: STL headers, boost, and any other big external APIs. I always prefer not to include any headers from the project itself, although if you have a header that is included in every file, you might as well include it in the precompiled set (or, even better, change it so it’s not included everywhere and improve the physical structure).

The gains from using precompiled headers are quite dramatic. The game library we mentioned in an earlier section took over 14 minutes to compile without pre-compiled headers, but only 2:30 when using them. Those are huge savings! Minimal rebuilds are also improved because we avoid parsing some of the common headers for one file, but the results aren’t as dramatic as for full builds.

Precompiled headers are not without their downside though. The first problem is that precompiled headers often end up forcing the inclusions of more headers than it is absolutely necessary to compile each individual file. Not every file needs or <windows.h> included, but since a fair amount of them do and those are considered expensive includes, they’ll invariably end up in the precompiled header section. That means that any compilation unit taking advantage of precompiled headers will be forced to include those as well. Logically, the program is the same, but we have worsened the physical structure of the source code. In effect, we are trading extra physical dependencies between files for a faster compile time.

The second problem is that precompiled headers are not something you can rely on from compiler to compiler and platform to platform. The only compilers I’m aware of that implement them are Microsoft’s Visual C++ and Metrowerks’ CodeWarrior (although I just did a Google search and apparently gcc also supports precompiled headers--that’s great news!). For the rest of us using different compilers, we’re out of luck as far as this technique goes. Considering how important multi-platform development is becoming in the games industry (and elsewhere), this is a big blow against them.

Finally, by far the worst aspect of precompiled headers, is what happens when you combine the first two problems: Take a set of source code developed on a compiler where precompiled headers were available, and try to build it on a different platform. The code will compile since everything is standard C++, but it’ll compile at a glacial pace. That’s because every file is including a massive precompiled header file, and it is being parsed over and over for every compilation unit without taking advantage of any optimizations in the part of the compiler. If the code had been developed without precompiled headers in the first place, each file would only include those headers that it absolutely needs to compile, which would result in much faster compile times.

So earlier, when I said that the game library without precompiled headers took over 14 minutes to build, that’s because it was written with precompiled headers in mind. Otherwise, I estimate it would only take about 5-6 minutes (still much longer than the 2:30 it took with precompiled headers).

Personally, I have never yet worked on a set of code that was developed to be compiled on multiple platforms and where some of them did not support precompiled headers. I suppose the best approach is to use #ifdefs to only include the precompiled headers in one platform and the minimal set of includes for the rest, but it seems like an extremely error-prone approach where programmers are going to be breaking the other platform’s builds all the time. I’d be interested to know how teams working in such an environment deal with it.

Result: Huge gains both for full builds and minimal builds if your compiler supports them. Much worse physical structure.

Recommendation: Definitely use them if you’re only compiling in a platform that supports them. If you need to support multiple platforms, the gain is still too big to pass up. It probably is worth if you manage to separate the includes for precompiled headers with lots of #ifdefs and try to keep the physical structure sane for platforms that don’t support them.

Single Compilation Unit

This is an interesting trick that you won’t find in most books. I first read about it in the sweng-gamedev mailing list about a couple of years ago. Be warned, this is hackish and ugly, but people claimed really good results. I just had to find out for myself how it stacked up against the other techniques to reduce build times.

This technique involves having a single cpp file (compilation unit) that includes all the other cpp files in the project (yes, that’s right, cpp files, not header files). To compile the project we just compile that one cpp file and nothing else. The contents of this file are simply #include statements including all the cpp files we’re interested in. Something along these lines:

// everything.cpp

#include “MyFile1.cpp” #include “MyFile2.cpp” #include “MyFile3.cpp” //….

As you can imagine by now, I wrote a script to create that file from a directory containing the source code for a project. I just created a file including all the cpp files in that directory, although a better way of doing it would be to parse the make (or project) file and only include those files that are actually part of the project. That way, as I discovered, you avoid including outdated files or files that are in that directory but are not part of the project.

I created this file (everything.cpp), compiled it and… get ready: The build time went down from 2:32 minutes to 43 seconds! That’s a 72% decrease in build time!! Not only that, but the .lib file it created from that library went from 42MB down to 15MB, so it should help with link times down the line. People in the mailing list reported even better results with gcc than with Visual Studio.

What is the reason for such reduction in build times? I can only speculate. I suspect part of it is due to avoiding the overhead of starting and stopping the compiler for every compilation unit. However, the biggest win probably comes from the reduced number of included files. Because everything is one compilation unit, we only include every file once. The second time any other file attempts to include a particular header file, the compiler will have already cached it (and it’ll have include guards so there’s no need to parse anything). To test this theory, I again turned on the /showincludes switch. Indeed, the number of includes during a full build went down from 10,568 to 3,197. That’s a 70% reduction of included files, which is, probably not coincidentally, the same reduction in build time.

One very interesting observation from this experiment is that build times are probably more dependent on the number of actual includes performed by the compiler than I thought at first. All the more reason to keep a really watchful eye on the physical structure of the program. The third part of this article will cover what architectural choices we can make to improve the physical structure and keep the overall number of includes down.

Unfortunately this method also has its share of problems. One of the biggest problems is that there is no such a thing as a minimal build anymore. Any modification to any file will cause a full rebuild. Of course, the full build takes a only a fraction of the time it took before, so this might not be much of an issue.

As with precompiled headers, we’re adding a lot of physical dependencies between files. In this case, files will have a physical dependency with any files that were included before it in the everything.cpp file.

However, the most objectionable of all problems is that we can now run into naming conflicts. Before we assumed each cpp file was a separate compilation unit. Now they’ve all been forcefully added to the same one. Any static variables or functions, or anything on an anonymous namespace will be available to every cpp file that comes after it on the large everything.cpp file. This means that there’s potential for having conflicting symbols, which is one of the things that anonymous namespaces were supposed to solve in the first place. If you have decided to use this technique, you will want to keep everything as part of a separate namespace or part of the class itself and avoid global-scope symbols completely.

Result: Huge improvement in full-build times, but minimal-build times become much worse. Potential for clashing of static and anonymous namespace symbols.

Recommendation: The gains of this technique are simply huge so it would be a shame to ignore it. It is probably no good for regular builds, but you might want to have it as an option when you just care about doing full builds (automated build machine or building someone else’s code). If so, make sure to wrap symbols in namespaces or classes.

analyze_includes.pl add_external_guards.pl add_pragma_once.pl create_everything.pl

The next (and final, I promise) part of this article will look at architectural choices that can greatly influence build times as well as looking briefly at link times and see what we can do about them.

A few days ago, Tom Whittaker (a friend who is currently working at Firaxis) emailed me with some surprising facts about the behavior of the #include directive in the C++ compiler of Visual Studio .NET. It turns out that in my book, I hinted that compilers often optimize the include step for header files and so they don’t pay the costs of opening a file and loading it. When Tom turned on the /showincludes flag on the compiler, he saw that the same file was repeatedly being included by the compiler, even if it had internal include guards. He ran several interesting tests and I’ll add a link from here whenever he gets around to putting them up on his web site (hint, hint, Tom).

All of that made me think again about the physical structure of a C++ program, how important it is, and how often it is overlooked. This two-part article will attempt to explain why the physical structure of a program is so important, present some useful guidelines, and show its effect on compile times.

Physical Structure

We are all familiar with the logical structure of a program. It deals with classes and functions and namespaces and templates. That’s what you learn about in school, and what you read about in most C++ books. Design patterns, object oriented programming, etc, etc, all deal with the logical structure of a program. And, truth be said, it really is the most interesting part.

The physical structure of a program deals with the files that make up its source code. The .cpp and .h files, how they include each other, how they’re subdivided into directories, etc. While not as interesting and sexy as the logical structure, it is crucial to understand the consequences of the physical layout of a program for any real-world project of any significant size. That includes just about every modern PC and console game, but probably not handheld devices because of their small code size.

Because it’s not a particularly hot topic, not many books talk about the physical structure of a program. The best book in the subject is Large Scale C++ Software Design by John Lakos. No technical lead should work on a game without at least having read parts of that book. Yes, some of the advice is a little outdated by today’s standards (it’s an almost antique book in the computer world–almost 10 years!), and some parts are a bit long winded with detailed measurements. Skip those in your first read and you’ll still get a lot of gems along the way. Every time I come back to that book I end up getting something new out of it. I haven’t read it in about 3-4 years, so it’s just about due for another read.

A good physical structure will result in files without many dependencies to other files, and with files clearly grouped in cohesive modules or libraries. On the other hand, a bad physical structure is one where files are related to other files from all over the project, without clear delimitations or boundaries. This is a clear example of the “blob” antipattern. Unfortunately, if left unchecked, this is the type of physical structure that develops over time.

Benefits of a Good Physical Structure

Why would anybody care about the physical structure? After all, the things we care about when we’re writing a program are that it does what it’s supposed to do, and that it does it fast. That might be true for a demo, or a throwaway project, but for a large project, maintainability is also a very important requirement. It’s no good to have a very efficient class if we can’t change it, and it’s also no good if iteration to test a change takes a long time.

The benefits of a good physical structure of a program are:

• Better logical structure. Usually, keeping an eye on the physical structure of a program results in a better logical structure. By reducing the dependencies between files, we will probably reduce dependencies between classes. It is often the case that unexpected connections will grow between classes and even different libraries if their header files are already included. Programmers won’t have anything to remind them that they’re just adding a new dependency when they decide to call some global function.One of my pet-peeves when programming for Windows is having windows.h included in every single .cpp file, either directly or indirectly. In the type of programs I write, most classes don’t need anything in windows.h, yet it is forced down their throats. At one point I attempted to remove a global windows.h include in a library since it was supposedly not needed anywhere, just to have to give up because of the many unnecessary DWORD, BOOL, and screwed up min and max calls scattered everywhere on the source code.
• Easier to refactor. If you think of each file as a little box literally connected with a string to all the other files it depends or is dependent on, the more of those strings there are, the harder it is to untangle it from the overall mess and separate it. It is the same thing with refactoring: The worse the physical structure, the harder it is to make any refactoring changes that involve separating or isolating sections (which, in my experience, it’s one of the most crucial refactorings you have to do to prevent programs from growing into the “blob” antipattern).
• Easier to test. Not surprisingly, the more modular and independent the project is, the easier it becomes to write unit tests for it since each piece can be tested separately from the rest. Unit tests really benefit from having very few dependencies, so one of the many benefits reaped by doing test-first development is a very modular design with a logical and physical structure that has very few dependencies.
• Faster compile times. This might come as a surprise to some people, but it is usually the most tangible and objective result of having a good (or bad!) physical structure. Usually, the worse the physical structure, the longer the compile times will be. Compile times are an issue for large projects. Even with the fastest machines today, full builds on large projects can easily take hours. More importantly, builds caused by changing just a file or two can trigger builds that last almost that long. Needless to say, having such a delay every time a change is made is not exactly encouraging programmers to test their work and iterate it to make it better. Part two of this article will look exclusively at compile times.

Guidelines

Here’s a distilled set of guidelines from Lakos’ book that minimize the number of physical dependencies between files. I’ve been using them for years and I’ve always been really happy with the results.

1. Every cpp file includes its own header file first. This is the most important guideline; everything else follows from here. The only exception to this rule are precompiled header includes in Visual Studio; those always have to be the first include in the file. More about precompiled headers in part two of this article.
2. A header file must include all the header files necessary to parse it. This goes hand in hand with the first guideline. I know some people try to never include header files within header files claiming efficiency or something along those lines. However, if a file must be included before a header file can be parsed, it has to be included somewhere. The advantage of including it directly in the header file is that we can always decide to pull in a header file we’re interested in and we’re guaranteed that it’ll work as is. We don’t have to play the “guess what other headers you need” game.
3. A header file should have the bare minimum number of header files necessary to parse it. The previous rule said you should have all the includes you need in a header file. This rule says you shouldn’t have any more than you have to. Clearly, start by removing (or not adding in the first place) useless include statements. Then, use as many forward declarations as you can instead of includes. If all you have are references or pointers to a class, you don’t need to include that class’ header file; a forward reference will do nicely and much more efficiently.

One unfortunate aspect of those guidelines is that the compiler doesn’t really care one way or another. As long as you provide enough includes, the compiler will happily churn away at the source code and come up with the desired object file. It is up to us to minimize the number of includes and to follow those rules. While it seems fairly straightforward at first (after all, it is only three rules), things get more complicated as soon as heavy refactoring starts. As you split classes, move functions, and consolidate functionality, there might be several unnecessary headers. The only quick way to verify whether they’re needed or not is to, gulp, comment them out and try to compile the file.

This situation is more common in large files, which are the ones you’re most likely going to be refactoring. If left unchecked, you’ll soon be left with a myriad little tendrils connecting your file to the rest of the code without getting any benefit from it. Wouldn’t it be great if there was an automated tool that would check that?

Automating the Guidelines

I did a quick search and I couldn’t find any tools or scripts that did exactly what I wanted. I suppose that a massive C/C++ style-checker tool might look for some of those things (like redundant includes), but nothing jumped out. Most programs are more concerned with checking logical errors and constructs than looking at the physical structure. I started from Scott Meyers’ summary of major C++ checkers. In particular I looked at CodeCheck, PC-Lint, and CodeWizard. I admit that I didn’t look too deep into them, so maybe they also check for some of these guidelines, but it’s certainly not their biggest selling point from reading their web sites.

As a quick challenge, I decided to try and write a quick script to check against those guidelines. Soon I realized that you can’t really have the word “quick” in anything related to parsing C++. I was quickly reminded how ugly the language is, how many quirks it has, how much baggage it a carries around. Java looks mighty tempting sometimes.

I decided that if I was going to have a chance to do this, I would need to leverage other software to parse the source code for me and the script could work directly on the abstract representation of the program. Not exactly what I had in mind, but I stumbled on the XML and perlmod output generated by Doxygen. I have been using Doxygen for years, but I always thought of it as a pretty documentation generator. I never realized what a powerful and robust C++ parser it was until now.

In a few hours I was able to put together a quick Perl script that hooks up to the perlmod output of Doxygen that checked against those rules. Guideline #1 was the easiest one to check against. Really, you don’t even need a fancy parser for that. Guideline #2 is already checked by the compiler (if you don’t have enough includes, the program won’t compile). Guideline #3 is the trickiest one. Still, the script makes a valiant effort and checks for the most common cases. It’ll try to detect whether an include is not needed at all, or whether it can be replaced with a forward declaration.

However, the script is a conservative one and will not always detect that a header is necessary. It deals fine with simple constructs such as member variables, enums, references, and pointers. However it seems that Doxygen has very little knowledge of preprocessor directives, so it won’t catch any #defines brought in from header files. Doxygen also seems very limited in how it deals with inline functions (probably because Doxygen is looking mostly at the logical rather than the physical structure of the program). It would probably be possible to deal with templates, but I didn’t bother making it that far. The script also detects duplicate includes, both in header files and cpp files.

Even with all those limitations, the script can easily deal with about 80% of the cases in most of the code I work with on a regular basis. It was certainly insightful running it on the source code for some of the libraries I work with. I saw plenty of instances where headers could in fact be removed and the physical structure of the program improved. I would love to see a robust tool along these lines that could be ran quickly enough after each compile, or at least once a night in our automated build machine.

analyze_includes.pl

Part two of this article will look at the consequences of physical structure on compile times, and what we can do to improve that.