Last year I wrote about the basics of Data-Oriented Design (see the September 2009 issue of Game Developer). In the time since that article, Data-Oriented Design has gained a lot of traction in game development and many of teams are thinking in terms of data for some of the more performance-critical systems.

As a quick recap, the main goal of Data-Oriented Design is achieving high-performance on modern hardware platforms. Specifically, that means making good use of memory accesses, multiple cores, and removing any unnecessary code. A side effect of Data-Oriented Design is that code becomes more modular and easier to test.

Data-Oriented Design concentrates on the input data available and the output data that needs to be generated. Code is not something to focus on (like traditional Computer Science), but is something that is written to transform the input data into the output data in an efficient way. In modern hardware, that often means applying the same code to large, contiguous blocks of homogeneous memory.

Applying Data-Oriented Design

It’s pretty easy to apply these ideas to a self-contained system that already works over mostly-homogeneous data. Most particle systems in games are probably designed that way because one of their main goals is to be very efficient and handle a large number of particles at high framerates. Sound processing is another system that is naturally implemented thinking about data first and foremost.

So, what’s stopping us from applying it to all the performance-sensitive systems in a game code base? Mostly just the way we think about the code. We need to be ready to really look at the data and be willing to split up the code into different phases. Let’s take a high-level example and see how the code would have to be restructured when optimizing for data access.

Listing 1 shows pseudocode for what could be a typical update function for a generic game AI. To make things worse, that function might even be virtual and different types of entities might implement it in different ways. Let’s ignore that for now and concentrate on what it does. In particular, the pseudocode highlights that, as part of the entity update, it does many conditional ray casting queries, and also updates some state based on the results of those queries. In other words, we’re confronted with the typical tree-traversal code structure so common in Object-Oriented Programming.

void AIEntity::Update(float dt)
{
    DoSomeProcessing();
    if (someCondition && Raycast(world))
       DoSomething();
    if (someOtherCondition && BunchOfRayCasts(world))
       DoSomethingElse();
    UpdateSomeOtherStuff();
}

Listing 1

Ray casts against the world are a very common operation for game entities. That’s how they “see” what’s around them and that’s what allows them to react correctly to their surroundings. Unfortunately, ray casting is a very heavy weight operation, and it involves potentially accessing many different areas in memory: a spatial data structure, other entity representations, polygons in a collision mesh, etc.

Additionally, the entity update function would be very hard to paralellize on multiple cores. It’s unclear how much data is read or written in that function, and some of that data (like the world data structure) might be particularly hard and expensive to protect from updates from multiple threads.

If we re-organize things a bit, we can significantly improve performance and paralellization.

Break Up And Batch

Without seeing any of the details of what’s going on inside the entity update, we can see the raycasts sticking out like a sore thumb in the middle. A raycast operation is fairly independent of anything else related to the entity, it’s heavyweight, and there could be many of them, so it’s a perfect candidate to break up into a separate step.

Listing 2 shows how the broken up entity update code would look like. The update is now split in two different passes: The first pass does some of the updating that can be done independently of any ray casts, and decides which, if any, raycasts need to be performed sometime this frame.

void AIEntity::InitialUpdate(float dt, RayCastQueries& queries)
{
    DoSomeProcessing();
    if (someCondition)
        AddRayCastQuery(queries);
    if (someOtherCondition)
        AddBunchOfRayCasts(queries);
}

void AIEntity::FinalUpdate(const RayCastResults& results)
{
    UpdateSomeOtherStuff(results);
}

Listing 2

The game code in charge of updating the game processes all AI entities in batches (Listing 3). So instead of calling InitialUpdate(), solve ray casts, and FinalUpdate() for each entity, it iterates over all the AI entities calling InitialUpdate() and adds all the raycast query requests to the output data. Once it has collected all the raycast queries, it can process them all at once and store their results. Finally, it does one last pass and calls FinalUpdate() with the raycast results on each entity.

RayCastQueries queries;
for (int i=0; i<entityCount; ++i)
    entities[i].InitialUpdate(dt, queries);

// Other update that might need raycasts

RayCastResults results;
for (int i=0; i<queries.count; ++i)
    PerformRayCast(queries[i], results);

for (int i=0; i<entityCount; ++i)
    entities[i].FinalUpdate(results);

Listing 3

By removing the raycast calls from within the entity update function, we’ve shortened the call tree significantly. The functions are more self-contained, easier to understand, and probably much more efficient because of better cache utilization. You can also see how it would be a lot easier to parallelize things now by sending all raycasts to one core while another core is busy updating something unrelated (or maybe by spreading all raycasts across multiple cores, depending on your level of granularity).

Note that after calling InitialUpdate() on all entities, we could do some processing on other game objects that might also need raycast queries and collect them all. That way, we can batch all the raycasts at compute them all at once. For years, we’ve been drilled by graphics hardware manufacturers how we should batch our render calls and avoid drawing individual polygons. This is the same way: By batching all raycasts in a single call, we have the potential to achieve much higher performance.

Splitting Things Up

Have we really gained much by reorganizing the code this way? We’re doing two full passes over the AI entities, so wouldn’t that be worse from a memory point of view? Ultimately you need to measure it and compare the two. In modern hardware platforms, I would expect performance to be better because, even though we’re traversing through the entities twice, we’re using the code cache much better and we’re accessing them sequentially (which allows us to pre-fetch the next one too).

If this is the only change we make to the entity update, and the rest of the code is the usual deep tree traversal code, we might not have gained much because we’re still blowing the cache limits with every update. We might need to apply the same design principles to the rest of the update function to start seeing performance improvements. But at the very least, even with this small change, we have made it easier to parallelize.

One thing we’ve gained now is the ability to modify our data to fit the way we’re using it, and that’s the key to big performance gains. For example, after seeing how the entity is updated in two separate passes, you might notice that only some of the data that was stored in the entity object is touched from the first update, and the second pass accesses more specific data.

At that point we can split up the entity class into two different sets of data. One of the most difficult things at this point is naming these sets data in some meaningful way. They’re not representing real objects or real-world concepts anymore, but different aspects of a concept, broken down purely by how the data is processed. So what before was an AIEntity, can now become a EntityInfo (containing things like position, orientation, and some high-level data) and AIState (with the current goals, orders, paths to follow, enemies targeted, etc).

The overall update function now deals with EntityInfo structures in the first pass, and AIState structures in the second pass, making it much more cache friendly and efficient.

Realistically, both the first and second passes will have to access some common data (for example the entity current state: fleeing, engaged, exploring, idle, etc). If it’s only a small amount of data, the best solution might be to simply duplicate that data on both structures (going against all “common wisdom” in Computer Science). If the common data is larger or is read-write, it might make more sense to give it separate data structure of its own.

At this point, a different kind of complexity is introduced: Keeping track of all the relationships from the different structures. This can be particularly challenging while debugging because some of the data belonging to the same logical entity isn’t stored in the same structure and it’s harder to explore in a debugger. Even so, making good use of indices and handles makes this problem much more manageable (see Managing Data Relationships in the September 2008 issue of Game Developer).

Conditional Execution

So far things are pretty simple because we’re assuming that every AI entity needs both updates and some ray casts. That’s not very realistic because entities are probably very bursty: sometimes they need a lot of ray casts, and sometimes they’re idle or following orders and don’t need any for a while. We can deal with this situation by adding a conditional execution to the second update function.

The easiest way to conditionally execute the update would be to add an extra output parameter to the FirstUpdate() function indicating whether the entity needs a second update or not. The same information could be derived form the calling code depending on whether there were any raycasts queries added. Then, in the second pass, we only update those entities that appear in the list of entities requiring a second update.

The biggest drawback of this approach is that the second update went from traversing memory linearly to skipping over entities, potentially affecting cache performance. So what we thought was going to be a performance optimization ended up making things slower. Unless we’re gaining a significant performance improvement, it’s often better to simply do the work for all entities whether they need it or not. However, if on average less than 10 or 20 percent of the entities need a ray cast, then it might be worth avoiding doing the second update on all the other entities and paying the conditional execution penalty.

If the number of entities to be updated in the second pass is fairly small, another approach would be to copy all necessary data from the first pass into a new temporary buffer. The second pass can then process that data sequentially without any performance penalties and it would completely offset the performance hit of copying the data.

Finally, another alternative, especially if the conditional execution remains fairly similar from frame to frame, is to relocate entities that need raycasting together. That way the copying is minimal (swapping an entity to a new location in the array whenever it needs a raycast), and we still get the benefit of the sequential second update. For this to work all your entities need to be fully relocatable, which means working with handles or some other indirection, or updating all the references to the entities that swapped places.

Different Modes

What if the entity can be in several, totally different modes of execution? Even if it’s the same type of entity, traversing through them linearly calling the update function could end up using completely different code for each of them, so it will result in poor code cache performance.

There are several approached we can take in a situation like that:

• If the different execution modes also are tied to different parts of the entity data, we could treat them as if they were completely different entities and break each of their data components apart. That way, we can iterate through each type separately and get all the performance benefits.
• If the data is mostly the same, and it’s just the code that changes, we could keep all the entities in the same memory block, but rearrange them so that entities in the same mode are next to each other. Again, if you can relocate your data, this is very easy and efficient (it only requires swapping a few entities whenever the state changes).
• Leave it alone! Ultimately, Data-Oriented Design is about thinking about the data and how it affects your program. It doesn’t mean you always have to optimize every aspect of it, especially if the gains aren’t significant enough to warrant the added complexity.

The Future

Is thinking about a program in terms of data and doing these kind of optimizations a good use of our time? Is this all going to go away in the near future as hardware improves? As far as we can tell right now, the answer is a definite no. Efficient memory access with a single CPU is a very complicated problem, and matters get much worse as we add more cores. Also, the amount of transistors in CPUs (which is a rough measure of power) continues to increase much faster than memory access time. That tells us that, barring new technological breakthroughs, we’re going to be dealing with this problem for a long time. This is a problem we need to deal with right now and build our technology around it.

There are some things I’d like to see in the future to make Data-Oriented Design easier. We can all dream up of a new language that will magically allow for great memory access and easy paralellization, but replacing C/C++ and all existing libraries is always going to be a really hard sell. Historically, the best advances in game technology have been incremental, not throwing away existing languages, tools, and libraries (that’s why we’re still stuck with C++ today).

Here are two things that could be done right now and work with our existing codebases. I know a lot of developers are working on similar systems in their projects, but it would be great to have a common implementation released publicly so we can all build on top of them.

Language

Even though a functional language might be ideal, either created from scratch or reusing an existing one, we could temporarily extend C to fit our needs. I would like to see a set of C extensions where functions have clearly defined inputs and outputs, and code inside a function is not allowed to access any global state or call any code outside that function (other than local helper functions defined in the same scope). This could be done as a preprocessor or a modified C compiler, so it remains very compatible with existing libraries and code.

void FirstEntityUpdate(input Entities* entities, input int entityCount, output RayCastQueries* queries, output int queryCount);

Dependencies between functions would be expressed by tying the outputs of some functions to the input of other functions. This could be done in code or through the use of GUI tools that help developers manage data relationships visually. That way we can construct a dependency diagram of all the functions involved in every frame.

Scheduler

Once we have the dependencies for every function, we can create a directed acyclic graph (DAG) from it, which would give us a global view of how data is processed every frame. At that point, instead of running functions manually, we can leave that job in the hands of a scheduler.

The scheduler has full information about all the functions as well as the number of available cores (and information from the previous frame execution if we want to use that as well). It can determine the critical path through the DAG and optimize the scheduling of the tasks so the critical path is always being worked on. If temporary memory buffers are a limitation for our platform, the scheduler can take that into account and trade some performance time for a reduced memory footprint.

Just like the language, the scheduler would be a very generic component, and could be made public. Developers could use it as a starting point, build on top of it, and add their own rules for their specific games and platforms.

Even if we’re not ready to create those reusable components, every developer involved in creating high-performance games should be thinking about data in their games right now. Data is only going to get more important in the future as the next generation of consoles and computers rolls in.

This article was originally printed in the September 2010 issue of Game Developer.

We’ve all had things nagging us in the back of our minds. They’re nothing we have to worry about this very instant, just something we need to do sometime in the future. Maybe that’s changing those worn tires in the car, or making an appointment with the dentist about that tooth that has been bugging you on and off.

Dynamic memory allocation is something that falls in that category for most programmers. We all know we can’t just go on allocating memory willy-nilly whenever we need it, yet we put off dealing with it until the end of the project. By that time, deadlines are piling on the pressure and it’s usually too late to make significant changes. With a little bit of forethought and pre-planning, we can avoid those problems and be confident our game is not going to run out of memory in the most inopportune moment.

On-Demand Dynamic Memory Allocation

The easiest way to get started with memory management is to allocate memory dynamically whenever you need it. This is the approach many software engineering books consider as ideal and it’s often encouraged in Computer Science classes.

It’s certainly an easy approach to use. Need a new animation instance when the player is landing on a ledge? Allocate it. Need a new sound when we reach the goal? Just allocate another one!

On-demand dynamic memory allocation can help to keep memory usage to a minimum, because, you only allocate the memory that you need and no more. In practice it’s not quite as neat and tidy because there can be a surprisingly large amount of overhead per allocation, which adds up if programmers become really allocation-happy.

It’s also a good way to shoot yourself in the foot.

Games don’t live inside a Computer Science textbook, so we have to deal with real world limitations, which make this approach cumbersome, clunky, and potentially disastrous. What can go wrong with on-demand dynamic memory allocation? Almost everything!

Limited Memory

Games, or any software for that matter, run on machines with limited amounts of memory. As long as you know what that limit is, and you keep extremely careful track of your memory usage, you can remain under the limit. However, since the game is allocating memory any time it needs it, there will most likely come a time when the game tries to allocate a new block but there is no memory left. What can you do then? Nothing easy I’m afraid. You can try to free an older memory block and allocate the new one there, or you can try to make your game tolerant to running out of memory. Both those solutions are very complex and difficult to implement correctly.

Even setting memory budgets and sticking to them can be very difficult. How can a designer know that a given particle system isn’t going to run out of memory? Are these AI units going to create too many pathfinding objects and crash the game? Hard to say until we run the game in all possible combinations. And even then, how do you know it isn’t going to crash five minutes later? Or ten? It’s almost impossible to know for certain.

If you insist in using this approach, at the very least, you should tag all memory allocations, so you have an idea of how memory is being used. You can either tag each allocation based on what system initiated it (physics, textures, animation, sound, AI, etc) or even on the filename where it originated, which has the advantage that it can be automated and should still give you a good picture of the overall memory usage.

Memory Fragmentation

Even if you take lots of pain not to go over your the available memory, you might still run into trouble because of memory fragmentation. You might have enough memory for a new allocation, but in the form of many small memory blocks instead of a large contiguous one. Unless you provide your own memory allocation mechanism, fragmentation is something that is very hard to track on your own, so you can’t even be ready for it until the allocation fails.

Virtual Memory

Virtual memory could solve all those problems. In theory, if you run out of real memory, the operating system swaps out some older, unused pages to disk and makes room for the new memory you requested. In practice, it’s just a bad caching scheme because it can be triggered at the worst possible moment, and it doesn’t know about what data it’s swapping out or how your game uses it.

Games, unlike most other software, have a “soft realtime” requirement: The game needs to keep updating at an acceptable interactive rate, which is somewhere around 15 or more frames per second. That means that gamers are going to make a trip to the store to return your game if it pauses for a couple of seconds every few minutes to “make some room” for new memory. So relying on virtual memory isn’t a particularly attractive solution.

Additionally, lots of games run in platforms with fixed amounts of RAM and no virtual memory. So when memory runs out, things won’t get slow and chuggy, they’ll crash hard. When the memory is gone, it’s really gone.

Performance Problems

There are some performance issues that are relatively easy to track down and fix. Usually ones that occur every frame and are happening in a single spot: some expensive operation, a O(n3) algorithm, etc. Then there are performance problems introduced by dynamic memory allocations, which can be really hard to track down.

Standard malloc returns memory pretty quickly, and usually doesn’t ever register on the the profiler. Every so often though, whenever the game has been running for a while and memory is pretty fragmented, it can spike up and cause a significant delay for just a frame. Trying to track down those spikes has caused more than one programmer to age prematurely. You can avoid some of those problems by using your own memory manager, but don’t attempt to write a generic one yourself from scratch. Instead start with some of the ones listed in the references.

Malloc spikes are not the only source of performance problems. Allocating many small blocks of memory can lead to bad cache coherence when the game code access them sequentially. This problem usually manifests itself as a general slowdown that can’t be narrowed down in the profiler. With today’s hardware of slow memory systems and deep caches, good memory access patterns are more important than ever.

Keeping Track Of Memory

Another source of problems with dynamic memory allocation are bugs in the logic that keeps track of the allocated memory blocks. If we forget to free some of them, our program will have memory leaks and has the potential to run out of memory.

The flip side of memory leaks are invalid memory access. If we free a memory block and later we access it as if it were allocated, we’ll either get a memory access exception, or we’ll manage to corrupt our own game.

Some techniques, such as reference counting and garbage collection can help keep track of memory allocations, but introduce their own complexities and overhead.

Introducing Pre-allocation

On the opposite corner of the boxing ring is the purely pre-allocated game. It excels at everything that the dynamically-allocated game is weak at, but it has a few weaknesses of its own. All in all, it’s probably a much safer approach for most games though.

The idea behind a pre-allocation memory strategy is to allocate everything once and never have to do any dynamic allocations in the middle of the game. Usually you grab as big a block of memory as you can, and then you carve it out to suit your game’s needs.

Some advantages are very clear: no performance penalties, knowing exactly how your memory is used, never running out of memory, and no memory fragmentation to worry about. There are some other more subtle advantages, such as being able to put data in contiguous areas of memory to get best cache coherency, or having blazingly-fast load times by loading a baked image of a level directly into memory.

The main drawback of pre-allocation is that is more complex to implement than the dynamic allocation approach and it takes some planning ahead.

Know Your Data

For preallocation to work, you need to know ahead of time how much of every type of data you will need in the game. That can be a daunting proposition, especially to those used to a more dynamic approach. However, with a good data baking system (see last month’s Inner Product column), you can get a global view of each level and figure out how big things need to be.

There is one important design philosophy that needs to be adopted for preallocation to work: Everything in the game has to be bounded. That shouldn’t feel too restrictive; after all, the memory in your target platform is bounded, as well as every single resource. That means that everything that can create new objects, including high-level game constructs, should operate on a fixed number of them. This might seem like an implementation detail, but it often bubbles up to what’s exposed to game designers. A common example is an enemy spawner. Instead of designing a spawner with an emission rate, it should have a fixed number of enemies it can spawn (and potentially reuse them after they’re dead).

Potentially Wasted Space

If you allocate enough data for the worst case in your game, that can lead to a lot of unused data most of the time. That’s only an issue if that unused data is preventing you from adding more content to the game. We might initially balk at the idea of having 2000 preallocated enemies when we’re only going to see 10 of them at once. But when you realize that each of those enemies is only taking 256 bytes and the total overhead is 500 KB, which can be easily accommodated in most modern platforms today.

Preallocation doesn’t have to be as draconian as it sounds though. You could relax this approach and commit to having each level preallocated and never having dynamic memory allocations while the game is running. That still allows you to dynamically allocate the memory needed for each level and keep wasted space to a minimum. Or you can take it even further and preallocate the contents of memory blocks that are streamed in memory. That way each block can be divided in the best way for that content and wasted space is kept to a minimum.

Reuse, RecycleM

If you don’t want to preallocate every single object you’ll ever use, then you can create a smaller set, and reuse them as needed. This can be a bit tricky though. First of all, it needs to be very much specific to the type of object that is reused. So particles are easy to reuse (just drop the oldest one, or the ones not in view), but might be harder with enemy units or active projectiles. It’s going to take some game knowledge of those objects to decide which ones to reuse and how to do it.

It also means that systems need to be prepared to either fail an allocation (if your current set of objects is full and you don’t want to reuse an existing one), or they need to cope with an object disappearing from one frame to another. That’s a relatively easy problem to solve by using handles or other weak references instead of direct pointers.

Then there’s the issue that reusing an object isn’t as simple as constructing a new one. You really need to make sure that when you reuse it, there’s nothing left from the object it replaced. This is easy when your objects are just plain data in a table, but can be more complicated when they’re complex C++ classes tied together with pointers. In any case, you can’t apply the Resource Acquisition Is Initialization (RAII) pattern, but it doesn’t seem to be a pattern very well suited for games, and it’s a small price to pay for the simplicity that preallocation provides.

Specialized Heaps

Truth be told, a pure pre-allocated approach can be hard to pull off, especially with highly dynamic environments or games with user-created content. Specialized heaps is a combination of dynamic memory allocation and pre-allocation that takes the best of both worlds.

The idea behind specialized heaps is that the heaps themselves are pre-allocated, but they allow some form of specialized dynamic allocation within them. That way you avoid the problems of running out of memory, or memory fragmentation globally, but you still can perform some sort of dynamic allocation when needed.

One type of specialized heaps is based on the object type. If you can guarantee that all objects allocated in that heap are going to be of the same size, or at least a multiple of a certain size, memory management becomes much easier and less error prone, and removes a lot of the complexity of a general memory manager.

My favorite approach for games is to create specialized heaps based on the lifetime of the objects allocated in them. These heaps use sequential allocators, always allocating memory from the beginning of a memory block. When the lifetime of the objects is up, the heap is reset and allocations can start from the beginning again. The use of a simple sequential allocator bypasses all the insidious problems of general memory management: fragmentation, compaction, leaks, etc. See the code in http://gdmag.com/resources/code.htm for an implementation of a SequentialAllocator class.

The heap types most often used in games are:

• Level heap. Here you allocate all the assets and data for the level at load time. When the level is unloaded, all objects are destroyed at once. If your game makes heavy use of streaming, this can be a streaming block instead of a full level.
• Frame heap. Any temporary objects that only need to last a frame or less get allocated here, and destroyed at the end of the frame.
• Stack heap. This one is a bit different from the others. Like the other heaps, it uses a sequential allocator and objects are allocated from the beginning, but instead of destroying all objects at once, it only destroys objects up to the marker that is popped fro the stack.

What About Tools?

You can take everything I’ve written here, and (almost) completely ignore it for tools. I fall in the camp of the programmers who consider the runtime as a totally separate beast from the tools. That means that the runtime can be lean and mean and minimalistic, but I can relax and use whatever technique makes me more productive when writing tools. That means you can allocate memory any time you want, you can use complex libraries like STL and Boost, etc. Most tools are going to run on a beefy PC and a few extra allocations here and there won’t make any difference.

Be careful with performance-sensitive tools though. Tools that build assets or compute complex lighting calculations might be a bottleneck in the build process. In that case, performance becomes crucial again and you might want to be a bit more careful about memory layout and cache coherency.

On the other hand, if the tool you’re writing is not performance sensitive, you should ask yourself if it really needs to be written in C++. Maybe C# or Python are better languages if all you’re doing is transforming XML files or verifying that a file format is correct. Trading performance for ease of development is almost always a win with development tools.

Next time you reach out for a malloc inside your main loop, think about how it can fail. Maybe you can pre-allocate that memory and stop worrying about what’s going to happen the day you prepare the release candidate.

This article was originally printed in the February 2009 issue of Game Developer.

Let me get one thing out of the way to stop all the trigger-happy, const-bashing, would-be-commenters: const doesn’t make any guarantees that values don’t change.

You can change a const variable by casting the constness away, or referencing it through a pointer, but you really had to go out of your way to do that. If it helped with that, const would solve (or improve) the memory aliasing problem like Hitler pointed out. It doesn’t, so const is pretty weak as far as promises go. It just says “I, the programmer, promise not to change this value on purpose (unless I’m truly desperate)”. Still, even a promise like that goes a long way helping with readability and maintenance.

With that out of the way, what exactly do I mean by using const everywhere?

Const non-value function parameters

Any reference or pointer function parameters that are pointing to data that will not be modified by the function should be declared as const. If you’re going to use const just for one thing, this is the one to use. It’s invaluable glancing at a function signature and seeing which parameters are inputs and which ones are outputs.

void Detach(PhysicsObject& physObj, int attachmentIndex, const HandleManager& mgr);

Marking those parameters as const also serves as a warning sign in case a programmer in the future tries to modify one of them. Imagine the disaster if the calling code assumes data never changes, but the function suddenly starts modifying that data! const won’t prevent that from happening, but will remind the programmer that he’s changing the “contract” and needs to revisit all calling code and check assumptions.

Const local variables

This is a very important use of const and one of the ones hardly anyone follows. If I declare a local (stack) variable and its value never changes after initialization, I always declare it const. That way, whenever I see that variable used later in the code, I know that its value hasn’t changed.

const Vec2 newPos = AttachmentUtils::ApplySnap(physObj, unsnappedPos);
const Vec2 deltaPos = newPos - physObj.center;
physObj.center = newPos;

This is one of the reasons why I did a 180 on the ternary C operator (?). I used to hate it and find it cryptic and unreadable, but now I find it compact and elegant and it fulfills my const fetish very well.

Imagine you have a function that is going to work in one of two objects and you need to compute the index to the object to work on. You could do it this way:

int index;
if (some condition)
	index = 0;
else
	index = 2;

DoSomethingWithIndex();

Not only does that take several lines not to do much, but index isn’t const (argh!). So every time I see index anywhere later on in that function, I’m going to have to spend the extra mental power to make sure nothing has changed (and, with my current coding style, I would assume it has changed).

Instead, we can simply do this:

const int index = (some condition) ? 0 : 2;
DoSomethingWithIndex();

Ahhhh… So much better!

Const member variables

This one doesn’t really apply to me anymore because I don’t use classes and member variables. But if you do, I strongly encourage you do mark every possible member function as const whenever you can.

The only downside is that sometimes you’ll have some internal bit of data that is really not changing the “logical” state of an object, but it’s still modifying a variable (usually some caching or logging data). In that case, you’ll have to resort to the mutable keyword.

Const value function parameters

Apparently I’m not a total Const Nazi because this is one possible use of const that I choose to skip (even though I tried it for a while because of Charles).

Marking a value function parameter as const doesn’t make any difference from the calling code point of view, but it serves the same purpose as marking local stack variables as const in the implementation of the function. You’re just saying “I’m not going to modify that parameter in this function” so it makes the code easier to understand.

I’m actually all for this, but the only reason I’m not doing it is because C/C++ makes it a pain. Marking parameters as const in the function declaration adds extra verbosity and doesn’t help the person browsing the functions at all. You could actually put the const only in the function definition and it will work, but at that point the declaration and the definition are different, so you can’t copy and paste them or use other automated tools or scripts.

The concept of const is one of the things I miss the most when programming other languages like C#. I don’t understand why they didn’t add it to the language. On something like Python or Perl I can understand because they’re supposed to be so free form, but C#? (Edit: How about that? Apparently C# has const. It was either added in the last few years or I completely missed it before). It also really bugs me that Objective C or the Apple API doesn’t make any use of const.

Frankly, if it were up to me, I would change the C/C++ language to make every variable const by default and adding the nonconst or changeable (or take over mutable) keyword for the ones you want to modify. It would make life much more pleasant.

But then again, that’s why the call me the Const Nazi.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

Every developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

• All my code is written in C and C++ (except for a bit of ObjC and assembly).
• It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
• I always try to write important code through Test-Driven Development.
• I’m the only programmer (and only designer).
• Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

• I’m showing all the data there is and how big it is. Nothing is hidden.
• I’m making it clear that it’s free of pointers and temporary variables.
• I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

If you haven’t seen the presentation, go download it right now. It’s really well put together and he has some great detailed examples on how caches are affected with traditional object-oriented design vs. a more data-oriented approach.

[1] Thanks to Christer Ericson for pointing that out.

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

Figure 1a. Call sequence with an object-oriented approach

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

Figure 1b. Call sequence with a data-oriented approach

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

Thinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,” with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,” but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is There Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

This article was originally printed in the September 2009 issue of Game Developer.

Here’s a Korean translation of this article by Hakkyu Kim.

Size

Charles and I were priding ourselves in keeping things small and minimal. But truth be told, it’s not like we were keeping track of how many lines of code we had written. Were things as small as we hoped they were?

The most convenient way of counting lines of code that I know is CLOC. It’s an extremely easy to use open source program which counts the lines of code in a codebase, gives very detailed information, strips out whitespace, breaks things down by language, and does just about everything you’d want from a program like that.

Running it on the latest version of our code (not including any 3rd party libraries) produces this:

    1621 text files.    1579 unique files.    3721 files ignored.
-------------------------------------------------------------------------------
Language                    files          blank        comment          code
-------------------------------------------------------------------------------
C++                            485          13577            303          46181
C#                             324           4935            712          22966
C/C++ Header                   407           4153             95          11975
MSBuild scripts                 18              0            126          1490
-------------------------------------------------------------------------------
SUM:                          1234          22665           1236          82612

Almost 60K lines of C++ code seemed very high. At first I thought it was because CLOC was counting files twice: once in their regular location and once in the .svn directory, but apparently it’s already removing all duplicates, so that wasn’t it.

Almost more scary than the amount of C++ code (which is all our runtime and some of our tools) is the amount of C# code. For a language that claims to be of significantly higher level than C++, that’s quite a mouthful of code!

Another surprising count in there is the number of lines with comments. Since we make heavy use of TDD, I really didn’t expect more than a couple dozen lines of code in the whole codebase. Still, I’m kind of proud that we have less than one line of code per file on average

Here’s a more detailed breakdown, with the line count just for our runtime (engine and game):

1089 text files.    1053 unique files.    2338 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            441          11997            245          40943
C/C++ Header                   385           3964             90          11405
-------------------------------------------------------------------------------
SUM:                           826          15961            335          52348

and for our tools:

532 text files.     531 unique files.    1383 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                             324           4935            712          22966
C++                             44           1580             58           5238
MSBuild scripts                 18              0            126           1490
C/C++ Header                    23            199              5            591
-------------------------------------------------------------------------------
SUM:                           409           6714            901          30285

Tests

Then I realized that a good chunk of those were tests. So excluding all directories matching *Tests* gets the following result:

1206 text files.    1187 unique files.    4199 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            283           6464            150          22464
C#                             213           2636            534          12402
C/C++ Header                   380           3824             94          10782
MSBuild scripts                 12              0             84            978
-------------------------------------------------------------------------------
SUM:                           888          12924            862          46626

A bit more than half the C++ code consisted of tests. That’s pretty consistent with my experience with TDD. C# seems to follow a similar percentage as well.

As for the exact number of tests, running a grep for TEST shows all the C++ tests:

C:\pow2>grep -r TEST SweetPea Engine Tools | grep -v svn | wc   2163    3620  221953

And doing the same thing with [Test] brings up all C# tests:

C:\pow2>grep -r \[Test\] SweetPea Engine Tools | grep -v svn | wc   735    1470  52717

That means that our average C++ test is about 11.5 lines long, and C# tests 14.4. Frankly, that sounds rather high. We make heavy use of fixtures whenever possible and each test usually only checks for a single condition (even if it involves a couple check statements). I suppose that number is higher than expected because it probably includes all the lines from #include statements and all the fixtures as part of the average.

* If we only count cpp files, that goes down to 49%

I was curious about that last part of checking a single thing per test, so I ran a grep for the number of CHECK statements in our code:

C:\pow2>grep -r CHECK SweetPea Engine Tools | grep -v svn | wc   3886   15079  399598

That’s 1.8 CHECK statements per TEST, which is about right. Even though we’re checking for a single condition, we’ll often check a couple things about it (i.e. the camera stopped and it reached its final destination).

Build Times

So, given that amount of code, how long does it take to build it? Clearly it depends on your hardware. Since we’re not exactly rolling in money, we don’t have particularly powerful machines. Here at home, I’m using a modest Core 2 Duo E4300 (overclocked to 2.6 GHz) with fast memory and a relatively fast SATA hard drive, so that’s what I used for all my timings.

A full build of our game, plus all the libraries, all the tests, and running all the tests takes exactly 1 minute and 10 seconds. That’s pretty good for two reasons:

• When we work with the game we don’t build and run the unit tests for the engine. We have a separate solution for that. A full build of just the engine, the game, and the game unit tests only takes 43 seconds.
• The game itself is a fairly large project and devenv doesn’t know how to paralellize that build, so it’s only using half the available CPU power for about half the build time.

An incremental build after changing a single cpp file takes slightly over a second (including half a second of unit test execution).

As you can imagine, working with that codebase is a dream come true. Snappy, responsive. Nothing is hard enough that can’t be changed.

Unfortunately that’s where the fairy tale ends. The tools are another story altogether. Our C# tools, with all their unit tests, build in a mere 18 seconds, and the C++ tools in 1 minute and 10 seconds. That’s not too bad, except that it’s a surprisingly large amount of time for the C++ tools since there aren’t that many of them.

Here’s the kicker, doing another build without changing a single thing take 38 seconds. Whoa! We’re doing some C++/CLI trickery and apparently dependency checking is totally broken in VS2005 (either that, or we just don’t know how to set it up right).

Keeping things fast

What’s the secret of a lighting-fast build? Clearly, keeping the code size down is crucial. If your codebase is 2 million lines of code, builds are going to be painful no matter what. But they can be a little less painful with some gentle care.

One of the main build-time killers that we’re avoiding is the use of STL or Boost. Those libraries pull in everything and the kitchen sink, and their heavy use of templates make build and link time go through the roof. No thanks.

Our template use is pretty minimal. We have a couple containers (which I love and I’ll write about it one of these days) and that’s about it.

We’re pretty anal when it comes to keeping physical dependencies to a minimum. We forward declare aggressively, and we only include the headers that are necessary for each cpp file (PC Lint is “kind” of enough to remind us every time we have unnecessary #includes). We’re not using external include guards or #pragma once.

Precompiled headers are either not used, or kept to a minimum. I think the only project that uses them is the game and only for Havok headers. We don’t even have windows.h in a precompiled header (which would be a really bad idea because you’d be putting all the junk in windows.h available to your whole program).

Finally, we are using incremental links whenever possible. I remember a few versions of Visual Studio ago they were pretty broken, but they’re not giving us any problems. The only caveat is that if you modify a static library your program is linking with, it will force a full link. So they’re really only good for modifying the executable itself.

We’re not using any distributed builds. First of all, we don’t have enough computers to make it worthwhile. And second, I had horrible experiences with distributed builds in the past. They would help with a badly structured codebase, at the cost of longer incremental builds and mysterious spurious bad builds. Besides, once they’re in place, they tend to encourage even further disregard for keeping dependencies to a minimum.

How about you?

So, that’s it for the Power of Two codebase. How about you? Want to share your size, build times, or any other data?

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions.

None of that would be an issue except that TDD creates lots of unit tests and encourages almost constant incremental builds. In the past I’ve stressed how important it is to keep them fast. But this week that I was able to really feel what a dramatic difference slow unit tests make.

For many months, our unit tests ran under one second. Each library has its own set of unit tests, and so does the game and each tool. So even though we were writing lots of unit tests, we weren’t always running them all at the same time. There was a definite snappiness to coding: write test, compile, fail, write code, compile, pass, move on. Go, go, go.

As the game project got larger, we started accumulating more unit tests and testing times started creeping up into the second range. At the time, I couldn’t quite put my finger on what had changed, but there was a definite change in feel. Things weren’t quite as snappy.

Then, about a couple of weeks ago, we hit the two second mark and things definitely changed.

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 2.13 seconds.

As the tests were running, my mind would wander, thinking about what test I would write next, or whether the code I had just written would be cache-friendly, or whether we’d do Honey’s or Manhattan for lunch. I would be jerked out of my state of flow. It wasn’t just me, Charles felt it too. The difference in productivity was huge.

The Zones

For those of you laughing at those times and dismissing them as insignificant, you need to keep in mind that our build times are very short:

• An incremental build with just a change in a cpp file takes under a second (including link times).
• A full build of *all* our runtime code, including libraries, is around 50 seconds in a dual-core machine [1].

So adding on 2 seconds to the incremental build time is going from “practically instant” to “OK, I’m waiting…”.

Of course, none of this has anything to do with unit tests in particular. It’s total incremental build times (including compiling, linking, and running unit tests) that matter. An incremental build that takes 3 seconds with no unit tests is just as bad as one that is under one second with 3 second unit tests. They both interrupt the flow the same way.

I’ve often thought about how a company size affects communication and “feel”. It’s close to a step function with logarithmic scale: there’s a distinct jump from a company of 3 to a company of 10, to one of 35, to one of 100+.

I think something similar applies to build times [2]

• 0 to 2 seconds. Programmer stays in the flow. Productivity is way high.
• 2 to 8 seconds. Flow is broken. Definite feel of things being a bit painful and sluggish.
• 8 to 30 seconds. Attention wanders to other parts of the code, email, etc. Multitasking kicks in and overall productivity plummets.
• 30+ seconds. Builds seen as batch processing. After each build is started, some other action is started. Maybe even physically get up, get a drink, start a conversation. Full brain reboot between coding tasks.
• 5+ minutes. Programmers bang their heads against the desk, actively consider the merits of different forms of suicide, or at the very least start polishing their resumes.

The Fix

Having been spoiled by sub-second build times, neither one of us wanted to put up with the broken flow of 2+ second builds, so we decided it was time to fix things [3].

The good news is that our slow build times were solely due to running the unit tests. 2.13 seconds to run only about 1100+ tests is really bad, even on a so-so desktop machine like the ones we can afford. That’s almost 2ms per test! Remember that these are unit tests, not functional tests. So I expect to be able to run several thousands of these under one second. Something was way off.

Things had gotten much worse in the last couple of weeks, so it wasn’t a slow, steady creep up. Chances are we had introduced something very slow recently, which meant it was probably easy to fix.

As a first pass, we turned on the option to fail any test that takes over a certain amount of time in UnitTest++:

RunAllTests(reporter, UnitTest::Test::GetTestList(), NULL, 5); // fail any test over 5ms

That brought about tons of failing test, most of them added recently and creating one particular object in their fixtures. Digging a big deeper, that particular class had an array of 25,000 (yes, that many zeros) objects. Those objects were plain-old-data structures, but had a default constructor. Even though the constructor was simply initializing its members to zero, that was causing a significant slow down when so many objects were created. To make things worse, some of the code I had written was iterating over the array resetting the entries or shifting them up or down, repeatedly calling the constructor. Ouch!

A few minutes and lots of code hacking later, and the object didn’t call a single constructor anymore. Our test times went down by almost half. Victory!

We were already at 1 second, which is in the acceptable range, but since we were here, maybe we could improve things a bit more. After all, 1ms per test is still too slow.

UnitTest++ wasn’t reporting failing tests over 1ms very reliably. Different runs would cause very different results. Things were too spread out or other things in the computer were interfering too much.

About a fifth or so of our tests use Havok. Not with mocks, but directly. They each initialize Havok, create a world, and deal with a simple rigid body, constraint, or something of the sort. I suspected all along that the Havok system initialization and shutdown for each test was probably causing quite a slow down, so I changed it to be initialized only once before we ran any tests.

And that did the trick. With that change we were down to 0.16 seconds. Back to snappy, happy land!

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 0.16 seconds.

In release mode things are even better, but we do most development in debug mode, so that’s the really important one:

> bin\SweetPea.exe -unittestSuccess: 1145 tests passed.Test time: 0.06 seconds.

Perhaps sub-second build times are not realistic for all projects, but I think there’s a lot to be gained by keeping compile, link, and test times as short as possible. For tips on how to keep compile times down, check out these articles I wrote a while ago. If your unit tests are slow and you really can’t speed them up anymore, consider splitting them into two sets: one that is important or recent ones that runs with every build, and another, more comprehensive one that runs in the build server.

Even if you cut down from 20 seconds down to 10, you’ll be making a huge shift in how you’re able to work and iterate on your programming.

[1] We’d love to keep detailed metrics with each build: executable size, number of lines of code, number of tests, etc, etc. Unfortunately, when the rubber hits the road, that ends up in the “nice to have category” and we simply have no resources to spare on a two-man startup.

[2] For a whole whooping statistical population of one since I’m only talking about my experience. Still, I suspect that the same applies to many other programmers by scaling the function by some constant.

[3] Perfect use of agile mentality. Fix things when you need them fixed, not before. But make sure you do fix them when you need them. Otherwise it’s not agile, it’s just lazy.

enum GameFlowType
{
    Run,
    Exit,
    Restart,
    Restore,
};

Now functions can return a GameFlowType to indicate that the main loop should do. So far so good.

Except that, look at how it’s used:

GameFlowType DoMainLoop(bool render)
{
    //...
    if (someCondition)
        return Exit;
    return Run;
}

Ouch! We’ve managed to pollute the global namespace with totally generic keywords such as Run and Exit. What if I want to have an enum that controls what action an AI is going to take:

enum AIAction
{
    Enter,
    Exit,
    Stop,
    Walk,
    Run,
};

That won’t even compile because the symbols conflict with the GameFlowType ones. Even worse, it will compile just fine if they don’t happen to be included in the same compilation unit. So you might be fine all along until you write some AI code that has control over the game flow. Oops!

OK, so you can prefix all your enums like this:

enum GameFlowType
{
    GameFlowTypeRun,
    GameFlowTypeExit,
    GameFlowTypeRestart,
    GameFlowTypeRestore,
};

and

enum AIAction{
    AIActionEnter,
    AIActionExit,
    AIActionStop,
    AIActionWalk,
    AIActionRun,
};

That will technically fix the problem, but it’s a bit ugly, and we’re still polluting the global namespace (what if I wanted to have a class called AIActionEnter?).

Another potential solution is to score the enum inside the class that is using them. So we could have:

class AIAction{
public:
    enum AIActionType
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };
};

Much cleaner, do doubt, but also with its share of problems: First of all, it forces you to associate a set of enums with a class, which is not something you always want to do. But the biggest problem is that as soon as you have more than one set of enums per class, they all get promoted to the same scope:

class AIAction
{
public:
    enum AIActionBehavior
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };

    enum AIActionGroup
    {
        None,
        Self,
        Team,
        All,
    };
};

When we see one of those enums in code referred to as AIAction::Self, we really have no idea that it’s referring to that group the action is applied to, as opposed to AIAction::Enter, which is the type of action.

Our solution? Move all enums into a descriptive namespace.

namespace GameFlowType
{
    enum Enum
    {
        Invalid,
        Run,
        Exit,
        Restart,
        Restore,
    };
}

When you have a variable of that enum type, it’s listed as GameFlowType::Enum type, which clearly indicates what it does. And its values are referred to as GameFLowType::Exit. You get the ideal scoping, you’re not tied to any particular class, things are explicit but not verbose, you’re no polluting the global namespace, and it’s doesn’t affect the runtime or compilation times.

Sometimes the best solutions are the simplest ones.

I had been using a modified version of CppUnitLite for quite a while, slowly fixing the parts in need of mending, and adding new functionality as it became needed. Eventually I released most of those changes as CppUnitLite2, and I thought that would be the end of that. It turns out that was just the beginning.

Shortly after that, several people started emailing me with new functionality, fixes, patches, suggestions, etc. At the same time, Charles Nicholson was starting to get quite a bit of response as well for his own testing framework, sometimes with similar fixes and suggestions.

If this was going to grow to be a real project, developing over time, supporting users’ needs, and accepting code contributions, clearly it would make sense to join forces. And that’s exactly what we did. We grabbed the best features of each framework and created what we think it’s the best C++ unit-testing framework out there (for our needs anyway).

We took the results and put them up in Sourceforge under a very unrestrictive license, and that’s how UnitTest++ was born.

Our ultimate goal is to apply test-driven development to game development. All of the existing frameworks fell short in one area or another. Specifically, the driving forces behind the design of UnitTest++ are:

• Portability. As game developers, we need to write tests for a variety of platforms, most of which are not supported by normal software packages (all the game consoles). So the ability to easily port the framework to a new platform was very important.
• Simplicity. The simpler the framework, the easier it is to add new features or adapt it to meet new needs, especially in very limited platforms.
• Development speed. Writing and running tests should be as fast and straightforward as possible. We’re going to be running many tests hundreds of times per day, so running the tests should be fast and the results well integrated with the workflow.

UnitTest++ is a fully featured testing framework, with many of the features you may expect from Xunit-style frameworks such as fixtures and reporting. Additionally, here are some of the specific features that make UnitTest++ unique:

• Minimal work required to create a new test. For TDD development, we write lots and lots of tests, so creating a new test should be as quick and painless as possible. UnitTest++ does not require explicit test registrations, and creating new tests requires minimal work.
• Good assert and crash handling. When automated tests are executed, it’s very important not to hang the process or abort the tests any time the program crashes or an assert fails. UnitTest++ will catch and report exceptions as failed tests and print a lot of information about them. It will translate signals to exceptions as well as be able to catch invalid memory accesses and other problems.
• Minimal footprint and minimal reliance on heavy libraries. When you intend to run tests on a Cell SPU with a measly 256 KB of RAM, you really want to be as lean and mean as possible. UnitTest++ is very lightweight, and it will get even lighter weight once strstream is replaced.
• No dynamic memory allocations done by the framework, which makes it much easier to track memory leaks and generally more attractive for embedded systems.
• Multiplatform support. Right now it supports Win32, Linux, and Mac OS X out of the “box,” and other platforms will probably work without any changes. The source code makes use of platform-specific functions whenever necessary, so it is easy to hook up special timers, allocators, or reporters for specific platforms. The code comes with a makefile as well as with project and solution files for Visual Studio 2003 and 2005.
• Full set of unit tests for itself. UnitTest++ was TDD’d from the ground up, including some of the ugly macros (imagine how much fun it was bending the framework to test itself that way!). It goes without saying, all the tests are included with the source code, so you can go totally wild with them

This was just the first release. UnitTest++ is ready for full production work, but we already have plans beyond this release. This is a peek at some of the features we have in mind for upcoming versions:

• Out-of-the-box support for game console platforms (Xbox 360, PS3 PPU and SPU, and Nintendo Revolution). Of course, this is dependent on us getting permission from the console manufacturers to release some code that works with their SDK.
• Ability to trade some features (such as rich check reporting) for some extra memory. In some platforms, like Cell SPUs, you really need to go barebones.
• Suites to group tests together.
• Different reporters (HTML, GUIs, etc). Don’t worry, all those modules will be optional and well separated, so they won’t affect the simplicity of the framework.
• Per-test timings. Maybe the ability to flag some tests as not exceeding a certain amount of time, or just failing any test that goes over a threshold.
• Built-in memory leak detection.

So that’s it. Download the source code and give it a try. We welcome any comments and suggestions (best channels would be through the project mailing list or comments here). We hope you find it useful and that it makes your TDDing even more productive.

CppUnitLite2_1_1.tar.gz

About a year ago, I wrote an article comparing unit test frameworks that went to become one of the most popular in this site. The article concluded with the recommendation of one of the less well known frameworks: CxxTest.

Shortly after writing that article, it came time to roll out test-driven development at work. Interestingly, once I weighted in all the requirements, I ended up not following my own advice. Instead, I decided to go with a customized version of CppUnitLite, which was listed in the article as the second choice. The idea of an extremely simple unit testing framework was too attractive to pass up in an environment where we have to support a large variety of environments and target platforms.

At this point, we have been using CppUnitLite2 for a year at High Moon Studios to do TDD on Windows, Xbox 360, and some PS3 (we even have a stripped-down version that runs on a PS3 SPU). It has been used to unit test libraries of an engine, pipeline tools, GUI applications, and production game code.

CppUnitLite2 was originally released along with my article a year ago. It was a heavily modified version of the original CppUnitLite by Michael Feathers. This version introduces the changes we made at work as different issues came up and we needed more functionality from the framework. In particular, Jim Tilander and Charles Nicholson were responsible for a lot of those changes. By now, I doubt there’s a single line left from the original code. Additionally, CppUnitLite2 itself has a set of tests using itself in an interestingly recursive way.

What’s new

Fixtures created and destroyed around each test.

I can’t believe that the original version of CppUnitLite2 didn’t have this feature. It was the biggest glaring problem with the framework (I even pointed it out in the article). Each fixture used to be created at static initialization time, which, apart from being a really bad idea, caused a lot of objects to be created and live in memory at the same time. Considering that some of those objects would initialize/shutdown some important game systems, we clearly needed to manage their lifetime a bit better.

The test macro was changed to register a temporary test with the TestRegistry, which in turn will create the fixture at execution time. Now fixtures and tests are created just before each test, and destroyed right after, just as you would expect from any sane framework.

For those masochists out there, here’s the ugly macro for a test with fixture in all its glory. Notice how the TestRegistrar registers the dummy test class that we created as part of the macro.

 #define TEST_F(fixture, test_name)                                                      \ struct fixture##test_name : public fixture {                                        \ fixture##test_name(const char * name_) : m_name(name_) {}                       \ void test_name(TestResult& result_);                                            \ const char * m_name;                                                            \ };                                                                                  \ class fixture##test_name##Test : public Test                                        \ {                                                                                   \ public:                                                                         \ fixture##test_name##Test ()                                                 \ : Test (#test_name "Test", __FILE__, __LINE__) {}                       \ protected:                                                                      \ virtual void RunTest (TestResult& result_);                                 \ } fixture##test_name##Instance;                                                     \ TestRegistrar fixture##test_name##_registrar (&fixture##test_name##Instance);       \ void fixture##test_name##Test::RunTest (TestResult& result_) {                      \ fixture##test_name mt(m_name);                                                  \ mt.test_name(result_);                                                          \ }                                                                                   \ void fixture ## test_name::test_name(TestResult& result_)

Improved check statements

Another one of the major weaknesses of CppUnitLite as it stood was the check statements. You needed to have a custom check macro for each data type you cared to check. The generic CHECK_EQUAL macro will now work on any data type as long as it can be compared with operator == and output to a stream.

This version of CppUnitLite2 also includes some special macros to check arrays with one and two dimensions. Again, this works on any data type that supports operator == and operator[]. It came it very handy for checking equality of vectors and matrices in a game engine. For example, the following code checks that two matrices are equal:

Ironically, in some heavily restricted environments (like the PS3 cell processors), we can’t use streams, so in that case we’re forced to go back to the old specialized check macros.

Invariant statements in checks

Another really annoying feature of the previous version of CppUnitLite2 was that the code in a check statement was called multiple times, which could cause strange side effects.

Consider the following code:

The way the check statement was expanded out, both those functions could be evaluated multiple times. If the functions had side effects, they would be executed several times, causing unexpected results (for example, the check could fail but the printed result could be different). The fix wasn’t trivial because we couldn’t save off the value of each statement because they could be of any type and we have no way of finding the type from within the macro.

The check macro now expands out to a templated function, which is able to evaluate both statements only once and work with the saved value to avoid any surprising side effects.

Optional fixture initialization

As we continued hammering on the unit test framework for all our development, one clear pattern appeared that wasn’t handled by the framework: Being able to create fixtures with parameters defined in the test. For example, the fixture might do ten different steps, but we want to set a particular value in one of those steps, and by the time control reaches the test code itself it’s too late.

We started getting around that by creating different fixtures, and then by inhereting from other fixtures and changing some parameters. Both of those solutions were less than ideal (lots of code duplication and added complexity). Things got out of hand when we made the fixtures a templated class on a value, and created them by specifying the value as part of the texture name. Now it was a pain to debug and things were even less clear.

So we finally implemented it natively in CppUnitLite2 as fixtures with initialization parameters. We introduced a new macro, TEST_FP (test with fixture and parameters), which takes any parameters you want to pass to the fixture constuctor.

Now you can declare tests like this:

We should prob
ably simplify that to avoid repeating the fixture name, but that works fine for now.

Include cleanup

The original version of CppUnitLite was pretty casual about what headers it included. Unfortunately, that means that all test code were automatically exposed to a variety of standard headers, whether you wanted it or not. This version is a lot more strict, and the only header that will get pulled in from using it will be iosfwd, which is even a fairly lightweight one.

Memory allocations

We have very strict memory allocation wrappers around our tests, and if any memory leaks, it is reported as a failed test. Since checking for memory allocations is very platform specific, it is left out of the framework (although it could be added pretty easily in the platform-specific sections). In any case, some of the static-initialization time allocations were confusing the memory tracker, so we removed them completely. All it means is that we use an array to register tests instead of a std::vector and we cap the maximum number of tests to a fixed amount. If you need more than the default 5000 tests, go right ahead and add a few zeros.

New license

Previous versions of CppUnitLite were not released with an explicit license. To make things clear, I explicitly added the license text for the MIT license with an added restriction. The MIT license allows you to use the code in any project, commercial or otherwise, without any restrictions. The added restriction to the license prohibits its use in any military applications or projects with any military funding.

Ratings

How does this release of the CppUnitLite2 compare with respect to my initial set of features I used to rate the different unit tests frameworks?

• Minimal amount of work needed to add new tests. Nothing has changed there. Still passes with flying colors.
• Easy to modify and port. That has always been the strong suite of this framework.
• Supports setup/teardown steps (fixtures). Much improved now by creating them correctly around each test.
• Handles exceptions and crashes well. No changes there. Still one of the best I’ve seen.
• Good assert functionality. The check macros have been much improved to match the best frameworks out there.
• Supports different outputs. Yup. Still there.
• Supports suites. It still doesn’t. It just shows that I really haven’t needed this feature yet.

Future work

So, what’s next for CppUnitLite2? It’s really quite usable right now, but I could see a few changes happening in the next year, and maybe a few others will come up and surprise me.

Test registration is based on static initialization of global variables, which is not very reliable across compliers. For example, if you try to put the tests in a static library, MSVC will strip out the registration code leaving you with no tests. To get around that, we could introduce a custom build step in a scripting language that would generate a simple file with explicit registration of tests. This could also be used to collect tests from many different libraries into a single executable.

A couple of times it would have been convenient to have parameterized tests (I think that’s the correct term for it). Basically, to have tests that could run with different data. Something like this:

 TEST_PARAM(MyCustomTest, int a, int b) { CHECK (something); CHECK(something else); CHECK(yadda yadda); }  TEST(MyTest) { CALL_TEST(MyCustomTest, 10, 5); } TEST(MyTest) { CALL_TEST(MyCustomTest, 12, 7); }

The key point is that a test fails in the parameterized test, it should report the error correctly indicating where it came from.

This is actually pretty easy to implement with a couple of macros. We simply haven’t needed it more than a couple of times, so we haven’t bothered yet. But I imagine next time we run up against this we’ll bite the bullet and implement it.

Wrap-up

Charles had so much fun working with this framework, he decided to write his own from scratch. He made it available on his web site, so go check it out.

Thanks to High Moon Studios for being open and letting us release the changes we made and share them back with everybody. If you end up using the framework and you make any changes, drop me an email and I’ll add them for the next release.

Jumbo shrimp. Instant classic. Military intelligence. C++ refactoring browser. Spot the pattern yet? Up until recently, there have been more sightings of Nessy and Bigfoot than of working C++ refactoring browsers. After months of using refactoring intensively in C++, my fingers are screaming for mercy and threatening me with repetitive stress syndrome. Fortunately, things seem to be changing a bit. I recently learned of SlickEdit‘s support for C++ refactoring, so I couldn’t resist taking it for a test drive.

C++ refactoring is hard. There’s no doubt about that. Just parsing C++ is hard, as I quickly found out when I wrote some scripts a couple of years ago to analyze the structure of a C++ program. Even with the help of Doxygen pre-processing, my analysis was full of holes and exceptions. Writing a robust program to correctly interpret and modify the structure of a C++ program is a nightmare come true.

Frankly, I had pretty much given up all hope of every having good automated refactoring tools for C++. My best hope was that new language designers keep refactoring in mind as a primary goal of their languages, so at least we’ll have those tools in whatever language we’re programming with in ten years.

So along comes SlickEdit version 10 with great promises of C++ refactoring. Surprisingly enough, that’s just one small bullet point completely hidden by a bunch of bullet-point items of no great interest or uniqueness (smart line selections? keyboard emulation? what year is this, 1992?). As a matter of fact, refactoring isn’t even mentioned in SlickEdit’s “cool features” web page! That’s not an auspicious beginning.

Fortunately, things get better. The installation for the Linux demo version of SlickEdit v10 worked without a hitch. The refactoring actions feature prominently in the Tools menu and the right-click context menu. I loaded a small project I was working on (just a handful of classes-nothing big) and started applying them one at the time.

Rename

This is the big one. If I can only have one refactoring, I’ll take this one. Being able to rename variables, functions, and classes takes care of 75% of the refactorings I do. A tool that could reliably do that would be a major productivity boost (reliably being the key word there).

How does SlickEdit’s rename work? Member variables were a dream to rename. But frankly, I can usually do that with “replace in file” and doing one edit in the header file, so that’s not very challenging. Still, good start.

Trying to rename member functions was not quite as successful. Sometimes it would work like a charm, and rename the function in the cpp file, the header file, and everywhere else it was called from. Sometimes however, it would look at my code and proclaim that there was nothing to rename. Nothing? I had just right clicked on top of the function to rename! Some other times it would rename the function in a couple of classes and totally miss where it was being used.

After some experimentation, it turns out that “Quick Rename” (which supposedly is not nearly as robust) worked time and time again without a single problem. That’s good I guess, but it left me with a very uneasy feeling.

Thumbs up for the user interface for refactoring too. As soon as you enter your refactoring and your parameters, you are presented with a list of the files that will be modified and a diff of each of them, so you can cancel the operation if something went horribly wrong. It certainly makes you feel a lot more in control.

Extract method

This is a fairly common refactoring. I actually thought it would be fairly complicated, but in every test I did, SlickEdit handled it admirably. The tricky part is to figure out what arguments the new function needs to take, and every time it figured it out correctly. You even get an option to name the new function and tweak the arguments it takes before applying the refactoring. Again, great user interface.

My only quibble with this refactoring is that the new methods are all made public. Uh? Would it have been so difficult to give us an option? If anything, I have just moved code from a member function to another member function in that class. By default it should be private, not public. Oh well, I just have to move one line. But it’s a bit annoying to have to tweak the header file by hand.

Modify parameter list

Another big one. Sometimes it’s quite painful to add a new parameter to a function that is called from dozens or hundreds of tests. So much so that it really discourages us from doing the “right” thing and is tempting to create a new function instead. Sometimes, after the second time you need to add a new parameter, we just create a struct with the parameters that the function takes and save some pain in the future. Not a particularly elegant solution, so this refactoring tool would come in very handy.

Unfortunately this one completely failed for me. Most of the time, trying to add a new parameter to a function would just come back and tell me there’s nothing to do. Same thing with rearranging the order of parameters. At best, once I got it to actually add the new parameter, it only changed the function and the header file, not the calling code. Doh!

Misc inheritance chain refactorings

SlickEdit has a set of refactorings intended to do operations in the inheritance chain: push down, pull up, extract super class, etc. I don’t work with big inheritance chains these days, so they’re not very useful to me and I didn’t test them at all. Actually, now that I think about it, the only inheritance I use are interfaces and mock objects. Then again, I suspect this would be extremely handy for manipulating legacy code (like some currently popular game engines).

Extract class

This seems like a really large refactoring. Extracting a whole class out of a set of code? Rename didn’t work consistently, and modify parameter list didn’t work at all. I certainly wouldn’t want to try something of this magnitude. Let’s get the other ones working before we jump on this one.

Convert local to field

This one seems marginally useful, and it’s not something I do in a regular basis. All it does is to prepend an m_ to the variable and add it to the header file. What if you don’t prefix your member variables with m_? Is there a way to turn that off? Also, the new member variable wasn’t added to the constructor(s) initialization list.

Move method

At first I thought that it was a totally pointless refactoring since it’s describing as moving a method from one class to another (I can’t remember when was the last time I did that). But it turns out it can be quite useful to move a non-member function into a class. That’s something I do sometimes when a helper function in an anonymous namespace starts needing more and more state from the object that uses it, and eventually it becomes clear that it wants to be a member function.

Move method will make it a static member function, and then you can follow that up with Convert static method to instance method and voila.

Encapsulate field

I’m sure some people will rave about this one, but I’m not too thrilled about it. It creates set/get methods of a member variable. Frankly, I don’t like set/get methods. I think that, most of the time, they’re a smell of bad design. I’m not sure I want to make it any easier for people to create them, so I would be happier without this refactoring tool at all. Besides, what’s with making them inline by default? At least give us an option to put them in the cpp file.

Replace literal with declared constant

This one takes a literal (magic number/string) and replaces with with a constant or a define. Now we’re really reaching the bottom of the barrel. Is it really faster to navigate a three-level menu to select this than to do it by hand? It’s good that you get an option to create the new constant as const, static const, or an evil #define, but it doesn’t place it in an anonymous namespace, even if one already exists in that compilation unit.

Create standard methods

This one will add a constructor, destructor, copy constructor, and assignment operator to the class. At first I thought it wasn’t a big deal. I already have macros that do that for new classes. Then I realized you can do this with an existing class, and it will take care of assigning all member variables. That’s pretty cool, but of limited practical use. I haven’t had to go back to many existing classes and add assignment operators. Still, I could see how it could be useful.

All in all, SlickEdit managed to impress me because I wasn’t really expecting functional C++ refactorings. On the other hand, it is clearly far from being robust enough to be relied upon for production work. Maybe the next version? For me, they just need to fix the rename and I’m almost there.

One cool aspect of SlickEdit is its powerful scripting language, where it has access to all the commands you can do from the menu. That means that I can write a script that, as part of what it does, kicks off some refactoring step. That’s extremely useful to adapt the refactorings to my own environment. One thing I didn’t see is whether the script language has access to the logical structure of the C++ program or whether it’s just limited to calling the refactoring function. It would be great to have more control and be able to create custom refactorings.

SlickEdit is a bit puzzling because it parses all the header files during each refactoring. And by all, I mean all. Not just the ones in your projects, but all the standard include files that you might have included. I don’t really understand why it does that. I’m clearly not going to change standard header files, so I think that limiting a refactoring only to files that are presently loaded in the project would be good. With my toy projects with a handful of classes, the refactorings were very fast (a second or two), but I fear how it might scale to a large project. Next time they have an improved version I’ll have to load it up with a large codebase and see how it fares.

It seems as if refactoring tools for C++ are just around the corner, but still tantalizingly out of reach. At this point I’m not ready to switch to SlickEdit based on the strength of its refactoring tools alone, although I’m still going to take a closer look at it. Right now I would prefer a standalone command-line refactoring tool that could be easily integrated in any environment. Anybody knows of any other decent C++ refactoring tools?

I’ve been following the discussion on the evils of assert started by Len Holgate. Poor old assert was getting beaten up from every side, so I’m going to have to step forward and defend it.

Yes, I do find assert useful. Yes, I’m doing full-out test-driven development. Yes, I’m a hardcore C++ programmer.

Before we even get started, let me recap what I see as the main reasons to use assert.

• Detect a program error or bug right away. The sooner the better. There’s nothing more frustrating than trying to track down bugs that occurred several function calls away, especially if they’ve unwound the stack and started a new series of calls. Even worse are errors that propagate and don’t become apparent until several frames later.Example: Creating a new character from a spwan point during gameplay should always work, so asserting on the pointer returned by the factory is a good idea.
• Document a decision, limitation, or assumption of particular functions. Nothing like calling a function and getting an assert right away to realize that’s not the way it was intended to be used.Example: A function that takes a pointer that can never be NULL and we don’t want to take a reference (maybe because the object is going to store and keep ownership of the pointer).
• Catch bad situations that will soon lead to trouble.Example: Asserting that the matrix passed to a rotate function is orthonormal. Everything will work fine for a little while if it isn’t, but soon things will degenerate if errors keep accumulating.

One key point about the use of assert is that it’s a message from the programmers to the programmers and nobody else. The idea is that whatever caused that assert is a top priority and should be fixed right away. An end user should never see an assert message because a) they don’t care and b) it means their program is toast.

This distinction between programming errors and “user” errors is particularly important in game development. User errors are anything that a user of the tools or game could cause just by using it. In our case that includes the content creators (designers and artists). The last thing we want to do is stop everything just because they typed the wrong file, or tried to load an old version of a model. Those errors need to be caught and dealt with. They are expected.

Programming errors are the ones that can only be caused by something done by a programmer in the code. As an example, in game development we constantly have functions that look like this: void Something::Update(float deltaTimeInSec). Unless you have a simulation system that can go both forwards and backwards, trying to call that function with negative seconds is nonsensical. An assert there tells the programmer that something either went horribly wrong, or they’re trying to use Update in a way it wasn’t intended. Clearly, no user of the program will ever be able to call Update with a negative time value just by using the program. So in this case, I do want to assert and have that fixed right away.

With that out of the way, let’s look at Len’s five major complaints about assert. Let’s take them one at the time.

1. The check will go away in a release build.

From my point of view, an assert is a way to tell a programmer (probably even myself in the future) that something has gone completely wrong. If an assert ever goes off, it’s a bug in the code and a programmer needs to fix it.

So what if asserts go away in release builds? We’re writing “shrink-wrapped” software, so that’s a good thing because there will be no programmers around with their handy debugger to fix the problem. If a user ever manages to come up with a way to cause the code to get in a bad state, then all we can do is pray. Especially in the case of console games, trying to capture traces and exception reports is not going to help anybody.

2. The assert is covering poor design.

I think this is really the crux of the discussion. Is assert really just a crutch for “poor design”? If you define good design as creating bullet-proof code that will work under any circumstance then it’s probably right. But you know, my job is not to write bullet-proof code. My job is to write whatever needs to be written to satisfy our customer stories every two weeks, and eventually ship a game with it.

This is very much in the agile development mindset. I’m not going to try to bomb-proof a piece of code for all foreseeable uses in the future. I’ll make sure it does what it needs to do now and leave it at that. Under this view, asserts are post-it notes saying “This code doesn’t even handle this situation. You either screwed up, or you better update the code to deal with it.”

Let’s go back to the Update() function. What are our options to make it work without assert?

• Implement Update() to go backwards in time as well as forwards. That will easily more than double development effort for something that will have no effect in the final product (unless you’re implementing a mechanic like Sands of Time, although I suspect that not even that game had a negative-time update).
• Make it physically impossible to pass negative numbers into the Update function. We could create a delta time class that cannot be manipulated in any other way than to have positive values. While this is clearly possible, I feel it’s a tradeoff between simplicity and “bullet-proofness.” I’ll take simplicity any day of the week for most situations.
• Have the update function check for negative values and return a failing value. That would be fine except that now everywhere we call Update we need to check for return values. Did you notice that Update(), the way it was declared, returned no values at all? That’s a function that is supposed to work and never fail. What if an Update() function fails? What do we do next? Quietly move on to the next one, or try to fix the system in some way? I’d rather slap the programmer with an assert and have him fix it right away than quietly continue to work while things are not working as expected.

So I see it more as a tradeoff between simplicity, time, and robustness. There’s no single correct answer, and it will depend on your particular situation. If you’re writing flight-control software, to hell with simplicity and time constraints, and go for 100% robustness and throw a few redundant systems for good measure in case of cosmic rays interfering with the hardware. In my case, turnaround time and simplicity trump absolute robustness.

Also, I admit this might not work in every environment. It works for us because we’re the only users of our own code. When something isn’t good enough, we make it good enough. That’s very different from writing a generic library for thousands of users to apply in their production code.

3. The assert makes it very hard to write a unit test for the failure condition.

I don’t understand where this comes from. Maybe with a crappy unit-test framework it’s a problem, but the unit-test framework I use deals with assertions just like any other exception, and fails the current test case. If your unit test framework barfs when your code tries to access an invalid pointer, you really need to look into getting a better one (I keep meaning to put up my version one of these days).

You should be able to hook up assert to anything you want (by using a custom assert or any other way you want). So the test runner first hooks up a special assert handler that throws a particular exception. Not only does this let us fail a test if an exception is triggered, but it even allows us to write tests to check that asserts are thrown. For example, I could write something like this:

 TEST (UpdateWithNegativeTimeAsserts) { Whatever stuff; CHECK_ASSERT(stuff.Update(-0.1f)); }

4. If the object is running without a user interface, then the assert’s dialog box may cause issues.

What dialog box? Maybe I’ve been doing game development too long, but every single project I’ve worked on has had a custom assert macro that gives us a lot more control than the standard one. Seriously, are people calling the standard assert() everywhere? No wonder they have issues.

Game Programming Gems 1 had a great introduction to using custom asserts by Steve Rabin with lots of great ideas (here’s an implementation based on his article). At the very least, you should be able to have full control of what you want the assert to do. A lot of the time that will simply be _asm int 3, but many other times you want to throw an exception, write out a log, send out an email, update a database, or even put up a dialog box to allow the user to bypass it. As a side note, I actually hate the option of continuing after an assert and I would never allow it in my own implementation. Assert means dead. Kaput. Done.

A custom assert takes five minutes to set up and will pay back in gold-pressed latinum in no time.

5. The assert is really just an “exploding comment.”

Yes, that’s completely true. It’s more like a programmer yelling in your ear “YOU JUST SCREWED UP!” But then again, that’s one of the things that make unit tests so great. They’re like comments that complain every time you change the behavior of the system in a way that violates what the tests are checking. Those are the best kind of “comments.”

That takes care of Len’s objections. Clearly, I don’t see any of them being an issue other than the second one, which is a matter of tradeoffs. Having said all that, I do use asserts less than I did before I used TDD. Unit tests take care of some of the things I would have checked with asserts in the past (did this loop really iterate over all the members that needed to be updated?), but unit tests still can’t protect against misuse of functions or simply unexpected situations.

So no, I’m not willing to give up on assert any time soon. It might be an ugly, rusty, oversized tool, but it still has a very definite spot in my toolbox.

A couple of months ago I looked at various build systems in the hopes of finding an ideal one for C++ development. In particular, the most important criteria I was using was iteration time for incremental builds. Jam was the clear winner and things looked good.

Fast-forward a few months and one aborted attempt at implementing that build system, and there are now more unanswered questions than answers. It turns out that Jam and its successors were far from the perfect solution I had envisioned, so I was back to square one. I took this opportunity to look into all the build systems I had left out of the first article, plus all the ones that other people asked me about since then, to come up with a much more comprehensive evaluation.

The Contenders

Jam

Since Jam is the culprit that caused this second go-around, I figured I would start with it. What’s wrong with Jam? Isn’t it the fastest kid on the block? It still is the fastest, but when it comes to supporting new platforms or adding new settings to builds, it’s not the most straightforward of all systems.

Jam is a bit too complex for my tastes. Even after sitting for a while with the Jam tutorial (which is a horrible introduction, by the way), reading the skimpy documentation many times, and playing around with some sample Jam files, it was still very difficult to create non-trivial Jam files. At that point, my best allies were the Jambase and Jamrules files themselves. When it came time to start adding support for features such as precompiled headers, different build configurations, or Xbox 360 support, things got a lot harder very quickly.

Jam is a bit of a puzzle. Out of the “box” it claims to be able to have platform-independent Jam files, and that’s the source of the problem, because it makes it a lot harder to support new platforms. In my case, since I’m targeting a very specific set of platforms, I would much rather specify each build rule myself and have a simple, clean set, instead of a monster set of rules that applies to everything.

At the same time, Jam doesn’t support some really basic features, such as spaces in filenames. Not like I advocate having spaces in filenames, mind you, but that’s an unfortunate reality of the setup of many Windows computers. Yes, I know I can patch it, but this is where the next, and biggest, problem of Jam comes in.

Jam is dead.

Yes, Jam is completely dead. The Jamming mailing list has less traffic than a side road in Siberia during the winter. My attempts at asking there for help went completely unanswered (except for one kind soul who emailed me privately, apparently too embarrassed to post to a dead mailing list).

The fact that the latest official version of Jam dates back to 2002 should have been a good tipoff, but somehow I didn’t catch on to that at first. The nail in the coffin was when I asked where the latest set of patches, or the latest unofficial version of Jam, was located and nobody, *nobody*, answered.

So where is everybody who developed and used Jam? Oh, they must have gone to one of Jam’s successors, like BoostBuild. That’s good, right? Right?…

BoostBuild v2

BoostBuild v2 is promising on paper. It builds on top of Jam to add several much-needed features such as multiple configurations, correct filename parsing, etc. Or at least so it seems.

As soon as I start digging into it it’s very clear that BoostBuild is much more complex than Jam. It’s not like Jam was a walk in the park, but BoostBuild is close to unacceptable. I’ll take a Makefile any day of the week. Still, I would be willing to overlook that, bury myself deep in the documentation, and learn the system inside out if BoostBuild lived up to expectations: Jam speed with more features.

Unfortunately it doesn’t. It doesn’t just fall short either, it misses by several miles and destroys a friendly nearby town in the process. BoostBuild was the slowest of all the build systems I tested. Slower even than Scons! Talk about taking something small and fast and turning it into a slow, lumbering beast.

On the up side, I won’t have to spend months learning the overly complex system. I had to find a silver lining somewhere, right?

Is BoostBuild v1 any better? It could be. But frankly, after my experience with v2 I wasn’t in the mood to go check it out. Besides, it sounds like they’re going to phase it out. If I want a fast, dead build system, I’ll choose plain Jam.

Microsoft Visual Studio 2005

I had heard (from the comments in the previous articles and other posts on the internet) that Visual Studio 2005 had an improved build system. Even though it seems that C++ is getting the shaft and not being rolled in the MSBuild system for this release, apparently it was still improved somewhat. Since a beta version of Visual Studio 2005 fell in my lap at the GameFest conference earlier this year, I decided to give it a try.

Incremental build times were definitely improved, although they are still far from Jam times. The one-second delay for each project is reduced to a bit less than half a second. Better for sure, but I don’t see why it can’t be faster. Still, that’s getting in the range of being more usable.

One interesting addition is that, by default, Visual Studio 2005 will use multiple threads to compile C++ code. That will be a very welcome addition for full builds. Now I really need a quad-processor machine for our build server.

VCBuild

MSBuild, the Ant clone that Microsoft has been hailing as the next great thing to come, doesn’t yet support C++ builds. It’s a matter of priorities, I suppose. In the meanwhile, MSBuild will use the standalone VCBuild tool to compile Visual Studio solutions.

VCBuild promises to be a lightweight, command-line version of devenv.com, ideally suited for build servers. Indeed, VCBuild had slightly better build times than MSVC2005, which is a huge improvement over 2003. That actually puts VCBuild as the fastest build system after Jam and Make.

At this point, I was interested enough to think about starting to use VCBuild for all of our command-line builds (at the server and the quick verification build before submitting code). Unfortunately, I immediately ran into a couple of problems. The first one is clearly listed in the Readme, so I knew it was coming: VCBuild will not add a dependent project to the link list like MSVC does. This is actually a good thing. I always hated that dependencies implied linking in Visual Studio. Unfortunately, since that’s the default behavior, we are relying on it It’s easy to change, but it’s just one small obstacle in the way.

The more serious problem was that VCBuild didn’t seem to parse all the project settings correctly (warning levels for example), so it created very different results than a build with devenv.com.

Finally, I was disappointed to see that VCBuild does not support multiple processors. I had started thinking of VCBuild as a standalone MSVC2005 project compiler, but that’s clearly not the case.

Fast Solution Build MSVC plugin

This one isn’t really a standalone build system like the others covered in this article. However, it directly addresses the problem of speeding up Visual Studio builds, so it was worth including it in. In the past, I had very mixed experiences with it, but after hearing some really good things about it from some co-workers, I decided to give it a whirl.

As promised, Fast Solution Build does speed up incremental Visual Studio builds to speeds similar to Visual C++ 6.0. That is to say, doing an incremental build with no changes on a large solution takes about a second, which is really how it should be. This time around, I didn’t have any problems with strangely broken builds or anything, so it seems pretty stable.

Unfortunately, Fast Solution Build is not without its set of problems. The first few problems are relatively minor. It doesn’t seem to play well at all with C#. If I ever make the mistake of attempting a fast build on a C# project, Visual Studio will start a build and never end. The only way to get out of that state is to kill the process and restart Visual Studio. I guess I just need to train my fingers not to press F7 in C# projects.

The other problem is more of a user problem than anything related to the plugin. In spite of its name, Fast Solution Build, it does not build a solution, but it builds the selected project. That alone caused more problems than anything because most people are used to doing solution builds, not project builds. It’s especially problematic if you want to have several unrelated projects build simultaneously (like in the case of unit tests projects and a game executable in addition to the libraries they depend on). More than once, we ended up checking in code that seemed to build fine, but broke when the server built the full solution.

The lack of a command-line interface also kills Fast Solution Build as a long-term solution. It might be OK while you’re working with the IDE, but you can’t use it for any command-line builds such as in the build server or for a quick sanity check we have before committing any code.

However, the biggest problem of all with Fast Solution Build is that it’s just a plugin to Visual Studio, so you’re left with many of the problems in Visual Studio itself, such as being hard to set up large solutions with lots of dependencies, having the visual representation mixed in with the real dependencies, suspect dependency checking (try deleting a file and then doing a build), or being tied to a single platform.

All in all, if you want a quick fix to your build times, Fast Solution Build is a good temporary band-aid provided you don’t work with C# much and you don’t have very complex solutions.

Scons

As I was wrapping up the new measurements, I noticed the announcement in for the new version Scons (0.96.91 pre-release). Since the last version was many moons ago, and since there was talk about improving Scons’ performance, I decided to give it a try.

The new version is indeed faster, but only by a tiny amount. Scons still lags far, far behind the curve in terms of performance. Unless the developers manage to squeeze performance improvements of several orders of magnitude, it’ll remain unusable as a build system for fast iterations.

Ant

Ant is one of the build systems I consciously left out of my first roundup. I had used it a bit in a build server, but I had the impression it was mostly intended as a “build sequencer,” with high-level actions like archiving files or emailing, and intended for full, nightly builds, rather for fast C++ incremental builds. It turns out that even though Ant was developed primarily with Java development in mind, it does have C++ support through some of the extra contribution tasks, with full dependency checking.

Another fact that put me off a bit from trying Ant was its level of verbosity. Writing XML files is anything but fun, and I much prefer the terseness of a Make file than the verbosity of an Ant file. Still, if a tool does the job well, I’ll put up with a lot.

Ant’s dependency checking was not bad, but not great either. Incremental build times with Ant were a bit worse then MSVS2005, but not by much. Frankly, that’s faster than I expected with my initial preconceptions.

Full build times on Windows were blazingly fast. So much so, that I was convinced I had configured something wrong and I was only doing half the build. It turns out that everything was fine, but Ant will send all cpp files to be compiled to the compiler at once, whereas other build systems will process them one at a time. Under Linux with gcc, this doesn’t make much of a difference, but in Windows using the MSVC compiler, it makes a huge difference. Other build systems should take note of that and take batching much more seriously.

Nant

Nant is Ant’s evil twin written with the .Net framework instead of Java. Frankly, I don’t really see the point of Nant when Ant is already a mature project and runs in many more platforms. To make matters worse, the Nant developers decided to make their syntax different from Ant. Was that really necessary? If at least it were a straight Ant clone, it would be easier to migrate from one to the other. Personally, I would have preferred to see all that energy put into improving Ant instead of creating a .Net clone.

Having said that, one of the early surprises of Nant was its native support for C++ compilation. In Ant it was a custom task, here it works out of the box.

Nant also has a good set of extra contributed tasks, including support for Perforce operations (although for a lot of the operations I wanted to do, I ended up having to fall back to command-line scripts because the tasks didn’t have enough parameters or support for exactly what I wanted to do).

Another potentially interesting feature of Nant is support for Visual Studio solution files. This could be very useful because it would move the dependency checking out of Visual Studio and into Nant. Unfortunately, I tried running it on the solutions at work and it failed miserably. It seems that lots of compiler options are not parsed correctly, so it was impossible to build our existing solutions. I also suspect that C++ isn’t their main target, so C++ project support is probably lagging behind C# and Visual Basic. Although I didn’t try it, I imagine that Nant would choke at trying to build an Xbox 360 solution file.

Just for kicks, I tried running it on Linux with Mono, but apparently it only works with the Microsoft cl.exe compiler, not with gcc.

Build times were almost identical to those of Ant. Nant does very good batching as well, so full builds were blazingly fast, but incremental builds were only middle-of-the-road.

Rake/Rant

I don’t do any Ruby development, so I had never heard of Rake until Ivan-Assen Ivanov told me about it. In for a penny, in for a pound, so I went ahead and ran the tests on Rake as well.

Rake is a build system written in Ruby (and I assume mostly intended for Ruby users as well). Like Scons, it brings the full power and expressiveness of a scripting language (Ruby in this case) to the build script. It looks pretty simple and clean. The Ruby syntax is a bit strange at first sight, but it’s easy to get used to (especially with lots of examples).

Unfortunately, Rake doesn’t have C++ dependency checking built in. I could have probably cobbled something together with makedepend, but that would take more Ruby-foo than I’m comfortable with.

Of course, all of this being open source, somebody, somewhere has had the same itch to scratch, and Rant was born. Apart from the cool name (after all, who doesn’t like to write Rantfiles?), Rant looks very similar to Rake. Don’t be fooled by their parallels to Make and [N]Ant. It’s much more Make meets Scons in Rubyland than a derivative of Ant.

One feature of Rant that I really liked was the good support for error messages. If the Rantfile had a problem in it, it would correctly identify the problem and point me to it. After working with Ant/Nant, that was a welcome change.

The build times were a bit of a mixed bag. Full incremental builds were pretty good, definitely in the top third of the build systems I looked at. Unfortunately, building a single library without any changes was extremely slow. It seems that Rant had to do a lot of work before it could even look at an individual library. If it weren’t because of that, I would be a lot more excited about it.

Make and MSVC2003

Nothing new in this front. For details, just read the first article. Everything there still applies.

The Method

I used the same test case and hardware as the last time. As a quick recap, the source code was made up of 50 static libraries with 100 classes each (2 files per class), 15 includes per cpp file to other files in that library, and 5 includes to files in other libraries.

The Python script generates a tree with a full code base for every build system along with all the necessary build files. Whenever the build files for two platforms are different, two separate trees are created. Thanks to Jim Tilander to writing the Python scripts to generate the BoostBuild files.

The computer was a P4 2.8GHz CPU with 1GB of RAM and a 7200 rpm hard drive. The Linux distro was Mandrake Linux 10.2 with the 2.6 kernel and Windows XP (without any antivirus running–which makes a huge difference!).

generate_libs.zip

The Results

Linux with gcc 3.4.1

Windows XP with MSVC 2003 compiler (except for the MSVC2005 run).

Update: Since several people asked, I timed Scons with MD5 checksum checking turned off in the hopes of getting a better time. I followed the advice described here. Just like last time, turning off MD5 checksum made it go slower if that makes any sense.

Using the option to indicate unchanged dependencies made it go a bit faster, but that’s not an option we can realistically use during development, since that would be changing constantly with the structure of our program. Still, the fact that without doing any dependency checking it’s still taking 47 seconds. What is it doing??

With multiprocessor machines being so common these days, a good build system needs to support parallel execution of tasks in multiple processors. Surprisingly, not all of the build systems compared here support it. Of the usable build systems, only Make, Jam, and MSVC2005 actually support parallel builds.

Conclusion

Jam still remains the best-performing build system of all by quite a lot. Unfortunately, the total lack of support and community around it make it very difficult to recommend. There has been some talk recently in the sweng-gamedev mailing list to do some work for build systems. Maybe we could take over the continued development of Jam. If that happened, I wouldn’t hesitate to recommend it. Until then, unless you’re a Jam expert or are willing to dig through user-created patches in the Perforce Public Depot, Jam is not a realistic option.

MSVS2005 has improved a lot from 2003, and along with VCBuild, it’s one of the best performing build systems under Windows for incremental links. Personally, I hate to be locked in to proprietary tools and single platforms, so I would opt for the next best systems, which are Ant/Nant. Pick your flavor of choice, they’re about the same.

MSBuild can be promising, especially once Microsoft rolls out their XNA Studio extensions specifically designed for asset building. It’ll only run in Windows, but at least they claimed that they’re making it general enough you can use for building any type of assets, not just their own formats. That’s definitely one to watch for the future.

If you’re comfortable with Make files, it still remains as a great option for very fast builds. You’ll probably want to wrap Make in a more comprehensive build system (such as Ant) for your build server, packaging builds, etc. But Make can be your everyday workhorse during development.

Whatever you do, stay far, far away from BoostBuild and Scons. Their performance is simply so horrible there’s no excuse to use them unless build times are of no importance to you at all.

I remain hopeful that we might be able to resurrect Jam. If you’re interested in contributing, contact me through email, comments on this page, or participate in the sweng-gamedev discussion directly

generate_libs.zip

I define an iteration cycle as the time elapsed between making a trivial change and being able to see the results of that change. In particular I’m concentrating on large-scale (around one million lines of code) C++ projects, which is representative of modern PC and console games today.

Of course, there’s more to fast iteration than just the speed of the build system. How quickly you’re able to get in the game and see the results is a big factor (another reason why I wouldn’t want to live without unit tests). The physical dependencies of your program are going to affect how quickly your code builds. And if you’re always working with a full game executable that takes forever to link, your iteration times are going to be shot (yet another reason to use unit tests!).

But let’s put that aside and concentrate on the build system itself.

Fast iteration is more than just about time and speed. It’s also about how you feel about the code and what you dare do with it. When things are slow and painful, you’re going to be a lot less likely to try new things, or fix one last thing to clean up the code, or refactor something out of a header file into its own module. Over time, this is going to accumulate into cruft, hacks, and unmaintainable code. It’s also about not breaking up the flow, the mental state you’re in while you’re writing software. Interruptions of more than just a few seconds are much more detrimental than their time value alone.

When you add test-driven development to the mix, fast iteration becomes even more crucial. With test-driven development, you end up doing micro-cycles of modify-compile-test, sometimes several times per minute. Unless you have very fast build times, you’re dead in the water.

Before we go any further, let’s crunch a few numbers. It’s not so much to show specific improvements, but to have as a reference point going forwards. Think about the project you’re currently working on. How long does it take to build when you modify a single cpp file (or even no files at all)? I’ve seen projects that took over two minutes to build, and anywhere between 30 seconds to one minute is fairly typical. Let’s say 30 seconds for this example. How often do you need to do a build? It’s not very fast, so maybe once every 5 minutes, 8 hours a day. That adds up to a staggering 48 minutes per day, or 10% of your full work day!

Now reduce the build time to two seconds instead. And to make things more interesting, let’s do a build every two minutes. That adds up to be 8 minutes per day, and, most importantly, they feel like almost instant builds, so they don’t bring you out of the flow state. That’s what our goal should be for a build system.

The Goals

As I started looking into different build systems (there are a lot of them out there!), I noticed that they have very different sets of goals, and a lot of them are fairly irrelevant to my particular needs.

As a game developer, I work with a varied, but limited number of platforms (most of which are not supported out of the box by build tools anyway). I’m also not planning on releasing the source code any time soon to let players compile the game in any platform, so I have no need to have a build system that automatically detects all the correct settings and does the right thing for any possible platform.

Some other build systems had useless features built in, such as access to version control or the ability to send emails. I consider those type of tasks to be totally beyond what the core build system should do, and I prefer to have those features in a wrapper system that does build/test/deployment of builds by calling the build system itself.

So what exactly do I want in a build system?

• Super-fast incremental builds (around two seconds). This is the key to fast iteration and I want this at almost any cost.
• Customizable. I’m going to be using unusual compilers and environments. I want to be able to easily set my own rules and actions and not be tied to any particular platform or compiler.
• Correctness. I want the build system to build the minimal amount of files and still do the right thing under most normal circumstances.
• Multiprocessor support. I also care about the speed of a full build, and with multiprocessor machines finally becoming popular, using multiple processors is a great way to speed up build times.
• Scalable. I want all the related source code to be tied to the same build system. I don’t want to create a separate build file for every minor tool so they build at a reasonable speed. I’d like to simply build any target and have the minimal amount of files rebuilt.

The Contenders

Visual Studio .NET

Most game developers doing Windows or Xbox development will be familiar with this build system. I’ve been stuck with it for many years, and while things were not great in Visual Studio 6.0 and earlier versions, it became decidedly unusable when it turned into “.NET”. I don’t know what happened, but I suspect that in their effort to cram all those languages under a single IDE they crippled the C++ build system even more.

My main gripe is how slow Visual Studio .NET is when doing an incremental build on a solution with many projects. It’s roughly a second per project. If you have 50 projects, there goes a full minute for nothing. That’s simply unacceptable, so I’ve always had to work around it by creating many solutions with the minimum amount of projects necessary, or keeping the dependencies in my head and forcing builds by hand. Alternatively you could throw all the code in a couple of projects instead of breaking it up into many different ones, but that’s like jumping off a cliff to avoid being stung by a bee.

Because Visual Studio mixes the build system with the visual representation of the files, large solutions are not only slow to build, but are a positive pain to work with, making browsing the source code extremely difficult.

The problems with Visual Studio don’t end there. The solution and project files are a pain to generate, they’re full of magical GUIDs and references to the registry, and they’re extremely verbose. The .NET framework offers an API to create those files, but the fact remains that they’re much more complex to create than any of the other build systems.

If you don’t generate the project files by hand, you soon enter configuration hell. Anybody who’s had to make sweeping changes to lots of projects with multiple configurations through the IDE will know what a painful process I’m talking about.

Some of my other complaints are not being able to easily set the build order for different projects (it appears to be determined by the order in which they appear in the solution file), or the fact that setting a dependency between two projects forces an implicit linking.

All in all, Visual Studio seems very well suited to small, toy projects. Projects with just a couple of libraries and a few thousand lines. Anything bigger than that and it becomes an exercise in frustration.

On the bright side, it is possible to use “makefile projects” in Visual Studio, which completely bypasses Visual Studio’s own build system and simply calls an external command to build the project. Visual Studio also offers a command-line line interface, so at least it is possible to do builds from the command line without launching the IDE. Finally, there are some third-party plug-ins that can speed up the dependency checking, which can help with some of the problems (unfortunately, FastSolutionBuild doesn’t seem to have a command-line interface, which renders it useless for my needs).

Other third-party add-ons like Incredibuild claim to speed up full builds, but they do so at the expense of incremental build speed, which I consider much more important.

Make

Make is the granddaddy of all the build systems. With a distinguished history of over 20 years, it has certainly proved its worth in the real world by building hundreds of thousands of projects over the years.

But make is far from perfect; otherwise there would be no need for other build systems. However, things are not as bad as people make them out to be, especially with modern versions of make (GNU make for example). The lack of portability is not an issue for us because we can just trivially write a new makefile, or a variant, for every platform we support.

The claims about make not being scalable are more serious. The article “Recursive Make Considered Harmful” certainly did much harm to make’s reputation. Specifically, that paper claims that it’s very hard to get the order of recursion right because using recursive make has no global project view. I think that’s only true if you’re dealing with self-modifying source code files or autogeneration of source code. If that’s not the case, I can’t see how the order of recursion can matter at all as long as the dependencies are met.

As for the claim that make is slow, let’s put that off until we compare it to the other build systems.

Personally, I really like make. It’s small, clean, and elegant. It does one thing and it does it very well. It’s easy to extend and modify. At first I was surprised that make didn’t do implicit dependency checking of C files (building a C file when an included header file changes), but it fits perfectly with the simplicity of make. It’s just a dependency graph with actions. If you don’t tell it about a rule, it won’t know about it. Fortunately, we can use tools like makedepend to generate the C dependencies with extreme ease.

One of my only gripes with main is the silly tab syntax. The fact that action commands have to be preceded by a tab character is awkward and out of place today, but it’s a small quirk I’m willing to adapt to. Fortunately gnu make’s error messages are very clear and it even asks “Did you forget to put a tab before the command?”

Jam

Jam was born an an improved make. It tried to keep all the good things about make and fix all the problems. And to a large extent, it succeeded.

Like make, Jam is small and easily portable. It deals better with inter-project dependencies by avoiding recursive Jam invocations (while still allowing individual sections to be built separately). The Jam language, even though it’s still fairly restrictive, is more expressive than make and it’s easier to write complex functionality.

One of the main differences from make is that Jam actually provides a fair amount of base functionality in its Jambase file. Jam out of the box knows about some of the most popular development environments and languages (including implicit dependency checking for C/C++ files), so it simplifies the build files for the simple cases. In the other cases, you can add your own rules and actions very easily.

I find it funny that while Jam fixes make’s weird tab requirement, it adds its own “space semicolon” weird command terminator (although I know some programmers who think that “space semicolon” is the one and true way). Either way, it really doesn’t matter since it’s such a small thing.

Jam also has spawned some forks that add extra functionality but are fully backwards compatible: FTJam and BoostJam.

Scons

Scons has been hailed as the next step in the evolution of build systems. It is supposed to be a much improved make-like system, not only written in Python, but using Python as the language used to define the build itself. Python is a fully general, object-oriented language, so it’s extremely expressive. It also has the advantage of being a well-established language with a great set of documentation, debuggers, and tools, which can make creating and debugging complex build scripts easier.

Scons also claims to be extremely accurate when it comes to determining what files need to be built. It doesn’t rely in the time stamp for a file, but it uses the MD5 signature instead (a type of checksum approach). Another very intriguing feature I didn’t get around to testing is the network cache of built object files.

Walking into this test I was a bit afraid of what I might find. I had read some reports of several people having problems with Scons performance on large data sets. However, the latest version (0.96.90), released just a couple of months ago, is supposed to have some performance improvements.

The Method

As a test, I decided to run each of the different build systems on the same codebase. Instead of using some real-world codebase, with its own set of quirks and problems (and the difficulty of easily building it with the different systems), I wrote a script to generate a simple C++ codebase. The code structure is based on what I expect to see in my own projects with many different projects. The physical dependencies in the generated codebase are extremely well contained, and header files never include other header files. Real code bases would have more complicated dependencies and would make the tendencies we see here even more exaggerated.

The specific parameters I used for this test were:

• 50 static libraries
• 100 classes (2 files per class, .h and .cpp) per library
• 15 includes from that library in each class
• 5 includes from other libraries in each class

This is by all accounts still a small or at most medium-sized codebase. A full game engine and tools can easily become much larger than this.

Thinking back, I really should have done the test with at least 100 libraries, not 50, because all my libraries have an extra associated project for unit tests. No big deal. I don’t think it would have changed the results very much. The important thing was to get enough code to make measurements noticeable (if we just build 10 files every build system is going to be really snappy).

For each of the build systems, I measured three operations:

• Full rebuild. Compiling the full source code for the first time. I didn’t expect this time to change much at all from build system to build system or even across platforms. I was quite wrong!
• Incremental build: Doing another build without any changes. This is the really interesting measurement that will tell us a lot about potential for fast iteration.
• Incremental build on a single library: Doing a build of a library without any changes.

I did the measurements in both Linux (2.6 kernel) and Microsoft Windows XP for different systems. Clearly some build systems only run in one platform (Visual Studio). But I decided to run some of the other build systems under Windows as well to provide a more fair comparison.

The specific hardware I ran these tests in is not as important since all we’re comparing are their relative merits. But for the curious it’s a P4 2.8 GHz CPU with hyperthreading, 2GB of fast RAM, and a 7200 rpm EIDE hard drive. The most important part is that I had enough memory to prevent thrashing.

GNU make, Jam, and Scons all support parallel builds. While it won’t speed up incremental builds any, this can reduce the time for full builds dramatically. Since this test was done in a single-CPU machine (and the primary measure was incremental builds), I restricted all the builds to use a single process.

The Results

We can make lots of very interesting observations from this table.

First of all, it confirms what I had seen all along, that Visual Studio is horrible for incremental builds with many projects. My off-the-cuff estimate of one second per project ended up being extremely accurate (54 seconds for 50 projects). That’s simply not acceptable for me.

As I feared, Scons, failed the fast iteration test as well. It actually ended up being slower than Visual Studio in all accounts, even for individual library rebuilds. It might do the “right” thing under all conditions, but frankly, that’s not a price I’m willing to pay to get absolutely correct results. I really don’t think I encounter any situation in everyday work in which Scons would do the right thing and make or Jam wouldn’t.

At this point, I was afraid that I just wasn’t going to be able to get the type of iteration I wanted out of file-based, compiled languages. Fortunately that’s not the case. Both make and Jam do a great job and fall in the range of what I consider acceptable (around a couple of seconds).

There are two interesting observations to be made about full build times from the chart above. First of all, Scons with g++ under Linux is twice as slow as Jam or make for a full rebuild. I find that extremely surprising. Although I guess that’s the extra minute of dependency checking plus some extra overhead of its own. I tried some of the suggestions to get faster Scons builds (by trading off accuracy for speed), but they just improved incremental build times by a couple of seconds. Clearly, Scons needs to do some catching up before it can play with the big boys.

The other one is comparing g++/Linux with VC++/Windows XP. Jam is over twice as slow with VC++/Windows XP than it is with g++ under Linux. Is it Windows XP or is it Visual C++? I don’t know. It would be interesting to try the experiment with g++ or some other compiler under Windows and see if the times are reduced at all. I suspect the Windows file system might have something to do with that.

Conclusion

This little experiment cleared up a lot of doubts for me. I’m ready to ditch Visual Studio as a build system and replace it completely with Jam or make. Make is a simpler but Jam probably edges it out because it’s a bit nicer, it doesn’t have any recursive problems, and the default functionality is pretty handy. It’s hard to go wrong with either one.

Since most programmers still expect to work from within the Visual Studio IDE, you can easily create a “makefile” project type and hook it up to the build system of your choice.

One interesting idea that came up during this research in the Scons mailing list is that of a background process that monitors which files change and updates dependency graphs on the fly. So whenever you initiate a build, all the work has already been done and the build can start right away. A variation on this idea that has been brought up in some TDD mailing lists is that of the build system not just computing dependencies in the background, but actually attempting to compile the code and run the unit tests in the background. If any of the tests fail, they can even be highlighted in the source code editor. Sort of like an on-the-fly, smart code checker on steroids.

Of course, we could also choose a language that has much smaller build times. I haven’t worked on a large-scale C# project yet, but the small tools I’ve created have impressed me with how fast the iteration can be. The same can be said for scripting languages such as Python or Lua. Unfortunately, we’re stuck with C++ for the foreseeable future in game development, so we better learn to deal with it the best we can.

For now, I’ll be happy to stick with Jam and two-second incremental builds. Let’s start jamming!

generate_libs.py