Let me get one thing out of the way to stop all the trigger-happy, const-bashing, would-be-commenters: const doesn’t make any guarantees that values don’t change.

You can change a const variable by casting the constness away, or referencing it through a pointer, but you really had to go out of your way to do that. If it helped with that, const would solve (or improve) the memory aliasing problem like Hitler pointed out. It doesn’t, so const is pretty weak as far as promises go. It just says “I, the programmer, promise not to change this value on purpose (unless I’m truly desperate)”. Still, even a promise like that goes a long way helping with readability and maintenance.

With that out of the way, what exactly do I mean by using const everywhere?

Const non-value function parameters

Any reference or pointer function parameters that are pointing to data that will not be modified by the function should be declared as const. If you’re going to use const just for one thing, this is the one to use. It’s invaluable glancing at a function signature and seeing which parameters are inputs and which ones are outputs.

void Detach(PhysicsObject& physObj, int attachmentIndex, const HandleManager& mgr);

Marking those parameters as const also serves as a warning sign in case a programmer in the future tries to modify one of them. Imagine the disaster if the calling code assumes data never changes, but the function suddenly starts modifying that data! const won’t prevent that from happening, but will remind the programmer that he’s changing the “contract” and needs to revisit all calling code and check assumptions.

Const local variables

This is a very important use of const and one of the ones hardly anyone follows. If I declare a local (stack) variable and its value never changes after initialization, I always declare it const. That way, whenever I see that variable used later in the code, I know that its value hasn’t changed.

const Vec2 newPos = AttachmentUtils::ApplySnap(physObj, unsnappedPos);
const Vec2 deltaPos = newPos - physObj.center;
physObj.center = newPos;

This is one of the reasons why I did a 180 on the ternary C operator (?). I used to hate it and find it cryptic and unreadable, but now I find it compact and elegant and it fulfills my const fetish very well.

Imagine you have a function that is going to work in one of two objects and you need to compute the index to the object to work on. You could do it this way:

int index;
if (some condition)
	index = 0;
else
	index = 2;

DoSomethingWithIndex();

Not only does that take several lines not to do much, but index isn’t const (argh!). So every time I see index anywhere later on in that function, I’m going to have to spend the extra mental power to make sure nothing has changed (and, with my current coding style, I would assume it has changed).

Instead, we can simply do this:

const int index = (some condition) ? 0 : 2;
DoSomethingWithIndex();

Ahhhh… So much better!

Const member variables

This one doesn’t really apply to me anymore because I don’t use classes and member variables. But if you do, I strongly encourage you do mark every possible member function as const whenever you can.

The only downside is that sometimes you’ll have some internal bit of data that is really not changing the “logical” state of an object, but it’s still modifying a variable (usually some caching or logging data). In that case, you’ll have to resort to the mutable keyword.

Const value function parameters

Apparently I’m not a total Const Nazi because this is one possible use of const that I choose to skip (even though I tried it for a while because of Charles).

Marking a value function parameter as const doesn’t make any difference from the calling code point of view, but it serves the same purpose as marking local stack variables as const in the implementation of the function. You’re just saying “I’m not going to modify that parameter in this function” so it makes the code easier to understand.

I’m actually all for this, but the only reason I’m not doing it is because C/C++ makes it a pain. Marking parameters as const in the function declaration adds extra verbosity and doesn’t help the person browsing the functions at all. You could actually put the const only in the function definition and it will work, but at that point the declaration and the definition are different, so you can’t copy and paste them or use other automated tools or scripts.

The concept of const is one of the things I miss the most when programming other languages like C#. I don’t understand why they didn’t add it to the language. On something like Python or Perl I can understand because they’re supposed to be so free form, but C#? (Edit: How about that? Apparently C# has const. It was either added in the last few years or I completely missed it before). It also really bugs me that Objective C or the Apple API doesn’t make any use of const.

Frankly, if it were up to me, I would change the C/C++ language to make every variable const by default and adding the nonconst or changeable (or take over mutable) keyword for the ones you want to modify. It would make life much more pleasant.

But then again, that’s why the call me the Const Nazi.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

What becomes significantly more difficult is learning to effectively apply TDD to code with more dependencies. A question that I’m often asked from people trying to use TDD for the first time is “How can you possibly use TDD for high-level game code? It’s impossible!”.

When The Going Gets Tough

At that point, the temptation to give up on using TDD for high-level code becomes very strong. It seems that all the time spent writing mocks and hooking objects together is a waste of time and that you could be writing game code much faster without it. “Maybe it would be best to skip TDD, just for that part”.

Bad idea!

By giving up on TDD for high-level code, you’ll be missing out on the main benefit of TDD: designing better code. I’ve said this many times, but it bears repeating: TDD is not a testing technique. It’s a design technique. You get lots of benefits from applying TDD [2], but one of the main one is much more modular and dependency-free code.

That’s because TDD turns our weaknesses into advantages. We’re humans and we’re lazy. We don’t want to repeat ourselves constantly or write complex code. If we’re writing a test for some code we’ll write in the future, we’re going to create a very simple test. We’re probably going to want to put some object or data on the stack and make a function call and nothing else. We certainly don’t want to initialize the graphics system, create a new world, add some level data, start the physics simulation, and then call the function! So by being lazy, we end up writing very simple code with the least amount of dependencies.

What does that mean for our high-level code? The AI logic seems to need access to every single game system, at least the way we implemented it in the past. Using TDD to implement AI logic doesn’t mean writing tests to write the same code you would have written before. It means writing tests and then writing some code that makes those tests pass and nothing else. For example, we might realize that the AI doesn’t need to access the physics system, cast a ray in the world, find out what entities is interesting, and return that information. Instead, all it might need to do is output some data saying that it wants a ray query done at a later time (which in turn is the key to batching those ray casts and achieving high performance).

You soon realize that the code you write is very different (and better!) to what you would have written before. Once you get over that hump, TDDing high-level code is not about twisting your code with mocks, but becomes simpler code that takes simple input data and creates simple output data. By pushing through and forcing yourself to apply TDD, you made a breakthrough and reaped all the benefits.

That’s the main reason why TDD books push the idea that no code can be written without a test first. Otherwise, inexperienced TDD programmers would be too quick to quit and would miss out on the technique completely.

Not only that, but code that is not created with TDD tends to have the bad habit of spreading. When code was not designed through TDD, it means that other code that uses it will probably be difficult to TDD itself. So it’s a bit like const-correctness: You’re either in our out, and there’s very little room for half measures.

On the other hand, I would argue that using TDD on a math library is a bad idea. It’s essential to write good unit tests for a math library, but probably not to design it through TDD. Are you really going to implement a cross product differently just because you wrote tests before? The emphasis there has to be on correctness and performance, not on creating the interface or implementation through tests.

It’s A Tool To Help Development

A few days ago, I received an email from Caleb Gingles with a great question:

I’ve started a new game project and have been making an effort to strictly follow TDD from the beginning. Which has been a challenge, as much of the coding at the beginning of the project has consisted of operating system requirements and framework stuff. [...] However, there are other situations that seem much harder to test. Like the main loop in a game. According to Kent Beck’s book, everything begins with testing, and no code is written unless a test requires it. But how can a test require the main loop framework in a game? The loop just… is. It exists to allow you to do certain other things, it doesn’t have much purpose in and of itself.

Applying what I just talked about, we could use TDD to create the main loop of a game. It definitely requires thinking about it differently and breaking preconceptions. It’s about writing code to pass the tests, not writing tests to fit the code we want to write.

So we could do something like this:

TEST(MainLoopExecutesCorrectNumberOfTimes)
{
    MainLoopState loopState;
    MainLoopUtils::Execute(loopState, 4);
    CHECK_EQUAL(4, loopState.frameCount);
}

All of a sudden we see a way to test something that before it seemed impossible to test. We can add tests to make sure certain functions are getting called (maybe we add those function pointers to the loopState), that the main loop exits under certain conditions, etc.

For some games with complex GUIs and different game modes, it might make a lot of sense to have a more flexible main loop and develop it completely through TDD. On the other hand, for a simple iPhone game that has a single main loop that never changes, there might be no point at all in using TDD. The loop code is going to end up looking more or less the same as it would without TDD, just more complex and generic. TDD didn’t help any in that case.

How about code that calls into OpenGL, or some other non-TDD-friendly library? Do we have to TDD that? Frankly, I wouldn’t bother. I did give it a good try at one point several years ago and I can say it is possible, but it’s a pain and you gain almost nothing from it. The main benefits is trusting that the library you’re using really does what it’s supposed to do, but it won’t affect much the design of your code. So what you’re really doing is adding tests to that library (which might or might not be of value).

My advice in a situation like that: Call the library from as few places as possible (I didn’t want to say make a thin wrapper because that implies isolation which is not the goal here), and just test that your code is called at the right time with the right data. The actual code that calls the library should be small and simple enough that you should feel OK not having any tests for it.

Even though it seems to go against what I said in the previous section, TDD is not all or nothing. It’s not a religion.

When you’re starting out, I encourage you to push yourself to think how you could apply TDD to situations you don’t think are possible. Once you’re experienced enough, you’ll know when to say enough is enough, and use TDD only when it actually benefits you. At that point you’ll also know how to write code in a way that plays well with TDD, even if the code itself was not developed through TDD.

In the end, TDD is a tool to help you develop better and faster. Don’t ever let it get in the way of that.

[1] That’s actually a horrible place to apply TDD. More on that in a second.

[2] Usable API, simple code, unit tests, documentation, safety net for refactoring among others.

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

Every developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

• All my code is written in C and C++ (except for a bit of ObjC and assembly).
• It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
• I always try to write important code through Test-Driven Development.
• I’m the only programmer (and only designer).
• Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

• I’m showing all the data there is and how big it is. Nothing is hidden.
• I’m making it clear that it’s free of pointers and temporary variables.
• I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

This article was first published in the 2005 Career Guide issue of Game Developer Magazine.

I lead the R&D team here, and my team’s primary responsibility is to come up with the technology that game teams use for different projects. Nowadays, that means putting a lot of middleware programs through their paces and choosing the ones that suit our needs. But it also means getting down and dirty and writing a lot of code for our engine and tools.

With that in mind, come on and follow me through a typical workday.

8:10 AM

I roll in to work on my bicycle like I do every day. Even though I’m an early bird, Jim, a programmer in my team, arrived a few minutes earlier and is already at his desk.

I quickly catch up with my email. I also notice that the PCLint pass on our codebase last night caught a couple of minor warnings, so I quickly fix those and check them in.

8:20 AM

Today is Tuesday and our two-week iteration ends on Friday. An iteration consists of a fixed period (usually two weeks in our case) during which the team commits to deliver a set of functionality described through customer stories. The customer (in our case the other internal teams in our company) creates and prioritizes a set of stories. My team then breaks down those stories into tasks and estimates how long they will take to complete (tasks vary between 1 and 8 hours).

Jim and I walk over to the “war room” (the room with all the task cards corresponding to user stories pinned to the wall) and we choose the task that reads “Blend ragdoll and animation,” which is part of the user story listed as “Throwing a rigid body at a character and seeing a hit reaction.” The task was originally estimated at 3 hours, but now that we know that the ragdoll system is a bit more complicated than we thought, we re-estimate it at 4 hours.

8:23 AM

In addition to our own personal desk areas, we have pair-programming stations in the R&D lab, with two monitors, two keyboards, two mice, two chairs, and plenty of room for two people. All production code is written by pairs of programmers. We grab a station and start working on the task.

Since we’re using test-driven development, we first write a very simple unit test for what we want to do, and only then write the code to make it pass. In this case, our first test checks that we create a blender object without inputs and that it produces no output. Then we write the blender class and make the test pass. It’s a tiny step, but it’s a step in the right direction. The whole cycle of write test, write code to make test pass, refactor, takes only 5-10 minutes, and we do it over and over.

8:39 AM

We have implemented a small amount of functionality, and all the code builds and all tests pass, so we check it in source control. This is called continuous integration, and it requires that programmers work on the latest version in source control and check in their own changes many times throughout the day.

8:50 AM

Other people from the team roll in, grab other pair programming stations, and start going at it. On their way in we have a quick chat and find out what we’re all working on.

Work hard…

9:37 AM

I overhear Joel and Gary discussing how they’re going to test something that requires updating the physics simulations. I just did that a couple of days ago so I jump in the discussion. It turns out they need to do something that is almost the same as what I already did, so I point them to what I wrote and they will modify it to suit their needs.

10:05 AM

Things are moving along very nicely. We’ve checked in four times already this morning. When a pair gets really going, they might check in as many as 20 or more times in a single day. At this rate we might be done sooner than the four hours we had estimated.

10:14 AM

We have the daily Scrum meeting at 10:15, so we head over to the war room. Scrum meetings are very short, standup meetings with the whole team (eight people plus Brian, our producer). We quickly go around the room discussing what people are doing to get everybody up to speed.

10:23 AM

During the meeting the topic of how we’re loading physics assets came up. So we return to the pair programming area and have a quick discussion with everybody involved. We draw some quick UML charts on a whiteboard, think about how the data is going to be passed around, and after ten minutes we reach an agreement and go back to work on our tasks.

11:15 AM

We got the blending working correctly. All the unit tests pass, although we haven’t implemented it in the demo yet. There’s another task card for that. We check the code in right away. The code definitely can stand to improve in a few places because we were just concentrating on getting things working, so we spend some time refactoring. We have lots of unit tests, so we’re confident our refactoring isn’t introducing any bugs.

11:56 AM

The code is now in a much better state. We check it in and wait a few minutes for the build server to report that everything built correctly and all tests passed. During that time we talk about what the next task should be.

12:05 PM

We have nice shoulder-high surf today, so I’m going out surfing at lunch with several of my teammates. If it’s not surfing, it’s a basketball game, cycling, running, or even yoga. If all else fails, there’s always a game of Guild Wars with the rest of the High Moon clan.

1:10 PM

Back at my desk I eat a quick lunch while I catch up on my email (which I haven’t read since this morning because I’ve been at the pair programming station). I read an email from a middleware developer in response to one of our queries earlier in the day and fire back a quick response.

1:25 PM

Sean stops by my desk. He’s ready to go back to work, but the programmer he was pairing with in the morning got pulled to work on some last-minute issues with Darkwatch, which is about to go gold and has priority over anything we’re doing.

Sean quickly brings me up to speed on what they had been working on that morning, which was to display the exact memory usage for our physics system. I remember them mentioning that in this morning’s Scrum meeting. I also worked on the physics system last week, so I’m pretty familiar with the code. In a couple of minutes we’re already making progress.

… rock hard

2:07 PM

After writing several unit tests and implementing some functionality, we’re ready to add the memory display to the demo program. A couple of minutes later, we have a display in the demo indicating how much memory the physics system is using, and we see it go up and down as we add and remove rigid bodies from the demo.

But something is wrong. When we remove rigid bodies, the memory doesn’t come down to the same level it was before. We exit the demo and we see a long memory leak dump. First things first, we check in our changes, and then we dive in and look for the code that is leaking memory. It’s probably not caused by the code we just wrote, but we have “collective code ownership,” which means that everybody is expected to fix anything that needs fixing anywhere.

2:12 PM

The build just broke! The build server detected a failed build and notified us through a system tray application. I bring up the latest build log and I see that one of the unit tests failed in release mode. Tyson, who is sitting at the station next to ours, says “Oh yeah, I know what that is. I’ll fix it right now.” In less than 30 seconds he makes the change, and checks it in. A few minutes later, the build system reports a passing build and everything is back to normal.

2:17 PM

We found the memory leak. It was a misuse of reference counting. We first wrote a unit test that showed it failing, and then we fixed in the physics library. We check in our code.

2:18 PM

We go to the war room and grab the next task. This one has to do with being able to expose different variables and functions on the demo to tweak them through a GUI. We sign up for the task and start working on it.

3:43 PM

We’re totally in the zone. We’ve been writing tests and code like crazy and this task is going really well. We’ve done three check-ins for this task alone in the last hour.

4:12 PM

Another pair is discussing how to handle errors for some particular case. This is an important topic and it should be done consistently across all the code, so we have a quick discussion about it involving most of the team. Five minutes later we’ve made a decision and we all resume working on what we were doing before.

5:40 PM

We do our last check in for the day. We’re almost done with the task, but not quite there yet. Even though we could stay another hour and try to finish it, we’re both quite tired by now and we’re starting to not think as clearly and make some mistakes. We can wrap this up tomorrow morning as soon as we get in. The important part is that we got to a state where we could check in.

We have a rule that says nobody can check in code and leave. You have to wait for the build server to build the code successfully and pass all the tests. We keep build times short, so that usually means hanging around an extra 4-5 minutes. If anything breaks, you need to fix it or revert what you did, but there’s no excuse to leave with a broken build.

While I’m waiting for the build server to finish, I check the pile of email that accumulated in my inbox during the day.

5:44 PM

After a few minutes we get the green go ahead from the build server. Today was a pretty productive day, and at this pace we will definitely complete all the user stories by Friday. The demo we’re putting together is also starting to look very cool.

One of the things that agile development, and especially pair programming, does is to make each day very intense. There are no little breaks to read email, check a web site, or just goof around. We get a lot accomplished in a work day, but we can’t keep that pace for a long time, so it’s important to call it quits and go home. That leaves me with time at home to read technical books, prototype different ideas, or work on side projects in addition to unwinding, spending time with my family, and enjoying other hobbies.

I hop on my bike and head home with a big smile on my face.

This talk discusses how agile development can help in all those scenarios. In particular we look at methodologies like XP and Scrum, and techniques such as test-driven development and pair programming.

Slides from the Montréal Game Summit 2004 talk (pdf format)

As I write this there are still a few cardboard boxes strewn across our home as we finish unpacking everything. My wife and I just finished a surprisingly painless cross-country move to sunny San Diego. The town house we’re moving into is definitely larger than our previous condo, but it wasn’t until we had to pack everything, move it, and then unpack it that we realized how much stuff we had accumulated over the years. Yes, there was a reason for everything we had, but it was still a lot of stuff. Then we came to the realization that we prefer a more open, simpler environment with fewer pieces of furniture, fewer paintings covering the walls, and less clutter in general.

That is an interesting insight because it applies to many different things, including programming: There is a cost associated with having a complex, cluttered environment, whether it be the furniture in your house, your class hierarchy, or your build process. In particular it will create resistance to change, which can be in the form of a cross-country move, or a simple class refactoring.

Goals

The ultimate reason to write code is to have the computer carry out a sequence of instructions. In our case, that sequence of instructions is a game or a tool to help create the game. It is tempting to think of the computer as the “audience” for our program, but it is not a very good audience. It is way too forgiving, and as long as things compile, it’ll be happy with anything you throw at it. We can do better than that.

We can still write code that is correct, but with people in mind as the primary audience. The usual reason for choosing people as your audience is that software development is a team effort, and it’s very important that other people be able to understand and modify your code. True, but there’s more to it than that. Even if you were the only person writing code, you should still write for people, not computers. Assuming it’s more than a toy project, you will have to refactor your code as you learn new things about it. Probably not once, not twice, but many times over the course of the project’s lifetime. You will have to keep it up to date, fix bugs, add new features, etc. I am convinced that refactoring (or lack thereof) is one of the major influences on code quality, and, as a consequence, product quality, so anything to help with that is extremely important in my book.

Lately, I’ve been taking it a step further and thinking of my audience not just other programmers, but other people who are not programmers. If you manage to write some code that a non-programmer can more or less understand, then you know you’re there. The code should be simple and self-explanatory enough that should be a breeze to refactor in the future.

But wait, as if ease of refactoring wasn’t enough, there are more reasons for writing really simple code. Edsger W. Dijkstra in his lecture “The Humble Programmer” advocated the avoidance of clever tricks and urged programmers to be fully aware of the limitations of the human mind when writing programs.

Dijkstra’s reason for keeping clever tricks away is not so much for ease of refactoring, but the quality of the program itself. Programmers are all too eager to tackle something too ambitious at once, without breaking it down into smaller, more understandable sub-problems, and, as a result, create a disastrous program that crashes often and doesn’t even implement all the required features.

Today, this sounds like a surprisingly modern attitude. After all, it’s right up the alley of agile development. But amazingly, this lecture took place in 1972. Talk about timeless advice (OK, OK, in the really short scale of computer history, not in the grand scheme of things).

Brian Kernighan had also something to contribute to the idea of writing simple code, but coming from a very different angle:

While I totally agree with the sentiment, I feel it is not as important as the other two reasons for aiming towards code simplicity. Frankly, if you use test-driven development and take steps of the correct size, you should hardly ever find yourself knee-deep in the debugger trying to make sense of your program. However, a decade or two ago that made a lot more sense, especially given that debuggers were less sophisticated then as well. Still, anything that encourages programmers to keep things simple is a step in the right direction. So if that convinces some people, so be it.

Attaining Simplicity

Great. Simple is good. How do we write “simple” code? That’s not something they teach in school and it’s not something that is pushed for in the workplace, so it’s probably not something you’re used to thinking about. The rest of the article will be a series of suggestions and techniques on how to make code as simple as possible. Specific examples are given in C++, but the general techniques should apply to most object-oriented languages.

Avoid unnecessary language tricks.

Listen to Dijkstra and go easy on “clever code.” If you feel the need to write a comment to explain to other people what that cute line of code you just wrote does, you should very strongly consider re-writing it so a comment is not necessary. I’m not saying to avoid taking advantage of language features, just the ones that can be done in another, simpler way. For example, don’t use a template when a non-templated implementation is perfectly fine. Don’t rely on a clever arrangement of variables so they’re initialized in the correct order even though the language standard says it should work.

Avoid or postpone performance optimizations that obscure the meaning of the code for as long as possible.

You’re all familiar with the Donald Knuth quote: “Premature optimization is the root of all evil.” Programs have a way of twisting into evil, misshapen fiends when they’re optimized, and they usually become much harder to refactor afterwards. Specifically in this case, I suggest avoiding (or delaying at least) optimizations that affect how simple or readable a section of code is. There are many ways to skin a cat, and there are also many ways to optimize a program. I find that there is often a way to optimize things that doesn’t cause a major loss in code simplicity. But if that’s not possible, at least defer that optimization as long as possible. Maybe you won’t need it, and maybe the code will change before then and you’ll save yourself a lot of rework.

Functions should do one thing only.

The secret to a good function is that it should do one thing and one thing only. Functions that follow that rule are usually quite small (at most 10-15 lines in C++) and their whole meaning can be easily understood at a glance. Because the function has only one purpose, refactoring becomes very easy: No need to worry about how changes in that function will affect other parts of the function, no pesky local variables reused in many parts of the function, etc.

Maybe this is a bit extreme, but ideally, functions should be so simple that I should be able to look at a function, delete it and re-write it from scratch without any difficulty.

I follow two guidelines to keep functions as simple as possible:

• If a function goes over 10 to 15 lines of code, I look at it really hard and try to split it. Chances are it’s doing more than one thing.
• If I ever feel the need to write an explanatory comment about what a few lines of code do, I move those lines into a function of their own and give it a name that explains what it does (usually the same as the what the comment would have been). By following this rule, the only comments in my code are the ones that explain *why* the code does things the way it does, and deal with higher-level such as the purpose of a class or a library.

Classes should do one thing only.

This should come as no big surprise. If we want our functions to be small, simple, and just one thing, we also want the same qualities from our classes. When classes are left unchecked, they can easily grow into unmanageable blobs of many thousands of lines. The largest class I had the misfortune of encountering had a staggering 10,000 lines in it, and the problem was compounded by inheriting from another class with over 8,000 lines! As you can imagine, having to do any work near that class was quite a punishment.

Personally, I feel that C++ classes should ideally be no more than about 200-300 lines (remember, with virtually no other comments). Anything that goes beyond that, and I start thinking hard about what exactly the class does. If a class reaches 1000 lines, it’s time to get the axe out.

One technique I’ve been following to keep classes as simple as possible is to use the controversial suggestion from Scott Meyers on how to decide on the scope of a function. The quick summary is that functions should be pushed out of a class as much as possible: if a function can be implemented as a non-member function, then do it; if it can be a class static, make it so; otherwise, leave it as a member function.

One immediate consequence of following that advice is that a class is stripped of anything that is not essential. Helper functions (both public and private to the class) are pushed out, and the meaning of the class is made more clear. The second consequence is that refactoring is a lot easier. By using non-member functions in anonymous namespaces whenever possible, we avoid the need to edit any header files at all, which results in faster compile and iteration times. Also, since those functions are completely independent of the class, they’re trivial to move around, promote to higher levels, etc. while member functions require some massaging to move around.

I’m suggesting that we keep functions small, but also that we keep classes small. We’re not going to end up with any less code than we would have before, but we’re going to end up with many more classes scattered all over our program. We have pushed the complexity from the function and class level up to the program organization and architecture level. The next article in this series will deal with how to manage this newfound complexity, but here’s a preview: Use hierarchies.

Timings, logging, and error handling

Ideally, I’d like my code to be really small, simple, and to the point so its meaning is always clear and obvious. Unfortunately, in the real world, that’s is not always possible. We can start with a very clean 5-line function, but then we add timings around it to get an idea of its performance, and logging statements to know what is going on, and error handling code to deal with the unexpected, and before we know it, we have a monster function whose meaning is lost in a mass of details.

What can we do about it? We could argue that timings should be done with non-intrusive methods such as using profilers, but that’s not always possible or desirable. We often want very specific timings that only affect a particular section of the code (load times), or we want to continuously monitor performance of certain sections while the game is running. For example, we probably want to get an idea of how much time we’re spending each frame on each of the major tasks: entity updates, AI calculations, physics, animation, rendering, sound, etc. We need to wrap each of those areas in timing statements. The more detailed we want those timings to be, the more junk we’ll have to add to the code. We can try making that as clean as possible by using macros that only require one line, but it’s still junk that obscures the meaning.

Logging doesn’t even have a non-intrusive counterpart unless you want to automatically add prologue and epilogue code to every function (which would generate way too many messages and would affect performance too much). So we’re stuck writing messages by hand to the log system.

Error handling at least as has the possibility of using exceptions, which add a lot less clutter than returning and checking error codes. Unfortunately, we have to deal with several real-world issues which often make exception-handling impractical. The first one is performance. As much as I hate bringing this up, exception handling can easily have a significant performance impact even in modern hardware. This is something that will hopefully go away in the near future as PC hardware and consoles become more powerful. The second problem is compiler support. Even today, some console manufacturers are admitting that exception-handling support in their compilers is not usable and we shouldn’t rely on it. So much for that idea. Finally, and probably most importantly, is complexity and programmer knowledge. If you have read Exceptional C++ you how complicated it can be to write exception-safe code, and not every C++ programmer out there today is going to feel comfortable using exceptions. All of this is a vicious circle, because until programmers demand robust and efficient exception handling, compiler and system creators aren’t going to provide them, which means a lot of programmers are not going to be exposed to them, etc, etc.

So, what great solutions do I have to this problem? None, I’m afraid. This is where I’d like to hear from some of you. How do you deal with timing, logging, and error handling in a way to minimize the clutter and leave your code as clear as possible? Email me and I’ll edit this article or make a new one with any great solutions I receive.

Striking a balance

In the end, even if we have the best goals in mind about simplicity, we will be forced to make compromises. Error handling is one case. Another case is often performance. Sometimes, having many scattered functions can add up to a noticeable performance hit in some inner loop of a performance-critical section. In that case simplicity needs to give way to practicality and do whatever is necessary. Just make really sure that having many functions is the cause of the performance problems before you make any changes.

It is also important not to confuse code simplicity with algorithmic simplicity. Writing simple code doesn’t mean that you have to use a silly bubble sort to order a list. Always choose the most appropriate algorithm for the task at hand, just implement it with the simplest possible code. Implementing it in a very simple way will also make possible to easily change it down the line with a more efficient algorithm.

The next article will deal with how to manage all the complexity we’ve pushed up into the program structure. Until then, keep your functions short, your classes clean, and remember The Humble Programmer:

Session 1 | Session 2 | Session 3

Friday 26th: Languages and Tools

The vast majority of game developers seem to be using C++. Some reasons cited for using other languages were better integration with their environment (especially for Java and mobile gaming), faster development time with a higher-level language, or the availability of a refactoring browser. C# this year had several people using it for tools development. Some people claimed very positive experiences, but others said that doing managed C++ wrappers around their C++ code was much too cumbersome.

GUI-development APIs

Most people using C++ were also using MFC for building GUI tools. Some of the other GUI APIs mentioned were Qt, Borland Builder, Fox Toolkit, Perl with Tk and Python with wxPython.

Asset management

The tools people were using for asset management were:

• Visual SourceSafe: 40%. Horror stories of constant database corruptions, but it seemed to work (more or less).
• Alienbrain: 40%. People weren’t thrilled with it. Felt too heavy weight. Only good if you use some of the more advanced features.
• CVS: 8%. They were using a small amount of data.
• Perforce: 5%. Artists were happily using the default GUI without any trouble.
• No management at all: 5%. Considering switching to an asset management system.

The amount of data people had in their asset management systems varied between 2GB all the way up to 20 GB. One handheld developer only had 300 KB of assets total. Ah, the old times!

Source code version control

The breakdown for source code was very different than for asset management.

• Perforce: 40%. Most people here were using branching.
• Visual SourceSafe: 36%. Hardly anybody used branching (not a surprise).
• CVS: 20%. Tortoise GUI apparently is very good, although several people were using it through the command line only.
• Accurev: 4%. An interesting two-step check-in is the main difference of this tool.

Unfortunately, nobody was using Subversion, but several people had looked into it in the past. They just released version 1.0 so hopefully next year someone will have tried it.

Documentation

Doxygen was mentioned as a good documentation tool. It creates documents from specially-marked comments in the source code as well as the structure of the source code itself. However, not many people referred to the documents that were generated for the code they were working on. It seems that Doxygen’s primary use was to explore other people’s code or a third-party API.

Nearly 50% of the participants were using some version of Wiki in their development, which is much more than in previous years. People were using it for design documents, coding standards, and tool documentation. Some were even using it with their artists and designers, so it’s not limited to programmers only.

The main advantage of Wiki are that it is a permanent but interactive collaborative medium, so it has more structure than an email thread and it stays there permanently. It makes bringing new people in the team much easier. Some of the most popular Wikis are: Wiki (the original), TWiki, OpenWiki, MoinMoin, and of course, ScrumWiki gets a special mention.

A participant found a plugin for Microsoft Word to create chm (help) files very useful to go from Word documents to online help for the tool. Some other people used Robohelp to create the help files instead.

Some of the tools mentioned to explore the relationships between classes were:

• Understand C++
• TogetherC++
• Rational Rose code analyzer

Finally, a simple yet very effective tool was a simple whiteboard plus a cheap digital camera to capture the output and put it up on an internal web site (either that, or the fancy whiteboards with built-in printer).

Bug Tracking

The majority of the participants were using a bug-tracking program. Some of the ones mentioned were:

• Bugzilla: Ugly but effective and free. Hard to convince management though.
• FogBugz: Simple, lightweight, but limited reports.
• Mantis
• Plain databases
• Many publisher-specific bug-tracking

Some teams reported having conflicts between their own internal bug-tracking system and what the publisher wanted. Some people ended up duplicating entries on both, or only starting to use the publisher one after code complete.

Testing

Many people were doing unit tests this year. Most of them were using some form of XUnit (CppUnit, JUnit, or NUnit mostly). A few people were using CppUnitLite or another, lightweight custom unit-testing framework.

Several teams did some sort of higher-level test on their games. All of these tests were driven with custom scripts in Perl, Python, or some other high-level language. Some of the tests were done with scripting languages internal to the game itself. Some people were playing back input (including random input, aka, the monkey), and some were even using doing networking tests as part of their automated testing.

Previous GDC roundtables

• GDC 2003. By the Books: Software Engineering in the Games Industry
• GDC 2002. By the Books: Software Engineering in the Games Industry

Session 1 | Session 2 | Session 3

Thursday 25th: Processes and Methodologies

From the initial discussions and show of hands it was clear the most people did not have one specific methodology that they followed during development. The majority of the participants admitted to using some sort of “code and fix” (or “hack and slash”, depending on the severity) approach. About 20% of the participants claimed to be using a more controlled iterative approach, while only about 6% were doing some form of agile development.

Agile development is a hot topic now and people seem very curious about how it can be adopted for game development since it seems to fit well with the chaotic and ever-changing nature of our industry. The rest of the session covered several aspects of agile development in detail.

SCRUM

SCRUM is a specific agile development process (like its more famous cousin extreme programming). It is best described in the book Agile Software Development with SCRUM by Ken Schawaber and Mike Beedle. SCRUM is not as radical a departure from traditional iterative development as extreme programming, so it might be easier to adopt for more conservative shops. SCRUM is currently being used at Sammy Studios with very positive results (but it has only been in place for slightly over a month). I’m really looking forward to hearing from them next year to see how it worked once they have more experience with it.

Perhaps the most characteristic part of SCRUM is its emphasis on 30-day iterations (called sprints). Like other agile methodologies, it requires small tasks (4-16 hours long), task estimates from the programmers, and continuous re-evaluation and prioritization. Meetings are frequent but very quick (no sitting allowed!). Applied to game development, the priorities for the tasks should come from the designers and the producers.

SCRUM allows for very good project visibility by keeping track of what tasks are completed and which ones are left for each iteration. It also allows to measure the development velocity (how many hours of real programming are done each day), which is invaluable when making estimates and schedules for the next 30-day sprint.

Test-driven development

Test-driven development is an integral part of many agile-development methodologies. For every feature, test-driven development requires that a unit test be written first before the feature is implemented. This is applied to all features of the program, and tests are refactored mercilessly as needed. Some good books on this subject are Test Driven Development: By Example by Kent Beck, and Test Driven Development: A Practical Guide by David Astels.

A surprisingly-high 18% of the participants were doing some form of test-driven development (especially considering that last year only a handful of people were doing any sort of unit testing at all).

The unit tests generated are a great form of regression tests for the code. Whenever anything breaks that has a unit test, you’ll know it right away (assuming the unit tests are executed very frequently). A good rule is that whenever a bug is found, first a test must be written that shows the bug, and only then the bug is fixed. That ensures it won’t pop up again accidentally. Other benefits of test-driven development is that it narrows down where to look for existing bugs, it serves as a form of documentation that can never get out of date, it helps tremendously with refactoring, and it forces a more modular design.

On the down side, it requires a large amount of “extra” code. People reported ratios anywhere from 1:1 to 2:1 of test code to application code. Some systems can be very difficult to test with unit tests, especially high-level systems or non-deterministic ones. Some people were using unit tests that compared the output of the renderer pixel by pixel with previous screenshots for some very specific rendering features.

Extreme programming

Two people in the roundtable were doing full extreme programming, but almost 20% of the people were using one or more features from extreme programming. The best introduction to extreme programming is the white book: Extreme Programming Explained: Embrace Change by Kent Beck.

Pair programming is one of the most controversial extreme programming features. A lot of people were doing it but only at some specific times (like near a milestone). One participant tried doing full pair programming for a small tool project and reported very positive results with it. Some of the advantages listed for pair programming was dissemination of knowledge, and training of junior developers. Interestingly, someone pointed out that pairing an experienced developer with a junior one was often a great combination because you would get the stability of the experienced developer but the fresh ideas and perspectives of the junior one. Just about everybody agreed that pair programming was much more useful than code reviews, which are often too late to do anything about any code problems. Also, pairing with another person could force developers to concentrate for longer periods of time without checking their email, answering the phone, or browsing the web.

Of course, not everybody likes pair programming. Some people feel they can be more productive by themselves, and especially if they are not comfortable with the other person or they think in totally different ways. The final output of two people pairing won’t be 2x the amount of code, but it’s supposed to be more than 2x as reliable (which means less maintenance and bugs down the road).

One interesting question was how to convince management to adopt extreme programming. The best answer was to make sure you don’t use the words extreme programming (and certainly don’t write it eXtreme programming or you’ll be doomed! ). Talk about what specific processes you want to put in place, or even talk about agile development instead. Another view was to start applying some of the techniques of extreme programming without direct management approval, such as test-driven development, refactoring, or pair programming. Showing management a cost-analysis study comparing extreme programming and plain code-and-fix could be another, more direct approach.

Office layout

Extreme programming promotes an open-office environment (the bullpen environment). The possible noise and distractions are supposed to make up for the increase in communication. The participants were clearly divided. Some of them found such an environment very useful (to the point of putting the computers on desks with wheels to arrange things in whichever way made more sense for that particular project), but others wanted to have the peace and quiet of a single office that allows them to concentrate and be productive. Some of the people who had an open environment also had private offices where people could go sometimes when they needed to concentrate more. Finally, nobody seemed to like cubicles very much. They’re almost the worst of both worlds: noisy, but not as open to communication as a fully open environment.

Previous GDC roundtables

• GDC 2003. By the Books: Software Engineering in the Games Industry
• GDC 2002. By the Books: Software Engineering in the Games Industry

Session 1 | Session 2 | Session 3

Overview and Attendance

Thanks again to everybody who attended the roundtables this year. All in all we had about 150 people show up (with the first two sessions being completely packed), which means the interest in software engineering continues to increase as projects get more complicated, and team sizes and budgets continue to increase.

We continued the trend from previous years and we had about 30-35% of managers and project leads. That’s a good thing because for things to change management needs to get involved, and the sooner they do, the better.

The large majority of the attendees were working on PC and console games (surprisingly, about the same number of both), with very few handheld and web game representation.

C++ this year was the undisputed king of the programming languages (so much that I couldn’t count the number of hands). There were only a handful of people using straight C, but a higher representation of Java and higher-level scripting languages like Python or Lua. Several people were starting to use C# for their tools development.

One new piece of data I collected this year was team size. I really wish I could compare this against previous years, but the results were very telling even by themselves. Specifically, I asked for the number of programmers in their teams. These are the results:

• 1-5: 35%
• 6-10: 45%
• 10-20: 20%
• > 20: 0.05%

All of the programmers who were working on a team of 20+ people were either from Electronic Arts or from outside of the games industry. Still, the numbers are pretty high already, and those are probably just signs of things to come. Even John Carmack lamented the increasing number of people required to make a game in his keynote speech.

Wednesday 24th: Software Engineering and Techniques

The session started with a discussion of how can software engineering help game development. Lots of reasons were brought up:

• Reducing risk
• Reducing bugs
• Helping with reuse
• Easier multiplatform development
• Repeatability
• Reduce crunch time
• Easier transfer between projects
• Better support of their content teams

All of those are reasons that will continue becoming more and more important in the near future.

A few people had concerns about over-engineering things and brought up the need to strike a middle ground. Even so, nobody admitted to doing too much engineering in their teams (although people did have over-engineered, or over-complicated, pieces of code).

Coding standards

About 75% of the attendees actually had coding standards in their companies (or, in some cases, multiple coding standards!). Only about half of them actually followed them, but most people who had a coding standard found them useful. The reasons given were that it was a good leveler by making the code more similar throughout the project, and that it saved programmer time and headaches.

However, not everybody was in favor of them, and certainly not everybody agreed on what they should cover and in what detail. Some people argued that a coding standard should only cover interfaces and not interfere with implementation details. Some found it difficult to maintain a standard when interfacing with third-party code or APIs. A good point that was brought up was that a coding standard shouldn’t be too detailed

Code ownership

Participants identified three main types of code ownership:

• Team ownership: The whole team owns the code.
• Loose code ownership: One or two people are mostly responsible for an area of code, but other people can still modify it.
• Strict code ownership: Only one person can modify the code.

Very few people had team code ownership (the few who did were doing pair programming). Most people either had loose or strict code ownership. The benefits of team code ownership are avoiding the “hit-by-a-bus syndrome” (where you might lose your most important programmer because of some unexpected turn of events), and spreading the knowledge of the system to everybody and making all the team members more valuable. The disadvantages were that sometimes people did not take responsibility for their own actions. Team code ownership requires a lot of communication and might work best in smaller teams (5-6 programmers).

Code reviews

About 20% of the participants had some form of code review in place, but the specifics varied a lot. Some people’s reviews were just a quick look by someone else before the code was checked-in (code buddy). This was done more frequently around milestone time to avoid people checking in code that would break the build.

A few people had more formal processes in place, printing out the code, and reviewing it in more depth. However, a lot of people found this type of reviewing more tedious than it was worth, and it has the potential to quickly degenerate into pointing out formatting problems or allowing people to go off in tangents about their favorite language feature.

The question of what exactly was the purpose of the code review came up. It clearly doesn’t help any with code design because the review happens after the code is written and working. The main reasons listed were bug prevention and avoiding horribly inefficient code.

Automated builds

About 40% of the participants had some sort of automated build system in place (way up from the 15-20% of last year!). Most of these automated builds were done by custom scripts in a high-level language such as Perl or Python. How and when the builds are done also varied a lot: some people had them triggered with each check-in, others had a machine building continuously, and others were simply doing nightly builds.

Some of the benefits claimed for automated builds were checking that the code could always compile and link, and easier version labeling and tracking. A lot of people, in addition to building the source code, were also doing asset builds. As an additional step, many people were doing some form of validation on their executables by loading a few levels or playing back some input (especially random input). People collected the results of the build by either sending emails or posting results on an internal web site.

Two points of caution were raised: A fast turnaround time was important, otherwise the QA and the engineering team could get out of synch and make bug fixing more difficult. The other point is that bad automation could be worse than no automation.

UML

Only a few people were using some form of UML (about 10%). Most of them were just using it in the initial stages of development to sketch out some ideas and discuss them, but then the models were quickly discarded and not kept up to date. Interestingly, some people were generating UML after the project was complete to pass it along to another team.

Only one person was using many of the UML features (sequence diagrams, use cases, etc), and everybody else was mostly using it for static class design. Several other people reported doing initial design just using “boxes and lines” instead of formal UML, which worked very well for their projects.

Some of the tools people were using were ArgoUML(free and open source), Enterprise Architect, and UMLNotepad (free, open source). Not exactly UML, but several people mentioned Doxygen as a tool to generate some visual representation of the code, which was particularly useful to explore code without existing documentation.

Previous GDC roundtables

• GDC 2003. By the Books: Software Engineering in the Games Industry
• GDC 2002. By the Books: Software Engineering in the Games Industry

You know one of the reasons I refuse to live near most cities in the US? Traffic. I’m not talking about the “this road is crowded” type of traffic that lets you zip along at a good speed, or even about the “I can’t get out of my lane because this is so packed” type of traffic that moves slowly along. No, I’ve come to accept and deal with that. It’s the “we might as well get out of the car and enjoy the sunshine because we aren’t moving” type of traffic that I can’t stand. Amazingly enough, it seems to happen around most major cities in the US during rush hour, and sometimes this “rush hour” stretches from 7AM until 8PM. It’s a bad sign when car manufacturers have advertisements telling you how much more comfortable you’ll be in their car when you’re stuck in traffic. Under those conditions it can easily take over an hour just to cover a distance of 3 or 4 miles.

Other than pointing out that you’re much better off walking, cycling, or using public transportation, what’s the point of all this and how does it relate to the physical structure of a program? There are some things that just don’t scale well. They appear to work perfectly fine for a small number of units, but as soon as a certain threshold is reached, things seem to bog down and eventually collapse under their own weight. Just adding more lanes doesn’t appear to solve the problem either, judging by the number of clogged-up 5-lane highways everywhere. Sometimes, you need to take a step back and deal with the problem in a different way. Either that, or buy a nice music system and enjoy your time in traffic.

We might not have much of a say over how traffic should be dealt with where we live, but we certainly have a lot of choices when it comes down to structuring our C++ source code. For small projects, we can blissfully code away without paying any attention to physical structure and we won’t be any worse off for it. However, as a project grows, it reaches a critical point, and compilation times start getting slower and slower, to the point where tiny changes could make you wish you were stuck in traffic instead of staring powerlessly at your monitor. Adding a faster CPU, more memory, or a better hard drive can help make things faster, but is usually not a good long-term solution.

Build Types

We are usually concerned with the time for two types of builds:

• Full builds. In this case we care about the time it takes to build the whole project from scratch, starting from a totally clean build. This situation comes about when we just want to use the result of the build of a project we’re not actively modifying. For example, an automated build machine will most likely be doing full builds of the game, so the turnaround time before a build is ready will depend on the full build time. Another example might be if you need to link your code with a library for which you have the source code.
• Minimal builds. Once we have done a full build on a project, we then make a very small change to its source code and build it again. That’s the time for a minimal build. This is what you really care about when you’re actively working on a project, making modifications and compiling constantly. Ideally, building the project after a small change should require very little time This allows for very fast turnaround time for debugging, or even to get feedback from the compiler on silly syntax errors we just typed.

Improving the physical structure of a program often reduces the time of both types of builds. Unfortunately things don’t always work out so neatly and there are times where some changes will make one type of build faster and the other slower. Understanding what affects each compilation time allows us to optimize our compilation strategy and strike a balance that fits our needs.

Clearly, the time for both types of builds depends on the number of files and the complexity of those files. Both types of builds are also affected by the number of files each file depends on (the number of #include statements in each file). However, as we’ll see in a moment, in the case of a full build there is the chance of caching the includes of some files and reusing them for other files.

There is something very different about minimal builds. Their build time is usually dominated by the number of files that depend on the modified files. In the worst situation, every file will depend on the file that changed and a full build will be triggered. In the ideal case, only the file with the changes itself will be compiled and no other files will have been affected. In one case the build could take less than a second, and in the other it could easily take multiple hours.

The rest of this article will look at different techniques to reduce build times and how they affect each of those two build types.

Counting Includes

It is easy to underestimate how quickly include statements can compound. If file A includes file B, and file B includes file C and D, every time someone includes file A they’re including three other files for the ride. Add a few more levels of inclusion with header files including many other header files, and you have a recipe for disaster (or for really long build times at least).

As an experiment, I added one more feature to the script I wrote last week. The script analyzes a set of source code files and determines how many times each file is included by other files in a recursive way. So, in our trivial example above, file C will be reported as being included twice (once by B directly, and once by A indirectly). I then decided to test it on the source code for a high-level game library (I’m not going to be any more specific since it wasn’t particularly good code and it had a pretty hideous physical structure). I wouldn’t be surprised if it’s not very different from the level of complexity of a lot of game code out there. As a point of reference, the library was composed of 300 cpp files and 312 header files.

Before I ran the script, I tried to guess how many times the most included file in the whole library was included by other files. My guess was around 600 times, just because I knew that the physical structure of that code wasn’t pretty. I figured maybe almost half the files included that one header file, and a few others included it indirectly. Boy was I wrong! Here are the shocking results:

That means that during the course of a full build for those 300 cpp files, one header file could be included over 10,000 times! No wonder this particular library seemed to take a long time to compile. Notice that the other top files quickly drop to being included around 800 times each (which is still even higher than my initial estimate).

As a comparison, I tried running that same script on another, much smaller library, but also one with a much better physical structure and many fewer dependencies between files. This second library was only made up of 33 cpp files and 39 header files. The most included file was only included a total of 23 times (with the second one being included less than 10 times). So, having the number of classes grow by a factor of 10, caused the number of includes to grow by a factor of 1000. Clearly not a very scalable situation.

Things aren’t quite that bad though. Header files typically have a set of include guards in them, to prevent the compiler from adding duplicate symbols if it encounters the same header file multiple times during the compilation of one cpp file. This is what include guards look like:

With every header file having include guards around it, I turned on the /showincludes switch in Visual C++ and performed a full build. The total number of includes during the course of building the 300 classes in the library was an astounding 15,264. Better than the worst-case-scenario we calculated earlier, but still tremendously high.

Apparently some C++ compilers try to optimize this situation by automatically caching header files and avoiding hitting the disk to reload them over and over. Unfortunately, there is very little hard data about that, and you’re always at the mercy of your current compiler writer. Was that true for Visual Studio .NET 2003?

Redundant Guards

To test if I could speed up the compilation any, I added redundant include guards to the whole library. Redundant include guards are like the regular include guards, but they are placed around the actual #include statement. I first saw them mentioned in the book Large Scale C++ Software Design by John Lakos (written in 1996), but popular wisdom claims that they are unnecessary with modern compilers. Well, time to test that.

This is what redundant include guards look like:

I wrote a quick script to add redundant guards to all the source code and did a full build again. The number of includes reported by the compiler went down to 10,568 (from over 15,000). That means that there were about 5,000 redundant includes in a full build. However, the overall build time didn’t change at all.

Result: Zero. Apparently Visual Studio .NET 2003 (and probably most of the major compilers) does a pretty good job caching those includes by itself.

Recommendation: Stay away from redundant include guards. I never liked having the including files know about the internal define, and if the guard ever changes it can easily break things. Besides, the code looks a lot messier and unreadable. It might have been worth it if we could define #include to expand to a redundant guard automatically, but I don’t think that’s possible with the standard C preprocessor.

#pragma once

Just in case, I decided to test another strategy and see if I obtained similar results. Instead of using redundant include guards, I added the #pragma once preprocessor directive to all header files. Visual C++ will treat files with that directive differently and it’ll make sure that those files are only included once per compilation unit. In other words, it accomplishes the same thing as the external guards, just in a non-portable way. Here’s another really simple script to add #pragma once to all the header files.

Result: No difference. Just as with redundant include guards, it seems that the compiler was smart enough already to optimize that case.

Recommendation: Don’t bother with it. It’s a non-standard construct that doesn’t get any apparent benefit. If you still feel compelled to use it, at least wrap it up with #ifdef checks for the correct version of Visual Studio.

Precompiled Headers

During a full build, every cpp file is treated as a separate compilation unit. For each of those files, all the necessary includes are pulled in, parsed, and compiled. If you look at all the includes during a full build, you’re bound to find a lot of common headers that get included over and over for every compilation unit. Those are usually headers for other libraries that the code relies on, such as STL, boost, or even platform-specific headers like windows.h or DirectX headers. They are usually also particularly expensive headers to include because they tend to include many other header files in turn.

From our findings in the previous two sections, it is clear that some compilers cache the headers encountered for each compilation unit. However, they don’t do anything about duplicated headers found across multiple cpp files, and that’s where precompiled headers come in.

When using precompiled headers, we can flag a set of headers as being part of the precompiled set. The compiler will then process them all at once and save those results. Every compilation unit will then automatically include all the headers that were part of the precompiled set at the very beginning, but at a much lower cost than parsing them from scratch every time.

The catch is that if any of the contents of the precompiled headers changes, a full rebuild is necessary to compile the program again. This means that we should only add headers that are included very often throughout our project but that don’t change frequently. Perfect candidates are the ones we mentioned earlier: STL headers, boost, and any other big external APIs. I always prefer not to include any headers from the project itself, although if you have a header that is included in every file, you might as well include it in the precompiled set (or, even better, change it so it’s not included everywhere and improve the physical structure).

The gains from using precompiled headers are quite dramatic. The game library we mentioned in an earlier section took over 14 minutes to compile without pre-compiled headers, but only 2:30 when using them. Those are huge savings! Minimal rebuilds are also improved because we avoid parsing some of the common headers for one file, but the results aren’t as dramatic as for full builds.

Precompiled headers are not without their downside though. The first problem is that precompiled headers often end up forcing the inclusions of more headers than it is absolutely necessary to compile each individual file. Not every file needs <vector> or <windows.h> included, but since a fair amount of them do and those are considered expensive includes, they’ll invariably end up in the precompiled header section. That means that any compilation unit taking advantage of precompiled headers will be forced to include those as well. Logically, the program is the same, but we have worsened the physical structure of the source code. In effect, we are trading extra physical dependencies between files for a faster compile time.

The second problem is that precompiled headers are not something you can rely on from compiler to compiler and platform to platform. The only compilers I’m aware of that implement them are Microsoft’s Visual C++ and Metrowerks’ CodeWarrior (although I just did a Google search and apparently gcc also supports precompiled headers–that’s great news!). For the rest of us using different compilers, we’re out of luck as far as this technique goes. Considering how important multi-platform development is becoming in the games industry (and elsewhere), this is a big blow against them.

Finally, by far the worst aspect of precompiled headers, is what happens when you combine the first two problems: Take a set of source code developed on a compiler where precompiled headers were available, and try to build it on a different platform. The code will compile since everything is standard C++, but it’ll compile at a glacial pace. That’s because every file is including a massive precompiled header file, and it is being parsed over and over for every compilation unit without taking advantage of any optimizations in the part of the compiler. If the code had been developed without precompiled headers in the first place, each file would only include those headers that it absolutely needs to compile, which would result in much faster compile times.

So earlier, when I said that the game library without precompiled headers took over 14 minutes to build, that’s because it was written with precompiled headers in mind. Otherwise, I estimate it would only take about 5-6 minutes (still much longer than the 2:30 it took with precompiled headers).

Personally, I have never yet worked on a set of code that was developed to be compiled on multiple platforms and where some of them did not support precompiled headers. I suppose the best approach is to use #ifdefs to only include the precompiled headers in one platform and the minimal set of includes for the rest, but it seems like an extremely error-prone approach where programmers are going to be breaking the other platform’s builds all the time. I’d be interested to know how teams working in such an environment deal with it.

Result: Huge gains both for full builds and minimal builds if your compiler supports them. Much worse physical structure.

Recommendation: Definitely use them if you’re only compiling in a platform that supports them. If you need to support multiple platforms, the gain is still too big to pass up. It probably is worth if you manage to separate the includes for precompiled headers with lots of #ifdefs and try to keep the physical structure sane for platforms that don’t support them.

Single Compilation Unit

This is an interesting trick that you won’t find in most books. I first read about it in the sweng-gamedev mailing list about a couple of years ago. Be warned, this is hackish and ugly, but people claimed really good results. I just had to find out for myself how it stacked up against the other techniques to reduce build times.

This technique involves having a single cpp file (compilation unit) that includes all the other cpp files in the project (yes, that’s right, cpp files, not header files). To compile the project we just compile that one cpp file and nothing else. The contents of this file are simply #include statements including all the cpp files we’re interested in. Something along these lines:

As you can imagine by now, I wrote a script to create that file from a directory containing the source code for a project. I just created a file including all the cpp files in that directory, although a better way of doing it would be to parse the make (or project) file and only include those files that are actually part of the project. That way, as I discovered, you avoid including outdated files or files that are in that directory but are not part of the project.

I created this file (everything.cpp), compiled it and… get ready: The build time went down from 2:32 minutes to 43 seconds! That’s a 72% decrease in build time!! Not only that, but the .lib file it created from that library went from 42MB down to 15MB, so it should help with link times down the line. People in the mailing list reported even better results with gcc than with Visual Studio.

What is the reason for such reduction in build times? I can only speculate. I suspect part of it is due to avoiding the overhead of starting and stopping the compiler for every compilation unit. However, the biggest win probably comes from the reduced number of included files. Because everything is one compilation unit, we only include every file once. The second time any other file attempts to include a particular header file, the compiler will have already cached it (and it’ll have include guards so there’s no need to parse anything). To test this theory, I again turned on the /showincludes switch. Indeed, the number of includes during a full build went down from 10,568 to 3,197. That’s a 70% reduction of included files, which is, probably not coincidentally, the same reduction in build time.

One very interesting observation from this experiment is that build times are probably more dependent on the number of actual includes performed by the compiler than I thought at first. All the more reason to keep a really watchful eye on the physical structure of the program. The third part of this article will cover what architectural choices we can make to improve the physical structure and keep the overall number of includes down.

Unfortunately this method also has its share of problems. One of the biggest problems is that there is no such a thing as a minimal build anymore. Any modification to any file will cause a full rebuild. Of course, the full build takes a only a fraction of the time it took before, so this might not be much of an issue.

As with precompiled headers, we’re adding a lot of physical dependencies between files. In this case, files will have a physical dependency with any files that were included before it in the everything.cpp file.

However, the most objectionable of all problems is that we can now run into naming conflicts. Before we assumed each cpp file was a separate compilation unit. Now they’ve all been forcefully added to the same one. Any static variables or functions, or anything on an anonymous namespace will be available to every cpp file that comes after it on the large everything.cpp file. This means that there’s potential for having conflicting symbols, which is one of the things that anonymous namespaces were supposed to solve in the first place. If you have decided to use this technique, you will want to keep everything as part of a separate namespace or part of the class itself and avoid global-scope symbols completely.

Result: Huge improvement in full-build times, but minimal-build times become much worse. Potential for clashing of static and anonymous namespace symbols.

Recommendation: The gains of this technique are simply huge so it would be a shame to ignore it. It is probably no good for regular builds, but you might want to have it as an option when you just care about doing full builds (automated build machine or building someone else’s code). If so, make sure to wrap symbols in namespaces or classes.

analyze_includes.pl
add_external_guards.pl
add_pragma_once.pl
create_everything.pl

The next (and final, I promise) part of this article will look at architectural choices that can greatly influence build times as well as looking briefly at link times and see what we can do about them.

A few days ago, Tom Whittaker (a friend who is currently working at Firaxis) emailed me with some surprising facts about the behavior of the #include directive in the C++ compiler of Visual Studio .NET. It turns out that in my book, I hinted that compilers often optimize the include step for header files and so they don’t pay the costs of opening a file and loading it. When Tom turned on the /showincludes flag on the compiler, he saw that the same file was repeatedly being included by the compiler, even if it had internal include guards. He ran several interesting tests and I’ll add a link from here whenever he gets around to putting them up on his web site (hint, hint, Tom).

All of that made me think again about the physical structure of a C++ program, how important it is, and how often it is overlooked. This two-part article will attempt to explain why the physical structure of a program is so important, present some useful guidelines, and show its effect on compile times.

Physical Structure

We are all familiar with the logical structure of a program. It deals with classes and functions and namespaces and templates. That’s what you learn about in school, and what you read about in most C++ books. Design patterns, object oriented programming, etc, etc, all deal with the logical structure of a program. And, truth be said, it really is the most interesting part.

The physical structure of a program deals with the files that make up its source code. The .cpp and .h files, how they include each other, how they’re subdivided into directories, etc. While not as interesting and sexy as the logical structure, it is crucial to understand the consequences of the physical layout of a program for any real-world project of any significant size. That includes just about every modern PC and console game, but probably not handheld devices because of their small code size.

Because it’s not a particularly hot topic, not many books talk about the physical structure of a program. The best book in the subject is Large Scale C++ Software Design by John Lakos. No technical lead should work on a game without at least having read parts of that book. Yes, some of the advice is a little outdated by today’s standards (it’s an almost antique book in the computer world–almost 10 years!), and some parts are a bit long winded with detailed measurements. Skip those in your first read and you’ll still get a lot of gems along the way. Every time I come back to that book I end up getting something new out of it. I haven’t read it in about 3-4 years, so it’s just about due for another read.

A good physical structure will result in files without many dependencies to other files, and with files clearly grouped in cohesive modules or libraries. On the other hand, a bad physical structure is one where files are related to other files from all over the project, without clear delimitations or boundaries. This is a clear example of the “blob” antipattern. Unfortunately, if left unchecked, this is the type of physical structure that develops over time.

Benefits of a Good Physical Structure

Why would anybody care about the physical structure? After all, the things we care about when we’re writing a program are that it does what it’s supposed to do, and that it does it fast. That might be true for a demo, or a throwaway project, but for a large project, maintainability is also a very important requirement. It’s no good to have a very efficient class if we can’t change it, and it’s also no good if iteration to test a change takes a long time.

The benefits of a good physical structure of a program are:

• Better logical structure. Usually, keeping an eye on the physical structure of a program results in a better logical structure. By reducing the dependencies between files, we will probably reduce dependencies between classes. It is often the case that unexpected connections will grow between classes and even different libraries if their header files are already included. Programmers won’t have anything to remind them that they’re just adding a new dependency when they decide to call some global function.One of my pet-peeves when programming for Windows is having windows.h included in every single .cpp file, either directly or indirectly. In the type of programs I write, most classes don’t need anything in windows.h, yet it is forced down their throats. At one point I attempted to remove a global windows.h include in a library since it was supposedly not needed anywhere, just to have to give up because of the many unnecessary DWORD, BOOL, and screwed up min and max calls scattered everywhere on the source code.
• Easier to refactor. If you think of each file as a little box literally connected with a string to all the other files it depends or is dependent on, the more of those strings there are, the harder it is to untangle it from the overall mess and separate it. It is the same thing with refactoring: The worse the physical structure, the harder it is to make any refactoring changes that involve separating or isolating sections (which, in my experience, it’s one of the most crucial refactorings you have to do to prevent programs from growing into the “blob” antipattern).
• Easier to test. Not surprisingly, the more modular and independent the project is, the easier it becomes to write unit tests for it since each piece can be tested separately from the rest. Unit tests really benefit from having very few dependencies, so one of the many benefits reaped by doing test-first development is a very modular design with a logical and physical structure that has very few dependencies.
• Faster compile times. This might come as a surprise to some people, but it is usually the most tangible and objective result of having a good (or bad!) physical structure. Usually, the worse the physical structure, the longer the compile times will be. Compile times are an issue for large projects. Even with the fastest machines today, full builds on large projects can easily take hours. More importantly, builds caused by changing just a file or two can trigger builds that last almost that long. Needless to say, having such a delay every time a change is made is not exactly encouraging programmers to test their work and iterate it to make it better. Part two of this article will look exclusively at compile times.

Guidelines

Here’s a distilled set of guidelines from Lakos’ book that minimize the number of physical dependencies between files. I’ve been using them for years and I’ve always been really happy with the results.

1. Every cpp file includes its own header file first. This is the most important guideline; everything else follows from here. The only exception to this rule are precompiled header includes in Visual Studio; those always have to be the first include in the file. More about precompiled headers in part two of this article.
2. A header file must include all the header files necessary to parse it. This goes hand in hand with the first guideline. I know some people try to never include header files within header files claiming efficiency or something along those lines. However, if a file must be included before a header file can be parsed, it has to be included somewhere. The advantage of including it directly in the header file is that we can always decide to pull in a header file we’re interested in and we’re guaranteed that it’ll work as is. We don’t have to play the “guess what other headers you need” game.
3. A header file should have the bare minimum number of header files necessary to parse it. The previous rule said you should have all the includes you need in a header file. This rule says you shouldn’t have any more than you have to. Clearly, start by removing (or not adding in the first place) useless include statements. Then, use as many forward declarations as you can instead of includes. If all you have are references or pointers to a class, you don’t need to include that class’ header file; a forward reference will do nicely and much more efficiently.

One unfortunate aspect of those guidelines is that the compiler doesn’t really care one way or another. As long as you provide enough includes, the compiler will happily churn away at the source code and come up with the desired object file. It is up to us to minimize the number of includes and to follow those rules. While it seems fairly straightforward at first (after all, it is only three rules), things get more complicated as soon as heavy refactoring starts. As you split classes, move functions, and consolidate functionality, there might be several unnecessary headers. The only quick way to verify whether they’re needed or not is to, gulp, comment them out and try to compile the file.

This situation is more common in large files, which are the ones you’re most likely going to be refactoring. If left unchecked, you’ll soon be left with a myriad little tendrils connecting your file to the rest of the code without getting any benefit from it. Wouldn’t it be great if there was an automated tool that would check that?

Automating the Guidelines

I did a quick search and I couldn’t find any tools or scripts that did exactly what I wanted. I suppose that a massive C/C++ style-checker tool might look for some of those things (like redundant includes), but nothing jumped out. Most programs are more concerned with checking logical errors and constructs than looking at the physical structure. I started from Scott Meyers’ summary of major C++ checkers. In particular I looked at CodeCheck, PC-Lint, and CodeWizard. I admit that I didn’t look too deep into them, so maybe they also check for some of these guidelines, but it’s certainly not their biggest selling point from reading their web sites.

As a quick challenge, I decided to try and write a quick script to check against those guidelines. Soon I realized that you can’t really have the word “quick” in anything related to parsing C++. I was quickly reminded how ugly the language is, how many quirks it has, how much baggage it a carries around. Java looks mighty tempting sometimes.

I decided that if I was going to have a chance to do this, I would need to leverage other software to parse the source code for me and the script could work directly on the abstract representation of the program. Not exactly what I had in mind, but I stumbled on the XML and perlmod output generated by Doxygen. I have been using Doxygen for years, but I always thought of it as a pretty documentation generator. I never realized what a powerful and robust C++ parser it was until now.

In a few hours I was able to put together a quick Perl script that hooks up to the perlmod output of Doxygen that checked against those rules. Guideline #1 was the easiest one to check against. Really, you don’t even need a fancy parser for that. Guideline #2 is already checked by the compiler (if you don’t have enough includes, the program won’t compile). Guideline #3 is the trickiest one. Still, the script makes a valiant effort and checks for the most common cases. It’ll try to detect whether an include is not needed at all, or whether it can be replaced with a forward declaration.

However, the script is a conservative one and will not always detect that a header is necessary. It deals fine with simple constructs such as member variables, enums, references, and pointers. However it seems that Doxygen has very little knowledge of preprocessor directives, so it won’t catch any #defines brought in from header files. Doxygen also seems very limited in how it deals with inline functions (probably because Doxygen is looking mostly at the logical rather than the physical structure of the program). It would probably be possible to deal with templates, but I didn’t bother making it that far. The script also detects duplicate includes, both in header files and cpp files.

Even with all those limitations, the script can easily deal with about 80% of the cases in most of the code I work with on a regular basis. It was certainly insightful running it on the source code for some of the libraries I work with. I saw plenty of instances where headers could in fact be removed and the physical structure of the program improved. I would love to see a robust tool along these lines that could be ran quickly enough after each compile, or at least once a night in our automated build machine.

analyze_includes.pl

Part two of this article will look at the consequences of physical structure on compile times, and what we can do to improve that.

Have you ever tackled a problem that was supposed to be solved by writing some code in half an hour, but that code haunted you (or your coworkers, much to their chagrin I’m sure) for months or years to come? Then you have felt first hand the consequences of maintenance, the hidden cost of software.

The situation usually starts our very innocently. Maybe someone brings up an idea during a meeting, or maybe your lead asks you to estimate how long something will take, or maybe it’s you who decides to implement a cool new feature you’ve been itching to add to the game. In any case, you end up agreeing to taking a couple of hours to write some quick code that will solve that particular problem.

If someone expresses some doubts about whether it’s worth doing it, you quickly answer “Don’t worry. It’ll only take me a couple of hours.” Famous last words.

Let’s take it from the top. Unfortunately, nobody works at full efficiency. Once you take into account the other meeting scheduled for that morning, a co-worker coming over to talk about yesterday’s football game, the obligatory mid-morning break to answer email and browse your favorite web site, and the time to synch to the latest code and do a full build while you get a fresh cup of coffee, we’re talking at least four hours instead of two. No big deal, it’s still really quick and it’s definitely worth doing.

Of course, chances are the initial estimate of two hours was totally inaccurate. It just felt like a task that should take two hours: you just sit there, type away for two hours, and it’s done. After all, you’re a really good programmer and if you can deal with writing super-optimized [insert your favorite complex algorithm here] functions, then you can write that simple code three times over before breakfast. Software engineers are eternal optimists, especially as far as our own abilities is concerned. Once you sit down and start banging away at the keyboard (because it’s so simple that you felt no need to do any thinking before start programming) you realize that it’s going a bit slower than you thought, and the “couple” of hours quickly turn into four or five hours. Add the interruptions and other distractions we mentioned earlier to that, and we’re talking a full work day taken up by that simple task. Maybe you even end up staying for an hour or two after work to finish the task. Professional pride; after all, you said it was going to be done in just a few hours. No big deal; it’s just one day.

You come in the next morning with a big smile on your face ready to continue with the work you had to put aside before you started with this task. You’re just wrapping your head around the code to figure out where you left it off a couple of days ago, when your lead walks in and tells you that the code you wrote yesterday doesn’t exactly do what he had in mind. He wanted it to output things in columns instead of comma-delimited rows. Sigh. No problem. In an hour you give him the new version he wanted. We’re up to 10 hours total. Let’s keep counting.

Life is good for a while and you manage to put that task behind you. It left a bad taste in your mouth somehow, but at least it’s working and you’re moving along. Then, one day, you’re checking your list of assigned bugs and you see a high-priority bug. Something with that code you wrote is causing a tool to crash in all the designers’ machines with the latest release of the tools. Damn! You try to reproduce the problem in your machine so you can debug it, and, of course, it works (that’s the curse of the programmers’ computers–your code will always work flawlessly there… except when you’re trying to demo it someone). So you head over to the office of a designer, fiddle with it for a while until the crash happens again, hook it up with remote debugging, and eventually you see the obvious bug. Doh! Quick fix, new release of the tools. Another wasted morning. Without counting the time the designers were not able to work because of the crash, we’re up to 14 hours.

Weeks later, the story repeats itself. This time it turns out your code is too slow. “What do you mean too slow? It was very fast when I wrote it!” Then your heart sinks when you see that you were testing it with a couple dozen objects, but the designers are applying it to a level with 50,000 objects, and your O(n^2) algorithm isn’t cutting it anymore. This means you pretty much have to re-write it completely from scratch, but now you need to use a more complicated algorithm. This time it takes two full days to write, plus another day to figure out why the changes you made broke another tool that was using the same code. 38 hours so far.

Does the situation sound familiar yet? Maybe a little too painfully familiar? So we’ll skip over the 4 hours that will take to keep up to date when somebody else changes an interface that your code uses (and nobody knew it did because you wrote it by yourself without discussing with with anybody), and the full day it took Bob to fix it when your code broke the build and you were out sick.

The point of this drawn-out example is that we all do maintenance, and a lot more than we’d like to think. It is unfortunately rare for a piece of software to be written and to be left alone and untouched for years to come. You’ll have to update it to reflect changes to the underlying libraries, you’ll have to speed it up when its speed becomes critical, you’ll have to make modifications to it to implement the new features the artists are asking for.

I’d love to collect some statistics gathered from the source control program of a mature code base and see how often certain files are changed, what the hotspots are, and which projects are mostly left alone and never touched (and then learn something about what makes those files different). I’m sure someone has already written a script to collect exactly that type of statistics, so if I get a chance I’ll try it out and I’ll make sure to report back here.

What can we learn from all this?

The first thing is that the cost of writing some software is a lot more than the initial time it’s going to take a programmer to implement it. Depending on where and how that code is used, it could be many times that of the initial implementation cost. Unfortunately people (both programmers and managers) tend not to think of that when they’re deciding what features to implement or whether to give the green light to a new tool.

What’s the best way to reduce that cost? Having no software to maintain in the first place! The best code is no code at all. That’s the easiest code to implement, debug, and maintain! If we can avoid writing something, and have no negative impact to the project, then we shouldn’t write it. On the other hand, if having a programmer implement that feature is going to save artists and designers a huge amount of hours for the duration of the project, then we clearly should go ahead with it. We should simply be aware of the true cost of what we’re doing.

Sometimes we won’t know at the beginning whether we should implement something or not. If there is a chance that we don’t want or need to implement a feature, it’s probably worth our time to spend some time up front to decide whether we really need it. For instance, I was in charge of adding a texture caching system to our current game with the objective of reducing the amount of texture memory used and/or let us use more textures in a single level. I could have jumped straight into the task, coded some cool systems, optimized them, and had something acceptable by the end. Instead, I spent a couple of days running some tests: I measured the throughput from the disk while reading from a background thread and I wrote a really simple cache simulator running on the actual game data. By the end, it was clear that we were not going to get much benefit from texture caching unless we were willing to change how our levels were put together (which was not something we were willing to do at this stage of the project). Too bad that we couldn’t get that feature, but better that, than having spent a long time to write a bunch of code that is going to complicate things and not get us much of a benefit.

If we have decided that we need to implement a feature, then we should aim for the simplest possible solution that will do what we want. There are many reasons for this (probably a topic for another article), but in general, the simpler a solution is, the easier and faster it is going to be to maintain. You just don’t know what you’ll need to do to that code in the future: maybe you’ll need to make it faster, maybe you’ll need to extend it, or maybe you’ll be lucky and you’ll just have to keep it up to date. Trying to optimize it early or make it very general before it’s needed is going to be a waste of time now, and a waste of time when it comes time to maintain it, even if it makes you feel better about all the cool code you got to write.

Another interesting consequence of the cost of maintaining software is that the rate at which a team can write new code is going to decrease as the size of the code base increases. As more code is added, more time is spent maintaining existing code and making sure new code works well with existing code, and less time can be devoted to writing new code. This effect will be particularly noticeable in the first couple of years right after starting a project from scratch. At the beginning the project is a small, cute snowball that everybody understands. Development moves on at a really high pace and everybody’s spirits are high. However, as the months go by, the result of that pace makes the snowball grow and grow. Eventually it will become frustrating how hard it has become to make a small change that before would have been a trivial task.

Some of this can be mitigated by maintaining a good architecture: keep modular, independent subsystems, reduce dependencies, etc. Still, even with a good system, the weight of all the existing code makes it more difficult to change or add code in the future.

Something that can help the burden of software maintenance is using third-party libraries as much as possible. When people make decisions to use third-party libraries, they’re usually thinking of right now. They can pay X amount of money, and then have a programmer spend two weeks integrating those features into the game engine. What they might not realize is that they’re also saving costs in the future. Those libraries are going to be debugged separately, they’re going to be improved, and most importantly, they’re going to be forcefully separated from the project itself, maximizing the independence between the two. Yes, you’ll still have to retrofit your engine to make use of any interface changes of new versions of the libraries, but it’s still a big win.

Finally, in this age where reusability is looked upon as the Holy Grail, this could be a surprising concept for some: Throwaway code can be your friend. Throwaway or temporary code completely liberates us from any maintenance costs. We just write a something, use it, and once it fulfills its objective, we throw it away never to be seen again (OK, it’s fine to leave it in the dark depths of the version control program so we can refer to it if we ever need to). Perfect examples of throwaway candidates are quick scripts and tools to convert between file formats, or “scaffolding” code to help us get somewhere or help us get started while we wait for some other functionality to become available. It is very important to keep in mind what type of code we’re dealing with when we’re developing it though, and we should also clearly label it in some way so everybody knows it’s throwaway. That way a class labeled Tmp3DView won’t become deeply rooted in your tools, or will it?