The Quest for the Perfect Build System

First there were punch cards, and people somehow managed to write software. Then came interactive computing with mainframes and personal computers, and people wrote even more software and become even more productive. There is no doubt that our development environments today are light-years ahead of what the computer pioneers had half a century ago. Yet I constantly see projects suffer with horrible environments that force slow iteration cycles on programmers.

I define an iteration cycle as the time elapsed between making a trivial change and being able to see the results of that change. In particular I’m concentrating on large-scale (around one million lines of code) C++ projects, which is representative of modern PC and console games today.

Of course, there’s more to fast iteration than just the speed of the build system. How quickly you’re able to get in the game and see the results is a big factor (another reason why I wouldn’t want to live without unit tests). The physical dependencies of your program are going to affect how quickly your code builds. And if you’re always working with a full game executable that takes forever to link, your iteration times are going to be shot (yet another reason to use unit tests!).

But let’s put that aside and concentrate on the build system itself.

Fast iteration is more than just about time and speed. It’s also about how you feel about the code and what you dare do with it. When things are slow and painful, you’re going to be a lot less likely to try new things, or fix one last thing to clean up the code, or refactor something out of a header file into its own module. Over time, this is going to accumulate into cruft, hacks, and unmaintainable code. It’s also about not breaking up the flow, the mental state you’re in while you’re writing software. Interruptions of more than just a few seconds are much more detrimental than their time value alone.

When you add test-driven development to the mix, fast iteration becomes even more crucial. With test-driven development, you end up doing micro-cycles of modify-compile-test, sometimes several times per minute. Unless you have very fast build times, you’re dead in the water.

Before we go any further, let’s crunch a few numbers. It’s not so much to show specific improvements, but to have as a reference point going forwards. Think about the project you’re currently working on. How long does it take to build when you modify a single cpp file (or even no files at all)? I’ve seen projects that took over two minutes to build, and anywhere between 30 seconds to one minute is fairly typical. Let’s say 30 seconds for this example. How often do you need to do a build? It’s not very fast, so maybe once every 5 minutes, 8 hours a day. That adds up to a staggering 48 minutes per day, or 10% of your full work day!

Now reduce the build time to two seconds instead. And to make things more interesting, let’s do a build every two minutes. That adds up to be 8 minutes per day, and, most importantly, they feel like almost instant builds, so they don’t bring you out of the flow state. That’s what our goal should be for a build system.

The Goals

As I started looking into different build systems (there are a lot of them out there!), I noticed that they have very different sets of goals, and a lot of them are fairly irrelevant to my particular needs.

As a game developer, I work with a varied, but limited number of platforms (most of which are not supported out of the box by build tools anyway). I’m also not planning on releasing the source code any time soon to let players compile the game in any platform, so I have no need to have a build system that automatically detects all the correct settings and does the right thing for any possible platform.

Some other build systems had useless features built in, such as access to version control or the ability to send emails. I consider those type of tasks to be totally beyond what the core build system should do, and I prefer to have those features in a wrapper system that does build/test/deployment of builds by calling the build system itself.

So what exactly do I want in a build system?

• Super-fast incremental builds (around two seconds). This is the key to fast iteration and I want this at almost any cost.
• Customizable. I’m going to be using unusual compilers and environments. I want to be able to easily set my own rules and actions and not be tied to any particular platform or compiler.
• Correctness. I want the build system to build the minimal amount of files and still do the right thing under most normal circumstances.
• Multiprocessor support. I also care about the speed of a full build, and with multiprocessor machines finally becoming popular, using multiple processors is a great way to speed up build times.
• Scalable. I want all the related source code to be tied to the same build system. I don’t want to create a separate build file for every minor tool so they build at a reasonable speed. I’d like to simply build any target and have the minimal amount of files rebuilt.

The Contenders

Visual Studio .NET

Most game developers doing Windows or Xbox development will be familiar with this build system. I’ve been stuck with it for many years, and while things were not great in Visual Studio 6.0 and earlier versions, it became decidedly unusable when it turned into “.NET”. I don’t know what happened, but I suspect that in their effort to cram all those languages under a single IDE they crippled the C++ build system even more.

My main gripe is how slow Visual Studio .NET is when doing an incremental build on a solution with many projects. It’s roughly a second per project. If you have 50 projects, there goes a full minute for nothing. That’s simply unacceptable, so I’ve always had to work around it by creating many solutions with the minimum amount of projects necessary, or keeping the dependencies in my head and forcing builds by hand. Alternatively you could throw all the code in a couple of projects instead of breaking it up into many different ones, but that’s like jumping off a cliff to avoid being stung by a bee.

Because Visual Studio mixes the build system with the visual representation of the files, large solutions are not only slow to build, but are a positive pain to work with, making browsing the source code extremely difficult.

The problems with Visual Studio don’t end there. The solution and project files are a pain to generate, they’re full of magical GUIDs and references to the registry, and they’re extremely verbose. The .NET framework offers an API to create those files, but the fact remains that they’re much more complex to create than any of the other build systems.

If you don’t generate the project files by hand, you soon enter configuration hell. Anybody who’s had to make sweeping changes to lots of projects with multiple configurations through the IDE will know what a painful process I’m talking about.

Some of my other complaints are not being able to easily set the build order for different projects (it appears to be determined by the order in which they appear in the solution file), or the fact that setting a dependency between two projects forces an implicit linking.

All in all, Visual Studio seems very well suited to small, toy projects. Projects with just a couple of libraries and a few thousand lines. Anything bigger than that and it becomes an exercise in frustration.

On the bright side, it is possible to use “makefile projects” in Visual Studio, which completely bypasses Visual Studio’s own build system and simply calls an external command to build the project. Visual Studio also offers a command-line line interface, so at least it is possible to do builds from the command line without launching the IDE. Finally, there are some third-party plug-ins that can speed up the dependency checking, which can help with some of the problems (unfortunately, FastSolutionBuild doesn’t seem to have a command-line interface, which renders it useless for my needs).

Other third-party add-ons like Incredibuild claim to speed up full builds, but they do so at the expense of incremental build speed, which I consider much more important.

Make

Make is the granddaddy of all the build systems. With a distinguished history of over 20 years, it has certainly proved its worth in the real world by building hundreds of thousands of projects over the years.

But make is far from perfect; otherwise there would be no need for other build systems. However, things are not as bad as people make them out to be, especially with modern versions of make (GNU make for example). The lack of portability is not an issue for us because we can just trivially write a new makefile, or a variant, for every platform we support.

The claims about make not being scalable are more serious. The article “Recursive Make Considered Harmful” certainly did much harm to make’s reputation. Specifically, that paper claims that it’s very hard to get the order of recursion right because using recursive make has no global project view. I think that’s only true if you’re dealing with self-modifying source code files or autogeneration of source code. If that’s not the case, I can’t see how the order of recursion can matter at all as long as the dependencies are met.

As for the claim that make is slow, let’s put that off until we compare it to the other build systems.

Personally, I really like make. It’s small, clean, and elegant. It does one thing and it does it very well. It’s easy to extend and modify. At first I was surprised that make didn’t do implicit dependency checking of C files (building a C file when an included header file changes), but it fits perfectly with the simplicity of make. It’s just a dependency graph with actions. If you don’t tell it about a rule, it won’t know about it. Fortunately, we can use tools like makedepend to generate the C dependencies with extreme ease.

One of my only gripes with main is the silly tab syntax. The fact that action commands have to be preceded by a tab character is awkward and out of place today, but it’s a small quirk I’m willing to adapt to. Fortunately gnu make’s error messages are very clear and it even asks “Did you forget to put a tab before the command?”

Jam

Jam was born an an improved make. It tried to keep all the good things about make and fix all the problems. And to a large extent, it succeeded.

Like make, Jam is small and easily portable. It deals better with inter-project dependencies by avoiding recursive Jam invocations (while still allowing individual sections to be built separately). The Jam language, even though it’s still fairly restrictive, is more expressive than make and it’s easier to write complex functionality.

One of the main differences from make is that Jam actually provides a fair amount of base functionality in its Jambase file. Jam out of the box knows about some of the most popular development environments and languages (including implicit dependency checking for C/C++ files), so it simplifies the build files for the simple cases. In the other cases, you can add your own rules and actions very easily.

I find it funny that while Jam fixes make’s weird tab requirement, it adds its own “space semicolon” weird command terminator (although I know some programmers who think that “space semicolon” is the one and true way). Either way, it really doesn’t matter since it’s such a small thing.

Jam also has spawned some forks that add extra functionality but are fully backwards compatible: FTJam and BoostJam.

Scons

Scons has been hailed as the next step in the evolution of build systems. It is supposed to be a much improved make-like system, not only written in Python, but using Python as the language used to define the build itself. Python is a fully general, object-oriented language, so it’s extremely expressive. It also has the advantage of being a well-established language with a great set of documentation, debuggers, and tools, which can make creating and debugging complex build scripts easier.

Scons also claims to be extremely accurate when it comes to determining what files need to be built. It doesn’t rely in the time stamp for a file, but it uses the MD5 signature instead (a type of checksum approach). Another very intriguing feature I didn’t get around to testing is the network cache of built object files.

Walking into this test I was a bit afraid of what I might find. I had read some reports of several people having problems with Scons performance on large data sets. However, the latest version (0.96.90), released just a couple of months ago, is supposed to have some performance improvements.

The Method

As a test, I decided to run each of the different build systems on the same codebase. Instead of using some real-world codebase, with its own set of quirks and problems (and the difficulty of easily building it with the different systems), I wrote a script to generate a simple C++ codebase. The code structure is based on what I expect to see in my own projects with many different projects. The physical dependencies in the generated codebase are extremely well contained, and header files never include other header files. Real code bases would have more complicated dependencies and would make the tendencies we see here even more exaggerated.

The specific parameters I used for this test were:

• 50 static libraries
• 100 classes (2 files per class, .h and .cpp) per library
• 15 includes from that library in each class
• 5 includes from other libraries in each class

This is by all accounts still a small or at most medium-sized codebase. A full game engine and tools can easily become much larger than this.

Thinking back, I really should have done the test with at least 100 libraries, not 50, because all my libraries have an extra associated project for unit tests. No big deal. I don’t think it would have changed the results very much. The important thing was to get enough code to make measurements noticeable (if we just build 10 files every build system is going to be really snappy).

For each of the build systems, I measured three operations:

• Full rebuild. Compiling the full source code for the first time. I didn’t expect this time to change much at all from build system to build system or even across platforms. I was quite wrong!
• Incremental build: Doing another build without any changes. This is the really interesting measurement that will tell us a lot about potential for fast iteration.
• Incremental build on a single library: Doing a build of a library without any changes.

I did the measurements in both Linux (2.6 kernel) and Microsoft Windows XP for different systems. Clearly some build systems only run in one platform (Visual Studio). But I decided to run some of the other build systems under Windows as well to provide a more fair comparison.

The specific hardware I ran these tests in is not as important since all we’re comparing are their relative merits. But for the curious it’s a P4 2.8 GHz CPU with hyperthreading, 2GB of fast RAM, and a 7200 rpm EIDE hard drive. The most important part is that I had enough memory to prevent thrashing.

GNU make, Jam, and Scons all support parallel builds. While it won’t speed up incremental builds any, this can reduce the time for full builds dramatically. Since this test was done in a single-CPU machine (and the primary measure was incremental builds), I restricted all the builds to use a single process.

The Results

We can make lots of very interesting observations from this table.

First of all, it confirms what I had seen all along, that Visual Studio is horrible for incremental builds with many projects. My off-the-cuff estimate of one second per project ended up being extremely accurate (54 seconds for 50 projects). That’s simply not acceptable for me.

As I feared, Scons, failed the fast iteration test as well. It actually ended up being slower than Visual Studio in all accounts, even for individual library rebuilds. It might do the “right” thing under all conditions, but frankly, that’s not a price I’m willing to pay to get absolutely correct results. I really don’t think I encounter any situation in everyday work in which Scons would do the right thing and make or Jam wouldn’t.

At this point, I was afraid that I just wasn’t going to be able to get the type of iteration I wanted out of file-based, compiled languages. Fortunately that’s not the case. Both make and Jam do a great job and fall in the range of what I consider acceptable (around a couple of seconds).

There are two interesting observations to be made about full build times from the chart above. First of all, Scons with g++ under Linux is twice as slow as Jam or make for a full rebuild. I find that extremely surprising. Although I guess that’s the extra minute of dependency checking plus some extra overhead of its own. I tried some of the suggestions to get faster Scons builds (by trading off accuracy for speed), but they just improved incremental build times by a couple of seconds. Clearly, Scons needs to do some catching up before it can play with the big boys.

The other one is comparing g++/Linux with VC++/Windows XP. Jam is over twice as slow with VC++/Windows XP than it is with g++ under Linux. Is it Windows XP or is it Visual C++? I don’t know. It would be interesting to try the experiment with g++ or some other compiler under Windows and see if the times are reduced at all. I suspect the Windows file system might have something to do with that.

Conclusion

This little experiment cleared up a lot of doubts for me. I’m ready to ditch Visual Studio as a build system and replace it completely with Jam or make. Make is a simpler but Jam probably edges it out because it’s a bit nicer, it doesn’t have any recursive problems, and the default functionality is pretty handy. It’s hard to go wrong with either one.

Since most programmers still expect to work from within the Visual Studio IDE, you can easily create a “makefile” project type and hook it up to the build system of your choice.

One interesting idea that came up during this research in the Scons mailing list is that of a background process that monitors which files change and updates dependency graphs on the fly. So whenever you initiate a build, all the work has already been done and the build can start right away. A variation on this idea that has been brought up in some TDD mailing lists is that of the build system not just computing dependencies in the background, but actually attempting to compile the code and run the unit tests in the background. If any of the tests fail, they can even be highlighted in the source code editor. Sort of like an on-the-fly, smart code checker on steroids.

Of course, we could also choose a language that has much smaller build times. I haven’t worked on a large-scale C# project yet, but the small tools I’ve created have impressed me with how fast the iteration can be. The same can be said for scripting languages such as Python or Lua. Unfortunately, we’re stuck with C++ for the foreseeable future in game development, so we better learn to deal with it the best we can.

For now, I’ll be happy to stick with Jam and two-second incremental builds. Let’s start jamming!

generate_libs.py