How Incredible Is Incredibuild?

There is no doubt that Xoreax’s Incredibuild will speed up most full builds of C++ projects using Microsoft Visual C++ to varying degrees. I’m not going to argue that. But is using Incredibuild in your project really a good idea?

Chances are that if you’re reading this, you already know what Xoreax’s Incredibuild is. If not, here’s the quick summary: Incredibuild is a distributed build system for Microsoft Visual C++ that recruits other machines in the network to parallelize your build; it claims build time reductions of up to 90%. There is no doubt that Incredibuild will speed up most full builds of C++ projects using Microsoft Visual C++ to varying degrees. I’m not going to argue that. But is using Incredibuild in your project really a good idea?

Let’s start with the positives. Incredibuild does work as advertised and can produce significant speed ups from distributing your build. It is well polished and it has clearly been tested in real-world application development. It has good load balancing, you won’t notice your machine being used as a slave, it has the concept of backup coordinators, it minimizes network usage, etc, etc. It does have a few technical glitches, but I’ll get to those later. All in all, it’s pretty robust and well-thought-out product, and there are clearly lots of people happy with it (interestingly, the list includes many game developers).

So, what’s not to like? When I started using Incredibuild last year, I was initially floored by the results. A project that would take 45 minutes to compile, would only take about 5-8 minutes with Incredibuild. Wow! That’s what I call a “significant” speedup!

However, the more I used it, the more it started getting in the way and the more it made me think about all the areas where it was lacking. Be warned that this doesn’t pretend to be a rigorous review with objective data to back up what I’m saying. These are the issues I’ve encountered using Incredibuild without trying to become an expert at it. It’s very likely that are ways around some of the things I’m going to complain about. If so, feel free to let me know in the comments below. Maybe I’ll change my mind and I’ll become an Incredibuild fan after all.

Iteration time

Slow incremental builds. I don’t mean incremental linking (that comes next), I mean non-full builds, like the case where I change a single line in a cpp file and want to generate a new executable. I found that using Incredibuild for such builds is significantly slower than the regular build in MSVC (incidentally, this FAQ claims exactly the opposite). I like to compile my code after every single change. Actually, I like to compile it and run all the tests after every single change. I’ll often even do it while I’m thinking. It’s a (good) habit you get in after you’ve been doing test-driven development for a while. Anything that slows down that cycle is a major impediment. So what if I do incremental builds with the regular MSVC build and just use Incredibuild for full rebuilds? Read on…

Problems mixing and matching object files. Mixing and matching object files created by Incredibuild and MSVC seems to work… sometimes. I know in their FAQ they indicate a few changes you need to do to the debug information settings to prevent some errors. I’m talking about bigger issues. As in, the executable generated is total garbage and crashes right away. Usually doing a full rebuild with Incredibuild fixes it (sit and wait 5-8 minutes). When this is caused just by changing a single cpp file, it gets old very quickly and you start questioning the benefits you’re getting in the first place.

I found that I have similar problems matching object files with incremental linking turned on (this time in the form of errors during the build). The only way I’ve managed to more or less reliably mix both types of object files is by turning off incremental linking completely. Of course, link times are often the bottleneck of the change-build-test cycle, being as high as a full minute or more when you’re dealing with the whole game. Incremental linking, as long as you’re only modifying the executable project, is really a god-send and can reduce a 1+ minute link time to just a couple of seconds. Again, turning it off to be able to use Incredibuild seems a step backwards.

No edit and continue. I admit it, I’ve never been a fan of edit and continue. But some people swear by it, and it’s yet another feature you can’t have turned on with Incredibuild. Interestingly, their FAQ fails to mention the words “incremental linking” or “edit and continue” anywhere. (Actually, they had it under “known issues”).

Technical issues

Tied to Microsoft Visual Studio. Are you developing for other platforms than Windows or Xbox? Are you using other compilers than Visual C++? Then you’re out of luck. You’ll have to come up with a totally different system to build your code for those platforms. You can’t use the same build back end because Incredibuild is totally tied to MSVC. I don’t know exactly why, I just hope they had a very good technical reason and it wasn’t to get prettier integration with colorful progress graphics. At least it is possible to run Incredibuild from the command line (still using MSVC of course), which is essential for build machines. Update: msew just pointed out that version 2.20 has beta support for the Inteal compiler. Good to hear that.

Uses Microsoft Visual Studio’s build system. If I could rip one thing from Visual Studio, it would be the completely broken build system. I don’ t know what they were thinking when they built it, and I have a very hard time believing anybody at Microsoft uses it for complex projects. It seems a great quick way to get toy projects off the ground, but it doesn’t scale well at all to large projects (very much like those GUI wizards, but people seem to put up with them anyway). In a real build system I want to have multiple targets (and combinations of those targets), I want to be able to enforce build order, I want to be able to choose not to link a project just because I make it dependent on another project, I want to be able to have over a hundred different projects in an environment and not bog down because of it, I want to be able to view and change build settings easily across projects. In other words, I want primarily something like make, Jam, or Scons. Fortunately, MSVC allows you to have makefile projects, but Incredibuild doesn’t support them. This is particularly important to me because I would like to use a common back end for all the different platforms, and I’d love to have a single build file that I can use to build any project or platform combination.

Failing to build correctly. Every so often, a build with Incredibuild will completely fail because some dependency check failed along the way (this seems to happen particularly when changing header files outside of the project itself, like middleware or other external APIs). To be fair, this seems to happen just as frequently in MSVC with regular builds using pre-compiled headers, so you probably can’t blame Incredibuild for it. It certainly doesn’t fix the problem though.

Builds are done differently than local ones. I haven’t investigated this one thoroughly, but Incredibuild uses slightly different rules than regular, local builds. As a result, it is possible to set up a build with custom rules that builds correctly in Incredibuild, but fails as a regular build. If you’re trying to mix and match the two, that can’t be good news. I think the source of the problem is that Incredibuild runs custom build rules locally as the first step of the build, and then distributes the rest. Regular local builds will call the custom build rule when they get to the file in question, so you can see how there’s a slight potential for problems there (this, incidentally, is not idle nit picking, but happened in a real project).

Performance

Network issues can affect performance significantly. For a while, Incredibuild kept having all sorts of problems communicating across the network and was causing builds to either take forever or never complete (some slave machine would just never return a result or timeout). I don’t know if that was caused by glitches in our network, too many people using it, or bugs in Incredibuild, but it was highly annoying. I suppose you’re bound to have that happen in any distributed system, but I would expect them to always be able to fall back to local compilation and not take any longer than a non-distributed build.

How much faster than a regular full build? This is the big question. I have seen with my own eyes that using Incredibuild on a project without physical insulation and carefully managed dependencies, and without precompiled headers, can make the build be many, many times faster. But how does it compare to a project that has been carefully managed, with well-thought-out precompiled headers? As an experiment, I retrofitted a large codebase to use precompiled headers. The awful physical dependencies were still there, and I’m sure the precompiled headers were less than ideal. Still, build times went down from 45 minutes to something like 10 minutes. Incredibuild took 5-8 minutes in that same codebase. Throw a faster machine with multiple cores (the future of PCs), and a decent build system that can use multiple threads (not MSVC, go figure), and I think you can get similar build times without any of the drawbacks.

Architectural issues

Here is one argument that cuts both ways.

Programmers will often try to avoid making changes that cause full rebuilds (to avoid spending 30+ minutes waiting for a result), but they end up doing it by hacking things in the wrong place: duplicating code, declaring externs by hand on the wrong file, etc. Ideally, you shouldn’t have many files that can cause a full rebuild (if you do, you have a big problem), but if you need to make a change there, you really should do it. Incredibuild can ease the pain of a full rebuild and prevent the hack, leading to a cleaner codebase and architecture.

On the other hand, having Incredibuild around can cause developers to throw caution to the wind and think they don’t have to worry about physical dependencies anymore. As a result, they end up with the blob antipattern. Ironically, after a few months of that, they’ll probably end up with really slow build times again (because almost every file causes a full rebuild).

Is the time spent in managing and minimizing physical dependencies worth it? Absolutely! Low physical dependencies also means low logical dependencies. Even if build times were instant, you would still be well advised to pay very close attention to dependencies and minimize them (not just for reuse or maintenance, but even during the development of the game, since it makes it much easier to refactor or change any part of your program).

Alternatives

What are the alternatives to Incredibuild then? Are we stuck with really slow builds?

• Manage your physical dependencies. Be as careful as you can about them, and you’ll be way on your way towards minimizing build times. Use the pimpl idiom in any header that is used extensively throughout the codebase.
• Use precompiled headers. Stay away from that option of “automatic precompiled headers” in MSVC because it does more harm than good. Use Bruce Dawson’s paper as a starting point about precompiled headers. Also, precompiled headers are supported across the board for most popular C++ compilers: MSVC, gcc, Codewarrior, etc, so you can get a lot of benefits from a single approach.
• Get a fast dual-core machine and use multiple threads to build. You can’t do that with MSVC, but man, Jam and Scons can certainly do it. I stress the fast CPU part. In the past, I ran some tests, and, assuming you have enough memory, compilation time was not limited by disk space but by CPU speed. Once you start using multiple threads it’ll put more stress on the disk so it’ll probably be a good idea to get fast hard drives as well.
• distcc. I admit I’ve only tested distcc in toy sample projects, but it’s exactly what I would like on paper: a fairly generic distributed build system. It comes with gcc support out of the “box,” but it can be extended to support other compilers. It uses a very clean way of distributing the builds, by running the preprocessor locally and sending the output file to be compiled into an object file. I really don’t know how it scales for a real-world project, but I’d like to learn more about it.
• Scons‘ network cache. Scons can transparently cache object files on the network, so if you ever try to build a file that is already in the cache, you’ll get it instantly. This can be a reasonable solution to avoiding full rebuilds after one person (or the build machine) has done it once.

Conclusion

If you’re stuck with some legacy code that takes forever to compile, don’t hesitate to get Incredibuild. It’ll do wonders for your project and will cut down your build times by a huge amount. If you’re developing on a new code base, or your build times are short already, I would recommend against using it. For the project I’m starting now, we’re starting without Incredibuild (even though we already have the licenses, so it’s not a money issue) and we’re carefully managing physical dependencies and using precompiled headers. If at some point it grows very large and we see benefits from Incredibuild, we might switch, but until then, we’re a lot better off without it.