How Incredible Is Incredibuild?

There is no doubt that Xoreax’s Incredibuild will speed up most full builds of C++ projects using Microsoft Visual C++ to varying degrees. I’m not going to argue that. But is using Incredibuild in your project really a good idea?

Chances are that if you’re reading this, you already know what Xoreax’s Incredibuild is. If not, here’s the quick summary: Incredibuild is a distributed build system for Microsoft Visual C++ that recruits other machines in the network to parallelize your build; it claims build time reductions of up to 90%. There is no doubt that Incredibuild will speed up most full builds of C++ projects using Microsoft Visual C++ to varying degrees. I’m not going to argue that. But is using Incredibuild in your project really a good idea?

Let’s start with the positives. Incredibuild does work as advertised and can produce significant speed ups from distributing your build. It is well polished and it has clearly been tested in real-world application development. It has good load balancing, you won’t notice your machine being used as a slave, it has the concept of backup coordinators, it minimizes network usage, etc, etc. It does have a few technical glitches, but I’ll get to those later. All in all, it’s pretty robust and well-thought-out product, and there are clearly lots of people happy with it (interestingly, the list includes many game developers).

So, what’s not to like? When I started using Incredibuild last year, I was initially floored by the results. A project that would take 45 minutes to compile, would only take about 5-8 minutes with Incredibuild. Wow! That’s what I call a “significant” speedup!

However, the more I used it, the more it started getting in the way and the more it made me think about all the areas where it was lacking. Be warned that this doesn’t pretend to be a rigorous review with objective data to back up what I’m saying. These are the issues I’ve encountered using Incredibuild without trying to become an expert at it. It’s very likely that are ways around some of the things I’m going to complain about. If so, feel free to let me know in the comments below. Maybe I’ll change my mind and I’ll become an Incredibuild fan after all.

Iteration time

tools Slow incremental builds. I don’t mean incremental linking (that comes next), I mean non-full builds, like the case where I change a single line in a cpp file and want to generate a new executable. I found that using Incredibuild for such builds is significantly slower than the regular build in MSVC (incidentally, this FAQ claims exactly the opposite). I like to compile my code after every single change. Actually, I like to compile it and run all the tests after every single change. I’ll often even do it while I’m thinking. It’s a (good) habit you get in after you’ve been doing test-driven development for a while. Anything that slows down that cycle is a major impediment. So what if I do incremental builds with the regular MSVC build and just use Incredibuild for full rebuilds? Read on…

Problems mixing and matching object files. Mixing and matching object files created by Incredibuild and MSVC seems to work… sometimes. I know in their FAQ they indicate a few changes you need to do to the debug information settings to prevent some errors. I’m talking about bigger issues. As in, the executable generated is total garbage and crashes right away. Usually doing a full rebuild with Incredibuild fixes it (sit and wait 5-8 minutes). When this is caused just by changing a single cpp file, it gets old very quickly and you start questioning the benefits you’re getting in the first place.

I found that I have similar problems matching object files with incremental linking turned on (this time in the form of errors during the build). The only way I’ve managed to more or less reliably mix both types of object files is by turning off incremental linking completely. Of course, link times are often the bottleneck of the change-build-test cycle, being as high as a full minute or more when you’re dealing with the whole game. Incremental linking, as long as you’re only modifying the executable project, is really a god-send and can reduce a 1+ minute link time to just a couple of seconds. Again, turning it off to be able to use Incredibuild seems a step backwards.

No edit and continue. I admit it, I’ve never been a fan of edit and continue. But some people swear by it, and it’s yet another feature you can’t have turned on with Incredibuild. Interestingly, their FAQ fails to mention the words “incremental linking” or “edit and continue” anywhere. (Actually, they had it under “known issues”).

Technical issues

Tied to Microsoft Visual Studio. Are you developing for other platforms than Windows or Xbox? Are you using other compilers than Visual C++? Then you’re out of luck. You’ll have to come up with a totally different system to build your code for those platforms. You can’t use the same build back end because Incredibuild is totally tied to MSVC. I don’t know exactly why, I just hope they had a very good technical reason and it wasn’t to get prettier integration with colorful progress graphics. At least it is possible to run Incredibuild from the command line (still using MSVC of course), which is essential for build machines. Update: msew just pointed out that version 2.20 has beta support for the Inteal compiler. Good to hear that.

Uses Microsoft Visual Studio’s build system. If I could rip one thing from Visual Studio, it would be the completely broken build system. I don’ t know what they were thinking when they built it, and I have a very hard time believing anybody at Microsoft uses it for complex projects. It seems a great quick way to get toy projects off the ground, but it doesn’t scale well at all to large projects (very much like those GUI wizards, but people seem to put up with them anyway). In a real build system I want to have multiple targets (and combinations of those targets), I want to be able to enforce build order, I want to be able to choose not to link a project just because I make it dependent on another project, I want to be able to have over a hundred different projects in an environment and not bog down because of it, I want to be able to view and change build settings easily across projects. In other words, I want primarily something like make, Jam, or Scons. Fortunately, MSVC allows you to have makefile projects, but Incredibuild doesn’t support them. This is particularly important to me because I would like to use a common back end for all the different platforms, and I’d love to have a single build file that I can use to build any project or platform combination.

Failing to build correctly. Every so often, a build with Incredibuild will completely fail because some dependency check failed along the way (this seems to happen particularly when changing header files outside of the project itself, like middleware or other external APIs). To be fair, this seems to happen just as frequently in MSVC with regular builds using pre-compiled headers, so you probably can’t blame Incredibuild for it. It certainly doesn’t fix the problem though.

Builds are done differently than local ones. I haven’t investigated this one thoroughly, but Incredibuild uses slightly different rules than regular, local builds. As a result, it is possible to set up a build with custom rules that builds correctly in Incredibuild, but fails as a regular build. If you’re trying to mix and match the two, that can’t be good news. I think the source of the problem is that Incredibuild runs custom build rules locally as the first step of the build, and then distributes the rest. Regular local builds will call the custom build rule when they get to the file in question, so you can see how there’s a slight potential for problems there (this, incidentally, is not idle nit picking, but happened in a real project).

Performance

Network issues can affect performance significantly. For a while, Incredibuild kept having all sorts of problems communicating across the network and was causing builds to either take forever or never complete (some slave machine would just never return a result or timeout). I don’t know if that was caused by glitches in our network, too many people using it, or bugs in Incredibuild, but it was highly annoying. I suppose you’re bound to have that happen in any distributed system, but I would expect them to always be able to fall back to local compilation and not take any longer than a non-distributed build.

How much faster than a regular full build? This is the big question. I have seen with my own eyes that using Incredibuild on a project without physical insulation and carefully managed dependencies, and without precompiled headers, can make the build be many, many times faster. But how does it compare to a project that has been carefully managed, with well-thought-out precompiled headers? As an experiment, I retrofitted a large codebase to use precompiled headers. The awful physical dependencies were still there, and I’m sure the precompiled headers were less than ideal. Still, build times went down from 45 minutes to something like 10 minutes. Incredibuild took 5-8 minutes in that same codebase. Throw a faster machine with multiple cores (the future of PCs), and a decent build system that can use multiple threads (not MSVC, go figure), and I think you can get similar build times without any of the drawbacks.

Architectural issues

Here is one argument that cuts both ways.

Programmers will often try to avoid making changes that cause full rebuilds (to avoid spending 30+ minutes waiting for a result), but they end up doing it by hacking things in the wrong place: duplicating code, declaring externs by hand on the wrong file, etc. Ideally, you shouldn’t have many files that can cause a full rebuild (if you do, you have a big problem), but if you need to make a change there, you really should do it. Incredibuild can ease the pain of a full rebuild and prevent the hack, leading to a cleaner codebase and architecture.

On the other hand, having Incredibuild around can cause developers to throw caution to the wind and think they don’t have to worry about physical dependencies anymore. As a result, they end up with the blob antipattern. Ironically, after a few months of that, they’ll probably end up with really slow build times again (because almost every file causes a full rebuild).

Is the time spent in managing and minimizing physical dependencies worth it? Absolutely! Low physical dependencies also means low logical dependencies. Even if build times were instant, you would still be well advised to pay very close attention to dependencies and minimize them (not just for reuse or maintenance, but even during the development of the game, since it makes it much easier to refactor or change any part of your program).

Alternatives

What are the alternatives to Incredibuild then? Are we stuck with really slow builds?

  • Manage your physical dependencies. Be as careful as you can about them, and you’ll be way on your way towards minimizing build times. Use the pimpl idiom in any header that is used extensively throughout the codebase.
  • Use precompiled headers. Stay away from that option of “automatic precompiled headers” in MSVC because it does more harm than good. Use Bruce Dawson’s paper as a starting point about precompiled headers. Also, precompiled headers are supported across the board for most popular C++ compilers: MSVC, gcc, Codewarrior, etc, so you can get a lot of benefits from a single approach.
  • Get a fast dual-core machine and use multiple threads to build. You can’t do that with MSVC, but man, Jam and Scons can certainly do it. I stress the fast CPU part. In the past, I ran some tests, and, assuming you have enough memory, compilation time was not limited by disk space but by CPU speed. Once you start using multiple threads it’ll put more stress on the disk so it’ll probably be a good idea to get fast hard drives as well.
  • distcc. I admit I’ve only tested distcc in toy sample projects, but it’s exactly what I would like on paper: a fairly generic distributed build system. It comes with gcc support out of the “box,” but it can be extended to support other compilers. It uses a very clean way of distributing the builds, by running the preprocessor locally and sending the output file to be compiled into an object file. I really don’t know how it scales for a real-world project, but I’d like to learn more about it.
  • Scons‘ network cache. Scons can transparently cache object files on the network, so if you ever try to build a file that is already in the cache, you’ll get it instantly. This can be a reasonable solution to avoiding full rebuilds after one person (or the build machine) has done it once.

Conclusion

If you’re stuck with some legacy code that takes forever to compile, don’t hesitate to get Incredibuild. It’ll do wonders for your project and will cut down your build times by a huge amount. If you’re developing on a new code base, or your build times are short already, I would recommend against using it. For the project I’m starting now, we’re starting without Incredibuild (even though we already have the licenses, so it’s not a money issue) and we’re carefully managing physical dependencies and using precompiled headers. If at some point it grows very large and we see benefits from Incredibuild, we might switch, but until then, we’re a lot better off without it.

  • msew

    >Interestingly, their FAQ fails to mention the words “incremental linking” or “edit and continue” anywhere

    for the ‘edit and continue’ look under known issues:

    http://www.xoreax.com/support_knownissues.htm#ki02

  • msew

    heh more nitpicking of details: (feel free to delete them if you update the text accordingly :-) )

    >Are you using other compilers than Visual C++? Then you’re out of luck. You’ll have to come up with a totally different system to build your code for those platforms.

    2.20 has beta support for intel compiler. Not Codewarrior or other console compilers but least one more :-)

    >Throw a faster machine with multiple cores (the future of PCs), and a decent build system that can use multiple threads (not MSVC, go figure)

    VS 8.0 uses multiple threads :-)

  • msew

    Do you all have the multiple CPU license for your agents?

    That allows the incredibuild to use both of your logical / physical processors for linking. So with incredibuild your linking times should actually be faster if you have two procs.

    Overall I agree with the article. Sometimes incredibuild fails and you need to recompile. sub 10 minutes later off you go!

    And I completely agree with the position that: incredibuild hides compile issues due to poor physical dependencies. There is nothing like PUNISHING a developer with long compile times to make them look at what badness they have wrought. And compiling after every change is a good practice as you can watch the list of files being compiled in the output window and go: “Why on earth is FooBarBaz.cpp being recompiled for!!!”

  • http://www.gamesfromwithin.com Noel Llopis

    Good to hear VS 8.0 uses multiple threads. That will help a lot. I really should install the beta as soon as I get a chance at work.

    We do have the multiple CPU licenses, but I didn’t know Incredibuild actually helped any with link times. It never ocurred to me that linking would be an easily parallelizable operation. But now that I think about it, all it’s doing is going through the object files and updating a bunch of function addresses based on a table, so there’s no reason why it can’t be distributed on multiple processors. Neat! Does VS 8.0 use multiple cores for linking as well?

  • Glenn

    Yeah with the Visual Studio 2005 they ripped out the old build system and completely re-wrote it so now you got MS Build which parses like XML descriptors and stuff. So I wouldn’t be surprised if it handles multiple cores

  • Tod

    Yeah, your experience with Incredibuild is pretty much that same as mine. I work for a game company and we did the 30-day trial with our 12 developers. After a week nobody was using it beause the incremental build was so so slow. Switching back and forth between MSVC and Incredibuild never really works. We didn’t end up buying it because even though you want it to be useful, it really is a crappy useless product.

  • http://www.gamearchitect.net Kyle Wilson

    “Yeah with the Visual Studio 2005 they ripped out the old build system and completely re-wrote it so now you got MS Build which parses like XML descriptors and stuff. So I wouldn’t be surprised if it handles multiple cores.”

    It doesn’t. Or at least it doesn’t unless you write your own multiprocessor-aware C# build script. MS helpfully writes that “there is nothing to stop a custom task using multiple processors to do work.”

    http://msdn.microsoft.com/chats/transcripts/vstudio/vstudio_072804b.aspx

    Also, don’t expect MSBuild to handle C++. Or at least without some grief on your part. Alex Kipman, the MSBuild Program Manager, writes that “in VS 2005 (Whidbey) C++ won’t be supported natively.” There’s supposed to be some support for interoperability with C++ projects by the time the whole thing actually ships. Read his post at the bottom here:

    http://www.winfx247.com/247reference/msgs/0/1868.aspx

    Visual Studio seems to be becoming more and more a solution for lightweight web applet development as opposed to large-scale application programming.

  • plunket

    Incredibuild fixes problems with DevStudio, that much is true.

    I’ve never had interop problems between default DevStudio compile/link and IB compile/link. In fact, I used them pretty interchangeably when i was doing PC development. Losing Edit&Continue was a small price to pay in general, but what really killed the whole thing for me was the enormous link times. Incredibuild could do a full rebuild of this project in 6-8 minutes, plus 2 for the link. The same codebase built with multiple Jam processes on a single workstation was just as long, but with a much shorter link.

    The key to sane development of course is managing dependencies. Someone actually needs to care about it from the beginning, and keep them managed throughout the project. IMO you shouldn’t really need Incredibuild at all, unless you’re doing a lot of full rebuilds for some reason (why, though?). For the incremental style of development that both Noel and I practice, though, fast linking is truly far more important than anything else.

    Oh yeah, Visual Studio is a terrible code editor also. I’m not surprised that they’re rolling it into something targeting smaller applications… ;)

  • TomW

    Incremental linking is definetly more important than fast “full rebuilds”. Our team was using Incredibuild for a while, but I’ve consistantly shown that IB is slower for us. Personally, I use Fast Solution Build which seems to do a better job at incremental linking that MSVC

    http://www.workspacewhiz.com/OtherAddins.html#FastSolutionBuild

  • MayaP

    Latest IncrediBuild version (v2.30) actually includes a Fast Solution Build-style incremental link mode. Which means it speeds up both compilation and incremental link times. It’s not perfect, but personally I would be totally lost without it.