Games from Within

I also agree with CMake. I haven’t had to use it for any sort of resource handling, however it works very well for building complex projects, and the IDE independence is really nice if your working on multiple platforms, or if you decide you don’t want to use an IDE for the day .

From my experience, kjam is most suited for gamedev building tasks. It’s much faster than make (GNU make) and original jam.

For my current project at work (8 libs of 10..50 files + app of 200 files + >5000 resource files) it takes ~2..3 secs to check all the dependencies (if I hit build button and it’s nothing to do actually).

The main and only problem with kjam is its unclear future — last update was about year ago.

As a long time SCons user and promotor, I would like to:
1) give some explanation on why SCons is slow
2) comment on the goals of your quest for a good build system
3) give some suggestions on candidates to investigate

1) True, the null build is slow and true, the file system speed
cannot explain it all. One root cause is low speed of computing
signatures which involves a lot of string interpolation.
For example, the command lines (among others) are part of
the signature of the built files and the command lines are made
up with Perl-ish substitutions, like the C compilation command line
is the result of “$CC $CCFLAGS -c $TARGET ….” expansion.
That is both good and bad. The bad part, you already mentioned it:
it is slow. Moving character strings in memory is generally seen
as fast but when you have many, many such operations the time adds up
to something large in the end.
The good part is two fold. First, the command line part of the signature
makes SCons one of the safest build systems around. Jam and brothers
cannot come anywhere close (without a lot of glue or other external help).
Second, that Perlish way to describe command lines is considered
friendly by many. Of course, the casual builder couldn’t care less
but they are not the only stakeholders of a software build. Some people
are prepared to spend a more time building for less time writing/developing
the build description.

2) Faster is less safe and safer is less speedy. That’s so fundamental that
your desire to get to a eye blink null build will not take you anywhere.
It easy to get a system that checks less and is faster (say, gnu make, non-recursive)
or a system that checks a lot and is slow (SCons is the best example there).
A smarter system would allow you to decide per run and per target
how to compromise between speed and reliability. SCons allows that
but to an extent far from getting rid of the sloweness reputation.
To my knowledge, there is no opne source product to be at the same time
safe, fast and extension-friendly.

3) In your case, you have quite some build steps that are not
code compilation but data compilation. That means you have
(or need to write) your own build steps. Any advanced feature
related to C/C++ build is not very helpful in that case.
A lot of the tools that have been recommended to you are
de facto C/C++ build tools (me, I was a speech scientist busy
with statistical modeling, data compilation by excellence).
So my advice is two fold. First, try waf (will be faster than SCons),
try rake (may be slower, not sure). Second, be prepared to develop
the build (Most SCons enterprise users, have their own
wrapper for SCons). Since safe+fast+extension-friendly is not
yet readily available as FOSS, you will have to spend
some development time to get close. Or wait a bit more for your builds.
Up to you to make a balance.

Feedback welcome,
Adrian.

After emailing, it looks like KJam is still being supported. I heard from the developer this morning. He just hasn’t updated his website.

Dude, MSBuild blows scons out of the water for runtime, and it’s every bit as flexible. We have our entire asset pipeline built around it, and the _deep_ asset dependency scanning for our project is on the order of ~100ms total, and that number has been more or less stable as we’ve added more assets. We build a ‘level’ target, and we have custom tasks, scanners, and emitters (to use SCons parlance) that crack the level file, figure out which game entities were placed in it, cracks the game entities to figure out which models they use, cracks the models to figure out which textures to use, and adds them all to the build set. It has great diagnostics and does correct dependency scanning orders of magnitude more quickly than SCons. MSBuild is emissive, programmable, does everything correctly and quickly, and is as far as i can tell equal or superior to scons in every way….

except parallel builds. MSBuild doesn’t go wide easily at all, and it requires a 90-degree rotation of your data dependency expressions, which is its fatal flaw. SCons does this very well, and I’m pretty disappointed in the MSBuild team’s decision to go parallel on child MSBuild tasks instead of integrating concurrency natively and deeply with the known build dependency graph.

You’re so SCons-centric that you’ve accepted its dichotomy of speed vs. correctness. While that dichotomy may be true in SCons, it’s not true globally and that’s why we’re up in arms to find something that doesn’t suck. Jam and MSBuild are _significantly_ faster than SCons during the dependency analysis phase of identical projects. This isn’t because they’re incorrect, it’s because THEY DON’T SUCK. I agree that SCons is correct, and good for it. I need something correct _and_ fast, though, something that doesn’t take 30 seconds to tell me that there’s nothing to do.

Why should such dependencies be implicit and why should it be a problem?

We actually do have dependencies like .dae -> .mesh -> .pak (the same for animations, textures and lots of other stuff). We do NOT have problems with it — if some .DAEs are changed then build-system re-build mesh, re-pack package and re-build game .ISO (if I need it). The process is quite fast.

There is ony one situation in build-process that can cause real problem : one binary depends on second binary and you cannot detect such dependency without reading _content_ of this binary — source files can be easily scanned for ‘#includes’ (and some build-systems do it well), but binaries are harder to deal with.

Charles — can you expound a little on what you mean by “90-degree rotation of your data dependency expressions”? I’m not quite sure what you mean by that. As I stated earlier, I’m also trying to find some build system (mainly for data, but one that supports xplatform code and data would be the best). Msbuild is on the list, but I have my own cons on that particular app. Was curious about your comments about the data building aspect.

thanks.

Could you make the generator for your test code available, or email it to me? I’m guessing it’s C or C++, so the number of preprocessor defines is important too.

“50 libraries, 100 classes each, 15 internal includes, 5 external includes). I built everything once, then ran it again and it took 49 seconds.”

I’ve done a fair amount of work recently analyzing no-change (aka null, do nothing etc) build times with SCons.

I *HATE* (capital H) scons.

My current project and my last project used it. The #1 issue I have with it is no one seems to actually understand it and so because it’s just a python library and you have all of python with which to hang yourself every sub project ends up being some personal hack of the guy who setup the build for that sub project. Back in make days, editing or adding something to a makefile was always relatively easy. Writing my own build system in perl or python has been relatively easy but everytime I see someone go try to add something to scons it’s 4-8 hours of pain.

One thing that many build systems seem to have problems with is dependencies that can’t be known before the build starts. An example is you don’t know what textures need to be processed until you build the maya/max file. So what you really want is

.mb->.middleformat->.texturelist->[texture1, texture2, ...]

When .texturelist already exists and is up to date (ie, newer than both .middleformat and .mb) it’s easy for some build system to follow the rules and build the textures but most build systems I’ve used don’t like it when that list doesn’t exist yet and therefore has to be added after the build has started. Or to put it another way, most build systems start with only some programming language in mind and make the assumuption they can scan the text quickly to figure out implicit dependencies. That assumption is not true for assets.

My team recently migrated from the Autotools suite (*that’s* pain) to SCons. Our codebase isn’t that large (~175 files code, more for other stuff (Doxygen etc.)), but we found Scons was, while slower than ‘make’, easily fast enough for the ease of use difference.

I just has a quick look at your test script, and I notice it’s not using any of SCons’ speed features. Reading your problem domain, one in particular might be useful. SCons uses objects called ‘Deciders’ to figure out if a file/dependency has changed. The default Decider uses MD5 sums of the file contents, which adds a *lot* of time overhead. The reason this is the default behaviour is explained here: http://www.scons.org/doc/production/HTML/scons-user.html#AEN815, but the important point is that it can be easily changed (using one of several built-ins or writing your own). There’s one that will use the later-than-mtime instead, like make, (flaw: doesn’t catch introduction of older files into the file tree), and another that’ll use mtime-exact (any different mtime causes a rebuild), and several more. There are trade-offs involved with these, but we found that mtime-exact took 66% of the time of md5s, which is not as fast as others but takes a lot of the sting out.