Bad News for Scons Fans
We have been talking a lot about Scons recently at the Power of Two Games World Headquarters. MSBuild has proven to be quite a pain to work with for our asset builds and eventually left us dissatisfied (that’s material for a whole other entry). So we kept looking over to Scons as a possible solution.
On paper it looks great:
- Built from the ground up for parallel builds
- Uses a real programming language (Python) instead of overly-complicated XML files
- It allows for discovery of assets to build as other assets are parsed
The big question mark hanging over it was whether it was fast enough. I had done some experiments with Scons and other build systems a few years ago, and Scons totally failed by having ridiculously long incremental build times just checking if anything had changed. That was simply not acceptable. If no data has changed, I want the build system to detect it and come back in a second or less.
Since then, I’ve been told that Scons had finally fixed its dependency checking and it was much faster now. Music to my ears!
So I resurrected the old script I wrote back then to generate a nightmare stress test, downloaded the latest stable version of Scons (1.0.1) and fired it up.
I’m sorry to report that Scons is still as unusable as it was back then. I generated a codebase like the one I used to measure all the build systems (50 libraries, 100 classes each, 15 internal includes, 5 external includes). I built everything once, then ran it again and it took 49 seconds. I have some pretty nice hardware, including a fast SATA hard drive and a 2.8GHz Core2Duo with plenty of RAM, and 49 seconds to say that nothing needs to be changed just doesn’t cut it.
Now to be fair, right now we’re looking at Scons to build our assets, not our code. Maybe the nested project configuration is not particularly realistic, so I tried something simpler: 5000 files in a single library without any dependencies among each other at all. This is a very conservative example for an asset build of a large commercial game.
- It’s only 5000 files. A real game can easily have 10 times as many asset files.
- All the files are tiny (cpp files in this case). Assets can get very large when dealing with textures, sounds, and animations.
- There are zero dependencies among assets in this example. You’ll often have models depending on shaders and textures, levels on game entity definitions, etc.
Building assets doesn’t have to be the instant, no delay kind of build that I require building code. I expect most developers will be building assets locally, tweaking things, rebuilding, and trying them in the game. The faster the build is, the better, but a delay of a few seconds can be acceptable. If Scons can’t handle this build in less than 10 seconds, it’s certainly doomed in a full-scale game.
The result: 37 seconds!
For purely masochistic reasons, I decided to see how it scales, so I threw 20,000 files at it (again, without any dependencies). Incremental build time without any changes: 3 minutes and 46 seconds. That’s over 6 times longer for processing 4 times the number of files, so it doesn’t even scale linearly.
Scons is definitely too slow, and MSBuild has its share of problems. What’s left out there? Any recommendations for good parallel asset building systems? Hopefully something lightweight and easily configurable through a real language. Anything?
Related posts:
.
barog
Cam
Tom Plunket
Incredibuild XGE seems pretty good to me:
http://www.xoreax.com/technology_xge.htm
I haven’t got around to playing with these tools yet, but perhaps you might find some time if your get desperate enough?!
http://sham.sourceforge.net/
http://www.cs.berkeley.edu/~billm/memoize.html
They basically work off of the same idea: run the commands to do the build, monitoring the inputs and outputs with kernel hooks to generate dependencies information. No need to specify or calculate the dependencies explicitly via some heuristic hack.
I imagine it would be pretty easy to implement a parallel build system on top of these. You could simply write your build instructions sequentially in a list with certain barrier points. All commands between barrier N and N+1 could be run in parallel. e.g.
compile blah/foo.cpp blah/out/foo.obj
compile blah/bar.cpp blah/out/bar.obj
compile blah/baz.cpp blah/out/baz.obj
compile guff/foo.cpp guff/out/foo.obj
compile guff/bar.cpp guff/out/bar.obj
compile guff/baz.cpp guff/out/baz.obj
——————————————- # barrier
make-library guff/out/*.obj libs/guff.lib
make-library blah/out/*.obj libs/blah.lib
——————————————- # barrier
# etc …
Could you make the generator for your test code available, or email it to me? I’m guessing it’s C or C++, so the number of preprocessor defines is important too.
“50 libraries, 100 classes each, 15 internal includes, 5 external includes). I built everything once, then ran it again and it took 49 seconds.”
I’ve done a fair amount of work recently analyzing no-change (aka null, do nothing etc) build times with SCons.
Ping?
I *HATE* (capital H) scons.
My current project and my last project used it. The #1 issue I have with it is no one seems to actually understand it and so because it’s just a python library and you have all of python with which to hang yourself every sub project ends up being some personal hack of the guy who setup the build for that sub project. Back in make days, editing or adding something to a makefile was always relatively easy. Writing my own build system in perl or python has been relatively easy but everytime I see someone go try to add something to scons it’s 4-8 hours of pain.
One thing that many build systems seem to have problems with is dependencies that can’t be known before the build starts. An example is you don’t know what textures need to be processed until you build the maya/max file. So what you really want is
.mb->.middleformat->.texturelist->[texture1, texture2, ...]
When .texturelist already exists and is up to date (ie, newer than both .middleformat and .mb) it’s easy for some build system to follow the rules and build the textures but most build systems I’ve used don’t like it when that list doesn’t exist yet and therefore has to be added after the build has started. Or to put it another way, most build systems start with only some programming language in mind and make the assumuption they can scan the text quickly to figure out implicit dependencies. That assumption is not true for assets.
Waf.
It has evolved from Scons years ago to address several flexibility concerns and performance issues. Have a look:
http://code.google.com/p/waf
http://freehackers.org/~tnagy/testdoc/index.html
My team recently migrated from the Autotools suite (*that’s* pain) to SCons. Our codebase isn’t that large (~175 files code, more for other stuff (Doxygen etc.)), but we found Scons was, while slower than ‘make’, easily fast enough for the ease of use difference.
I just has a quick look at your test script, and I notice it’s not using any of SCons’ speed features. Reading your problem domain, one in particular might be useful. SCons uses objects called ‘Deciders’ to figure out if a file/dependency has changed. The default Decider uses MD5 sums of the file contents, which adds a *lot* of time overhead. The reason this is the default behaviour is explained here: http://www.scons.org/doc/production/HTML/scons-user.html#AEN815, but the important point is that it can be easily changed (using one of several built-ins or writing your own). There’s one that will use the later-than-mtime instead, like make, (flaw: doesn’t catch introduction of older files into the file tree), and another that’ll use mtime-exact (any different mtime causes a rebuild), and several more. There are trade-offs involved with these, but we found that mtime-exact took 66% of the time of md5s, which is not as fast as others but takes a lot of the sting out.
Is kjam still alive ? I have sent mail to him but no response.
Frankly, I had never really considered test-driven builds. We’re definitely test-driving our data conversion tools (take this model from the intermediate format to the final one).
The main problems I see with a test-driven tool is that the high-level build process is a well-understood, generic process. It’s also a process that is highly performance sensitive and it’s going to be stressed with Gigabytes of data, so you want to parallelize it as much as possible. So given that, it seems that trying to adopt someone else’s solution is the way to go. Yes, there’s still no perfect build system out there, but they’re probably better than what a single person can do in a month or so.
On the other hand, if your game doesn’t have too much data, then you can just grab any of them (make, Scons, MSBuild) and be happy with it and you can spend your efforts in the things that really matter (the game). In general I’m not a fan of writing build systems from scratch as a set of custom scripts though.
Noel, what’s your take on test-driven builds? I.e., don’t use generic build tools
but rather write a test that formalizes the requirement “I want to generate
data format xyz from source format abc.” This test can be tuned as much as necessary,
e.g. it could be rewritten as “Check for the abc file with the latest date. Read
the latest date of the target format’s file (from a cache, not considering every file).
Run task if required.”
The bad news: It’s more work.
The good news: It allows you to tune the build as much as necessary. Actually, you may only
have to tune
some parts of the build. The language doesn’t really matter, any generic scripting
language will do (say Python, or Groovy with its nice integration of Ant tasks).
Can anyone tell me what kind of license/cost KJam is?
That’s great.
Not that I need anything added or fixed in Kjam, but it’s good to know the tool is ‘alive’
Jam, the unsupported devil child of Perforce, may just be the ticket. Many people swear by the Boost version of Jam, I found the “ftjam” version easier to build and it takes the “precompiled header” extensions quite easily.
http://freetype.sourceforge.net/jam/index.html
Stephen Knight, creator of Scons, sits upstairs and works on building, among other things, Chrome. And yes, it sucks, but our previous system sucked more and knew nothing about being cross platform.
This page
http://www.scons.org/wiki/GoFastButton
says
The command ’scons –max-drift=1 –implicit-deps-unchanged’
will execute your build as fast as possible.
We’re using SCons for several years now to do all our asset builds, some code builds (lua wrapping classes), vcproj generation, test building, etc.
It’s horrible.
For one thing, it’s very slow;
Then, there is a constant stream of build-related problems every week (though perhaps we’re doing something wrong here).
Also SCons (as it seems) has a bug with complex dependency checking scheme – i.e. we have a .ma -> .dae -> binary file geometry pipeline, .dae file is scanned for textures, which are built before binary file, so binary file depends on them; if .ma has changed, then in some cases the dependency tree contains BOTH dependencies from old .dae file AND from new one (just generated).
And the best thing of course is that we’re not going to change it.
So you have .ma -> .dae -> bin. The specific aspect is that you have
a derived file with many implicit dependencies.
Bad news for you: this is a demanding file flow that all build systems
have and will always have a very hard time to deal with.
Some systems, make for example, can even hardly do a correct build
for such flow in only one pass (usually they do it in 2 passes).
AT the other extreme are systems that do it correctly in one pass
but in a prohibitive time because they read and scan those derived
files with implicit dependencies on every null build.
What you should know is that the task
of making such a build both fast and safe is way more complex
problem than most of us like to believe it is.
Even more bad news for you, SCons is one of the best at managing
such builds, from the corectness point of view and not only.
The other tools that I know fare much worse. For example, SCons
gives you the choice of caching implicit dependency info, caching some
or not caching at all. You probably rushed to use that –implicit-deps-unchanged
because it is displayed on every wall without considering carefully if
your build is in one of those corner cases that cannot afford that optimization
or not. In your particular case, you complain about some extra dependencies
and low speed. You should ask the question if checking those extra files
is faster or not compared to updating the the list of dependencies. You may get
a surprise…
I do know that most of us are in a position where we cannot change
the file flow to make it more build-friendly. The price to pay
is quite often longer time spent in builds (multiple passes, time to scan
files with information that in fact belongs to the build description, etc).
If you spot a good build system for a build like yours, I’m highly
interested. Do tell.
Adrian
Hi,
There are two viable alternatives to scons.
The first one is waf. It was written after attempts to convert the KDE project’s build system to scons. After a while, Thomas Nagy found too many limitations in scons (including but not limited to the one you witnessed) and wrote his own build framework in python. It’s used by several open source projects but is not really mainstream.
The other alternative is September 19, 2008 at 5:24 am