By now you’ve probably heard of Lasting Legacy: you’re managing a family around the 19th century through several generations, socializing, choosing good marrying prospects, and helping family members pick an occupation. Ah, occupations…

Occupations

Occupations and the life and blood of the game. It’s what gives the player lots of new actions and effects that can be combined in lots of different ways to achieve lots of powerful effects. The only downside of occupations is that… there are a lot of them! Right now we have about 50 and that’s just some really basic stuff. I expect the game will ship with about 100 occupations, and possibly more once we add country-specific ones.

50 doesn’t sound too bad you say. Sure, but each occupation has a lot of data that goes along with it: a name, whether it has an action or not, text for the action,text for the log, costs, whether it’s inherited… Hang on, let me tell you exactly what an occupation has:

struct OccupationInfo
{
     Occupation::Enum occupation;
     Rarity::Enum rarity;
     const char* maleName;
     const char* femaleName;
     const char* description;
     const char* logEntry;
     IconSpriteId::Enum spriteId;
     uint32_t periodFlags;
     int goldIncome;
     bool incomeMultiplied;
     int prestigeIncome;
     int legacyIncome;
     bool action;
     int goldCost;
     int prestigeCost;
     bool educationRequired;
     int targetCount;
     bool pickable;
     bool destructive;
     bool inherited;
     Occupation::Enum childrenOccupation;
     int personLike;
     uint32_t tags;
     AccessoryType::Enum maleAccessories[MaxAccessories];
     AccessoryType::Enum femaleAccessories[MaxAccessories];
     int maleAccessoriesCount;
     int femaleAccessoriesCount;
};

How do we set all that data for 50 different occupations? Back when I was working for big game companies I’m sure that someone’s job would have been to create a GUI tool to let designers enter those values. Then someone else would create a fancy XML format that could be exported from the tool. Then, if we were lucky, someone else would take that XML and crunch it during the asset conversion process into something binary that could be read directly into memory. Probably someone else would work on tech that would let you hot load new data while the game was running. Sweet!

Except that now I’m a lowly indie developer and I don’t have time for any of that stuff that doesn’t actually make the game better. Fortunately, I also don’t have a team of 50 designers to deal with, so my solution is this:

{
	OccupationInfo& info = OccupationInfos[Occupation::Banker];
	info = OccupationInfo(Occupation::Banker, "Town Banker", IconSpriteId::Banker);
	strcpy(info.description, "Gain 100 gold. Income decreases by 1.");
	info.logEntry = "%s %s gained %d gold.";
	info.tags = OccupationTags::Gold;
}
{
	OccupationInfo& info = OccupationInfos[Occupation::Poisoner];
	info = OccupationInfo(Occupation::Poisoner, "Cunning Poisoner", IconSpriteId::Poisoner);
	strcpy(info.description, "Kill another family member.");
	info.logEntry = "%s %s poisoned %s %s.";
	info.cost = 500;
	info.tags = OccupationTags::Danger;
	info.rarity = Rarity::Uncommon;
}
{
	OccupationInfo& info = OccupationInfos[Occupation::Tutor];
	info = OccupationInfo(Occupation::Tutor, "Private Tutor", IconSpriteId::Tutor);
	strcpy(info.description, "Another adult family member leaves their occupation and becomes educated.");
	info.logEntry = "%s %s sent %s %s to school.";
	info.educationRequired = true;
	info.tags = OccupationTags::Social;
}

“Oh the horror! Hardcoded data!” shriek all new CS students as they recoil from the screen.

But it really isn’t that bad. Actually, it’s pretty great: I don’t have any exporting/importing to do, I don’t have to load anything at runtime, and the compiler does a lot of checking for me. True, I can’t change it on the fly, but I don’t really need to when rebuilding the whole game takes about a second.

Unfortunately the problem is that we have 50 of those initialization blocks. Hopefully a lot more by the time we ship. They’re perfectly fine to work on each of them in isolation, but I’m not able to get a big picture. Have I created a similar one to this new one I’m thinking about? How much do other similar occupations cost? Can I get all the Social occupations together so I can compare them? How many rare occupations do we have in the late period? I can’t answer those questions very well by looking at that code. I needed something else.

Global View

Using some kind of text file wouldn’t help any. It would be a matter of changing that into an ini or an XML file, which would be much more painful without any of the benefits.

I started setting up a Google Spreadsheet, which is something we did for the specialists in Subterfuge. Spreadsheets have some nice capabilities like doing data validation on cells (for example, to enter enums), or letting you sort records based on some criteria. In Subterfuge the spreadsheet method worked reasonably well because we didn’t have as many specialists and each of them had less data, so the spreadsheet was pretty manageable. Here we just had too much data for each occupation and the spreadsheet was getting unwieldy.

Miguel mentioned SQLite, but I wasn’t too keen on linking with some extra library and loading that data at runtime. It also sounded like total overkill to have a full relational database for what amounts to a single table. Then I realized how wrong I totally was.

People have written GUI tools for SQLite that let you create, edit, and browse databases very easily. For example, we’re using DB Browser for SQLite and I think there are better ones out there (although maybe not free ones). Our main table looks something like this:

And the data itself like this:

It looks like a spreadsheet until you realize that you can very easily search any field and you can even create “views” with more complex queries ahead of time. So now I can quickly see how many rare occupations we have in the late period with a single click. Very sweet! It definitely gives me that global view I was hoping for.

How about loading this data from the database at runtime? I doubt it would be a big deal, but it’s still something I’m not crazy about. Fortunately, it’s super easy to extract the data from SQLite. We have this in our make file:

@sqlite3 -header -csv $(DB_SRC) 'SELECT * FROM Occupations' > $(DB_CSV)

And that spits out a nice comma-separated file with all the data. Still, loading csv files at runtime isn’t much of an improvement, so we do a bit more processing. Initially I started converting the csv file into an ini file (since we already have an ini-reader library), but then I realized there was no point in doing that. Unlike Subterfuge, I’ll never want to have different versions of the game data around and load them based on different rules versions. We just want one true version. What’s the easiest way to put that data in the game? C code!

So with a simple Python script, I convert that csv file into a C file that looks an awful lot like the one we started with, except that this time it’s completely derived from the contents of the database. One of the great benefits of this is that we can let the compiler do a lot of the checking instead of having to do it in the script or (gasp) at runtime. For example, some occupations force their children to have a particular occupation (King -> Prince). I don’t even have any code that checks that the occupation you enter there is valid. We just generate code like this:

info.childrenOccupation = Occupation::XXXXX;

If XXXXX doesn’t happen to be a valid Occupation enum, you get a build error until you fix it. And again, since the whole turnaround time between editing the database and building the game can be a second or two, it’s really no big deal at all.

Constants

I left one small detail: Constants. Look at some of the first code I put up early on:

strcpy(info.description, "Gain 100 gold. Income decreases by 1.");

What’s wrong with that? Yup, you got it. Somewhere else in the code, I have some constants like this:

const int BankerReward = 100;
const int BankerIncomePenalty = 1;

Now every time I change how much money the Banker gives, I need to change the string and the constant (let’s not even talk about localized strings yet).

So instead now I use global string replacement while we’re generating the C code. The string on the database is this:

Gain [[BankerReward]] gold. Income decreases by [[BankerIncomePenalty]].

Whenever we load the csv file, we look for any substrings with the pattern [[…]] and we try to match them from a header file with constants that we’ve loaded. If we find them, we replace them with their values, and if we don’t, we spit out an error and stop because clearly someone wanted to have a string replaced there and probably mistyped it.

Does all of that sound complicated? Not really. When I said earlier that the Python script was simple, I wasn’t kidding. This is the bulk of it, including the constant replacement (just a couple helper functions are missing):

def main():
	dir = os.path.dirname(os.path.realpath(__file__))
	baseDir = dir + "/../LastingLegacy/AssetsRaw/"
	reader = csv.DictReader(open(baseDir + 'db/occupations.csv', 'rU'))

	sourceCodeDir = baseDir + "../src/"
	globalVariables = readGlobalVars(sourceCodeDir)
	pattern = re.compile(r'\b(' + '|'.join(globalVariables.keys()) + r')\b')

	Rarities = {'C': 'Rarity::Common' , 'U': 'Rarity::Uncommon', 'R': 'Rarity::Rare', 'Q': 'Rarity::Unique'}
	Periods = {'E': 'Period::Early' , 'M': 'Period::Middle', 'L': 'Period::Late'}

	defFile = open(sourceCodeDir + "Occupations.inc", 'w')
	for entry in reader:
		type = entry['id']
		defFile.write('{\nOccupationInfo& info = OccupationInfos[Occupation::' + type + '];\n')
		defFile.write('info.occupation = Occupation::' + type + ';\n')
		defFile.write('info.spriteId = IconSpriteId::' + type + ';\n')
		defFile.write('info.maleName = "' + entry['maleName'] + '";\n')
		if entry['femaleName']:
			defFile.write('info.femaleName = "' + entry['femaleName'] + '";\n')
		else:
			defFile.write('info.femaleName = info.maleName;\n')
		
		description = multipleReplace(entry['description'], globalVariables)
		defFile.write('info.description = "' + description +'";\n')
		
		defFile.write('info.logEntry = "' + entry['logEntry'] + '";\n')
		WriteOptionalBool(defFile, entry, 'educationRequired', type)
		WriteOptionalKey(defFile, entry, 'goldCost', type)
		WriteOptionalKey(defFile, entry, 'prestigeCost', type)
		WriteOptionalKey(defFile, entry, 'goldIncome', type)
		WriteOptionalKey(defFile, entry, 'prestigeIncome', type)
		WriteOptionalKey(defFile, entry, 'legacyIncome', type)
		WriteOptionalBool(defFile, entry, 'incomeMultiplied', type)
		WriteOptionalBool(defFile, entry, 'action', type)
		WriteEnum(defFile, entry, 'rarity', Rarities, type)
		WritePeriodFlags(defFile, entry, 'periods', Periods, type)
		WriteOptionalKey(defFile, entry, 'targetCount', type)
		WriteOptionalBool(defFile, entry, 'destructive', type)
		WriteOptionalBool(defFile, entry, 'pickable', type)
		WriteOptionalBool(defFile, entry, 'inherited', type)
		if entry['childrenOccupation']:
			defFile.write("info.childrenOccupation = Occupation::" + entry['childrenOccupation'] + ";\n")
		WriteOptionalKey(defFile, entry, 'personLike', type)
		WriteOptionalTags(defFile, entry, 'tags', type)
		WriteOptionalAccessories(defFile, entry, 'maleAccessories', type)
		WriteOptionalAccessories(defFile, entry, 'femaleAccessories', type)

		defFile.write('}\n')

	defFile.close()	

Here’s the full script in case anyone is interested in the details.

Boom! Done!

Ship it.

Right now I have a couple of game ideas in hand that I’m pretty excited about. I prototyped one of those, but I quickly realized it might be a game better suited for desktops rather than iOS, so I decided to write my next prototype for that game on the Mac. Maybe another day I’ll write a post about prototypes, but what I want to write about today is my experience moving over my prototyping code to the Mac.

I’ve been working on MacOS since 2007, when I switched because of iPhone development. For the most part, I’ve been loving it as an OS, but I never actually targeted it as a platform in my development. I figured it wouldn’t be very hard at all to compile my simple C/C++ + OpenGL code that I use as the basis for all my iOS games on MacOS. After all, that code started on Windows and DirectX, and porting it to iOS was pretty quick and easy. On top of that, I can use Xcode 4 for Mac, just like iOS (I’m not sure that counts as a plus though). How card can it possibly be?

The answer is, not hard at all, as long as you’re aware of a couple gotchas along the way.

Xcode project

I figured I would start from scratch, so I created a new Xcode project targeting MacOS. First surprise was that, unlike iOS, there wasn’t a template to create an OpenGL application. Instead, I need to create an empty Cocoa application and set things up by myself. No big deal.

I used the samples from Apple to get me started. I could have grabbed those and started adding stuff, but for some reason they have all the logic built into the OpenGL views, and I like to keep my views as simple as possible: they hold the surface to render, and they collect input and other window events. Done. So I had to do some moving around of code and responsibilities, which at the same time allowed me to make it more like the structure I used on iOS.

The AppDelegate does most of the initialization and creation on applicationDidFinishLaunching, and the OpenGL view gets created through the minimal nib file (with just a menu, window, and a view). AppDelegate is also responsible for setting up the display link callback, and then all the rest of the execution happens in my own code, away from those classes.

That was all pretty smooth. The only weird gotcha I ran into is that, unlike with previous Xcode 4 projects on iOS, I was no able to add any files to it that were not under the project root. They would turn red and not be available in the project. I tried adding them in every possible way and always got the same result. So, in the end, I gave in and just moved my External dir under the same directory as the Xcode project. Some people on Twitter had the same experience, while others didn’t, so there’s something fishy going on in there.

I have all my game static libraries (I refuse to call it an “engine”) as a separate Xcode project. I was able to add the project to the workspace and just change the target to be MacOS, and everything worked without a hitch.

OpenGL

I knew I was going to have to make some changes to the graphics code because all my iOS code uses OpenGL ES 2.0, and the Mac supports OpenGL “non ES” version. But I never did any graphics programming on the Mac, so I had no idea what to expect.

I decided to start with this sample code from Apple. After all, it seemed to be exactly what I was looking for: A simple project that sets up OpenGL both on iOS and MacOS. Score!

Once I started digging into it, I realized that it wasn’t going to be as straightforward as I realized. It turns out there were lots and lots of differences between OpenGL ES 2.0 and the OpenGL version on the Mac. Some were easy to catch, like different syntax for the shaders (“varying” became “in”, “uniform” was “out”, etc). Some were a pain to track down (needed to bind some Vertex Array Objects, and I was forced to use Vertex Buffers).

The documentation wasn’t clear on this point at all, but fortunately I got some good tips from Twitter. It turns out that the sample I used as a starting point created an OpenGL 3.2 context, and OpenGL 3.2 is very different from OpenGL ES 2.0.

	NSOpenGLPixelFormatAttribute attrs[] =
	{
		NSOpenGLPFADoubleBuffer,
		NSOpenGLPFADepthSize, 24,
		NSOpenGLPFAOpenGLProfile,
		NSOpenGLProfileVersion3_2Core,
		0
	};

	NSOpenGLPixelFormat* pf = [[NSOpenGLPixelFormat alloc] initWithAttributes:attrs];
	NSOpenGLContext* context = [[NSOpenGLContext alloc] initWithFormat:pf shareContext:nil];
	int swapInt = 1;
	[context setValues:&swapInt forParameter:NSOpenGLCPSwapInterval];

It turns out that leaving out the last attribute (NSOpenGLPFAOpenGLProfile) creates an OpenGL 2.1 profile, which is very similar to OpenGL ES 2.0 (GLSL 1.2 is the shader version that OpenGL 2.1 uses and that’s also very similar to GLSL ES). With that change, all the OpenGL code worked perfectly, with the only exception of the precision modifiers in shaders. Fortunately, I we can wrap those up in a simple #ifdef like this

#ifdef GL_ES
precision lowp float;
#endif

Unit tests

Apart from input, which will require a total re-work, the last major thing I had left was running unit tests after each build. I’ve seen references to Xcode’s unit testing functionality, but I was hoping to reuse UnitTest++ and run it like I do on iOS projects. It turns out it was even easier because you can run them natively without the extra step of the simulator like on iOS projects.

All I had to do was create a new phase in the Debug scheme to run a script. The “script” is just the name of the executable that was just build with the -unittest parameter added to it. I suspect that there’s probably a “better” way than hardcoding the .app/Contents/MacOS/ string in the middle to find the binary itself, so if someone knows the correct variable that includes that, let me know.

The only other thing that came up is that apparently Xcode adds the parameter -NSDocumentRevisionsDebugMode when you debug it from the IDE. No big deal, unless you have your program exit with an error if you pass an unrecognized parameter. I just made it accept and ignore that parameter, and everything worked like a charm.

Conclusion

It took a bit longer than I expected, but I finally have my code base running on MacOS. Iteration time is faster because it doesn’t even launch the simulator, and I suspect graphics performance will also be a lot faster (the iOS simulator does all the graphics processing in software), so I might end up doing more prototyping on the Mac unless the touch input is an integral part of the game.

That future is a reality today for some games and apps (Netflix, Kindle), and I’m convinced that players will expect that in most games in the next year or so. So obviously, the next bit of new iOS tech I decided to try was iCloud. I would love to turn Flower Garden into that kind of seamless experience, independently of the device you use to access it.

As a quick spoiler, it turns out I won’t be able to make Flower Garden quite so seamless without a lot of extra work. But I learned a lot along the way and I should be able to take a small step in that direction.

Design considerations

iCloud promises to be the “available from everywhere” storage solution that will be a key component towards the scenario of playing on any device at any time. Unfortunately, it’s just a component of that whole scenario and still requires quite a bit of work on the part of the developer.

Since I’m just retrofitting Flower Garden to work with iCloud, I wanted a simple way to simply replicate the game state on iCloud. Then, whenever you play it from any device, you get the latest state and everything works as expected.

The big problem with that approach is that you can’t always count on having an active (or fast enough) internet connection. iPod Touches, iPads, or even iPhone in a plane or a cell tower black spot will make it so your device has no way to communicate with iCloud. So what happens if the player plays without an internet connection, then comes home, grabs another device, plays for a bit with internet connection, and finally launches the original device, this time with internet connection. You’ll end up with two conflicting game states. Not fun!

I can think of two different strategies to deal with this situation:

• Require internet connection to play. Some games do this today. It can be quite frustrating not being able to play your favorite game in the plane, but it does solve the game state conflict problem because they can always communicate with iCloud, so they usually provide seamless multi-device play.
• Detect changes on both game states and intelligently solve them. Sounds good in theory, but it’s a pain in practice. Unless you have a very simple game state (a percentage completion for example), it means not only do you have to record the state, but you have to record the events that led to that state, and be ready to examine them, compare them, and marge them in the case of a conflict. And the worst part is that even if you put lots of care into it, there will be cases where the merge is not ideal and the player will feel like he lost something in the merge.

For Flower Garden, neither solution is particularly attractive. It’s the proverbial rock and a hard place. So for the moment I’ve decided to implement just a small step in that direction: Sync things that only progress in one direction, without any danger of conflict. That includes purchased IAPs and unlocked seeds. Later on, once that’s working solidly without any problems, I can consider adding counts like Fertilizer, Color Dust, or Green Thumb points. That will be a bit trickier because there’s still chance for conflict since the count can go up and down, but it’s possible to change each amount into two different values that always go up: Earned amount, and spent amount. That way I can always grab the maximum value that I find either local or on iCloud and things should work smoothly. (Update: What I wrote there about separating them in two different values is totally wrong, and any DB programmer worth his salt would be laughing at that. Separating them in two different values won’t help at all. That’s what I get for doing brain-dump posts on the fly :-)

Apple provides two APIs to store data in iCloud: key-value storage and file storage.

Key Value storage

Initially, this seems like a great way to go. It’s a very simple API, and it’s very familiar to most iOS programmers because it’s just like NSUserDefaults.

Don’t be fooled into thinking you need to store data in strings or one field at the time. You can stuff whatever binary chunk of data you want with NSData, so it’s quite versatile.

It does have three pretty big drawbacks thought:

• Size. You can only store up to 64KB of data. That’s definitely a big deal if you’re planning on storing large data files. Even for Flower Garden, each pot is about 30KB, so I wouldn’t be able to save the full garden state this way.

• Syncing. The Apple docs say that “Keys and values are transferred to and from iCloud at periodic intervals.” Ouch! What does that mean? Clearly this is intended for non-crucial data (like settings), so the potential delay isn’t a big deal. In my tests, I found that data was often not available when starting the app, but would become available a few seconds later.

• No way to check state. When starting the app, there’s no way to find out if the values you’re reading are up to date. Combined with the slow syncing, it makes it less than idea for important data. On the flip side, you can be notified when the data changes, but that means having to deal with changes while the game is running (which I was hoping not to do).

File storage

File storage might be a better option then since it doesn’t have any of those drawbacks: there’s no size limit, data is synced much more aggressively, and you can check if the data is up-to-date, and wait until it is otherwise.

What’s not to like about file storage then? The cumbersome and intrusive API that Apple created around it. If you read the docs, they make it sound like you need to inherit from UIDocument and load all your data through that class. I like to keep things simple and portable, so I’d really rather not introduce UIDocument into the file-loading process if I don’t have to.

It turns out that Apple’s docs on the iCloud SDK are quite sparse and lacking details. I was able to put together a demo from those docs and some experimentation, so hopefully this will be useful to other devs as well.

The device has an iCloud storage daemon running, monitoring specific directories for changes. You get the directory assigned to your app by calling URLForUbiquityContainerIdentifier.

To add a new file, the docs recommend first creating the file somewhere else, and then calling NSFileManager:setUbiquitous:itemAtURL:destinationURL:error: to move it to iCloud. Interestingly, I accidentally skipped this step and just wrote to the iCloud directory directly, and the files were stored correctly anyway. Might as well leave it just in case that behavior changes later on, or there’s some side effect I didn’t notice.

After that, every time you write to the file, the iCloud daemon will detect the changes and push them out to iCloud. This is an important point because a) You don’t have explicit control to say “start send it now” or even “This file is ready to be sent to iCloud” (that would be my choice), and b) I don’t know what it does with partial updates, so I would be very careful about writing to those files and make sure it’s an atomic operation (save somewhere else, and them move the file in one operation).

To get the latest version of the files on iCloud, you can check whether they’re fully downloaded or not by calling NSURL:getResourceValue:&isDownloaded forKey:NSURLUbiquitousItemIsDownloadedKey error:. If they are up to date, you can move on, otherwise, you can initiate a download of the files by calling NSFileManager:startDownloadingUbiquitousItemAtURL:error:.

One weird thing is that the startDownloading function doesn’t have a callback (that I can see). So I had to set up a timer to check periodically if the files are synced.

Since all the syncing happens at startup, there’s no complexity involved with data changing while the app is running. If that’s a case you need to handle, you might be better off using UIDocument since it at least detects conflicts. If you do it this way, the latest one will overwrite any past changes.

Also, working this way, it seemed easier to keep the files in Application Support (or Documents), and only move them to/from iCloud when I wanted to. That has the advantage of iCloud not changing things from under you while the game is running, and the fact that, if for some reason the files in iCloud are corrupt, you always have good local files to fall back to.

Demo

iCloudTest.zip

The demo iCloudTest saves data both with the key value pair, and the file storage system directly like I described above. If you run it from different devices, you’ll see that it’s amazing how quickly file data gets propagated, but key-value data takes a few more seconds.

The demo project also works in the simulator (there’s no iCloud support, so it just uses local files), and it even works with iOS4 (also using local files, and it avoids using any iOS5 symbols while detecting iCloud support).

Sharing Data Between Multiple Apps

Once you’re at this point, sharing data between multiple apps is really straightforward.

First you need to make sure the app IDs of both apps start with the seam Team ID. For some unknown reason lost in the midsts of time, Flower Garden and Flower Garden Free use different prefixes, so that option is out for me. Hopefully most people having been using the Team ID only.

Then you need to decide which type of data you want to share. If you want to share files, you need to add the app ID of the other app to your iCloud entitlement, and then as for the correct app ID (including prefix!) when you call URLForUbiquityContainerIdentifier. If you just want the app’s own iCloud data, you can pass NULL as the parameter.

As far as I can tell, each application can only have one set of key-value data. So sharing it between two apps means changing one app to use the app ID of the other app in the com.snappytouch.icloudtest field of the iCloud entitlements file. As long as both apps use the same Team ID prefix, it should work fine without having to do anything else.

I use the URL shortener in Flower Garden to send bouquets through SMS. Space is limited in a text message, so the message just contains some text explaining what is it and the URL pointing to the bouquet image. (Yes, I would much rather send them through MMS, but Apple isn’t exposing that yet to developers).

In this case, the full URL is http://flowers.snappytouch.com/sms.php?id=949618b4b3c6f3d76e32b45446e238a0 which gets thankfully shortened to http://goo.gl/IV5cq.

Sending bouquets through SMS has been in Flower Garden for several months, but it was using bit.ly before, which is probably the most popular URL shortener out there. I like their web interface and their super-easy to use API, but unfortunately it seems that I hit some mysterious API limit during the Mother’s Day Flower Garden promotion. That limit isn’t public anywhere, and as far as I can tell, I can’t even see it myself through the web interface or through an API query.

Finding out that I reached the API limit was quite shocking, because sending bouquets through SMS isn’t a particularly popular feature. Unfortunately I don’t have good analytics hooked up to that step, but I can’t imagine there were more than a few thousand per day.

They were very nice and contacted me instead of shutting down my account since it was just a spike. They also tried to sell me their “Enterprise” account, but $995/month is a tad too expensive for me. By about $990 probably, so I had to look for other options.

After a very quick research, goo.gl was the perfect alternative. Not only is it very fast (and backed up by the giant Google no less), but they have an API limit of 1,000,000 queries/day. If I ever blow that budget, I’ll be able to afford the $995/month without a problem :-)

All URL shorteners are very easy to use. You need an API key, and figure out the exact format of the HTTP message you send and the response you get.

goo.gl

Goo.gl has a great Getting Started Guide that tells you everything you need to know. Get your private API key from here, and you’re ready to rock.

Drop this in your app and start shortening away. You’ll notice I used a synchronous HTTP request, which is usually a big no-no. Here I felt it was justified since the user is blocked waiting for the SMS to be prepared and sent. Besides, goo.gl is very, very fast, so it’s never even noticeable.

const NSString* GooGlAPIURL = @"https://www.googleapis.com/urlshortener/v1/url?key=YOUR_API_KEY_HERE";

NSString* ShortenURLWithGooGl(NSString* longURL)
{
	NSURL* apiUrl = [NSURL URLWithString:GooGlAPIURL];
	
	NSMutableURLRequest* req = [[NSMutableURLRequest alloc] initWithURL:apiUrl];
	[req setHTTPMethod:@"POST"];
	[req setTimeoutInterval:Timeout];
	[req setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
	
	NSString* body = [[NSString alloc] initWithFormat:@"{\"longUrl\": \"%@\"}", longURL];
	[req setHTTPBody:[body dataUsingEncoding:NSUTF8StringEncoding]];
	[body release];
	
	NSError* error = [[NSError alloc] init];
	NSHTTPURLResponse* urlResponse = nil;
	NSData* data = [NSURLConnection sendSynchronousRequest:req returningResponse:&urlResponse error:&error];
	[error release];
	if (data == NULL || ([urlResponse statusCode] < 200 && [urlResponse statusCode] >= 300))
		return NULL;

	NSString* response = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];
	NSDictionary* responseDict = [response JSONValue];
	NSString* shortURL = [[responseDict objectForKey:@"id"] retain];
	[response release];
	return shortURL;
}

I’m using a JSON framework to parse the answer, but it’s so simple I probably wouldn’t even have to. I only used it because it’s already part of the project because of Facebook Connect.

bit.ly

Even though it’s not my favorite shortener, I’m adding it here for completeness (and because I had the code already written).

The one thing that bit.ly has going for it is that it’s even easier to use than goo.gl. No JSON involved, and you don’t even need to send a body with your request. As usual, get your API key by signing up with bit.ly.

const NSString* BITLYAPIURL = @"http://api.bit.ly/v3/shorten?login=%@&apiKey=%@&format=txt&";

NSString* ShortenURLWithBitLy(NSString* longURL)
{
	NSString* urlWithoutParams = [NSString stringWithFormat:BITLYAPIURL, LoginName, APIKey];	
	CFStringRef encodedParamCF = CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault,
															 (CFStringRef) longURL, 
															 nil, (CFStringRef) @"&+", kCFStringEncodingUTF8); 
	NSString* encodedURL = (NSString*)encodedParamCF;
	NSString* parameters = [NSString stringWithFormat:@"longUrl=%@", [encodedURL stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding]];
	NSString* finalURL = [urlWithoutParams stringByAppendingString:parameters];
	
	NSURL* url = [NSURL URLWithString:finalURL];
	
	NSMutableURLRequest* req = [[NSMutableURLRequest alloc] initWithURL:url];
	[req setTimeoutInterval:Timeout];
	
	NSHTTPURLResponse* urlResponse = nil;  
	NSData* data = [NSURLConnection sendSynchronousRequest:req returningResponse:&urlResponse error:NULL];
	if (data == NULL || ([urlResponse statusCode] < 200 && [urlResponse statusCode] >= 300))
		return NULL;

	NSString* response = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];
	return response;
}

That’s it. You should be able to drop in either one of those snippets in your project and spend your time working on the things that really matter in your games.

Recording particular events as part of the analytics is only part of it. The most important part is how that data is presented to the developer. Having tables with millions of entries does me no good, and as a busy indie developer, I can’t afford to spend hours writing scripts to analyze it. I want something that allows me to easily visualize the data and makes sense out of it at a glance.

One possible snagging point about analytics is that Apple was cracking down on some applications with analytics enabled a while back. Specifically, the iOS developer agreement states:

3.3.9 You and Your Applications may not collect user or device data without prior user consent, and then only to provide a service or function that is directly relevant to the use of the Application, or to serve advertising. You may not use analytics software in Your Application to collect and send device data to a third party.

Where does that leave us? Reading it carefully, it seems that the restrictions are limited to “user or device data”. I’m interpreting that to mean things like UDIDs and emails, not anonymous player usage data (how long did it take to reach level 5, how long are play sessions, etc), so I think we’re clear there.

The puzzling thing is that a lot (all?) of the third party analytics libraries do report device information, like what kind of hardware or iOS version is running. That is extremely important information that developers really benefit from knowing, but it seems to go against point 3.3.9. Maybe “device data” only applies to information about that specific device? (as in, the UDID that is now going away). I hope so.

Not having analytics isn’t really an option. Unless you make a game that you plan to throw on the App Store, never touch again, and hope for the best, you’re flying blind without analytics.

What are some of the options we have then?

Home brewed

If you read this blog regularly, you probably know that I’m a low-level, do-it-myself kind of guy, with a deep mistrust and suspicion of middleware. So you would think that I would want to write my own analytics package. After all, how hard can it be? Collect the data you want and ping your server with it. If you get fancy, you can even use a scalable server back end like AWS or GAE. Done.

Not so fast. To do that well, it’s a lot more involved than that. You want to batch when you send out the information, and you might want to distinguish between WiFi and 3G connection (to avoid causing extra data usage for players on a limited data plan).

That in itself is not even that bad. The real pain comes in visualizing that data, and that’s where you can easily sink in days or weeks, and you would still not have something as good as some of the other alternatives.

The other drawback is that if you have some successful applications, you may be generating Terabytes of data per day. Think about the storage and bandwidth costs for that. Yes, I know that sounds insane, but Playfish reported generating that much analytics data at a GDC 2011 talk (video for paid GDC Vault members).

Flurry

Flurry Analytics appears to be the most common analytics package out there among my indie iOS dev friends. It’s free and it’s easy to integrate. The visualization page is pretty good, and it even offers some fancy features beyond session length and events, such as flow through the application.

So what’s not to like? I was never able to make heads or tails of the application flow. When you get at that level, you start needing to spend some serious time making sense of data, which is what I don’t have as an indie. The web page to visualize the data is written in Flash, so for those of you using an iPad to check it, it’s not a good option. Update: Flurry apparently added a Flash-free web page since I last looked at it a few months back. Thanks Charilaos Kalogirou for the tip.

The killer deal for me was bloat. Adding the Flurry analytics library to an app increased the executable size by 500KB. I’m sorry, but that’s completely unacceptable for me. Memory is tight, and half a MB is very significant. I would rather add another large texture or another music track. And if you think about it, why does it need 500KB to buffer and send some events? That’s simply ridiculous.

Google Analytics

I never actually tried out Google Analytics. They have an iOS SDK and it integrates quite well into the web page Google Analytics environment. The main drawback I heard from other developers is that it’s designed more for web pages rather than for apps and games, so it wasn’t a perfect fit.

Anyone who used it want to expand on this in the comments?

Localytics

Localytics is a relative newcomer to the iOS analytics field, but it was love at first sight for me.

It has the same ease of integration of Flurry, and it provides very similar functionality. Localytics, however, is completely open source, so instead of a black box library, you get the source code and you can add it directly to your game. How much does it increase executable size? 4KB! You have to wonder what the other 496KB were for in the Flurry library.

As a bonus, their visualization web page works great on an iPad, although it can be a bit slow for very large data sets sometimes.

One of their biggest selling points is that they report the analytics in real time, but I really don’t care one way or another. Waiting 12 or 24 hours to see the analytics doesn’t bother me one bit.

Unlike Flurry, you can only add strings as parameters to events. That works fine if I have a set of discrete options. For example, when someone sends a bouquet in Flower Garden, I can send a “bouquet sent” event with a parameters that is “email”, “facebook”, or “sms”. As a result, I can see a nice pie chart with the breakdown of how people are sending bouquets. Very useful stuff!

But how about things that don’t have discrete options? For example, in Casey’s Contraptions, we wanted to see how long players take to solve each level. It turns out you can’t have a number as a parameter, but you can easily get around that by discretizing it yourself, which in the end, is easier to visualize. So when we send the LevelXXFinished event, we look at how long the player took to finish it, and we break it down into ranges: under 30s, 30s-1min, 1min-2min, etc.

This is what the report looks like for one of Casey’s Contraptions levels:

It looks like a fairly balanced level with the majority of the people spending under 3 minutes to solve it.

Random tips

A couple random things we learned along the way about analytics:

• Less is more. Start with just a few events and go from there. If you have tons of data, you might never have the time to look at it.
• Use analytics during playtesting. One thing is what people tell you, and another thing is what they really do. Since most of our playtesting is done remotely and we can’t observe as people play (which is invaluable), we can at least gather some hard data about it.
• Turn off analytics reporting in debug mode. Trust me on that one.

How about you? What’s your favorite analytics package and why?

[1] For example, Flower Garden has been on the App Store for almost two and a half years, and it has changed radically in that time!

As programmers, we’re constantly reusing code. Sometimes it’s in the form of low-level OS function calls, or game middleware, and sometimes code that our teammates wrote. At the same time, unless you’re writing top-level game script code, chances are that the code you’re writing will be used by people as well.

I’m purposefully avoiding labeling reusable code as “libraries” or “middleware”. Those are just two of the many forms in which we end up reusing code. Copying some source files onto your project, calling an API function, or instantiating a class written by someone else in your team, are all different forms of code reuse.

Getting Started

Without a doubt, the most difficult job writing reusable code, is making sure it solves a problem correctly and meets everybody’s needs. That’s not a easy task when we’re writing code for ourselves, but it’s much more difficult when we’re targeting other programmers. Too many libraries get it only half right, and they solve some problems at the expense of introducing a bunch of new ones. Or they force the user to jump through all sorts of hoops to get the desired result in the end. We need a clear understand of the exact problem we’re trying to solve with our code.

Libraries often fall in the trap of presenting an implementation-centric interface. That is, their interface is based on the implementation details of the library, rather than how the users are going to use it in their programs.

The best way I’ve found to address both those shortcomings is to start by implementing code that solves the problem for one person. Just one. Forget about multiple users and code reusability for now. If you don’t have immediate and constant access to that one person, then you need to play that role and create a game or application that is as close as possible to what one of your users is going to be developing. By using this approach, I find that the interface to the reused code is much more natural, and it’s based on the experience of having solved the problem at least once. Otherwise, you run the risk of creating an interface that is not a good fit and forcing everything to conform to it in unnatural ways.

Once you have implemented a solution, take a moment to think before you dive into doing any more work. Many times you’ll find there is no reason to abstract it any further since your code is only going to be used in one place. If that’s the case, step away from the code and go do something more productive. You can always come back later whenever there is a real need to reuse it later in the project or in a future game.

There are exceptions to this approach of implementing a solution first, and abstracting it later. Some problems are very simple, or very well understood, so we might be able to jump in and implement the reusable solution directly (an optimized search algorithm, or a compression function for example). It’s also possible that you’ve implemented a similar system several times before, and you know exactly at what kind of level to expose the interface and how things should look like. In that case, it’s perfectly valid to draw on your past experience. Just try to avoid the second-system effect: The tendency to follow up a successful, simple first system, by an overly-complex system with all the ideas that didn’t make it into the first one. Setting Goals

Most successful reusable code was created with specific goals about how it was meant to be used. Sometimes those goals are explicitly stated, most of the time they are implied in the code design.

Some of the most common goals are flexibility, protection, simplicity, robustness, or performance. You can obviously not meet all those goals at once, and even if you could, you probably shouldn’t try. That would be a tremendous waste of time and resources. Stop thinking in the abstract and think about your one user. What does he or she need? What is the most important goal for them?

Many APIs and middleware packages are designed with protection as one of the primary goals: They don’t want the user to accidentally do anything wrong. In itself is not a bad goal, and it can often be implemented by having clean, unambiguous interfaces, clearly-named types and functions, and strongly-typed data types. Unfortunately, a design with protection as a main goal can often result in encumbered interfaces, verbose code, slow performance, and inflexible code. There is nothing more frustrating than wanting to do something that is explicitly being protected against, and having to work around the interface.

If your target users are professional game developers, give them the benefit of the doubt and don’t try to overprotect all your code. Save that for the scripting API exposed to junior designers and released to the customers with the game. If you’re concerned about programmers using your code correctly and not making mistakes, provide good sample code, tests, and documentation. If that’s not enough, and you feel that everybody would benefit from some level of protection, try to keep it to a minimum and maybe even provide lower-level functions that bypass it for power users.

Whatever the primary goal, the libraries and APIs I prefer to work with, help me get whatever I need done, while getting out of the way as much as possible.

Architecture

There are many different ways to architect code that is intended for reuse. The best approach will depend on the particular code: How complex is it? How much of it is there? What are its goals?

Unless your goals are to make quick, throwaway applications, I strongly recommend against a framework type of architecture (see Figure 1.) A framework is a system in which you add a few bits of functionality to customize your program into an existing system. They may sound like a clean and easy way to use complex code, but they’re inevitably very restrictive, and they make it very difficult, if not impossible, to do things with them beyond what they were intended to.

A more flexible approach is a layered architecture (see Figure 2.) Each layer is relatively simple and provides a well-defined set of functionality. Higher-level layers build on top of lower-level layers to create more complex or more specific functionality.

Keep in mind that it is not necessary, or even desirable, to have higher-level layers completely abstract out and hide the lower-level ones. By letting layers be fully transparent, they allow you to mix and match at what level you want to access the code. This can be very important, especially towards the end of a game when fixing some bugs or trying to squeeze some more performance out of the engine.

For example, one layer can expose functionality to create and manipulate pathfinding networks and nodes, another one can implement searches and other queries on those networks, while a third, higher-level layer, can expose functions to reason on the state of the network.

A toolkit architecture (see Figure 3), is the most flexible of all. It provides small, well-defined modules or functions with very few dependencies on other modules. This allows users to pull in whatever modules they need into their game to meet their needs and nothing else. Users can also start by reusing some modules, and but replace them down the line when they want to go beyond the existing functionality. Because of this, toolkit architectures are particularly well suited to game development.

Object Oriented

I tend to avoid complex class hierarchies in most of my code, but that’s especially important in the case of reusable code. Class hierarchies are very rigid, and impose a particular structure on their users.

If you want to remain object-oriented, a better approach is to emphasize composition of objects instead of inheritance. That allows users to much more easily pull in the functionality they need, and create their own objects based on their own constraints.

You can even go a step further and provide purely procedural interfaces. Plain static functions that operate on data types. Interfaces based on static functions are often much easier to understand and grasp than interfaces that involve classes and inheritance. They are also much more convenient for users to wrap and use in many different ways.

Remember what you learned about object-oriented design and having private data? Forget about it, and keep everything accessible. You may think you’re doing the user a favor by making some variables private and reducing the complexity of the interface. That’s partly true, but eventually, your users will want to have access to some of those variables that you took pains to hide. And if they really want to, they will get to them, even if it means direct addressing into an object or vtable.

It’s true that large codebases can be intimidating, and exposing all the internal details along with the regular interface would make it unwieldy for new users. An effective approach is to separate the public interface from what is intended for internal use only, but still make it available through some other means. For example, private data and functions could be wrapped in a different namespace, or simply in a different set of headers. Anything that clearly sets them apart, and doesn’t clutter the initial look at the interface, but that allows experienced developers to get to them and get their hands dirty. Extensibility

As soon as you make your code available to a wide range of developers, you’ll find that people want to use it in progressively more complex and bizarre situations. Your code might have completely solved the case of your first couple of users, and since it’s layered and modular, it can meet a lot of different requirements. But eventually, some people will start taking it to extremes you hadn’t imagined and it falls short for them. What to do?

You could start adding more options and more modules and more callbacks to your code. That way programmers can hook up into almost any part of the code and replace it with their own. The problem with that approach is that you’ve taken something that was relatively simple and made it into an large, ugly, fully-customizable, behemoth that tries to keep everybody happy. That sounds like a lot of common APIs we know and hate. Most successful products try to completely meet the need of some people rather than meet everybody’s needs part way. Otherwise you inconvenience 95% of your users for the benefit of 5% of them.

A better approach is to let those 5% users fend for themselves, but give them the means to do it. How so? With source code. Without source code, developers feel caged and constrained. They know they can’t look behind the interfaces, let alone modify anything in case something goes wrong (and we all know something will go wrong). The more code there is, and the more a project relies on it, the more important it is to have access to the full source code. Many teams will refuse to use some libraries or middleware unless the full source code is available. As soon as you make source code available to your users, you immediately put them more at ease because they feel more in control, and you allow those with special requirements to make whatever modifications they need to do.

Even those developers without a need to modify the code, they will be able to browse the code and see how certain functions are implemented. Not only will it make people much more likely to use your code, but they will probably also fix your bugs and suggest performance improvements, so it’s a win-win situation for everybody.

For extra bonus points, the source code should be accompanied by a set of tests. The more comprehensive the better (unit tests, functional tests, etc). Hopefully you created all those tests while you were developing the code, so distributing it along with the source code shouldn’t be any extra effort, but it will make a huge difference to your users. It will give them much more confidence modifying the code to suit their needs and still see that all the tests are passing.

Upgrades

As soon as you release some code and you have your first user, the question comes up of how to deal with new versions. There are many ways you can go about it, depending on how often you’ll release new versions, and how important it is to maintain backwards compatibility.

On one extreme, you can change the code and the interface to fit new features, changes in architecture, or any other reason. Whenever you release a new version, users will have to choose to remain with their current version or upgrade to the latest and make whatever changes are necessary. This is a common approach in open source projects and internal company code.

On the other extreme, once you release a version, you stick to that interface whether it’s a good idea or not. This can be good for users because they can get new versions without any extra work on their part, but it can be very constraining. It can make new features impossible to add, and it can prevent performance optimizations. This approach is more common on code that will become the foundation of many programs, like OS libraries and low-level APIs.

A good compromise is to keep interfaces the same during minor versions, and only change them whenever a major version is released. That way other developers only have to put in the time to upgrade to a major release if they really need the new features at that time.

Another approach is not to change existing functions or classes, but to introduce new ones and slowly deprecate the old ones over time. That way code continues to work, but users can take advantage of the new functions. After a few versions, you can completely drop off deprecated functionality, at which point most people will have upgraded already. If you do this, make sure to label deprecated functions with a #pragma warning or some other way. That way, developers know it will be phased out and they can start thinking about upgrading to the new interface.

The easier you make the transition to the new version, the better for your users, and the more likely they will be continue using your code. For example, you can provide scripts that parse their source code and upgrade it to match the new interfaces. That can be a bit risky, but it can be really worthwhile if you have a lot of required changes that are relatively mechanical (renamed functions, changed parameter order, etc).

Is there any point to all this talk of interfaces if you made your source code available? Yes, very much so. Most developers will look at the source code to know how things work under the hood, but they probably won’t modify it. Even if they do, they know that’s something they do at their own risk, so they’ll be more than willing to make a few changes whenever a new version is released. Everybody else will still definitely benefit from a relatively stable interface.

Conclusion

Writing reusable code starts with solving a problem and solving it well. The rest should all fall from there, and you can pick whichever method is more appropriate for your particular code and how you want to share it.

This article was originally printed in the March 2009 issue of Game Developer.

Over the years I’ve mellowed out quite a bit. Now I recognize my obsession with writing endless tools and technology was more of an escape from doing the really hard part of game development. In comparison to making all the decisions involved in the design and implementation of the game itself, writing the perfect resource streaming system sounds like a really good time. It’s amazing how early it started too: I still remember spending weeks writing a Basic-to-assembly translator in 1986, before I had even heard the word “compiler”.

Now I just want to make games.

I’ll gladly use code other people have written as long as they fit my needs. For example, I’m very happy with Box2d in Casey’s Contraptions. Was Box2d perfect? Far from it, but it was good enough and I’m very happy with the tradeoff.

So how come that Casey’s Contraptions, apart from Box2d, is all custom code, written in C and OpenGL? Why didn’t I use some of the great tools and libraries available like Cocos2d or Unity?

In the land of AAA console games, developers routinely pull off the performance card: Only our code will allow us to make something that is so tailored to the hardware that it will make the game stand out above all other games. But for Casey’s Contraptions that’s not much of an issue: Any of the middleware options would be able to handle that amount of performance on an iPad just fine.

It wasn’t a lack of features either. We’re not doing anything particularly fancy, and I’m sure it would be possible to implement it in almost any other framework.

So if it’s not performance of features, why not?

Creating a game is a craft. It’s a mix of art and technique, and the craftsman or artisan has a vision and slowly probes, guides, and implements the final product. As with most crafts, the tools you use are extremely important and will directly affect the final product. If I’m going to change my tools and my techniques, the payoff I get needs to be huge to make up for the change. It’s as if I give up a set of hand woodworking tools for a power tool. Maybe it will be faster when I’m cutting in a straight line, but it will be very different and will completely change what I create.

As a game developer, my main tool is the language I use (C with a dash of C++). Things like editors or IDEs are secondary, and I can easily adapt to different ones without much of a problem (like when going from Visual Studio to Xcode). The way I like to work involves fast iterations, unit tests, very explicit memory layout, and processing data at a global, rather than local (entity) level.

I feel that by going to Unity or Cocos2d, I would have to completely give that up. A lot of middleware are frameworks instead of toolkits. That means I’m restricted to implementing bits of code that are embedded in a larger system and called by the rest of the framework. I give up the ability to architect the game in the way I want. Maybe it wouldn’t be a big deal if I liked a component or inheritance-based, entity-centered approach like a lot of frameworks, but I don’t.

Will I get enough benefits from using one of those frameworks to make up for the loss of productivity and techniques I like to use? Doubtful. All of a sudden I can’t just use fwrite on a memory block for serialization, I have to parse data files to load game state, undo/redo isn’t as simple as copying a block of memory, rendering becomes much more generic and less tied to the game, I can’t easily do test-driven development, iterate quickly, etc.

Sometimes, shifting workflows and techniques is inevitable. You may be used to solving collisions in a particular way, but when you move to a full physics engine, you need to process things differently. The payoff you get from using a pre-made physics engine is huge, so it’s worth it in my book.

For 2D games, there isn’t that much that I need: cross-platform support, simple math functions, ways to load textures, play sounds, and render simple things. Throw in some kind of simple UI display and layout tool, and that’s it. The rest I can handle. Actually, that’s not true, let me rephrase that: The rest I prefer to do myself, because it’s always highly dependent on the game.

What I really want is a toolkit-based middleware that does exactly those things without imposing any workflow or architecture on me. It saves me from doing the boring bits, gets out of the way as much as possible, and lets me work the way I want to. That’s already what I have with my current code, minus the UI library/tool part, which I’m still stuck using mostly UIKit. The idea of using Unity just so I can have cross-platform UI seems total overkill.

Even in the case of middleware that fits my style, I still need to be weary of the quality of the code. The last thing I want to do is trade time implementing my own code for time debugging someone else’s crappy code. Also, having learned my lesson years ago, I’ve sworn that I will never use a middleware/library that I don’t have source code access to. Nothing more frustrating than having to reverse-engineer UIKit classes to work around their bugs just because I don’t have the source code.

I’m not even going to go in the debate about middleware lock-in, future portability, building tech vs. relying on it, and financial aspects. All those are important, but secondary to how using some middleware will affect my everyday development.

If my next game were a networked, 3D game involving moving some kind of character or vehicle around a world, and I had nothing to start from, then the pain of adopting a foreign workflow and architecture might be worth it. Might. Even though it might be a wash on the code side, the tools and pipeline might be the thing that sways me over to adopt it.

In the end, as an indie developer, I chose to make games because it’s something I enjoy doing. I get up every morning excited about it, and I enjoy the development process. Whichever platform, tool, middleware or process I end up choosing, it needs to be enjoyable. The day I’m not having fun making my own games, I shouldn’t be an indie anymore.

What do you think? Am I being overly paranoid? Is there a good solution out there that’s multiplatform and will double my productivity once I get used to it? Has anyone written their own code for a 2D game and later switched to Cocos2d/Unity and liked it?

For those of you who have been living under a rock and haven’t seen the Casey’s Contraptions video, go watch it now. I’ll wait. Or even better, here it is. You don’t even have to leave this page:

The core interaction loop of Casey’s Contraptions gameplay is placing some items, pressing the Play button, seeing the simulation, and repeating based on what you learned from seeing the simulation. We designed this loop to be very tight and without any penalty: Running/stopping the simulation is instant, there’s no limit on the number of times you run it, and you can even stop the simulation by tapping anywhere on the screen. Even if you painted yourself in a wall by creating a solution that needs a very specific placement of items on the screen, making small changes is very quick and painless.

There’s a very important, underlaying assumption in that loop: Running the same simulation multiple times will result in the same behavior. Imagine how frustrating it would be to create a complex chain reaction, just to find out it only works every other time you run it. That would truly deserve a ONE STAR WANT MY MONEY BACK!!

Determinism

That’s referred to as the game being deterministic: Given the same inputs, it produces the same outputs. It’s not the first time I’ve had to deal with that. Actually, it seems that I’ve had to deal with that in one form or another for all my games in the last 10 years (except for Flower Garden). Most recently, it was a key component of gameplay for the unreleased game we were working on at Power of Two Games.

To make a game deterministic, you need to remove the obvious sources of “accidentally different” inputs. That’s usually random number generators, and making sure the initial state is truly the same. It’s too easy to have some leftover state from the previous simulation that throws things off a little bit.

Another potential source of problems are uninitialized variables. If at some point the simulation depends on an uninitialized variable, different runs will cause different results based on whatever value happened to be in memory at that time.

Once you fix all of those little things, that should be it, right? Same input should create the same output. Not really. We’re missing the most crucial input: The timer.

Casey’s Contraptions runs at 60 fps on an iPad one, but the timestep isn’t set to a fixed 16.667ms. Instead, I use a high-resolution timer to measure how much time has really elapsed since last frame, and I advance the simulation by that much.

The problem is that the timer doesn’t always return the exact same amount. It’s not super-precise, and it can be slightly affected by other things going on in the device. It won’t be off by more than 1/10th of a ms, but that’s enough to cause different results and start diverging the simulation.

Fixed timestep

One way to get around this problem is to hardcode the timestep to always be 16.667ms, independently of what it really was. That would fix the determinism problem, but it would add its own drawbacks. If the simulation can’t keep up with 60 fps, the game will appear to slow down, which is not the effect we want. Casey’s Contraptions includes a level editor to make your own free-form contraptions, and even though we have a cap in the maximum number of items you can add, it’s probably possible to add enough of them to start slowing down an iPad 1.

Recording timesteps

A tempting solution (and one I’m ashamed I even briefly tried), is to record the exact timestep during the first run of the simulation. Then, in subsequent runs when nothing has changed, we use the initially recorded stream of timesteps instead of the real ones from the clock.

Apart from being a clumsy-looking solution (I hate having two modes for doing the same thing), the solution only works in the case of replaying the exact same input. This approach falls apart if you make a change to an item that doesn’t affect the simulation until after several seconds have passed. At that point, you’re running a new simulation with new timesteps and everything can diverge before the affected object is changed. Bad, idea. Bad.

Fixed simulation timesteps

This is the correct solution and it’s what we’re doing in Casey’s Contraptions.

You probably already know that physics simulations are a lot more stable if you take small steps, always of the same size. So you really never want to advance your physics simulation by your frame time. Instead, you want to have a small and fixed simulation step (we’re running at 120Hz, but 200 or even 300 are not unheard of). Then, you run the physics simulation as many times as you can fit into your larger timestep. It’s also important to keep around the amount of leftover time you haven’t simulated, to apply it to the next frame.

The simulation loop code looks something like this:

	const float timestep = 1.0f/120.0f;
	accumulator += dt;
	while (accumulator >= timestep)
	{
		// Do physics simulation by timestep
		accumulator -= timestep;
	}

Once we have this loop in place, our real frame time doesn’t matter anymore. The physics simulation will either run for a full step (or multiple steps), or it won’t. There’s no half-way states. So it doesn’t matter if in the first run of the simulation the loop gets executed 3 times in the first frame and 2 times in the second, and in the next simulation is 2 times in the first and 3 in the second. The state of the world will be the same.

Glenn Fiedler writes about this kind of simulation loop in much more detail in this excellent article.

Jitterbug

There’s still a slight problem with the above loop. Nothing horrible; the simulation is deterministic at this point (assuming everything else is taken care of correctly). But you might notice some annoying jittering as things move around.

That’s because how we’re sampling the simulation. We might render a frame after 3 steps of the physics simulation, but another one after 2. So movement isn’t going to be very smooth.

To get around it, we need to interpolate the state of the world to match the time at which the frame is rendered. To do that, we need to simulate one timestep in the future, and then interpolate between the previous one and the future one by the percentage amount left in the accumulator.

Something like this:

	const float timestep = 1.0f/120.0f;
	accumulator += dt;
	while (accumulator >= timestep)
	{
		GameState lastState = gameState;
		// Do physics simulation by timestep
		accumulator -= timestep;
	}

	const float t = accumulator/timestep;
	GamePhysicsUtils::LerpState(interpolatedState, prevState, gameState, t);

The interpolation is just the positions and rotations of every item in the world. That’s only used for rendering, so no need to worry about velocities or forces (unless those are affecting rendering in a very obvious way too).

This is when it comes really handy to have the game state as a contiguous, relocatable block of memory. You can really easily copy states around without having to worry about allocating any memory or fixing any pointers. Yes, I really like my simple, pre-allocated data.

One added benefit of this approach: If you do it right, you can have extreme slow-motion in your game and everything will be right. Even if you only run one physics simulation loop every 10 rendering frames, the interpolation will take care of making sure everything is super smooth.

Odds and ends

A couple gotchas to watch out for (that’s code speak for “I didn’t think of this and it bit me hard”).

If you have any game logic that affects the physical state of your game objects, it needs to be executed in the inner, simulation loop. Anything that deals with visual or audio stuff can go in the regular update loop and only be executed once per frame.

A good example of this was the balloon code. Every frame, I iterate through all existing balloons, and apply some upward force based on buoyancy and other factors. Initially that was done once per frame, after the physics simulation, but that’s obviously wrong because it will leave balloons (and everything else) in a different state depending on your frame delta time. I moved it to the inner simulation loop and everything became deterministic again.

This second gotcha sounds silly, but it was a big deal: Make sure you deal with the situation where you have different numbers of items in the previous game state than in the next game state. For example, if a balloon is destroyed, between the two states, the item count will be different. In my case I don’t free any memory, but I was initially making the assumption that I could interpolate based on the index of the items on both states. But since I removed the balloon and pushed all the items behind it forward, the interpolation was totally wrong.

I solved that problem by marking items as invalid, and removing them only after the inner simulation loop was complete. That allowed me to still interpolate in a very straightforward way and not have to worry about missing items.

Taking it further

Not only are we relying on this determinism while you play a level of Casey’s Contraptions, but also to let you replay solutions from your friends. We simply load their initial layout and run the simulation locally.

[Pause]

Ah, yes, astute reader, you’re right to be puzzled about that. How do we make the simulation deterministic across platforms and versions? Good question. Stay tuned.

We’ve all had things nagging us in the back of our minds. They’re nothing we have to worry about this very instant, just something we need to do sometime in the future. Maybe that’s changing those worn tires in the car, or making an appointment with the dentist about that tooth that has been bugging you on and off.

Dynamic memory allocation is something that falls in that category for most programmers. We all know we can’t just go on allocating memory willy-nilly whenever we need it, yet we put off dealing with it until the end of the project. By that time, deadlines are piling on the pressure and it’s usually too late to make significant changes. With a little bit of forethought and pre-planning, we can avoid those problems and be confident our game is not going to run out of memory in the most inopportune moment.

On-Demand Dynamic Memory Allocation

The easiest way to get started with memory management is to allocate memory dynamically whenever you need it. This is the approach many software engineering books consider as ideal and it’s often encouraged in Computer Science classes.

It’s certainly an easy approach to use. Need a new animation instance when the player is landing on a ledge? Allocate it. Need a new sound when we reach the goal? Just allocate another one!

On-demand dynamic memory allocation can help to keep memory usage to a minimum, because, you only allocate the memory that you need and no more. In practice it’s not quite as neat and tidy because there can be a surprisingly large amount of overhead per allocation, which adds up if programmers become really allocation-happy.

It’s also a good way to shoot yourself in the foot.

Games don’t live inside a Computer Science textbook, so we have to deal with real world limitations, which make this approach cumbersome, clunky, and potentially disastrous. What can go wrong with on-demand dynamic memory allocation? Almost everything!

Limited Memory

Games, or any software for that matter, run on machines with limited amounts of memory. As long as you know what that limit is, and you keep extremely careful track of your memory usage, you can remain under the limit. However, since the game is allocating memory any time it needs it, there will most likely come a time when the game tries to allocate a new block but there is no memory left. What can you do then? Nothing easy I’m afraid. You can try to free an older memory block and allocate the new one there, or you can try to make your game tolerant to running out of memory. Both those solutions are very complex and difficult to implement correctly.

Even setting memory budgets and sticking to them can be very difficult. How can a designer know that a given particle system isn’t going to run out of memory? Are these AI units going to create too many pathfinding objects and crash the game? Hard to say until we run the game in all possible combinations. And even then, how do you know it isn’t going to crash five minutes later? Or ten? It’s almost impossible to know for certain.

If you insist in using this approach, at the very least, you should tag all memory allocations, so you have an idea of how memory is being used. You can either tag each allocation based on what system initiated it (physics, textures, animation, sound, AI, etc) or even on the filename where it originated, which has the advantage that it can be automated and should still give you a good picture of the overall memory usage.

Memory Fragmentation

Even if you take lots of pain not to go over your the available memory, you might still run into trouble because of memory fragmentation. You might have enough memory for a new allocation, but in the form of many small memory blocks instead of a large contiguous one. Unless you provide your own memory allocation mechanism, fragmentation is something that is very hard to track on your own, so you can’t even be ready for it until the allocation fails.

Virtual Memory

Virtual memory could solve all those problems. In theory, if you run out of real memory, the operating system swaps out some older, unused pages to disk and makes room for the new memory you requested. In practice, it’s just a bad caching scheme because it can be triggered at the worst possible moment, and it doesn’t know about what data it’s swapping out or how your game uses it.

Games, unlike most other software, have a “soft realtime” requirement: The game needs to keep updating at an acceptable interactive rate, which is somewhere around 15 or more frames per second. That means that gamers are going to make a trip to the store to return your game if it pauses for a couple of seconds every few minutes to “make some room” for new memory. So relying on virtual memory isn’t a particularly attractive solution.

Additionally, lots of games run in platforms with fixed amounts of RAM and no virtual memory. So when memory runs out, things won’t get slow and chuggy, they’ll crash hard. When the memory is gone, it’s really gone.

Performance Problems

There are some performance issues that are relatively easy to track down and fix. Usually ones that occur every frame and are happening in a single spot: some expensive operation, a O(n3) algorithm, etc. Then there are performance problems introduced by dynamic memory allocations, which can be really hard to track down.

Standard malloc returns memory pretty quickly, and usually doesn’t ever register on the the profiler. Every so often though, whenever the game has been running for a while and memory is pretty fragmented, it can spike up and cause a significant delay for just a frame. Trying to track down those spikes has caused more than one programmer to age prematurely. You can avoid some of those problems by using your own memory manager, but don’t attempt to write a generic one yourself from scratch. Instead start with some of the ones listed in the references.

Malloc spikes are not the only source of performance problems. Allocating many small blocks of memory can lead to bad cache coherence when the game code access them sequentially. This problem usually manifests itself as a general slowdown that can’t be narrowed down in the profiler. With today’s hardware of slow memory systems and deep caches, good memory access patterns are more important than ever.

Keeping Track Of Memory

Another source of problems with dynamic memory allocation are bugs in the logic that keeps track of the allocated memory blocks. If we forget to free some of them, our program will have memory leaks and has the potential to run out of memory.

The flip side of memory leaks are invalid memory access. If we free a memory block and later we access it as if it were allocated, we’ll either get a memory access exception, or we’ll manage to corrupt our own game.

Some techniques, such as reference counting and garbage collection can help keep track of memory allocations, but introduce their own complexities and overhead.

Introducing Pre-allocation

On the opposite corner of the boxing ring is the purely pre-allocated game. It excels at everything that the dynamically-allocated game is weak at, but it has a few weaknesses of its own. All in all, it’s probably a much safer approach for most games though.

The idea behind a pre-allocation memory strategy is to allocate everything once and never have to do any dynamic allocations in the middle of the game. Usually you grab as big a block of memory as you can, and then you carve it out to suit your game’s needs.

Some advantages are very clear: no performance penalties, knowing exactly how your memory is used, never running out of memory, and no memory fragmentation to worry about. There are some other more subtle advantages, such as being able to put data in contiguous areas of memory to get best cache coherency, or having blazingly-fast load times by loading a baked image of a level directly into memory.

The main drawback of pre-allocation is that is more complex to implement than the dynamic allocation approach and it takes some planning ahead.

Know Your Data

For preallocation to work, you need to know ahead of time how much of every type of data you will need in the game. That can be a daunting proposition, especially to those used to a more dynamic approach. However, with a good data baking system (see last month’s Inner Product column), you can get a global view of each level and figure out how big things need to be.

There is one important design philosophy that needs to be adopted for preallocation to work: Everything in the game has to be bounded. That shouldn’t feel too restrictive; after all, the memory in your target platform is bounded, as well as every single resource. That means that everything that can create new objects, including high-level game constructs, should operate on a fixed number of them. This might seem like an implementation detail, but it often bubbles up to what’s exposed to game designers. A common example is an enemy spawner. Instead of designing a spawner with an emission rate, it should have a fixed number of enemies it can spawn (and potentially reuse them after they’re dead).

Potentially Wasted Space

If you allocate enough data for the worst case in your game, that can lead to a lot of unused data most of the time. That’s only an issue if that unused data is preventing you from adding more content to the game. We might initially balk at the idea of having 2000 preallocated enemies when we’re only going to see 10 of them at once. But when you realize that each of those enemies is only taking 256 bytes and the total overhead is 500 KB, which can be easily accommodated in most modern platforms today.

Preallocation doesn’t have to be as draconian as it sounds though. You could relax this approach and commit to having each level preallocated and never having dynamic memory allocations while the game is running. That still allows you to dynamically allocate the memory needed for each level and keep wasted space to a minimum. Or you can take it even further and preallocate the contents of memory blocks that are streamed in memory. That way each block can be divided in the best way for that content and wasted space is kept to a minimum.

Reuse, RecycleM

If you don’t want to preallocate every single object you’ll ever use, then you can create a smaller set, and reuse them as needed. This can be a bit tricky though. First of all, it needs to be very much specific to the type of object that is reused. So particles are easy to reuse (just drop the oldest one, or the ones not in view), but might be harder with enemy units or active projectiles. It’s going to take some game knowledge of those objects to decide which ones to reuse and how to do it.

It also means that systems need to be prepared to either fail an allocation (if your current set of objects is full and you don’t want to reuse an existing one), or they need to cope with an object disappearing from one frame to another. That’s a relatively easy problem to solve by using handles or other weak references instead of direct pointers.

Then there’s the issue that reusing an object isn’t as simple as constructing a new one. You really need to make sure that when you reuse it, there’s nothing left from the object it replaced. This is easy when your objects are just plain data in a table, but can be more complicated when they’re complex C++ classes tied together with pointers. In any case, you can’t apply the Resource Acquisition Is Initialization (RAII) pattern, but it doesn’t seem to be a pattern very well suited for games, and it’s a small price to pay for the simplicity that preallocation provides.

Specialized Heaps

Truth be told, a pure pre-allocated approach can be hard to pull off, especially with highly dynamic environments or games with user-created content. Specialized heaps is a combination of dynamic memory allocation and pre-allocation that takes the best of both worlds.

The idea behind specialized heaps is that the heaps themselves are pre-allocated, but they allow some form of specialized dynamic allocation within them. That way you avoid the problems of running out of memory, or memory fragmentation globally, but you still can perform some sort of dynamic allocation when needed.

One type of specialized heaps is based on the object type. If you can guarantee that all objects allocated in that heap are going to be of the same size, or at least a multiple of a certain size, memory management becomes much easier and less error prone, and removes a lot of the complexity of a general memory manager.

My favorite approach for games is to create specialized heaps based on the lifetime of the objects allocated in them. These heaps use sequential allocators, always allocating memory from the beginning of a memory block. When the lifetime of the objects is up, the heap is reset and allocations can start from the beginning again. The use of a simple sequential allocator bypasses all the insidious problems of general memory management: fragmentation, compaction, leaks, etc. See the code in http://gdmag.com/resources/code.htm for an implementation of a SequentialAllocator class.

The heap types most often used in games are:

• Level heap. Here you allocate all the assets and data for the level at load time. When the level is unloaded, all objects are destroyed at once. If your game makes heavy use of streaming, this can be a streaming block instead of a full level.
• Frame heap. Any temporary objects that only need to last a frame or less get allocated here, and destroyed at the end of the frame.
• Stack heap. This one is a bit different from the others. Like the other heaps, it uses a sequential allocator and objects are allocated from the beginning, but instead of destroying all objects at once, it only destroys objects up to the marker that is popped fro the stack.

What About Tools?

You can take everything I’ve written here, and (almost) completely ignore it for tools. I fall in the camp of the programmers who consider the runtime as a totally separate beast from the tools. That means that the runtime can be lean and mean and minimalistic, but I can relax and use whatever technique makes me more productive when writing tools. That means you can allocate memory any time you want, you can use complex libraries like STL and Boost, etc. Most tools are going to run on a beefy PC and a few extra allocations here and there won’t make any difference.

Be careful with performance-sensitive tools though. Tools that build assets or compute complex lighting calculations might be a bottleneck in the build process. In that case, performance becomes crucial again and you might want to be a bit more careful about memory layout and cache coherency.

On the other hand, if the tool you’re writing is not performance sensitive, you should ask yourself if it really needs to be written in C++. Maybe C# or Python are better languages if all you’re doing is transforming XML files or verifying that a file format is correct. Trading performance for ease of development is almost always a win with development tools.

Next time you reach out for a malloc inside your main loop, think about how it can fail. Maybe you can pre-allocate that memory and stop worrying about what’s going to happen the day you prepare the release candidate.

This article was originally printed in the February 2009 issue of Game Developer.

Google App Engine Costs

At the time I wrote that post, the bandwidth of Flower Garden (and Flower Garden Free) was just under the free bandwidth allowance, so I didn’t have to pay anything. However, this last week, after riding the Pocket Frogs rocket, Flower Garden Free shot up to the #56 overall app in the US. And boy, did that make a difference in server usage!

That screenshot shows the Google App Engine dashboard with the number of requests per second for a 4 day period. Can you guess when the Pocket Frogs cross-promotion started? It went from an average of 1.5 requests per second, to a peak of about 16. That’s a factor of 10 right there.

Let’s see what that means in terms of cost. With the Google App Engine, you get charged for different resources you consume: CPU usage, incoming bandwidth, outgoing bandwidth, data storage, and recipients emailed. Of those, the only one that Flower Garden makes any dent on is outgoing bandwidth (mostly the HTML pages and images in the Flower Shop).

For each of those resources, you get a free amount every day. In the case of outgoing bandwidth, we get 1GB before we have to start paying anything. Afterwards, it’s $0.12 per GB. The peak day for Flower Garden Free, when it was all the way at #56, it consumed 7.8 GB. Now that it’s back down, it’s using up at around 2-3 GB of outgoing data every day.

That means, that the busiest day I was charged $0.82 in bandwidth, but the profits for that day were over $1500! I can only hope for days just as busy!

As a reference, these are my Google App Engine charges for the busy week during the promotion. $3.28 for the week? Bring it on! :-)

Minimizing Bandwidth

When I first saw the first day’s bandwidth, I was pretty surprised it was taking up that much. I hadn’t tried to minimize it first, but 8 GB seemed like a significant amount.

The first thing I did was to forward any requests to the “More Games” and “News” section back to the Dreamhost account. After all, if that server goes down, it doesn’t really matter, and they can take up significant bandwidth with all the images.

To forward something, I couldn’t just do it from the configuration file. Instead, I set a handler for the moregames directory like this:

- url: /moregames/.*
  script: redirect.py

And the redirect script is very simple:

import os
import sys
import logging
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class RedirectMoreGames(webapp.RequestHandler):
	def get(self, params):
		url = "http://flowers.snappytouch.com/moregames/Snappy_Touch_More_Games/More_Games.html"
		self.redirect(url, permanent=True)
	
application = webapp.WSGIApplication(
							[(r'/(.*)', RedirectMoreGames),
							 ],
							debug=True)

def main():
	run_wsgi_app(application)

if __name__ == "__main__":
	main()

Another thing you can do is to turn on explicit caching of files in the app.yaml file for your Google App Engine app. I don’t know how much that helps with data requested from within an iPhone app, but it can’t hurt.

Finally, I made some changes to Flower Garden. I never had bandwidth minimization in mind, so for example, I was happily requesting updated news and other data at the beginning of every session. I wised up a bit and now it only requests that data at most once every few hours or days.

In case you’re ever getting close to using up your quotas, I learned that you can query them at runtime. I haven’t done anything about it yet, but I suppose I could start forwarding image fetches to a backup server whenever I get close to the limit. But for that, I’ll have to sell many more units of Flower Garden!

Google App Engine Shortcomings

Overall, I’m very happy with the Google App Engine. But it’s not all a bed of roses over here. I think the Google App Engine excels at scaling under heavy load, but surprisingly, it doesn’t have anything close to 100% uptime. I’m going to say it probably doesn’t even have 95% uptime! With such a massive system and all the redundancy underneath, I’m surprised they can’t have better uptime. Supposedly their status page makes you think everything is perfect, but it’s really far from it.

Apart from downtime, people are also reporting severe latency issues sometimes, and I’ve seen some errors like that in my log every so often. Chalk it up to being a beta I suppose. I imagine it’s only going to get better with time.

I would still use the Google App Engine for any new game I’ll release in the future, whether it has big or small backend needs. If nothing else, the ease of development and testing beats every alternative for me.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Lag In Games

I’m defining lag as the time elapsed between the moment the player performs an input action (press a button, touch the screen, move his finger), until the game provides some feedback for that input (movement, flash behind a button, sound effect).

Mick West wrote a great article on the causes of lag in games, followed up by another one in how to measure it. I’m going to apply some of that to the lag I was experiencing in my game.

Lag can be introduced in games by many different factors:

• Delay between gathering input and delivering it to the game. Delay updating the simulation to reflect new inputs.- Delay rendering simulation state.
• Delay displaying the latest rendered state on screen.

The new game runs on the iPad and involves moving objects around the screen with your finger. To make sure it wasn’t anything weird with the rest of the game code, I wrote a quick (and ugly!) program that draws a square with OpenGL that follows your finger on the screen. When you run the sample, the same lag becomes immediately obvious.

The iPad is a much larger device than the iPhone, and it encourages a physical metaphor even more. As soon as you attempt to move an “object” on screen, the lag kills that sense of physicality. Instead of moving an object around with your finger, you’re dragging it around with a rubber band. It moved the player from applying direct action on screen, to being removed and disassociated with the actions on screen.

Loop Structure

The place to start looking for lag is in my main loop. The main loop looks something like this:

	ProcessInput();
	UpdateSimulation();
	RenderWorld();
	PresentRenderBuffer();

So I was reading the input correctly before the simulation. Nothing weird there.

Touch input is delivered to the code as events from the OS. Whenever I received those events (outside of the main loop), I queue them, and then process them all whenever the main loop starts in ProcessInput().

The loop runs at 60Hz, so the lag here is at most 16.7 ms (if you’re running at 30Hz, then you’re looking at a delay up to 33.3ms). Unfortunately, the lag I was seeing in the game was way more than one frame, so there was to be something else.

Rendering

For some reason, I thought that iDevices were triple buffered. I ran some tests and fortunately it looks like it’s regular double buffering. That means that if I render a frame and call presentRenderBuffer(), the results of that render will be visible on screen at the next vertical sync interval. I’m sure there’s a slight lag with the iPad screen, but I’m willing to be is close to negligible when we’re talking about milliseconds, so we’ll call that zero.

Main Loop Calls

The game uses CADisplayLink with interval of 1, so the main loop is called once every 16.7 ms (give or take a fraction of ms). I thought that perhaps CADisplayLink wasn’t playing well with touch events, so I tried switching to NSTimer, and even to my old thread-driven main loop, but none of it seemed to make any difference. Lag was alive and well as always.

That the simulation and rendering in the game are very fast, probably just a few ms. That means the rest of the system has plenty of time to process events. If I had a full main loop, maybe one of the two other approaches would have made a difference.

It looks like the lag source had to be further upstream.

Input Processing

On the dashboard, press and hold on an icon, now move it around the screen. That’s the same kind of lag we have in the sample program! That’s not encouraging.

A touch screen works as a big matrix of touch sensors. The Apple OS processes that input grid and tries to make sense out of it by figuring out where the touches are. The iOS functions eventually process that grid, and send our programs the familiar touchesBegan, touchesMoved, etc events. That’s not easy task by any means. It’s certainly not like processing mouse input, which is discrete and very clearly defined. For example, you can put your whole palm down on the screen. Where are the touches exactly?

TouchesBegan is actually a reasonably easy one. That’s why you see very little lag associated with that one. Sensors go from having no touch values, to going over a certain threshold. I’m sure that as soon as one or two of them cross that threshold, the OS identifies that as a touch and sends up the began event.

TouchesMoved is a lot more problematic. What constitutes a touch moving? You need to detect the area in the sensor grid that is activated, and you need to detect a pattern of movement and find out a new center for it. In order to do that, you’ll need several samples and a fair amount of CPU cycles to perform some kind of signal processing on the inputs. That extra CPU usage is probably the reason why some games get choppier as soon as you touch the screen.

Measuring Lag

Measuring lag in a game is tricky. You usually can’t measure it from within the code, so you need to resort to external means like Mick did in his tests.

I decided to do something similar. I pulled out my digital video camera, and started recording my finger moving on the screen. The quality leaves much to be desired, but it’s good enough for the job. I can see how far my finger gets from the center of the square, but that’s not enough information to quantify the lag. How fast is my finger moving exactly? Fortunately, that’s something I can answer in code, so I added that information to the screen [1]. Now, for a given frame, I can see both how far the finger is from the center of the square and how fast it’s going.

The square is 100 pixels wide. When I move my finger at about 500 pixels per second, the center of my finger is on the edge of the square. That makes a rough 100 ms total delay from input until it’s rendered. That’s a whopping 6 full frames at 60 fps!

What Can We Do About It

As iOS developers, there isn’t much we can do. Make sure your loops are set up correctly to avoid an extra frame delay. Make sure you provide feedback as soon as you can and don’t delay it any longer than you have to. Other than that, there’s nothing much we can do.

I’ve been saying this for a while, but I’m a big fan of layers as long as you can get to the underlying layers when you need to. Here’s a perfect case where it would be fantastic if Apple gave us raw access to the touch matrix input. Apart from being able to process the input faster (because I know what kind of input to expect for the game), can you imagine the possibilities that would open up? Input wouldn’t be limited to touch events, and we could even sense how “hard” the user is pushing, or the shape of the push.

At the very least, it would be very useful if we had the option to allocate extra CPU cycles to input recognition. I’m not doing much in my game while this is going on, so I’d happily give the input recognition code 95% of the frame time if it means it can give me those events in half the time.

I’m hoping that in a not very far distant, iDevices come with multiple cores, and maybe one of those cores is dedicated to the OS and to do input recognition without affecting the game. Or maybe, since that’s such a specialized, data-intensive task, some custom hardware could do the job much faster.

Until then, we’ll just have to deal with massive input lag.

How about you? Do you have some technique that can reduce the touch event lag?

LagTest source code. Released under the MIT License, yadda, yadda, yadda…

[1] I actually shrank the OpenGL view to make sure the label wasn’t on top if it because I was getting choppier input than usual. Even moving it there caused some choppiness. This is exactly what I saw last year with OpenGL performance dropping when a label is updated every frame!

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Flower Garden was using a shared Dreamhost server to upload bouquets, host in-game news, and, most importantly, to serve the Flower Shop (which involves hosting the static shop files, redeeming promo codes, recording transactions, verifying receipts with Apple’s server, and delivering purchased content). All the Flower Shop functionality was implement in PHP and using a mySQL database.

After some stressful sorting through the logs, it looked like the server was running out of memory and killing PHP processes. Dreamhost’s technical support didn’t help much and just claimed that I was simply something that happened and they would be happy to sell me a virtual private server. I was surprised to hear it was using up a lot of memory, and certainly nothing I could see with the tools available to me showed that. I certainly hadn’t done any changes in a while and traffic was constant. Doing some Googling at the time brought up lots of other angry Dreamhost users in a similar situation, so I suspected server configuration problems.

Whatever the case, I couldn’t let it be that way and I had to fix it somehow. I briefly considered a virtual private server, or even a dedicated host in some highly recommended providers. But in the end, the simplicity, scalability, and affordability of the Google App Engine won me over.

Now that Flower Garden has been using the Google App Engine for several weeks, I still think it’s fantastic. I wish someone had whacked me on the head when I started writing server code and forced me to use it. Hopefully this post will be more pleasant than a hit to the head and will still have a similar (good) effect.

Google App Engine Overview

Even though I was pretty much completely new to the Google App Engine (I had looked into it briefly before the launch of Flower Garden but dismissed it because they couldn’t support the amount of email traffic I needed), it took me two days to port over everything. Actually flipping the switch from the old system to the new one took a while longer, but that required more courage than work. More on that later.

This is not a Google App Engine tutorial. Instead, it’s going to be an overview of what it can do and why you should consider it as a backend for your iPhone app instead of using a shared or even dedicated server.

The first shock (at least for me) when looking into the Google App Engine is that you can’t run PHP. You’re limited to Java or Python. It turns out Python is my go-to scripting language, so I was thrilled at the idea. PHP is fine, but Python is a whole class above.

Next, it took a bit of adjusting to the fact that the environment to run anything on the Google App Engine has to be carefully controlled. You need to create a configuration file indicating the entry points and their handlers. Fortunately, all of that is well covered in the extensive online documentation.

This is what the Flower Garden config file looks like, indicating both static files and several execution entry points:

application: st-flowershop
version: 1
runtime: python
api_version: 1

handlers:
- url: /catalog
  static_dir: catalog

- url: /emailassets
  static_dir: emailassets

- url: /news
  static_dir: news

- url: /moregames
  static_dir: moregames

- url: /.*
  script: purchase.py

If you hit one of the URLs listed as static_dir, you get those files directly (like the in-game news). Anything else is handled by the Python script purchase.py. Google even provides the webapp framework that easily allows you to connect different parts of the script to handle different requests and easily get to the input parameters.

Once you’re past that and you have the requisite “Hello World” up and running, then it’s all fun and games.

One of the great features of the Google App Engine is that it comes with a fully-featured, local environment. This environment is installed automatically with the Google App Engine SDK, and there’s nothing to configure. It’s certainly nothing like setting up Apache + mySQL + PHP in your dev station! That way, you can do all your work locally, and only update the servers when you’re confident everything is working.

Beyond this, the only other thing that is different from what you may be used to is the datastore. They provide a query language called GQL, which is not exactly SQL, but it’s very similar. Close enough that I had no trouble porting over my very simple queries anyway.

Adding data to a data store couldn’t be simpler. For example, to add a promo code, I need the following class definition:

class PromoCode(db.Model):
	code = db.StringProperty()
	productid = db.StringProperty()
	amount = db.IntegerProperty(default=1)
	singleuse = db.BooleanProperty(default=True)
	enabled = db.BooleanProperty(default=True)

And then I can add it this way:

	promocode = PromoCode()
	# fill it up here
	promocode.put()

How do I check if a promocode is valid? I can use the SQL-like syntax, or something even simpler:

	q = PromoCode.all()
	q.filter("code =", code)
	return q.get()

Once you know that, you can go to town with your Python programs.

Google App Engine Advantages

Scalability This was the big reason that forced me to move away from Dreamhost. I have no way of testing how scalable it really is, but I’m going to take Google’s word for it. They know a tiny little bit about scalability. And besides, Flower Garden is nothing compared to other apps (it serves about 1.5 requests per second on average).

Since I don’t know how things are implemented under the hood in the Google App Engine, I don’t have a good feel for performance best practices. So I’m hoping that my very simple queries (add to the end of a table, or see if a record is present) are just fine. If not, I can do some tuning down the line (and I won’t have to wait for any approval to make the code go live!).

Local environment The local environment is simply great. It makes developing a pleasure. You can add any test data you want to the local datastore through a web interface without affecting the live server. Probably my favorite feature is the console: Since the server is running locally, you can print out any debug info in the server code and you can see it live as you perform some action. That saved me hours of debugging compared to doing it on a remote server with php!

Server synching In the past, I’ve synced my server scripts through ftp and I kept meaning to write an automated script to do that. In this case, Google provides a script (and a GUI tool) to do the actual synching with the servers, which is great. It even waits a few seconds until it can verify that the new version is live on the server.

Python I’m not a big Java fan (although I used it a bit way back in the day when it first came out), but I do love Python. For a program with some complexity like this, it’s definitely a much better choice than PHP.

Real-time stats The Google App Engine web console displays real-time stats of access to your site. You see it expressed in requests per second and you can visualize it in the past few hours, days, or even month. You also have access to all the logs, and you can filter by type of log message (debug, info, warning, error).

Price This is one category where the Google App Engine definitely shines. Every day you’re given a free quota of bandwidth, disk usage, and CPU usage. If you stay under that quota you don’t pay anything. If you go over, you pay per unit as you go (at very reasonable prices). You can even set a maximum cap on daily spending so you don’t encounter any nasty surprises at the end of the month.

So far, Flower Garden has been hovering right at the edge of the outgoing bandwidth free quota (the rest of the metrics it doesn’t even come close). So even if traffic were to double, expenses would be very reasonable.

Development Tricks

Along the way, I figured out a couple of interesting tricks that helped me during development.

Bypass the App store for testing For me, few things are more annoying than doing development in the actual device. Iteration is slow and stepping with the debugger is mostly impossible (seems to depend on the mood of the debugger that day). So I try to do the bulk of the development on the simulator and just test on the device when things are ready.

Unfortunately, the device can’t access the App Store, which makes testing all the server functionality dealing with the App Store painful. What I did was to add a define that, when present, the program bypasses the App Store but still functions as usual.

Local server During development, you want to be using your local server. In my case, I set it up so that Debug versions of the code access the local server, but Release and Distribution ones access the live one.

There is one problem: The Google App Engine server, by default, binds itself to localhost, not to the IP address of your network card. That meant I was able to hit it from the simulator just fine accessing http://localhost:8080, but not from the device. In order to access the development server from the device, I had to explicitly tell the server to bind itself to another address like this:

dev_appserver.py --address 192.168.1.150 FlowerShop

Now I can access the server at 192.168.1.150:8080 from both the simulator and any device.

Switching servers Porting the code took just a couple of days. Making the switch on the live server was a lot scarier though. I couldn’t wait for a new update to be released because that seems to take over a week these days, so I forwarded requests from the old server to the new one.

I did it one system every day, avoiding weekends which is when there’s the most traffic. News was easy, and so was the web view with more games. But I ran into a snag with promo codes and Flower Shop purchases.

Initially, I was just redirecting requests in .htaccess this way:

Redirect permanent /Shop/catalog http://st-flowershop.appspot.com/catalog
Redirect 301 /Shop/catalog http://st-flowershop.appspot.com/catalog

As I learned the hard way, POST variables aren’t preserved in a redirect of that kind (I guess to avoid state being changed in multiple servers). I tried a bunch of things, but in the end, I wrote a quick php script that gathered all POST variables and re-posted them to the new request. A bit ugly, but it did the trick:

<?php
function PostRequest($url, $_data) {

    $data = array();    
    while(list($n,$v) = each($_data)){
        $data[] = "$n=" . urlencode($v);
    }    
    $data = implode('&', $data);
    // format --> test1=a&test2=b etc.

    $url = parse_url($url);
    if ($url['scheme'] != 'http') { 
        die('Only HTTP request are supported !');
    }

    $host = $url['host'];
    $path = $url['path'];

    $fp = fsockopen($host, 80);

    fputs($fp, "POST $path HTTP/1.1\r\n");
    fputs($fp, "Host: $host\r\n");
    fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
    fputs($fp, "Content-length: ". strlen($data) ."\r\n");
    fputs($fp, "Connection: close\r\n\r\n");
    fputs($fp, $data);

    $result = ''; 
    while(!feof($fp)) {
        $result .= fgets($fp, 128);
    }
    fclose($fp);

    $result = explode("\r\n\r\n", $result, 2);
    $header = isset($result[0]) ? $result[0] : '';
    $content = isset($result[1]) ? $result[1] : '';
    return $content;
}

print PostRequest("http://st-flowershop.appspot.com/purchase", $_POST);
?>

Conclusion

I hope this post is enough to at least make you interested in checking out the Google App Engine and help you get over a couple of the initial hurdles. I’ll definitely be using it in any future projects.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

A good game prototype is something fast/cheap that allows you to answer a specific question about your game. The key points there are fast/cheap and specific question. It’s not a level of a game, it’s not a “vertical slice”, and it’s certainly not an engine for the game.

Chris Hecker and Chaim Gingold gave one of the best presentations on the subject of rapid prototyping. It was hugely influential for me, and it made me re-think the way I do prototypes. If you get a chance, find the audio for the presentation, it’s definitely worth it.

Mistake #1: Going With The First Idea

Every company I’ve ever worked at has done this mistake. The team hashes out a bunch of ideas, and somehow they pick one (or create it by committee). Maybe they’ll create a prototype to show something about the game, or maybe they’ll dive straight and start writing a design document and developing technology. If you’re lucky, or you have an extremely talented game director, the game that comes out of the other end might be fantastic. In most cases, it’s just a so-so idea and the team only realizes it when the first level comes together, years later, at around alpha time. At this point the choice is canning a project after spending millions of dollars, or patching it up to try to salvage something. Neither idea is particularly appealing.

Creating a prototype for a game you know you’ve already committed to is pointless. It’s nothing more than an exercise to keep management happy. Frankly, I even made that same mistake at Power of Two, when we prototyped the game idea we had in mind and immediately moved on into pre-production (and yes, later we realized we had to change things to make it more interesting).

What I do now is to force myself to prototype several of my top ideas before committing to any one project. I have a page (actually, a wiki page) with every game idea or thought I have. That page has way over a hundred entries, and every so often I cull and reorganize it, bringing up the most promising ideas towards the top. Whenever I’m in prototyping mode, I start grabbing them from the top and prototype them.

With a good prototype it’s easy to see if an idea is worthwhile. If it’s not, I discard it and move on to the next one. If it has potential but it’s just so-so, I either choose to continue just a bit longer (to ask another, better question) or I shelve it back in the list of potential game ideas. Maybe at some later time, things might click in or I might have a new inspiration and the game idea might become a lot stronger.

Also, often times, after doing one prototype and deciding against it, a new idea will come up. Usually a variation on the original prototype or something directly sparked from it, so I’ll find myself jumping to that idea instead of one of the ones I had saved in my list.

Eventually, one idea will click and you’ll know that’s “the one”. If you’re lucky (or unlucky) enough to have that happen with the first one you try, I still recommend doing a few more prototypes. If nothing else, you might be prototyping future projects, so it’s certainly not wasted time. For my current project, I went through eight prototypes before finding “the one” (several of them were a collaboration with Miguel). Eight to ten prototypes per project is roughly what I’m hearing from other indies with this approach.

Mistake #2: Not having a good question

A good prototype attempts to answer a question about the game. But not all questions are created equal. First of all, a question needs to be relevant and crucial to the project. “Can I have a pretty settings screen?” isn’t a particularly difficult question to answer, so it doesn’t deserve its own prototype. “Can I control little planes by drawing smooth lines on the screen?” is a much bigger unknown (before Flight Control anyway, today you can just download the game and immediately answer yes).

Also, a good question is concise and can be answered in a fairly unambiguous way. “Is this game awesome?” isn’t a good question because “awesome” is very vague. A better question might be “Can I come up with a tilt control scheme that is responsive and feels good?”. Feels good is a very subjective question, but it’s concrete enough that people can answer that pretty easily after playing your prototype for a bit.

Even though most questions are about game design, they can also be about any other aspect of the game. Maybe you’re doing something tricky with technology and you want to make sure it’s feasible. If not, there’s no point in even starting. It’s more uncommon to think of prototyping art, but it’s also a very valid approach: “Will this art style allow foreground objects to stand out enough?” “Will the lighting approach allow the player to see important features in enough detail?”. Often you can do these art “prototypes” directly in Photoshop or a 3D modeling package.

In the case of Flower Garden, the main unknown was the technology behind the procedural flowers. So I created a prototype to answer the question “Can I create compelling procedural flowers that grow in real-time and the user can interact with them?”. The prototype had several parts to answer that question: the geometry generation, the animation, the simulation, and the rendering. There isn’t anything else particularly ground-breaking in the rest of the Flower Garden code, so as soon as I was able to answer “yes” to that question, I green-lighted the project and started production on it.

For larger projects, you might have several major, outstanding questions, so you’ll need to do multiple prototypes. Unless they’re very closely related, I find it easier to keep them separate instead of building on top of the same prototype.

Without a good question, it’s too easy for a prototype to go on for a long time. You feel you’re making progress because new things are added, but you have no real sense of when to stop or when it’s done. You really have to focus on the question, ignore everything else, and be merciless in your approach.

Mistake #3: Taking too long

One of the key concepts in the definition of a prototype was that it has to be fast/cheap (which are two sides of the same coin). What’s fast enough? It depends on the length of the project itself. It’s not the same thing to do a prototype for a two-month iPhone game, than for a three-year console game. Also, a larger, more expensive project probably has more complex questions to answer with a prototype than a simple iPhone game.

In my case, I shoot for one-day prototypes. If you already have the tech to create games with, one day allows you to focus 100% on answering the question. By the end of the day, or even before, I have a pretty good idea how the game is going to work out. My shortest prototype ever was a 2-hour one. I thought it was going to be longer, but I managed to complete everything to answer the question in two hours (for the record, I didn’t can that idea, but I shelved it for a possible future project).

Game jams are a great way to get over the mental block of doing quick prototypes. There you are focused on making the game on a very short time, surrounded by people trying to do the same thing. I can’t think of a more fun way to prototype than that!

Also, think beyond programming. Is there a faster way you can answer the prototype question? Maybe you can use the modding capabilities of an existing game, or even do a mockup with paper moving pieces around. Don’t fall in the trap of thinking you have to code a prototype if something simpler and faster will do.

If you find that a day is not enough, take a step back and really ask yourself why you need more time. Were you getting side-tracked on things that were irrelevant to the prototype (menus, tech, art, etc)? Are you asking a question that the prototype can’t answer? Do you have the game idea clearly defined in your head?

Mistake #4: Building a system, not a game

When you’re making a prototype, if you ever find yourself working on something that isn’t directly moving your forward, stop right there. As programmers, we have a tendency to try to generalize our code, and make it elegant and be able to handle every situation. We find that an itch terribly hard not scratch, but we need to learn how. It took me many years to realize that it’s not about the code, it’s about the game you ship in the end.

Don’t write an elegant game component system, skip the editor completely and hardwire the state in code, avoid the data-driven, self-parsing, XML craziness, and just code the damned thing.

When you’re prototyping, it’s a race to answer the main prototype question. Everything is expendable. Don’t even worry about memory leaks, hardwired numbers, or how you load resources. Just get stuff on the screen as quickly as you can.

And don’t ever, ever, use the argument “if we take some extra time and do this the right way, we can reuse it in the game”. EVER.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

One of the things you potentially give up by keeping the game runtime to a minimum is an editor built in the game itself. But that’s one of those things that sounds a lot better than it really is. From a technical point of view, having an editor in the game usually complicates the code a huge amount. All of a sudden you need to deal with objects being created and destroyed randomly (instead of through clearly defined events in the game), and you have to deal with all sorts of crazy inputs and configurations.

The worse part though, is having to implement some sort of GUI editing system in every platform. Creating the GUI to run on top of the game is not easy, requiring that you create custom GUI code or try to use some of the OpenGL/DirectX libraries available. And even then, a complex in-game GUI might not be a big deal on a PC, but wait and try to use that interface on a PS3 or iPhone. After all, making games is already complicated and time-consuming enough to waste more time reinventing the widget wheel.

Remote game editing manages to keep a minimal runtime and allow you to quickly create native GUIs that run on a PC. It’s the best of both worlds, and although it’s not quite a perfect solution, it’s the best approach I know.

Debug Server

Miguel Ángel Friginal already covered the basics of the debug server, so I’m not going to get in details there.

The idea is that you run a very simple socket server on the game, listening in a particular port. This server implements the basic telnet protocol, which pretty much means that it’s a line-based, plain-text communication.

The main difference between my debug server and Miguel’s (other than mine is written in cross-platform C/C++ instead of ObjC), is that I’m not using Lua to execute commands. Using Lua for that purpose is a pretty great idea, but goes against the philosophy of keeping the runtime as lean and mean as possible.

Instead, I register variables with the server by hand. For each variable, I specify its memory address, it’s type, any restrictions (such as minimum and maximum values), and a “pretty name” to display in the client. Sounds like a lot of work, but it’s just one line with the help of a template:

registry.Add(Tweak(&plantOptions.renderGloss, "render/gloss", "Render gloss"));
registry.Add(Tweak(&BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3))

And yes, if I were to implement this today, I would probably get rid of the templates and make it all explicit instead (ah, the follies of youth :-)

TweakUtils::AddBool(registry, &plantOptions.renderGloss, "render/gloss", "Render gloss");
TweakUtils::AddFloat(registry, &BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3);

The debug server itself responds to three simple commands:

• list. Lists all the variables registered in the server.
• set varname value. Sets a value.
• print varname. Gets the value for that variable.

For example, whenever the server receives a set command, it parses the value, verifies that it’s within the acceptable range, and applies it to the variable at the given memory location.

Telnet Clients

Because we used the standard telnet protocol, we can start playing with it right away. Launch the game, telnet into the right port, and you can start typing away.

However, most telnet clients leave much to be desired for this. They all rely on the history and cursor manipulation being handled by the shell they assume you’re connected to. Here we aren’t connected to much of anything, but I’d like to be able to push up arrow and get my last command, and be able to move to the beginning of the line or the previous word like I would do in any text editor. The easiest solution I found for that was to use a telnet client prepared for that kind of thing: A MUD client! Just about any will do, but one that works well for me is Atlantis.

So far, we’ve implemented the equivalent of a FPS console, but working remotely. And because the code is fully portable, our game can be in just about any platform and we can always access it from our PC. Not just that, but we can even open multiple simultaneous connections to various development devices if you need to run them all at once.

Custom Clients

Game parameter tweaking is something that is OK through a text-based console, but really comes into its own when you add a GUI. That’s exactly what we did at Power of Two Games. We created a generic GUI tool (based on WinForms since we were on Windows at the time), that would connect to the server, ask for a list of variables, and generate a GUI on the fly to represent those variables. Since we knew type and name of each variable, it was really easy to construct the GUI elements on the fly: A slider with a text field for floats and ints, a checkbox for bools, four text fields for vectors, and even a color picker for variables of the type color.

It worked beautifully, and adjusting different values by moving sliders around was fantastic. We quickly ran into two problems through.

The first one is that we added so many different tweaks to the game, that it quickly became unmanageable to find each one we wanted to tweak. So, in the spirit of keeping things as simple as possible (and pushing the complexity onto the client), we decided that the / symbol in a name would separate group name and variable name. That way we could group all related variables together and make everything usable again.

The second problem was realizing that some variables were changing on the runtime without us knowing it on the client. That created weird situations when moving sliders around. We decided that any time a registed variable changes on the server, it should notify any connected clients. That worked fine, but, as you can imagine, it became prohibitively expensive very quickly. To get around that, we added a fourth command: monitor varname. This way clients need to explicitly register themselves to receive notifications whenever a variable changes, and the GUI client only did it for the variables currently displayed on the screen.

During this process, it was extremely useful to be able to display a log console to see what kind of traffic there was going back and forth. It helped me track down a few instances of bugs where changing a variable in the client would update it in the server, sending an update back to the client, which would send it again back to the server, getting stuck in an infinite loop.

You don’t need to stop at a totally generic tool like this either. You could create a more custom tool, like a level editor, that still communicates with the runtime through this channel.

Flower Garden Example

For Flower Garden, I knew I was going to need a lot of knobs to tweak all those plant DNA parameters, so I initially looked into more traditional GUI libraries that worked on OpenGL. The sad truth is that they all fell way short, even for development purposes. So I decided to grab what I had at hand: My trusty tweaking system from Power of Two Games.

I’m glad I did. It saved a lot of time and scaled pretty well to deal with the hundreds of parameters in an individual flower, as well as the miscellaneous tweaks for the game itself (rendering settings, infinite fertilizer, fast-forwarding time, etc).

Unfortunately, there was one very annoying thing: The tweaker GUI was written in .Net. Sure, it would take me a couple of days to re-write it in Cocoa (faster if I actually knew any Cocoa), but as an indie, I never feel I can take two days to do something tangential like that. So instead, I just launched it from VMWare Fusion running Windows XP and… it worked. Amazingly enough, I’m able to connect from the tweaker running in VMWare Fusion to the iPhone running in the simulator. Kind of mind boggling when you stop and think about it. It also connects directly to the iPhone hardware without a problem.

VMWare Fusion uses up a lot of memory, so I briefly looked into running the tweaker client in Mono. Unfortunately Mono for the Mac didn’t seem mature enough to handle it, and not only was the rendering of the GUI not refreshing correctly, but events were triggered in a slightly different order than in Windows, causing even more chaos with the variable updates.

Here’s a time-lapse video of the creation of a Flower Garden seed from the tweaker:

Drawbacks

As I mentioned earlier, I love this system and it’s better than anything else I’ve tried, but it’s not without its share of problems.

Tweaking data is great, but once you find that set of values that balances the level to perfection… then what? You write those numbers down and enter them in code or in the level data file? That gets old fast. Ideally you want a way to automatically write those values back. That’s easy if the tool itself is the editor, but if it’s just a generic tweaker, it’s a bit more difficult.

One thing that helped was adding a Save/Load feature to the tweaker GUI. It would simply write out a large text-based file with all the variables and their current values. Whenever you load one of those, it would attempt to apply those same values to the current registered variables. In the end, I ended up making the Flower Garden offline seed file format match with what the tweaker saved out, so that process went pretty smoothly.

Another problem is if you want lots of real-time (or close to real time) updates from the server. For example, you might want to monitor a bunch of data points and plot them on the client (fps, memory usage, number of collisions per frame, etc). Since those values change every frame, it can quickly overwhelm the simple text channel. For those cases, I created side binary socket channels that can simply send real-time data without any overhead.

Finally, the last drawback is that this tweaking system makes editing variables very easy, but calling functions is not quite as simple. For the most part, I’ve learned to live without function calls, but sometimes you really want to do it. You can extend the server to register function pointers and map those to buttons in the client GUI, but that will only work for functions without any parameters. What if you wanted to call any arbitrary function? At that point you might be better off integrating Lua in your server.

Future Directions

This is a topic I’ve been interested in for a long time, but the current implementation of the system I’m using was written 3-4 years ago. As you all know by now, my coding style and programming philosophy changes quite a bit over time. If I were to implement a system like this today, I would do it quite differently.

For all I said about keeping the server lean and minimal, it could be even more minimal. Right now the server is receiving text commands, parsing them, validating them, and interpreting them. Instead, I would push all that work on the client, and all the server would receive would be a memory address, and some data to blast at that location. All of that information would be sent in binary (not text) format over a socket channel, so it would be much more efficient too. The only drawback is that we would lose the ability to connect with a simple telnet client, but it would probably be worth it in the long run.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

This was one of the first columns I wrote. At first glance it might seem a completely introductory topic, not worth spending that much time on it. After all, all experience programmers know about pointers and indices, right? True, but I don’t think all programmers really take the time to think about the advantages and disadvantages of each approach, and how it affects architecture decisions, data organization, and memory traversal.

It should provide a good background for this coming Thursday’s #iDevBlogADay post on how to deal with heterogeneous objects in a data-oriented way._

From a 10,000-Foot view, all video games are just a sequence of bytes. Those bytes can be divided into code and data. Code is executed by the hardware and it performs operations on the data. This code is generated by the compiler and linker from the source code in our favorite computer language. Data is just about everything else. [1]

As programmers, we’re obsessed with code: beautiful algorithms, clean logic, and efficient execution. We spend most of our time thinking about it and make most decisions based on a code-centric view of the game.

Modern hardware architectures have turned things around. A data-centric approach can make much better use of hardware resources, and can produce code that is much simpler to implement, easier to test, and easier to understand. In the next few months, we’ll be looking at different aspects of game data and how everything affects the game. This month we start by looking at how to manage data relationships.

Data Relationships

Data is everything that is not code: meshes and textures, animations and skeletons, game entities and pathfinding networks, sounds and text, cut scene descriptions and dialog trees. Our lives would be made simpler if data simply lived in memory, each bit totally isolated from the rest, but that’s not the case. In a game, just about all the data is intertwined in some way. A model refers to the meshes it contains, a character needs to know about its skeleton and its animations, and a special effect points to textures and sounds.

How are those relationships between different parts of data described? There are many approaches we can use, each with its own set of advantages and drawbacks. There isn’t a one-size-fits-all solution. What’s important is choosing the right tool for the job.

Pointing The Way

In C++, regular pointers (as opposed to “smart pointers” which we’ll discuss later on) are the easiest and most straightforward way to refer to other data. Following a pointer is a very fast operation, and pointers are strongly typed, so it’s always clear what type of data they’re pointing to.

However, they have their share of shortcomings. The biggest drawback is that a pointer is just the memory address where the data happens to be located. We often have no control over that location, so pointer values usually change from run to run. This means if we attempt to save a game checkpoint which contains a pointer to other parts of the data, the pointer value will be incorrect when we restore it.

Pointers represent a many-to-one relationship. You can only follow a pointer one way, and it is possible to have many pointers pointing to the same piece of data (for example, many models pointing to the same texture). All of this means that it is not easy to relocate a piece of data that is referred to by pointers. Unless we do some extra bookkeeping, we have no way of knowing what pointers are pointing to the data we want to relocate. And if we move or delete that data, all those pointers won’t just be invalid, they’ll be dangling pointers. They will point to a place in memory that contains something else, but the program will still think it has the original data in it, causing horrible bugs that are no fun to debug.

One last drawback of pointers is that even though they’re easy to use, somewhere, somehow, they need to be set. Because the actual memory location addresses change from run to run, they can’t be computed offline as part of the data build. So we need to have some extra step in the runtime to set the pointers after loading the data so the code can use them. This is usually done either by explicit creation and linking of objects at runtime, by using other methods of identifying data, such as resource UIDs created from hashes, or through pointer fixup tables converting data offsets into real memory addresses. All of it adds some work and complexity to using pointers.

Given those characteristics, pointers are a good fit to model relationships to data that is never deleted or relocated, from data that does not need to be serialized. For example, a character loaded from disk can safely contain pointers to its meshes, skeletons, and animations if we know we’re never going to be moving them around.

Indexing

One way to get around the limitation of not being able to save and restore pointer values is to use offsets into a block of data. The problem with plain offsets is that the memory location pointed to by the offset then needs to be cast to the correct data type, which is cumbersome and prone to error.

The more common approach is to use indices into an array of data. Indices, in addition to being safe to save and restore, have the same advantage as pointers in that they’re very fast, with no extra indirections or possible cache misses.

Unfortunately, they still suffer from the same problem as pointers of being strictly a many-to-one relationship and making it difficult to relocate or delete the data pointed to by the index. Additionally, arrays can only be used to store data of the same type (or different types but of the same size with some extra trickery on our part), which might be too restrictive for some uses.

A good use of indices into an array are particle system descriptions. The game can create instances of particle systems by referring to their description by index into that array. On the other hand, the particle system instances themselves would not be a good candidate to refer to with indices because their lifetimes vary considerably and they will be constantly created and destroyed.

It’s tempting to try and extend this approach to holding pointers in the array instead of the actual data values. That way, we would be able to deal with different types of data. Unfortunately, storing pointers means that we have to go through an extra indirection to reach our data, which incurs a small performance hit. Although this performance hit is something that we’re going to have to live with for any system that allows us to relocate data, the important thing is to keep the performance hit as small as possible.

An even bigger problem is that, if the data is truly heterogeneous, we still need to cast it to the correct type before we use it. Unless all data referred to by the pointers inherits from a common base class that we can use to query for its derived type, we have no easy way to find out what type the data really is. On the positive side, now that we’ve added an indirection (index to pointer, pointer to data), we could relocate the data, update the pointer in the array, and all the indices would still be valid. We could even delete the data and null the pointer out to indicate it is gone. Unfortunately, what we can’t do is reuse a slot in the array since we don’t know if there’s any data out there using that particular index still referring to the old data.

Because of these drawbacks, indices into an array of pointers is usually not an effective way to keep references to data. It’s usually better to stick with indices into an array of data, or extend the idea a bit further into a handle system, which is much safer and more versatile.

Handle-Ing The Problem

Handles are small units of data (32 bits typically) that uniquely identify some other part of data. Unlike pointers, however, handles can be safely serialized and remain valid after they’re restored. They also have the advantages of being updatable to refer to data that has been relocated or deleted, and can be implemented with minimal performance overhead.

The handle is used as a key into a handle manager, which associates handles with their data. The simplest possible implementation of a handle manager is a list of handle-pointer pairs and every lookup simply traverses the list looking for the handle. This would work but it’s clearly very inefficient. Even sorting the handles and doing a binary search is slow and we can do much better than that.

Here’s an efficient implementation of a handle manager (released under the usual MIT license, so go to town with it). The handle manager is implemented as an array of pointers, and handles are indices into that array. However, to get around the drawbacks of plain indices, handles are enhanced in a couple of ways.

In order to make handles more useful than pointers, we’re going to use up different bits for different purposes. We have a full 32 bits to play with, so this is how we’re going to carve them out:

• The index field. These bits will make up the actual index into the handle manager, so going from a handle to the pointer is a very fast operation. We should make this field as large as we need to, depending on how many handles we plan on having active at once. 14 bits give us over 16,000 handles, which seems plenty for most applications. But if you really need more, you can always use up a couple more bits and get up to 65,000 handles.

• The counter field. This is the key to making this type of handle implementation work. We want to make sure we can delete handles and reuse their indices when we need to. But if some part of the game is holding on to a handle that gets deleted—and eventually that slot gets reused with a new handle—how can we detect that the old handle is invalid? The counter field is the answer. This field contains a number that goes up every time the index slot is reused. Whenever the handle manager tries to convert a handle into a pointer, it first checks that the counter field matches with the stored entry. Otherwise, it knows the handle is expired and returns null.

• The type field. This field indicates what type of data the pointer is pointing to. There are usually not that many different data types in the same handle manager, so 6–8 bits are usually enough. If you’re storing homogeneous data, or all your data inherits from a common base class, then you might not need a type field at all.

struct Handle
{
    Handle() : m_index(0), m_counter(0), m_type(0)
    {}

    Handle(uint32 index, uint32 counter, uint32 type)
        : m_index(index), m_counter(counter), m_type(type)
    {}

    inline operator uint32() const;
    
    uint32 m_index : 12;
    uint32 m_counter : 15;
    uint32 m_type : 5;
};

Handle::operator uint32() const
{
    return m_type << 27 | m_counter << 12 | m_index;
}

The workings of the handle manager itself are pretty simple. It contains an array of HandleEntry types, and each HandleEntry has a pointer to the data and a few other bookkeeping fields: freelist indices for efficient addition to the array, the counter field corresponding to each entry, and some flags indicating whether an entry is in use or it’s the end of the freelist.

struct HandleEntry
{
	HandleEntry();
	explicit HandleEntry(uint32 nextFreeIndex);
	
	uint32 m_nextFreeIndex : 12;
	uint32 m_counter : 15;
	uint32 m_active : 1;
	uint32 m_endOfList : 1;
	void* m_entry;
};

Accessing data from a handle is just a matter of getting the index from the handle, verifying that the counters in the handle and the handle manager entry are the same, and accessing the pointer. Just one level of indirection and very fast performance.

We can also easily relocate or invalidate existing handles just by updating the entry in the handle manager to point to a new location or to flag it as removed.

Handles are the perfect reference to data that can change locations or even be removed, from data that needs to be serialized. Game entities are usually very dynamic, and are created and destroyed frequently (such as enemies spawning and being destroyed, or projectiles). So any references to game entities would be a good fit for handles, especially if this reference is held from another game entity and its state needs to be saved and restored. Examples of these types of relationships are the object a player is currently holding, or the target an enemy AI has locked onto.

Getting Smarter

The term smart pointers encompasses many different classes that give pointer-like syntax to reference data, but offer some extra features on top of “raw” pointers.

A common type of smart pointer deals with object lifetime. Smart pointers keep track of how many references there are to a particular piece of data, and free it when nobody is using it. For the runtime of games, I prefer to have very explicit object lifetime management, so I’m not a big fan of this kind of pointers. They can be of great help in development for tools written in C++ though.

Another kind of smart pointers insert an indirection between the data holding the pointer and the data being pointed. This allows data to be relocated, like we could do with handles. However, implementations of these pointers are often non- serializable, so they can be quite limiting.

If you consider using smart pointers from some of the popular libraries (STL, Boost) in your game, you should be very careful about the impact they can have on your build times. Including a single header file from one of those libraries will often pull in numerous other header files. Additionally, smart pointers are often templated, so the compiler will do some extra work generating code for each data type you instantiated templates on. All in all, templated smart pointers can have a significant impact in build times unless they are managed very carefully.

It’s possible to implement a smart pointer that wraps handles, provides a syntax like a regular pointer, and it still consists of a handle underneath, which can be serialized without any problem. But is the extra complexity of that layer worth the syntax benefits it provides? It will depend on your team and what you’re used to, but it’s always an option if the team is more comfortable dealing with pointers instead of handles.

Conclusion

There are many different approaches to expressing data relationships. It’s important to remember that different data types are better suited to some approaches than others. Pick the right method for your data and make sure it’s clear which one you’re using.

In the next few months, we’ll continue talking about data, and maybe even convince you that putting some love into your data can pay off big time with your code and the game as a whole.

This article was originally printed in the September 2008 issue of Game Developer.

[1] I’m not too happy about the strong distinction I was making between code and data. Really, data is any byte in memory, and that includes code. Most of the time programs are going to be managing references to non-code data, but sometimes to other code as well: function pointers, compiled shaders, compiled scripts, etc. So just ignore that distinction and think of data in a more generic way.

If you haven’t seen the presentation, go download it right now. It’s really well put together and he has some great detailed examples on how caches are affected with traditional object-oriented design vs. a more data-oriented approach.

[1] Thanks to Christer Ericson for pointing that out.

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

hinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is Thre Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

Figure 1a. Call sequence with an object-oriented approach

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

Figure 1b. Call sequence with a data-oriented approach

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

Thinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is There Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

This article was originally printed in the September 2009 issue of Game Developer.

Here’s a Korean translation of this article by Hakkyu Kim.

Identity Crisis deals with the horrors of a CRC clash on a resource right before submitting the gold master. And The Programming Antihero shows a fairly common trick in the games industry to get that extra memory after the artists and designers swear up and down that they can’t cut any more content.

Nothing like a good dose of reality to make everybody feel better about their own code :-)

Let’s look at some of the common problems that can trip up the playback system and what we can do about them. We’ll warm up by starting with the easiest ones and work our way to the hardest ones.

Game Versions

A playback of a game recorded with a different version is almost guaranteed to end up with different results than the original version. Any minor tweaks to AI parameters, pathfinding algorithms, or just about anything is most likely going to generate a different game state. It doesn’t even have to be a change in code either: adding a new character or moving a trigger volume will also cause different playbacks.

In general, you’re best off avoiding using recorded inputs from previous versions of the game. Inserting the game build version at the beginning of the recorded game state file allows you to detect version mismatches during playback. If a mismatch is detected, you can either print a warning or give up on the state verification altogether. This check will be particularly useful if you’re sharing recorded sessions across the office (perhaps through a bug database) and people are on slightly different versions.

The idea of recording a session in release mode (all optimizations enabled, few debugging checks in place), and playing it back in debug mode is very tempting. Unfortunately, mixing and matching configurations is almost guaranteed not to work. Middleware libraries will often have slightly different behaviors in debug and release, which will throw off playback right away. But most likely so does your own code, and often in ways you might not expect. For example, you might have different floating point rounding modes in debug and release. Or even a more subtle difference: depending on the optimizations you have turned on, in debug mode a float variable might be copied back to memory after each operation (which rounds it back to a float), whereas in release the program might execute several operations in a row on that value before writing it back to main memory and causing the rounding only once. The results are going to be very close, but they’ll often be cumulative and the simulation will quickly diverge.

As a general rule, treat debug and release builds as if they were different versions. Make sure to include which configuration it is along with the build number in the game state file so you can check for that as well.

Memory State

Reading data from uninitialized memory is probably the number one cause of non-determinism in games. Uninitialized memory will contain whatever data was stored there before. That can mean anything from sane-looking values to total garbage. What’s worse, they’re going to change from run to run, so if your game ever uses the values contained in uninitialized memory, playback is not going to be deterministic.

The most common case of using uninitialized memory happens by reading a variable that was declared and not initialized. This mistake is easy to see when we’re dealing with a standalone variable, but when it becomes a member variable hidden inside some class, it becomes a lot less obvious. It’s also important to notice that this will only happen when we declare variables on the stack or we create them dynamically on the heap. For global and static variables, the C runtime takes care of initializing them all to zero for us.

Cranking up the compiler warning level to the maximum will allow it to catch some of the most obvious cases of reading uninitialized variables. For extra checks, I recommend a static code analysis program such as PC Lint. That’s almost guaranteed to catch every use of uninitialized variables (along with ten thousand other things, some of them extremely useful, though most of them you probably won’t care about at all).

Another common mistake is to read past the end of a valid memory area, such as an array. The values read from memory in that case are going to depend on where that array is, and what happened before. In any case, it’s going to become a source of non-determinsim, and it’s a clear bug that should be fixed right away.

If you ever find that your game is deterministic in debug mode but not in release mode, start by suspecting access to uninitialized memory. In debug mode, a lot of platforms will fill the memory heap with specific bit patterns describing the memory status: allocated, freed, etc. Those patterns can be really useful to track down problems with memory allocations, but they also set memory values to a consistent state, which will make debug builds seem deterministic when they really aren’t.

Pointer Values

Unless you’re working on a platform over which you have total control, and you have a very strict memory allocation scheme, you should probably never rely on the actual numerical value of your pointer variables. The pointer values will change from run to run, and will depend on what happened before, including what other programs were running at the time.

If you’re puzzled at the idea of using the pointer values as part of the logic in your program (and you should be), you still need to watch out for this possibility. Some well-known, open source compression libraries rely on this technique by hashing pointer values. This means that two consecutive runs of the same program might end up with two slightly different results.

In general, it’s considered a good practice to avoid using pointer values for anything other than dereferencing them and accessing the data they’re pointing to. Making decisions based on their actual numerical values is asking for trouble, and you can almost always achieve the same result by using offsets between pointer values (as long as you know they’re coming from the same memory block), which has none of those problems.

Asynchronous File I/O

This is where things start to get fun. A lot of games perform background loads while the game is running. Whenever the game detects it will need an asset, it initiates an asynchronous load. When the load is complete, the game is notified and, optionally, does some processing on that asset. There is no guarantee about when exactly the load will complete, which means that each playback has the potential to be slightly different.

If the background load just brings in more mipmaps for a texture, it’s not going to affect the simulation whether it happened a frame earlier or a frame later. On the other hand, if it’s loading a more detailed AI navigation path, it can definitely affect the position and state of AI units. Even if you’re not loading data that affects the simulation, it’s very useful to have asynchronous file IO work deterministically to catch bugs more reliably. For example, the game might crash if a background texture load completes the same frame as a line of dialog sound is requested. As any programmer who’s ever had to debug a problem like this will tell you, being able to reliably reproduce that crash is worth a small fortune.

We can turn asynchronous file loading into a deterministic operation by considering the read completions as inputs to the game. Whenever an asynchronous file load completes, we record it in the game input file at the corresponding frame. During playback, the game will request background loads as usual, but completion events are made available only when they’re read from the input stream.

This can be easily implemented at the game level by buffering all background read completions. During recording and normal game operation, background read completion events are made available as soon as they happen. During playback, however, background read completion events need to be available the exact same frame they happened in the original session. If the read finishes earlier, it is simply buffered for a few frames until the correct time. If the read hasn’t completed by the time it finished in the original session, the game blocks until it’s done and it’s made available right away. That means that playback might be a bit choppy at times, while the game blocks for data to be read, but it will ensure that playback is fully deterministic. Notice that blocking the game while waiting for a read to complete is not going to affect subsequent frames with a larger delta time because we’re also reading the system clock from the input stream.

Network Data

What about online games? Our playback method has been concerned exclusively with local players. Playtesting and Q/A on online games is much more expensive and time consuming than single-player games because of the manpower required and the coordination necessary to set the games up. Wouldn’t it be great to be able to replay a 30-person game that resulted in a crash without having to get 30 players again?

Data received from the network most definitely influences the game itself, so it’s another input to the game. We could treat all network traffic like any of the other inputs. Record it along with the other game input and play it back by inserting the network packets as if they had come from the network card. That would work, but it’s a bit more complicated than it has to be. Fully emulating the network traffic, connection status, and everything else can be a bit tricky to get just right.

A simpler approach consists of translating the data received through the network into higher-level game actions. For example, whenever a player fires a weapon, it results in a network packet, which is translated into a game-level action whenever it’s received. Then, the local simulation applies the action that causes that player to fire locally. You probably already have a system like that in place in your multiplayer game anyway. Recording those actions is a much simpler task than the raw network traffic, and playing back a recorded multiplayer game is just a matter of inserting the game-level actions at the same time they occurred, just like any other input.

The game input file is going to grow significantly as soon as you start recording all the actions created by the players through the network. Fortunately, most games are extremely careful to minimize network bandwidth, so the input file will remain at a reasonable size even recording all that data.

Threads And Multiprocessors

In today’s games, multithreading is an inevitable reality. Even if your code isn’t explicitly multithreaded, parts of your engine and middleware are probably using multiple threads. The problem with threads is that you never know exactly when one thread stops working and another one resumes. So it is possible that two events in two different threads will happen in different order in two runs of the game. To make things even more fun, most of today’s platforms have multiple cores, making determinism even more complicated. Do we need to give up determinism in a thread-dominated world?

The totally honest answer is that we’ve already picked all the low-hanging fruit. If you really want your game to be 100 percent deterministic while running in multiple threads over multiple cores, get ready to roll up your sleeves and do some serious work. For the rest of us, we can still get most of the benefits of a playback system without total determinism. That means that there is potential for bugs caused by thread interactions that might not be repeatable from run to run. But at least the game simulation will be the same for every playback, so the recording technique should still be very useful.

In the easiest case, the simulation runs on a single thread, with the rest of the threads dedicated to graphics, sound, and other systems. If we apply all the techniques we’ve covered so far, the simulation itself should be deterministic and we shouldn’t have to do anything different than we did in the singlethreaded case.

However, as soon as the simulation is spread across multiple threads, we need to be a lot more careful. Even if we have no control over the thread context switches, we can at least ensure that major events happen in the same order no matter how they were executed. For example, if a worker thread creates a set of actions to be executed, we can sort those actions before processing them. If that’s not an option (because those actions are processed as soon as they are created, for example), we could record the creation order as part of the input recording, although enforcing that order during playback might have far reaching consequences for many systems deep inside the game.

If achieving 100 percent determinism in a threaded environment is important to your project, have a look at Replay Director. It’s a commercial tool and set of libraries that gives you very accurate recordings and playbacks of your game, including thread context switches, with little extra programming on your part.

More Than Just For Debugging

At this point, if you’ve applied all the techniques we’ve discussed so far, you should have a pretty bullet-proof recording and playback system: lightweight, reliable, and accurate. It will be a great help as a debugging tool, but there’s no reason to stop there. You could use the same system to record demo sequences to show off in presentations without the need to make a movie capture and degrade the image quality. You can also truthfully claim that it’s a live demo, running on the game itself, not a canned movie, which always carries extra weight with most audiences.

You can even integrate the playback system as a key feature in your game. Halo 3 already does this by allowing you to replay any play session, examine what happen exactly, and let you take screenshots of the juiciest moments. It can be a great feature for a lot of games for almost no extra effort over what you’ve already implemented. The main problem will be to make sure future updates to the game don’t affect the playback of older gameplay sessions. This is really hard to achieve, since the most insignificant change can end up causing the simulation to diverge in unexpected ways. So if you’re going to rely on it as a game feature, you need a suite of comprehensive tests verifying that nothing changes whenever the game is updated. Last month we saw how to verify that a playback results in the same game state as the original session, so again, there’s very little extra work there.

With all these techniques in your toolbox, you should be able to make your game pretty close to 100 percent deterministic—enough to have a solid playback system and use it to its fullest during production, and maybe even as a game feature.

This article was originally printed in the June 2008 issue of Game Developer.

How would you like to be able to reproduce every crash report that QA adds to the bug database quickly and reliably? How useful would it be to be able to put a breakpoint the frame before a crash bug happens?

You can do all that and more if your game is deterministic and you feed it the same inputs as an earlier run. Sounds easy? It is, if you implement it early on and you keep it that way during development. If you choose not to make your game deterministic, your team will go insane by Einstein’s definition, and maybe by a few other definitions as well by the time the project ends.

Determinism

A system is said to be deterministic if, given the same set of inputs, it produces the same set of outputs. Think of your game as a big, black box, with some inputs and outputs.

The two main input types for most games are:

• Player input devices. The state of the gamepad, keyboard, mouse, or any other input device is used to update the simulation.
• System clock. Most games query the system clock once per frame to retrieve a current time and delta time. The game then advances the simulation forward by that amount of time. Even console games that expect to run at a rocksolid 60 Hz are often implemented this way to be able to deal with different TV refresh rates or with slower builds during development.

The main outputs of a game system are the pixels on the screen and the generated sounds. Optionally there are other outputs such as network packets or force feedback.

To make the game deterministic we need to make sure that, for a given set of inputs, it always produces the same outputs. In other words, playing the game twice, entering the same button presses at the exact times, the game will end up in the same state both times (same location, same health, number of lives, NPC positions, etc).

What about random numbers, which we rely on so much in games? Aren’t random numbers inputs as well? Yes and no. In games we use pseudo random number generators. That means that the sequence of numbers from a particular seed is always the same. For a given seed, all runs of the game are fully deterministic, so the random numbers are not really an input into the system. The seed to the random number generator is an input, since it will greatly affect the state of the simulation.

Record and Playback

Recording all player inputs and the system clock is as straightforward as it sounds: At the beginning of the simulation loop, sample the clock and the player input devices. Then save those values to a file and continue the game simulation as usual. You want to make sure the file is flushed after writing the values for each frame. That way, if the game crashes, you’ll have all the input leading up to that frame and you’ll be able to play it back right up to the point where it crashed.

Recording the input has a negligible performance impact and the amount of data saved is very small. As an example, in our current game, every frame we save the frame number, the clock time, frame delta time, and a 20-byte structure for each gamepad (see Listing 1). Without any compression, that’s 52 bytes per frame for a two-player game. At 60 Hz, a 20-minute game would be a tiny 3.6 MB, which is small enough to attach to a bug tracking system, email to the team, or archive with the build itself.

Listing 1. Game input structures.

struct FameInput{    uint32 frameNumber;    float time;    float dt;};

struct RawControllerState{    uint32 buttons;    float leftStickX;    float leftStickY;    float rightStickX;    float rightStickY;};

Using the recorded input for playback is almost as easy. Every frame, before the simulation starts, we read the input data from the file and feed it to the game as if it came from the system clock and the input devices. A clean way to do that is to separate the reading of the data from where it comes from. In C++ we can use abstract base classes that define an interface describing how to retrieve the data. For example, a class IGameInput can define an interface, and the class GamepadGameInput reads the data from the hardware, while the class FileGameInput reads it from the recorded file (see Listing 2).

Listing 2. IGameInput, GamepadGameInput and FileGameInput

class IGameInput{public:    virtual void GetInputData(RawControllerState& state) = 0;};

class GamepadGameInput: public IGameInput{public:    virtual void GetInputData(RawControllerState& state);};

class FileGameInput: public IGameInput{public:    virtual void GetInputData(RawControllerState& state);};

In addition to recording and playing back those inputs every frame, we also need to record the seed to the random number generator before the game starts, and read it and apply it during playback. That will ensure that all the random numbers are the same in both runs of the game.

Verification

If you’re going to rely on this recording system, you need to make sure the playback produces the exact same results as the original session. If they aren’t the same, the whole system is worthless. Also, because we’re just recording input, if the state of the game during playback ever starts to diverge, it will continue getting more and more out of sync from there. We need a way to make sure the playback produces the same state as the initial recording.

Earlier we identified some of the outputs of the game as the pixels on the screen or the sound produced by the speakers. We could compare pixels with a previous run of the game, but the storage requirements would be enormous, and the performance less than ideal. Besides, subtle changes in shaders, lighting, or a simple texture change would throw the comparison off. An easier solution is to check the game state itself. If an enemy is in a different location in two runs of the game, of course it’s going to render to a different set of pixels. It will be a lot faster and easier to compare game states than raw output.

During recording, in addition to the game input, we can save some of the state of the game. There’s no need to record it all, just some of the most important and representative state. Good candidates include player and enemy positions, prop transforms, score, etc. Unless it’s crucial to your game, there’s no need to record things like player animations or enemy AI state. Chances are that if any of those diverge, the positions for those entities will also diverge right away.

Once we have all of this state recorded, we can verify the state does not change during playback. Every frame, we compare the current game state with the recorded game state. If they’re different, even by the smallest amount, we know something has gone wrong and we flag it right away. The game isn’t quite deterministic and something needs to be fixed. We can choose to record and verify the game state anywhere in the simulation loop (as long as they’re both done in the same spot), but if we do it after the simulation step instead of before, we’ll be able to see what the inputs were that caused the divergence this frame, which will help when debugging. The main loop is shown in Listing 3.

Listing 3. Main loop structure

while (!done){    SampleClockAndInputs();   // from different sources through the same interface    RecordInput();    UpdateSimulation();    if (recordGameState) RecordGameState();    if (verifyGameState) VerifyGameState();}

Unlike recording game input, recording the game state can be a much more expensive operation. Traversing the game structures can be a significant performance hit due to cache misses, and the data saved to disk often results in large files. Because of this, in our current game player input is recorded all the time, but recording game state is optional, and we can be controlled through a command-line parameter. Game state verification is also optional, since sometimes we want to play back a recorded set of inputs even knowing that the state of the game is going to diverge due to changes we’ve made.

For a few of you, checking the game state won’t be enough. If you’re writing a middleware graphics layer, you probably want to verify that the values you’re generating in the back buffer are the same ones from a previous pass. Or maybe that improving the performance of a rendering algorithms still generates the same image. In that case, you might want to consider something like Perceptual Image Difference, which will be a much less error-prone than comparing exact values for pixels.

Monkeying Around

Once the game is fully deterministic and the playbacks are rock solid, you want to make sure it stays that way. It’s all too easy to introduce bugs that will cause playbacks to diverge. For example, a single, innocent-looking, uninitialized variable can change the simulation depending on the value it happens to have for this run.

The best method I’ve found to stress test the playback system is to use a recorded session of monkey input. The monkey input method consists of feeding the game pseudo-random inputs (as if a monkey were playing the game). Recording both the input and game state of a monkey input play session, and then playing it back verifying the game state kills two birds with one stone: You get some nice automated testing of your game, and you verify that the game is fully deterministic. It sounds too simple to be useful, but you’ll be amazed at how many bugs your first session of monkey input will uncover.

A clean way to implement the monkey input, is writing a new class that implements the IGameInput interface. This new class will generate game input for all the buttons and axes of a game controller, but it won’t come from the hardware game controller or from a recorded session, but from randomly generated values. Apart from being a very clean way to insert input into the game, this approach has the advantage that the monkey input can be recorded just as if it were regular input coming from the controller. That means we can later replay a session and verify its game state, which makes for a very useful functional test.

It turns out that totally random input values are not ideal, as it becomes apparent as soon as you implement the naive random monkey input class. If the state of the jump button is truly random, it will be pressed and released almost every frame, which means the player will hardly ever jump high enough off the floor. A better implementation will use a range with a minimum and maximum press time durations, which will give much better results. You might also want to avoid pressing specific buttons (like the pause button, or the restart button combination). Resist the temptation to make the monkey input too smart and make it behave more like a real player, for example, by limiting the number of buttons it can have pressed simultaneously. Part of the benefit of the monkey input is that it will do very unexpected things, that no sane player will ever do on purpose, and by doing that, it will unearth many more problems and issues with the game.

One day, shortly after I implemented the monkey input system, the playback verification started failing. Some game entities were ending up in a different state than expected. After doing some digging, I realized that I was using the same random number generator for the game simulation and for the monkey input. Since the playback was just reading input values from the file and not generating them on the fly, the random number sequence was getting off sync right away, causing entities with some randomness in them to behave differently. Lesson learned: Make sure that nothing in the monkey input affects the game systems. In this case, I solved it by using two instances of the random number generator, one for the monkey and one for the game.

Playback in The Real World

At this point we can easily reproduce bugs. We can re-run the game and put a breakpoint right before the game crashes—except that the crash happens 20 minutes into the playback and nobody wants to wait that long.

Fortunately, we can make time go by faster. During playback we don’t care about tearing artifacts, so we can turn off the vertical sync, which will speed things up a bit. Most importantly, we often don’t even care whether we render anything or not, so if we turn off rendering completely, we can make the playback run significantly faster, particularly if you were graphics bound. One word of caution: If the rendering part of the game does significant work, and especially if you suspect it might be interacting with the rest of the game to cause a bug or some other source of non-determinism, you might want to do the same work in the rendering system, constructing the push buffer the way you would normally do, but never send it off to the graphics hardware.

Turning off the vertical sync and disabling the graphics rendering, is just reducing the amount of time we spend in each frame. But it still takes 10 minutes to get 10 minutes into the game, we just go through many more frames to get there. The last piece of the puzzle to make time go faster is to be able to set a fixed timestep. Now we can go through the simulation at a rate much faster than real time (and the beefier you computer, the faster it will go). In my current game, we can go through 10 minutes of game time in about 1 minute of real time.

One use for recording and playback that we haven’t mentioned is performance comparisons. When you’re optimizing the game, you can record a gameplay sequence and gather some performance statistics: average fps, longest frames, etc. Then you can apply your optimizations, playback the same input sequence, and compare the performance statistics to see how effective the optimizations really were. Just make sure you use real clock time and not the recorded clock time, otherwise you won’t see any differences, even with all the hard effort you put into the optimizations.

There are a few details we glossed over that might prevent some games from begin fully deterministic: network traffic, asynchronous file I/O, and threading issues. We’ll cover those in detail next month.

You’ll soon find that input recording and playback becomes an essential tool for in development process and you’ll wonder how you ever lived without it. Spend a few hours implementing it early in the development cycle and reap the benefits many times over during production. You’ll be the hero of the day when that dreaded crash bug from QA arrives minutes before a deadline.

Thanks to Jim Tilander for being a great idea bouncing-board and proofreading the article.

This article was originally printed in the May 2008 issue of Game Developer.

Size

Charles and I were priding ourselves in keeping things small and minimal. But truth be told, it’s not like we were keeping track of how many lines of code we had written. Were things as small as we hoped they were?

The most convenient way of counting lines of code that I know is CLOC. It’s an extremely easy to use open source program which counts the lines of code in a codebase, gives very detailed information, strips out whitespace, breaks things down by language, and does just about everything you’d want from a program like that.

Running it on the latest version of our code (not including any 3rd party libraries) produces this:

    1621 text files.    1579 unique files.    3721 files ignored.
-------------------------------------------------------------------------------
Language                    files          blank        comment          code
-------------------------------------------------------------------------------
C++                            485          13577            303          46181
C#                             324           4935            712          22966
C/C++ Header                   407           4153             95          11975
MSBuild scripts                 18              0            126          1490
-------------------------------------------------------------------------------
SUM:                          1234          22665           1236          82612

Almost 60K lines of C++ code seemed very high. At first I thought it was because CLOC was counting files twice: once in their regular location and once in the .svn directory, but apparently it’s already removing all duplicates, so that wasn’t it.

Almost more scary than the amount of C++ code (which is all our runtime and some of our tools) is the amount of C# code. For a language that claims to be of significantly higher level than C++, that’s quite a mouthful of code!

Another surprising count in there is the number of lines with comments. Since we make heavy use of TDD, I really didn’t expect more than a couple dozen lines of code in the whole codebase. Still, I’m kind of proud that we have less than one line of code per file on average :-)

Here’s a more detailed breakdown, with the line count just for our runtime (engine and game):

1089 text files.    1053 unique files.    2338 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            441          11997            245          40943
C/C++ Header                   385           3964             90          11405
-------------------------------------------------------------------------------
SUM:                           826          15961            335          52348

and for our tools:

532 text files.     531 unique files.    1383 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                             324           4935            712          22966
C++                             44           1580             58           5238
MSBuild scripts                 18              0            126           1490
C/C++ Header                    23            199              5            591
-------------------------------------------------------------------------------
SUM:                           409           6714            901          30285

Tests

Then I realized that a good chunk of those were tests. So excluding all directories matching *Tests* gets the following result:

1206 text files.    1187 unique files.    4199 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            283           6464            150          22464
C#                             213           2636            534          12402
C/C++ Header                   380           3824             94          10782
MSBuild scripts                 12              0             84            978
-------------------------------------------------------------------------------
SUM:                           888          12924            862          46626

A bit more than half the C++ code consisted of tests. That’s pretty consistent with my experience with TDD. C# seems to follow a similar percentage as well.

As for the exact number of tests, running a grep for TEST shows all the C++ tests:

C:\pow2>grep -r TEST SweetPea Engine Tools | grep -v svn | wc   2163    3620  221953

And doing the same thing with [Test] brings up all C# tests:

C:\pow2>grep -r \[Test\] SweetPea Engine Tools | grep -v svn | wc   735    1470  52717

That means that our average C++ test is about 11.5 lines long, and C# tests 14.4. Frankly, that sounds rather high. We make heavy use of fixtures whenever possible and each test usually only checks for a single condition (even if it involves a couple check statements). I suppose that number is higher than expected because it probably includes all the lines from #include statements and all the fixtures as part of the average.

* If we only count cpp files, that goes down to 49%

I was curious about that last part of checking a single thing per test, so I ran a grep for the number of CHECK statements in our code:

C:\pow2>grep -r CHECK SweetPea Engine Tools | grep -v svn | wc   3886   15079  399598

That’s 1.8 CHECK statements per TEST, which is about right. Even though we’re checking for a single condition, we’ll often check a couple things about it (i.e. the camera stopped and it reached its final destination).

Build Times

So, given that amount of code, how long does it take to build it? Clearly it depends on your hardware. Since we’re not exactly rolling in money, we don’t have particularly powerful machines. Here at home, I’m using a modest Core 2 Duo E4300 (overclocked to 2.6 GHz) with fast memory and a relatively fast SATA hard drive, so that’s what I used for all my timings.

A full build of our game, plus all the libraries, all the tests, and running all the tests takes exactly 1 minute and 10 seconds. That’s pretty good for two reasons:

• When we work with the game we don’t build and run the unit tests for the engine. We have a separate solution for that. A full build of just the engine, the game, and the game unit tests only takes 43 seconds.
• The game itself is a fairly large project and devenv doesn’t know how to paralellize that build, so it’s only using half the available CPU power for about half the build time.

An incremental build after changing a single cpp file takes slightly over a second (including half a second of unit test execution).

As you can imagine, working with that codebase is a dream come true. Snappy, responsive. Nothing is hard enough that can’t be changed.

Unfortunately that’s where the fairy tale ends. The tools are another story altogether. Our C# tools, with all their unit tests, build in a mere 18 seconds, and the C++ tools in 1 minute and 10 seconds. That’s not too bad, except that it’s a surprisingly large amount of time for the C++ tools since there aren’t that many of them.

Here’s the kicker, doing another build without changing a single thing take 38 seconds. Whoa! We’re doing some C++/CLI trickery and apparently dependency checking is totally broken in VS2005 (either that, or we just don’t know how to set it up right).

Keeping things fast

What’s the secret of a lighting-fast build? Clearly, keeping the code size down is crucial. If your codebase is 2 million lines of code, builds are going to be painful no matter what. But they can be a little less painful with some gentle care.

One of the main build-time killers that we’re avoiding is the use of STL or Boost. Those libraries pull in everything and the kitchen sink, and their heavy use of templates make build and link time go through the roof. No thanks.

Our template use is pretty minimal. We have a couple containers (which I love and I’ll write about it one of these days) and that’s about it.

We’re pretty anal when it comes to keeping physical dependencies to a minimum. We forward declare aggressively, and we only include the headers that are necessary for each cpp file (PC Lint is “kind” of enough to remind us every time we have unnecessary #includes). We’re not using external include guards or #pragma once.

Precompiled headers are either not used, or kept to a minimum. I think the only project that uses them is the game and only for Havok headers. We don’t even have windows.h in a precompiled header (which would be a really bad idea because you’d be putting all the junk in windows.h available to your whole program).

Finally, we are using incremental links whenever possible. I remember a few versions of Visual Studio ago they were pretty broken, but they’re not giving us any problems. The only caveat is that if you modify a static library your program is linking with, it will force a full link. So they’re really only good for modifying the executable itself.

We’re not using any distributed builds. First of all, we don’t have enough computers to make it worthwhile. And second, I had horrible experiences with distributed builds in the past. They would help with a badly structured codebase, at the cost of longer incremental builds and mysterious spurious bad builds. Besides, once they’re in place, they tend to encourage even further disregard for keeping dependencies to a minimum.

How about you?

So, that’s it for the Power of Two codebase. How about you? Want to share your size, build times, or any other data?