You would think that dealing cards at random would be trivially easy to implement. You would also be wrong.

Truly Random

Always the eternal optimist (and devout follower of the YAGNI principle), I start by implementing dealing cards by simply picking randomly among the types of cards available. Something along these lines:

for (int i=0; i<HandSize; ++i)
    hand[i] = Random_GetInt(0, CardType::COUNT-1);

The best part of this approach is that it’s simple. The bad part is… everything else.

Maybe that works for your game, and if so, congratulations, you can stop now and move on with the rest of the game. Unfortunately, this doesn’t cut it for most card games. The player might get a hand all of the same card. Or maybe never get a card type that she really needs. Most of the time, an algorithm like this will frustrate players quite a bit and get in the way of the enjoyment of the game.

Deck of Cards

The next approach I usually try is to simply copy physical games and build an actual deck. Remember that we’re talking about internal structures, so this doesn’t mean that the concept of a deck is exposed to the player at all. As far as they know, they’re just getting cards at “random”.

In this case, you create an array of card types, shuffle it, and give cards from the top to the player.

const int cardsPerType = 3;
for (int i=0; i<CardType::COUNT; ++i)
    for (int j=0; j<cardsPerType; ++j)
        Deck_Add(deck, i);

Deck_Shuffle(deck);

for (int i=0; i<HandSize; ++i)
    hand[i] = deck[i];

This is quite an improvement over our first approach. To start with, we’re guaranteed never to get more than a certain amount of cards of each type in the deck. That will prevent ridiculous hands of all cards of the same type that make the game unplayable.

It also allows us to vary the amount of cards of each type in the deck. That way we can make some cards more rare than others, or even enforce unique cards by only adding one of each of those types.

You may argue that this approach is perfect since this is how “true” (meaning “physical) card games are designed most of the time. While that is true, physical card games often involve just randomly shuffling a deck because the alternative would be very time consuming (or might need a third-party who isn’t playing to set up the deck). Because of that, most games need to be able to deal with very uneven distribution of cards, which might be too limiting for your designs.

A lot of games games will benefit of having a more controlled deck, and since we’re running on a computer, we can afford to do a lot more than just shuffle cards together. We can take advantage of the digital medium and do things we couldn’t easily do with physical cards.

Aside: Shuffling a Deck

Shuffling a deck might seem like a trivial algorithm, but as usual the details can get a bit tricky.

If you wanted to get very physical, you could try to simulate real riffle shuffling of a deck: randomly split the deck into more-or-less equal parts, and interleave them with each other with some randomness. Then you’d have to run it several times and even so I think you’d get very suspect results.

A simplified algorithm is to randomly grab two cards in the deck and swap their positions. Is that good enough? How many times do you need to run that loop in order to get a proper (i.e. fully randomized) shuffle? I haven’t analyzed that algorithm, but I get the feeling it would have to be done many times in order to get something acceptable (especially considering the not-very-random nature of computer “random” number generators).

The best and simplest approach I have found to shuffle a deck is the following: assign a random number to each card in the deck, then sort the cards based on that random number. It’s easy to understand how a single sort will shuffle the deck properly and works the same for any number of cards (and runs in O(n log n) if we want to get pedantic about it).

Multiple Decks

One way to have more control over the deck composition is to have multiple decks. We don’t need to communicate to the player that there are in fact multiple decks, this is something that happens under the hood.

This is the approach we took in Subterfuge. Players are presented with a choice of 3 specialists every X hours. Initially those specialists were completely random, but the results were very underwhelming. The approach we ended up settling on involved classifying each specialist as being in the categories of attack, defense, or other. The game prepares 3 decks, one of each category with 3 of each of the specialists in each category. Additionally, we make sure that we never have 2 of the same specialist in a row.

When it comes time to generate 3 “random” specialists for the players to choose from, we pick the top one from each of the decks.

Blocks

In my current prototype, the player has a hand of 4-5 “cards”, and as soon as they use one, a new one is drawn and their hand refilled. Cards can be roughly classified in 6 different categories, and it’s important that the player gets access to one of those categories within a few turns.

I wanted to come up with an approach that guaranteed access to tiles of all categories, but without making the pattern too obvious. For example, I didn’t want to have the deck be one card of each category.

The solution I can up with was to break up the deck in blocks of 10 cards. Within each of those blocks, I can make some guarantees about its composition. This approach was inspired by the composition of boosters in collectable card games like Magic: The Gathering, or even in the way the deck is prepared in Pandemic.

The way I implemented it, each block of 10 cards will contain one card of each category, plus 4 other random cards (as long as they are not already in the block). Then all the cards in the block are shuffled and added to the deck. This results in apparently “random” draws without feeling predictable, and yet players are never left without a key resource they may need.

This kind of enforced structure can also add some depth to the game. Expert players might notice this pattern (or read this, which is a lot easier) and use it to their advantage when they’re trying to make decisions based on what cards might appear in future terms.

Any of you out there use a different method for creating random-but-not-quite-random draws in your games?

When we started work on Lasting Legacy, I envisioned it as a quick 9-10 month project (that’s quick for me, but I know it’s long for other devs). As we worked on it, it felt like the game was interesting and rich enough to spend a little longer and flesh it out to its full potential. Maybe spend a total of a year and a half on it.

Now it’s been two full years since we started. We’ve had a playable game almost from the start, but when I have an honest look at it, it’s still far from launch. We still need lots more content, game balancing, polishing, different game modes, porting to different platforms, creating videos, and all the marketing. It’s that pesky last 20% that takes 80% of the time–OK, maybe not 80% but at least another 12 months. All of that is definitely doable, but it would bring the project to a full 3 years (at least!), or a total of 6 man-years.

The problem with that is that, for Lasting Legacy to be financially successful, it would have to sell a lot of copies. A lot more than I can realistic expect it to sell. Let’s reeeeeally stretch the meaning of “successful” and say that the goal is that we should each make $50K per year. That’s a total of $300K after the cut from the stores, or about $430K in sales. At $15 per sale, that’s almost 30K sales, and most Steam games don’t come even close to that number.

So we had a choice: We could focus really hard on the marketing aspect to make sure we hit that many sales, or we could stop development altogether, cut our loses, and move on to something else. The problem with focusing on marketing is that it would add even more time, making it necessary to sell even more copies to break even. It would also mean spending months doing work that I really dislike (making trailers, doing a Kickstarter, promoting the game, chasing reviewers and streamers…).

Would it be worth doing that marketing push? Was I really willing to bet that Lasting Legacy would be a big seller? If I’m honest, I think that Lasting Legacy has the potential to be a very good, original game, but I don’t think it’s a game that lots of people want (or if they do, I wasn’t able to connect with that audience). I’m fine making games that don’t sell very much, but at least I want to enjoy the work that I’m doing. So in the end, I’d rather stop working on Lasting Legacy and move on to other projects that I can enjoy and can be potentially more profitable.

Fortunately the time I spent on Lasting Legacy wasn’t completely wasted: Apart from developing some new tech that I didn’t have when I started (SQLite to organize data, or on-the-fly font rendering for example), I continued to grow as a designer, and I hope a lot of what I learned will carry over to my future projects.

Some people will take away from this that Lasting Legacy failed because we didn’t do market research to see if there was demand for it, or tailor it to the tastes of gamers. I disagree. My goal from the start was not to maximize profits, it was to make a game I found interesting and getting compensated for my time. The failure of Lasting Legacy was that we were just too slow. If we had managed to make the game in a year like we originally planned, just selling 5-6K units would have been enough to cover our time.

So what’s next for me? I’ve taken the lessons learned to heart and decided to make a game by myself in 6 months. That’s a hard deadline. I’m drawing inspiration from Michael Brough and Arnold Rauers, both of them excellent game designers who make small games that start with major artificial constraints and they design around them. The constraint I picked is to make a simulation game in a very small grid (single screen, around 6x6). So far I’ve been doing some prototypes and I’m pretty happy how it’s coming along. If all goes well I’ll announce it soon and release it in a few months.

As part of this shift towards making smaller games, I’m going to create a mailing list for my games in general, not for each game individually. If you’d like to keep up to date, please sign up for my mailing list and I’ll make sure you’re the first ones to know about my upcoming games.

Subscribe to my mailing list

Wish me luck!

So I decided to write a “rulebook” for Lasting Legacy. I put rulebook in quotes because Lasting Legacy isn’t 100% a board game. There’s a light simulation component behind the scenes that is opaque to the player, but everything else can be treated like a board game. I figured it would be a good exercise for me, and maybe a good reference for early testers so they know what’s going on without a fancy tutorial.

I’m happy with the final result. It’s about three pages of generously-spaced rules without any images, which beats a lot of board games out there.

A word of caution: This is not trying to be a funny, engaging rulebook. It’s a dry, to the point, description of all the rules in the game, arranged in the best way to understand all the concepts in a single read. It’s also not a “How to play” document. I think that could be another interesting exercise for down the line, where I just focus on the bare minimum to get a player playing.

Lasting Legacy Rulebook

1. Overview

Goal

Gain the highest Legacy possible.

Setting

Guide a European family around the 19th century through several generations. Socialize and find suitable marriage partners to grow you family, so you can earn Gold, Prestige, and ultimately, Legacy.

2. Concepts

Note: Any numbers shown here are just the defaults. There are always effects in the game that can modify those numbers up or down.

Family members

People in the family tree are called family members. Family members under 14 years old are considered children. They’re represented with round frames and don’t have any special abilities. Once they become adults they can choose between one or more occupations, depending on their education level. Adult family members will have children based on their age and fertility.

The head of the family, indicated with a special frame, defines which family members can be used at any given time. Only the head of the family, their spouse, and people under them (children and heirs) are enabled. Other family members are disabled and can’t be used.

Occupations

Every adult member can have an occupation. The clothes and items people have represents their occupation. Occupations can be passive or active:

• Passive occupations have a permanent effect on the game. All the current effects are shown in the effect window.
• Active occupations provide you with a potential action you can perform by clicking on the occupation button.

Some occupations require a minimum education to appear as a choice. Occupations come in 4 rarities: common, uncommon, rare, and unique. Unique occupations will appear at most once per game. Occupations also could have one or more time periods during which they can appear: early, middle, or late in the game.

Personality

Every adult family member and friend has personality traits that consists of a series of likes and dislikes.

Friends

Your family friends are shown along the bottom of the screen, in oval frames. The smiley face shows their friendship level towards the family. Each year friendships drop a fixed amount. When the friendship level for a friend drops to 0, the face turns blue, and the following year that friend leaves.

Friends can join your family through marriage. The heart in the friend’s frame represents how much they love someone in your family. Love is determined by many factors, including personality match, sexual preference, age similarity, and friendship amount. If that love level is above a certain threshold, it means they’re in love with someone and they’re willing to marry that person (see section 3: Proposing and Accepting Proposals).

Gold

Gold is one of the primary resources in the game. It can be earned through actions and effects of people. Additionally, you may have a recurring Gold income (shown in the green arrow), and a recurring Gold payment (shown in the red arrow) that happen every year.

You can spend more Gold than you have, automatically taking a loan. Every year your Gold is negative, your family pays 10% of that negative amount in interest payments. 200 Gold is the debt limit. If you ever owe more that amount, the game ends in bankruptcy.

Prestige

Prestige is the other primary resource of the game. It represents your social standing, and the more Prestige you have, the more friend slots you have available. You always have at least one friend slot. For every 10 Prestige you accumulate, you gain an additional friend slot up to a maximum of 10. Beyond that, you can accumulate more Prestige but you won’t gain any more slots.

If at any point you gain a friend but you have no empty slots available, the friend with the lowest friendship leaves and the new one takes its place.

Prestige can also be spent to do actions that are usually associated with negative social implications. Friends can leave you if your Prestige falls and you lose a slot occupied by a friend.

Unlike Gold, Prestige can’t even go below zero. However, you can still use actions that would cause your Prestige to go to zero. If you do, the person who did the action will be disowned by the family and won’t be part of the active family anymore.

Health

People normally are healthy. Sometimes they will become sick due to a variety of reasons. From there, each year they can either become healthy, stay sick, or turn terminally ill. Once someone is terminally ill, unless something is done about it, they will die the following year.

Heir

A yellow line in the tree indicates the next heir to the head of the family, which is the preferred child of the current head of the family. Whenever a family member does an action that’s related to one of those traits, their parents will change their opinion of that person, either in a positive or negative way.

You can see the current opinion of someone by selecting them and looking at the smiley faces over each of their children. The more filled in that face is, the higher the parent’s opinion of that child is.

When the head of the family dies, the current heir becomes the new head of the family. The new head of the family also presents you with a new family goal due to their tastes and whims. This goal will let you gain Legacy by doing certain things during their time as head of the family.

Heirlooms

Heirlooms will give you Legacy based on different criteria. You start the game with one random family heirloom in your collection.

Events

Every 30 to 70 years, an event will happen which will affect how things normally work. You will be notified 10 years before the event becomes active.

Countries

The majority of the people you’ll encounter during a playthrough come from the country you are playing in. You might encounter a few friends that come from neighboring countries. If you manage to add those friends to your family, that country will be unlocked and available for a future playthrough.

Countries add variety in the form of starting conditions, tweaked rules, and some unique occupations to those countries among other things.

3. How to play

Each turn, take one of the following actions. Each action will move the year forward one year.

Socialize

Gain one friend. The new friend’s age and education level is influenced by the person who socialized.

Visit Friend

Restore one friend’s friendship level to the maximum.

Propose in marriage

Men in the family can propose in marriage to friends. When you click on the propose in marriage button, only the friends who are willing to accept will be highlighted. Click on one of them to marry them.

Women will sometimes have a dowry made out of Gold and/or Prestige associated with them. When you marry a woman with a dowry, those resources are added to the family resources. On the other hand, women from the family will sometimes have to offer a dowry in order to marry a man. In that case, those resources will be taken away from the family resources.

Accept marriage proposal

People in the family can receive marriage proposals from friends (indicated by the hearts above their heads). These proposals are only available for one year. If you accept it, they’ll be married right away. Otherwise, the friend will feel rejected and their friendship will drop more than usual.

Choose an occupation

When a family member becomes an adult, they can choose an occupation (indicated by the question marks above their heads and their lack of occupation item and clothes).

If you don’t choose an occupation, they will pick one of the available ones after 5 years.

Use occupation action

You can click on the occupation action of any active family member to perform that action. Occupation actions are unique to each occupation. Sick people can’t do any occupation actions.

You can also use the occupation action of a friend. That’s like asking them for a huge favor, so they’ll do the action, but then leave.

Restore heirloom

Whenever someone in your family over the age of 50 dies, they will leave a family heirloom in the attic. Each heirloom in the attic has a cost associated with restoring it. You need to restore an heirloom to add it to your heirloom collection so it generates Legacy.

Pass

When there’s nothing else better to do, you can pass the turn and earn 5 Gold.

4. End of game

The game will end when one of these conditions is true:

• You run out of time and the war starts. The year display will always tell you how many years you have remaining.
• You reach your Gold debt limit.
• The head of the family dies without an heir.

The amount of Legacy you have at the end of the game is your final score.

My early-in-development creative process.

I don’t usually bother fleshing individual ideas out in design documents because it usually takes just as long for me to implement those things and see them in the game instead. And who would want to read about an idea when you can see how it works in the game directly?

During this phase I need to generate lots of different ideas because only some of them are going to stick. The more varied the better, so I like to approach my idea generation from different angles. The two most common approaches are starting from the theme, and starting from the mechanics

For example, in Lasting Legacy, we quickly came up with occupations like Family Doctor or Ball Organizer from the theme, and figured out what useful things they could do in the game (heal people, and attract new friends respectively).

We also came up with several occupations starting from a mechanics point of view. For example, we knew we wanted someone to increase the income of other people, so we came up with the Savvy Businessman occupation.

This time around I also used a third approach to generate ideas: Isomorphism.

Isomorphism is a word that comes from the greek composed of “isos” (same) and “morphe” (shape) and it means that two concepts are equal. This concept is very useful in mathematics. Imagine you have a very tough problem that you don’t know how to solve. Instead of having to go through the very difficult task of trying to solve that problem, you could instead show that it maps exactly (is isomorphic) to another problem you have already solved.

How does that help us with game design? If we can determine that our game, or at least parts of our game, map to parts of other games, we can look to see if the mechanics of those games can apply to ours.

I didn’t set out to design the Lasting Legacy this way, but it turns out there is a close correspondence between a lot of concepts in Lasting Legacy and Magic: The Gathering. That’s really good news because not only is Magic a great game, but it has been in active development for 25 years and it has explored a lot of different mechanics. So being able to draw parallels and tap into those ideas is a great resource and a very different angle of attack to a tough problem.

The mapping between Magic and Lasting Legacy goes something like this:

• Hand of cards = Friends
• Permanents = Active family tree
• Graveyard = Dead and inactive family members
• Library = Upcoming friends

I want to highlight that we’re talking about looking at concepts from another game, mapping them to our game, and then applying them, not using them straight. Those concepts couldn’t be used as it is, or wouldn’t make any sense, without the mapping from game to game. For example, in Magic: The Gathering, the player has a hand of cards but Lasting Legacy doesn’t, so the idea of discarding a card to achieve something doesn’t apply. Once we establish the mapping between the hand of cards and available friends, we could use the idea of dismissing a friend for similar purposes.

These are are some of the mechanics that we were able to map and apply to Lasting Legacy.

Bringing cards back from graveyard. Reanimating creatures has been a staple of black decks in Magic from the beginning. Thematically we could revive a dead family member, but that doesn’t fit very well with the more realistic approach of Lasting Legacy. Instead, we created a Genealogy Researcher, who can become the same occupation as a dead family ancestor.

Looking at cards from the top of the deck. Deck searching and manipulation is a very common blue Mechanic in Magic, but Lasting Legacy doesn’t really have the concept of a deck of cards, so we can’t do a lot of that. We can, however, look at the “top” X friends that would come up and pick one, which is exactly what the Dashing Socialite does. It’s a very useful occupation because it gives you a degree of filtering of your friends and reduces randomness.

Enter the battlefield effects. In Magic, many creatures have an effect that is triggered whenever they “enter the battlefield” (meaning, enter in play). This is a very useful tool, and we adopted it in Lasting Legacy. It gives you another reason to bring someone into the family, even if it’s not for their occupation or even to extend the main family branch. For example, the Charismatic Musician increases your prestige whenever he enters the family.

Blinking. Several cards in Magic allow players to “blink” permanents. That means those permanents go away and immediately enter into play again. That’s very useful to protect those permanents or re-triggering their enter the battlefield effects. We haven’t added this to Lasting Legacy yet, but it’s definitely towards the top of the lit of possibilities, especially if we end up with many other occupations that have effects when they join the family. This mechanic might make for an interesting subtheme for an expansion down the line.

There are many more mechanics that could be mapped to Lasting Legacy, but these are some of the most interesting ones we’ve done so far. If we end up making expansions, I’m sure we’ll revisit this topic as a source of new ideas and mechanics.

Apologies for the less than ideal sound and video quality. We’re working on fixing that for next time.

By now you’ve probably heard of Lasting Legacy: you’re managing a family around the 19th century through several generations, socializing, choosing good marrying prospects, and helping family members pick an occupation. Ah, occupations…

Occupations

Occupations and the life and blood of the game. It’s what gives the player lots of new actions and effects that can be combined in lots of different ways to achieve lots of powerful effects. The only downside of occupations is that… there are a lot of them! Right now we have about 50 and that’s just some really basic stuff. I expect the game will ship with about 100 occupations, and possibly more once we add country-specific ones.

50 doesn’t sound too bad you say. Sure, but each occupation has a lot of data that goes along with it: a name, whether it has an action or not, text for the action,text for the log, costs, whether it’s inherited… Hang on, let me tell you exactly what an occupation has:

struct OccupationInfo
{
     Occupation::Enum occupation;
     Rarity::Enum rarity;
     const char* maleName;
     const char* femaleName;
     const char* description;
     const char* logEntry;
     IconSpriteId::Enum spriteId;
     uint32_t periodFlags;
     int goldIncome;
     bool incomeMultiplied;
     int prestigeIncome;
     int legacyIncome;
     bool action;
     int goldCost;
     int prestigeCost;
     bool educationRequired;
     int targetCount;
     bool pickable;
     bool destructive;
     bool inherited;
     Occupation::Enum childrenOccupation;
     int personLike;
     uint32_t tags;
     AccessoryType::Enum maleAccessories[MaxAccessories];
     AccessoryType::Enum femaleAccessories[MaxAccessories];
     int maleAccessoriesCount;
     int femaleAccessoriesCount;
};

How do we set all that data for 50 different occupations? Back when I was working for big game companies I’m sure that someone’s job would have been to create a GUI tool to let designers enter those values. Then someone else would create a fancy XML format that could be exported from the tool. Then, if we were lucky, someone else would take that XML and crunch it during the asset conversion process into something binary that could be read directly into memory. Probably someone else would work on tech that would let you hot load new data while the game was running. Sweet!

Except that now I’m a lowly indie developer and I don’t have time for any of that stuff that doesn’t actually make the game better. Fortunately, I also don’t have a team of 50 designers to deal with, so my solution is this:

{
	OccupationInfo& info = OccupationInfos[Occupation::Banker];
	info = OccupationInfo(Occupation::Banker, "Town Banker", IconSpriteId::Banker);
	strcpy(info.description, "Gain 100 gold. Income decreases by 1.");
	info.logEntry = "%s %s gained %d gold.";
	info.tags = OccupationTags::Gold;
}
{
	OccupationInfo& info = OccupationInfos[Occupation::Poisoner];
	info = OccupationInfo(Occupation::Poisoner, "Cunning Poisoner", IconSpriteId::Poisoner);
	strcpy(info.description, "Kill another family member.");
	info.logEntry = "%s %s poisoned %s %s.";
	info.cost = 500;
	info.tags = OccupationTags::Danger;
	info.rarity = Rarity::Uncommon;
}
{
	OccupationInfo& info = OccupationInfos[Occupation::Tutor];
	info = OccupationInfo(Occupation::Tutor, "Private Tutor", IconSpriteId::Tutor);
	strcpy(info.description, "Another adult family member leaves their occupation and becomes educated.");
	info.logEntry = "%s %s sent %s %s to school.";
	info.educationRequired = true;
	info.tags = OccupationTags::Social;
}

“Oh the horror! Hardcoded data!” shriek all new CS students as they recoil from the screen.

But it really isn’t that bad. Actually, it’s pretty great: I don’t have any exporting/importing to do, I don’t have to load anything at runtime, and the compiler does a lot of checking for me. True, I can’t change it on the fly, but I don’t really need to when rebuilding the whole game takes about a second.

Unfortunately the problem is that we have 50 of those initialization blocks. Hopefully a lot more by the time we ship. They’re perfectly fine to work on each of them in isolation, but I’m not able to get a big picture. Have I created a similar one to this new one I’m thinking about? How much do other similar occupations cost? Can I get all the Social occupations together so I can compare them? How many rare occupations do we have in the late period? I can’t answer those questions very well by looking at that code. I needed something else.

Global View

Using some kind of text file wouldn’t help any. It would be a matter of changing that into an ini or an XML file, which would be much more painful without any of the benefits.

I started setting up a Google Spreadsheet, which is something we did for the specialists in Subterfuge. Spreadsheets have some nice capabilities like doing data validation on cells (for example, to enter enums), or letting you sort records based on some criteria. In Subterfuge the spreadsheet method worked reasonably well because we didn’t have as many specialists and each of them had less data, so the spreadsheet was pretty manageable. Here we just had too much data for each occupation and the spreadsheet was getting unwieldy.

Miguel mentioned SQLite, but I wasn’t too keen on linking with some extra library and loading that data at runtime. It also sounded like total overkill to have a full relational database for what amounts to a single table. Then I realized how wrong I totally was.

People have written GUI tools for SQLite that let you create, edit, and browse databases very easily. For example, we’re using DB Browser for SQLite and I think there are better ones out there (although maybe not free ones). Our main table looks something like this:

And the data itself like this:

It looks like a spreadsheet until you realize that you can very easily search any field and you can even create “views” with more complex queries ahead of time. So now I can quickly see how many rare occupations we have in the late period with a single click. Very sweet! It definitely gives me that global view I was hoping for.

How about loading this data from the database at runtime? I doubt it would be a big deal, but it’s still something I’m not crazy about. Fortunately, it’s super easy to extract the data from SQLite. We have this in our make file:

@sqlite3 -header -csv $(DB_SRC) 'SELECT * FROM Occupations' > $(DB_CSV)

And that spits out a nice comma-separated file with all the data. Still, loading csv files at runtime isn’t much of an improvement, so we do a bit more processing. Initially I started converting the csv file into an ini file (since we already have an ini-reader library), but then I realized there was no point in doing that. Unlike Subterfuge, I’ll never want to have different versions of the game data around and load them based on different rules versions. We just want one true version. What’s the easiest way to put that data in the game? C code!

So with a simple Python script, I convert that csv file into a C file that looks an awful lot like the one we started with, except that this time it’s completely derived from the contents of the database. One of the great benefits of this is that we can let the compiler do a lot of the checking instead of having to do it in the script or (gasp) at runtime. For example, some occupations force their children to have a particular occupation (King -> Prince). I don’t even have any code that checks that the occupation you enter there is valid. We just generate code like this:

info.childrenOccupation = Occupation::XXXXX;

If XXXXX doesn’t happen to be a valid Occupation enum, you get a build error until you fix it. And again, since the whole turnaround time between editing the database and building the game can be a second or two, it’s really no big deal at all.

Constants

I left one small detail: Constants. Look at some of the first code I put up early on:

strcpy(info.description, "Gain 100 gold. Income decreases by 1.");

What’s wrong with that? Yup, you got it. Somewhere else in the code, I have some constants like this:

const int BankerReward = 100;
const int BankerIncomePenalty = 1;

Now every time I change how much money the Banker gives, I need to change the string and the constant (let’s not even talk about localized strings yet).

So instead now I use global string replacement while we’re generating the C code. The string on the database is this:

Gain [[BankerReward]] gold. Income decreases by [[BankerIncomePenalty]].

Whenever we load the csv file, we look for any substrings with the pattern [[…]] and we try to match them from a header file with constants that we’ve loaded. If we find them, we replace them with their values, and if we don’t, we spit out an error and stop because clearly someone wanted to have a string replaced there and probably mistyped it.

Does all of that sound complicated? Not really. When I said earlier that the Python script was simple, I wasn’t kidding. This is the bulk of it, including the constant replacement (just a couple helper functions are missing):

def main():
	dir = os.path.dirname(os.path.realpath(__file__))
	baseDir = dir + "/../LastingLegacy/AssetsRaw/"
	reader = csv.DictReader(open(baseDir + 'db/occupations.csv', 'rU'))

	sourceCodeDir = baseDir + "../src/"
	globalVariables = readGlobalVars(sourceCodeDir)
	pattern = re.compile(r'\b(' + '|'.join(globalVariables.keys()) + r')\b')

	Rarities = {'C': 'Rarity::Common' , 'U': 'Rarity::Uncommon', 'R': 'Rarity::Rare', 'Q': 'Rarity::Unique'}
	Periods = {'E': 'Period::Early' , 'M': 'Period::Middle', 'L': 'Period::Late'}

	defFile = open(sourceCodeDir + "Occupations.inc", 'w')
	for entry in reader:
		type = entry['id']
		defFile.write('{\nOccupationInfo& info = OccupationInfos[Occupation::' + type + '];\n')
		defFile.write('info.occupation = Occupation::' + type + ';\n')
		defFile.write('info.spriteId = IconSpriteId::' + type + ';\n')
		defFile.write('info.maleName = "' + entry['maleName'] + '";\n')
		if entry['femaleName']:
			defFile.write('info.femaleName = "' + entry['femaleName'] + '";\n')
		else:
			defFile.write('info.femaleName = info.maleName;\n')
		
		description = multipleReplace(entry['description'], globalVariables)
		defFile.write('info.description = "' + description +'";\n')
		
		defFile.write('info.logEntry = "' + entry['logEntry'] + '";\n')
		WriteOptionalBool(defFile, entry, 'educationRequired', type)
		WriteOptionalKey(defFile, entry, 'goldCost', type)
		WriteOptionalKey(defFile, entry, 'prestigeCost', type)
		WriteOptionalKey(defFile, entry, 'goldIncome', type)
		WriteOptionalKey(defFile, entry, 'prestigeIncome', type)
		WriteOptionalKey(defFile, entry, 'legacyIncome', type)
		WriteOptionalBool(defFile, entry, 'incomeMultiplied', type)
		WriteOptionalBool(defFile, entry, 'action', type)
		WriteEnum(defFile, entry, 'rarity', Rarities, type)
		WritePeriodFlags(defFile, entry, 'periods', Periods, type)
		WriteOptionalKey(defFile, entry, 'targetCount', type)
		WriteOptionalBool(defFile, entry, 'destructive', type)
		WriteOptionalBool(defFile, entry, 'pickable', type)
		WriteOptionalBool(defFile, entry, 'inherited', type)
		if entry['childrenOccupation']:
			defFile.write("info.childrenOccupation = Occupation::" + entry['childrenOccupation'] + ";\n")
		WriteOptionalKey(defFile, entry, 'personLike', type)
		WriteOptionalTags(defFile, entry, 'tags', type)
		WriteOptionalAccessories(defFile, entry, 'maleAccessories', type)
		WriteOptionalAccessories(defFile, entry, 'femaleAccessories', type)

		defFile.write('}\n')

	defFile.close()	

Here’s the full script in case anyone is interested in the details.

Boom! Done!

Ship it.

I managed to keep things shorter than last time, so this one is only 12 minutes. For those of you who prefer text updates, don’t worry, we’ll have one of those next week.

It’s a whopping 19 minutes long, so I may have overdone it a bit! I’ll try to keep it shorter for future updates.

Since this is the first time we’re doing this, any feedback is appreciated. Would you like to see more of these in the future? What should the focus be? Any technical issues I should improve?

Lasting Legacy is a fairly unique game, so it doesn’t quite fit in any predetermined genre. The closest category would be single-player, turn-based simulation, although the simulation part in Lasting Legacy is very light (unlike something like Sim City or The Sims), and in that respect it’s more like a board game. So it’s more accurate to say that Lasting Legacy is a blend of simulation games and board games.

Intrigued? You’re in the right place. Read on.

What Do You Do in Lasting Legacy?

The description of the game says that you manage a family for about 200 years and build a lasting legacy. Every action you perform moves the time forward one year, and actions are performed directly on individual people.

There are some common, general actions available:

• Socialize, which helps you find more friends.
• Visit a friend, to increase that friendship.
• Propose in marriage or accept a marriage proposal.
• Pick an occupation for someone who recently became an adult.

Those actions allow you to tend to the family and continue growing the family tree. The meat of the game comes from each person having an occupation, and that occupation provides either a special action, or an ongoing effect.

For example, having a Doctor in the family gives you an action to heal a sick person.

On the other hand, a Moneylender in the family gives you the ongoing effect of not having maximum debt anymore (time to run those credit cards!).

The ultimate goal of the game is to achieve the highest legacy possible each time you play. You gain legacy in one of two ways:

1. By acquiring heirlooms from deceased family members that give you legacy for different reasons (for example, number of people in your family with a particular occupation).
2. By meeting the whims of some important family members. For example, someone in the family might have a thing for big weddings, so you’ll gain legacy for every big wedding you have while they’re around.

Sounds simple enough, but in order to do those things, you’ll have to balance the maintenance of your family (social needs, marriages, children, sickness), acquiring gold and prestige, and, to top it all, dealing with the rising military tension in your country. War is inevitable, but you may be able to deflate military tension and push off the start of the war (which brings the end of the game).

Game Mechanics

Stepping back and looking at the game from a high-level point of view, there are two major mechanics that appear:

1. Tableau building. The family tree is your tableau, because it represents the powers and actions that are available to you. A lot of card games have a similar mechanic (Race for the Galaxy is one of the best and most famous ones), but in Lasting Legacy the tableau has a twist: It’s dynamic. People can only make a contribution while they’re alive, so your tableau will change as decades go by, and you’ll have to adapt your strategy and plan ahead to take full advantage of it.
2. Hand management. Your friends are a very similar resource to a hand of cards in most games. You may be able to bring them into play as part of your family, or maybe you can do their action once and discard them. This hand of cards is also very dynamic and it changes with the friendship levels and your family prestige.

Those two mechanics are very common in card and board games. That’s not surprising since Lasting Legacy has strong board game influences. However, Lasting Legacy is not a board game, and it would be very difficult to make a straight translation into a physical game. There’s a light simulation under the hood that makes people do things by themselves: have children, propose in marriage, pick occupations (unless you pick them for them), etc. Hopefully it’s a good blend borrowing the best of both the digital and the physical worlds.

Want to find out more about Lasting Legacy as we announce it? Make sure you sign up for our mailing list and/or join the Facebook page.

Once you have defined that feel, you can run every single design decision by it. Every game feature should support those feelings in some way, if not, they’re a good candidate to cut. And if some contradict them directly, you can veto them right away and not go down that path any further.

For Lasting Legacy, we have two main feelings we want to evoke in players:

1. Experiencing the drama of complex family stories.

A playthrough of Lasting Legacy lasts about 20-30 minutes and the game is intended to be played over many times (more details on this for a later post). Each of those playthroughs is completely different, and the player experiences an entirely different story of a family through the generations. The combination of the player’s actions and the behaviors of the game characters should create funny, memorable, and sometimes bizarre situations.

For example, this is a recount of something that happened in just one of the generations of one of my recent plays of the game:

The Novak family was pennyless and without any income. Mr. and Ms. Novak spent all their effort into trying to gain prestige and money to secure a place in society. They had two children, Jakab and Andrea. Jakab, unfortunately, ended up being the Town Gossipmonger and never held any real jobs. Andrea managed to become a Baker, but the income was meager.

Things started looking up when Andrea, caught the eye of Mr. Kende, a handsome, rich friend of the family. They were quickly married and the family felt the relief as they had access to Mr. Kende’s fortune. The parents approved very strongly of this union and made sure that Andrea would become the heir of the family.

Years passed, and even though the disappointing Jakab was married and provided numerous grandchildren to Mr. Novak, Andrea and Mr. Kende didn’t have any children of their own yet. This was a grave cause of concern for her parents, who even encouraged them to see an herbalist to increase their fertility. A few more years passed and at this point it was clear that there would be no children out of that marriage. Something had to be done or the dynasty would end right there.

Jakab started spreading rumors about how Mr. Kende had come by his fortune through shady means, hoping to hurt his reputation and have his parents choose him as the new heir. His attempt failed, ended up being further distanced from his parents, and Andrea was still the sole heir.

Mr. and Ms. Novak died after some time, and Andrea and Mr. Kende, still childless, became the new head of the family. It looked like the dynasty would end here, but that’s when they became acquainted with Ms. Bognar, the orphanage director. She was able to facilitate the adoption of Henrietta, an adorable 6 year-old girl who would become the hope of the family’s future for years to come.

We want to be able to let players share their most memorable stories, but we haven’t fleshed out exactly how we’re going to do that. As you can see from the previous paragraphs, it’s not a trivial matter. One possibility is to export some kind of autogenerated scrapbook that players can annotate if they want to, but there are many other ways we can go wit this (Any great ideas? Let us know in the comments).

2. Feeling smart for discovering combinations of occupations that create really powerful outcomes.

The explicit goal of Lasting Legacy is to score as many points as possible (i.e. build the largest legacy). In order to do that, players will acquire gold, prestige, and eventually turn it into legacy through different means. To get really high scores, players will have to carefully combine the actions and abilities from the different members of their family and their friends. The more outrageous the combination, the bigger the score.

This is an example of the kind of thinking that goes behind finding combinations of powers:

The current head of the family likes prestige and will give 1 legacy for every 10 prestige the family earns. You have a Philantropic Benefactor in your family that gives you 5 prestige for every 50 gold you “donate”, but gold is running low. However, you also have access to a Ruthless Negotiator, who reduces all action costs by half, so each Philanthropic Benefactor activation gives you 5 prestige for 25 gold. This works for a few years, but you’re almost out of money.

Then you notice that one of your friends is a Radical Anarchist, whose power is to get rid of all debt but he goes to jail. So you bring him to the family, rack up the debt to the maximum with the Philanthropic Benefactor, and finally send off the Radical Anarchist to erase all the debt. As a result, you gained a large amount of legacy, greatly increased your prestige, and didn’t spend much gold in the process.

This is very similar to the kind of combination of powers you find in games like Magic: The Gathering (a huge influence on me in general), and also reminiscent of the specialists powers in Subterfuge.

Those two goals are orthogonal. I could see some players enjoying one but not caring about the other, so we’re making the game that can be enjoyable even if you just care about one of those aspects. We’re making sure that both kinds of players can enjoy the game to the fullest. Someone who mostly cares about the stories, can just focus on growing the family and dealing with the emerging drama. On the other hand, someone who cares about maximizing their score also need to maintain the family going because it means playing longer and scoring more points, but they can ignore the in-game meaning of the actions they’re taking and just visualize it in terms of how big of a legacy they’re creating.

We hope that once players have played through the game many times and they become really good at it, that they ignore the explicit game goals and make their own constraints. The game becomes more of a toy at that point (unconstrained play) which is an aspect I love in games (we did that with Casey’s Contraptions as well). For example, the player may want to decide to play through a family in extreme poverty that never earns any gold, or a family that is all made out of criminals and low-life characters, or maybe just try to breed the most extreme physical characteristics possible. We hope there’s a lot of room for exploration and enjoyment beyond just getting high scores.

You can read more about Lasting Legacy in the official web site. There isn’t a lot of information yet, but we wanted to announce it now so we can talk openly about it until release. We’ll definitely be revealing more information here over the upcoming weeks.

Platforms

The first thing that might surprise some people is that we’re going to release Lasting Legacy for Windows and Mac first (hopefully through Steam). This is a departure from all of my games as an independent developer, but I feel that PC is a better fit than mobile for Lasting Legacy. Also, I know Steam is flooded with new game releases, but it’s not quite the mad house that mobile is these days, especially for a game that follows the old “pay-once play-forever” approach.

It won’t be difficult to make a tablet/phone version later, so assuming it’s moderately successful, we’re likely to make an iOS version afterwards (Android is such a horrible development environment for native C games, that it probably won’t be worth our time). And I suppose there’s always the possibility of a console version if the opportunity presents itself.

Goals

Why are we making Lasting Legacy? There are several high-level goals I had for my next project.

1. Shorter development. When I look back at the development times of the games I released over the years, I see a disturbing trend.

   

   I really didn’t want my next project to take nearly as long as Subterfuge did. Part of it meant tackling a game with a smaller scope, and part of it meant making it single player. My hope is that Lasting Legacy takes us less than 18 months from start to launch. Let’s see if we can make it.

2. Potential for wide appeal. The nature of Subterfuge meant that it was always a very niche product. We went into it knowing that and we successfully created a game that a very small percentage of people really love. This time I wanted to change things and wanted to create a game that had potential for wide appeal. It doesn’t mean that it’s going to be a clone of existing games or that we’re going to dumb down anything. It just means that more people might like it, and some will hopefully love it as much as the devoted Subterfuge fans.

   So far, in the extremely tiny, non-statistically meaningful sample of our close families, my wife, who is definitely not a video gamer, really likes the idea of the game and is interested to hear more about it (which is more than I can say for any of my projects since Flower Garden).

3. Low money investment. Maybe it’s not quite the Indipocalypse but there’s no doubt it’s a lot harder to make money from selling independent, original games than it was 5 years ago. Because of that, I don’t feel comfortable investing a large amount of money into a project that might or might not end up being financially successful. That means picking a game that we can do almost completely between the two of us, minimizing the amount of contracting we have to do, and the scope of Lasting Legacy is perfect for that.

Roadmap

We still have a lot of work to do ahead of us.

• Steam Greenlight. Once we have a lot more content in the game, we’ll be able to create a video for the game and get it on Steam through the Greenlight process.
• Early access. Once we’re on Steam, we’ll have a period of Early Access. Because the Lasting Legacy is highly replayable, it’s going to be a good candidate for Early Access. That time is also really going to help us fine tune the balance of the game and take player feedback into account for the final release.
• Release. When will this be? Once the game is done (but it better be this year!!).

Want to help us make Lasting Legacy a success? Make sure you sign up for the mailing list and/or join the Facebook page. Apart from progress updates, we’ll announce when the game is on Greenlight so you can help vote us through it. Once we hit Early Access, we’d love it if you can play the game and send us your feedback.

Some interesting observations:

• I made zero income in my first two years as as independent developer! I was supplementing my income writing a column for Game Developer Magazine, teaching classes, and doing some contracting. I even started interviewing thinking I would have to give up the indie life and go back to work for a company. I always tell new game developers to be ready not to make any money for a while for this reason.
• Flower Garden seems to have a reasonable tail in spite of minimal updates. Every so often I feel a bit guilty for not doing more with it, but even if I wanted to spend time on it, I can’t think of anything that would increase sales.
• Subterfuge is a blip in the radar. But what’s even worse, Subterfuge took 3.5 years to make, whereas Flower Garden was around 2 years (including all updates).
• The US tax system isn’t really set up for people with really spikey income like this.
• I need to make games more quickly.

Here’s to releasing a game in 2017 that is visible on that chart!

Instead, I’m going to try something a bit different. I want to highlight the games that have had the most influence on me as a game maker. Not the best games, not the most memorable games, not the most impressive games, but the ones that have directly influenced me and the games I’ve made.

It was actually really hard coming up with this list. I initially had games that I absolutely love as a player and I spent many hours playing them. When I started fleshing things out, I realized they didn’t directly influence me as a designer, so I ended up cutting them, even though it was really hard to do that (anyone who knows me will wonder why Fallout or Master of Orion are missing from this list).

Sorcery (1985)

This game single-handedly set me in the course of making games. I saw Sorcery at a friend’s house, and I knew I had to make my own games, even before I owned a computer. I was blown away with everything about it: The sense of exploration and adventure, the atmosphere, the graphics, and the gameplay. I even dared replay this game recently and it mostly stands the test of time (although the tunnels over water are as frustrating as always).

The Hobbit (1982)

This is the oldest game in the list, but I actually didn’t discover until later. The Hobbit, along with other text adventures, changed my preconceptions of what games could do. I felt I was free to walk around and interact with the world in any way I wanted in a way that graphic adventure games wouldn’t let me.

I also learned the power of text in games, and how some good paragraphs of text can be much more evocative than graphics.

All of this led to the first serious game I completed: La máquina del tiempo (which was an adaptation of H.G. Well’s The Time Machine).

Macadam Bumper (1985)

This was the first game I played that allowed me to create my own levels (I suppose that in another world I should have played the earlier Pinball Construction Set). For a budding game creator, this was a dream come true. I remember spending many afternoons with my cousin making our own version of a pinball table at the local arcade. I didn’t play it much beyond that, but the seed of games with built-in level creation had been planted.

Casey’s Contraptions is clearly the game I created that was most directly influenced by Macadam Bumper.

Sim City (1989)

The idea of a game being a simulation you can interact with was completely new to me, and I was instantly drawn to Sim City (even with the horrendous initial graphics). Later versions became much prettier and refined, but the basic idea remained the same. As a game designer, simulations are probably my favorite genre, and I’m planning on exploring them more in the future.

Magic: The Gathering (1994)

Another love at first sight. So many ideas from this game were new to me! Customization, card interactions, expandability… I also saw first hand how a game can grow beyond being a game that you play and “finish” to a full hobby that becomes part of your life. I’ve taken some breaks from the game over the years, but even today I’m playing it quite actively.

In spite of its flaws, I consider it one of the best and deepest games I’ve ever played.

There’s a lot I could write about Magic, but I’m going to save it for another (or multiple) posts down the line.

Braid (2008)

Braid (along with World of Goo) really showed me how an independent developer can make not just similar, but better and different games than traditional companies. This was particularly relevant to me at the time since it was around the time I had quit my full-time job to go independent and make games with a friend.

Playing through Braid and listening to Jon’s talks also opened my eyes to the idea of deliberate design. In Braid everything has a reason to be where it is. Every puzzle explores a slightly different consequence of the rules of the game (even when they’re not very player friendly). That’s an important idea I’ve tried to carry with me since then, and it definitely had an impact in Subterfuge.

Binding of Isaac (2011)

Binding of Isaac is a bit of an odd duck in this list. I’m usually not a big fan of games that rely on speed and reflexes, yet Binding of Isaac is very much a twitch game. It’s also a game that was not love at first sight. I played it for a bit and put it down. I even remember asking on Twitter what’s so good about it. Fortunately some friends convinced me to give it another try, thinking about it less as an arcade game and more as the rogue-like it really is. At that point everything clicked in and I was hooked.

It was my first and strongest experience with rogue-like games (sorry Spelunky, I tried but never really got into you). I fell in love with the idea of games that can be completed relatively quickly, but are intended to be replayed over and over as things change from play to play.

Observations

I had no idea what list I was going to come up with when I started writing this, so it’s quite interesting to see the date distribution. There are more games from the 80s in this list than any other decade. That’s to be expected since that was my first contact with games and that’s when I had the strongest influences.

What’s more interesting is that all decades since then are represented. I consider myself to be pretty jaded with respect to games (and movies and books), so it takes a lot for me to take notice of something and get excited about it. So I’m really glad to see that there are standout games recently that wow me with something new and awesome.

Another interesting point is that there are no Japanese games in this list. Most game developers I know (especially in the US), grew up with Nintendo and Sega consoles and playing lots of Japanese games there. On the other hand, I never played more than 5 minutes of a Mario, Zelda, or Castlevania game! Instead, I grew up on Jet Set Willy, Roland in Time, and Dizzy. Whether you want it or not, that definitely changes the way I see and interpret games today (it never happened, but maybe I’ll revisit the idea in the future).

At one point I was even going to use that source of uniqueness to make a game based around my love of the old games I grew up with. The idea was to present something familiar but that would be different to most people that had a different gaming background than me.

Finally, the game I’m currently working on (which I hope we can announce any day now) has very clear influences from games on this list:

• Strong game element interactions and combinations(Magic: The Gathering).
• Some simulation elements (Sim City).
• Deliberate design (Braid).
• Rogue-like structure (Binding of Isaac).

Hopefully next post I can finally talk about this super-duper secret project.

Happy new year!

2015 was the year when I started focusing on playing more often the games I loved, and stopped chasing after trying all the hot new games.

Towards the end of 2014 it dawned on me that most new board games I was playing were ranging from bad to just OK, and only a selected few made it into the good or very good category. So why was I spending so much time and energy playing new games? It’s not like I had played the old ones too much and was done with them. As a matter of fact, I was frustrated with how little I was playing them, and how, if I only play a game once every six months or a year, I’m basically starting over from scratch. So I was always playing games at a very superficial level, without being able to get deeper into them. Not to mention it’s a pleasure to pull out a game and not have to worry about reading or explaining the rules!

2015 was the year of depth for me.

Of course depth comes at the cost of breath, so when it comes time to make a best-of list for 2015, I don’t have nearly as many different games to draw on as other years. Because of that, I’ll limit my list to just 3 games.

Honorable mention: Magic: The Gathering

Magic: the Gathering and I go a long way back. I’ve played Magic on and off for over 20 years and it’s irrevocably intertwined with a lot of the different stages of my life. It has also strongly influenced me as a game designer.

But why list Magic as one of the best games of this year? Because I started playing seriously it again and it’s an amazing game, in every way better than it was in 1995. Standard competitive play is constantly evolving and adapting, and booster draft is very balanced, fun, and skill-intensive. Wizards of the Coast did an amazing job with the Khans of Tarkir, Fate Reforged, Dragons of Tarkir, and Origins sets, and even though Battle for Zendikar was a big let down, I have my hopes up for Shadows Over Innistrad.

3. The Gallerist

Even though I’m moving away from complex games, one of my favorite games of 2015 was The Gallerist by Vital Lacerda. Kanban, by the same author, snuck in my top games of all time list that I created last year, and I like the Gallerist better. The theme is quite unique, the mechanics fresh and different, and you’re hardly ever locked out of a particular action, which is something I find very frustrating in games. The game feels like a giant puzzle with lots of gears where you need to figure out how to make the most money before the end.

To top it off, my wife Amy had a painting accepted in the abstract paintings category, so it’s really cool to play a game with a piece of her art in it.

2. Grand Austria Hotel

I love it when a neat theme meets good design into a very cohesive game. That’s exactly what happened in Grand Austria Hotel, and somehow, it manages to go beyond the usual resource-conversion-to-points-engine to something fairly unique and interesting. Two warnings with this game though: 1) It can be a bit brain-burning towards the end with all the bonus actions, and 2) I would avoid playing this at full player count. There’s a fair amount of downtime, and the board state would probably change too much between turns. It worked great at two players and I suspect it would be good with three as well.

1. Pandemic Legacy

(Spoiler free) And the #1 game for 2015 is… Pandemic Legacy! But… this is not the glowing review you may be expecting of the game that rocketed up the BGG rankings and was the first game to displace the Twilight Struggle from the top spot in many years.

I love Pandemic and I love the Pandemic expansions. My wife and I play it frequently and it’s always a fun experience, teetering between success and world collapse. We can play it over and over. None of this should be a surprise since Pandemic was #6 of my all-time games list from last year.

How about Pandemic Legacy? Frankly, if I have to choose, I’ll pick regular Pandemic. Pandemic Legacy is a nice little distraction. It’s a sequence of games where the rules keep changing a bit, sometimes half way through a game. If I didn’t already know and love Pandemic, I might even find it annoying. We’re only about half way through the campaign, but I feel that nothing we do has a meaningful impact in the overall storyline. Things are going to happen independently of what you do, and you’re just going to reveal the next card in the legacy deck.

Still, is Pandemic Legacy fun? Yes, but because it’s Pandemic. It’s good enough to take the #1 spot for 2015 for me. But plain Pandemic is better and it’s the place you should start if you haven’t already.

How about you? Any interesting picks for you top board games of 2015?

Yikes! It was over a year ago! That’s probably the longest gap ever in the history of this blog. What happened? And most importantly for the two of you still reading, what’s going to happen over here?

Subterfuge happened mostly. It was a long project that ran much longer than we expected. This last year I was focused on getting Subterfuge out the door. The little bit of writing energy I had went into writing for the Subterfuge development blog.

For those of you who missed it, here are some of the highlights of the stuff I wrote focused on game design. Also check it all out because Ron wrote a bunch of good stuff as well:

• A Board Game At Heart. Where I explain how Subterfuge is more like a board game without any of the physical manifestations of a physical board game.

• Luck in Subterfuge. Applying one of my favorite topics to Subterfuge.

• Balancing Different Paths to Victory. Balancing a competitive, multiplayer game was tricky and new for me. Lots of lessons learned.

• Emergent Depth. Why simplicity and depth are not mutually exclusive.

• Depth-First Design. More on how we first designed Subterfuge for depth.

• Gaming the Game. The pains of a multiplayer, competitive game. And that was even before launch!

• Cycling and Subterfuge. One of the least popular posts, but one of my favorites, highlighting how Subterfuge and road cycling have several similar dynamics.

OK, so I was busy. You get it. What’s going to happen over here in the future? To be honest, it wasn’t an easy decision.

I considered stopping writing new entries in the blog and just leaving it up as an archive. Interesting game design conversations happen over at Twitter fairly frequently and so I have a reduced impulse to write here. Unfortunately those conversations are a) hard to follow b) quickly lost. So if what I write gets split up in a bunch of small tweets, it would be an overall loss.

At the other extreme, I considered getting back on a regular schedule and writing one article every week while back in the days of #idevblogaday. It’s tempting, but I didn’t like the pressure to have something written by a particular day. Nor did I like to feel that I had to come up with a significant, polished article.

So instead I’ll take a middle road: I’ll write relatively short posts in a relatively frequent schedule. How’s that for vague? :-)

As for the content of the posts, it will all be centered around game design (probably not much pure programming anymore) and hopefully cover all the phases of my next project. I’m also curious about possibly doing some streaming, but I have to find some content that fits well with that medium.

If you have some topics you’d like me to cover, make sure to let me know on the comments or on Twitter.

Happy 2016 everybody!

This list was inspired by Casualgod’s 3rd Annual Top 20 Games of All Time. David and I have a lot of overlapping tastes, so if you like what you see here, make sure you check out his list as well.

20. Arkham Horror

I make no claims this is a good game. However, this game is an amazing experience. The cooperative aspect is very important, and it’s definitely a game that you get better the more you play.

Our first few games were way too long (3-4 hours), but once we got the rules down, now we can knock a game out in 2 hours or less.

The expansions add a lot of much-welcome variety. My favorite way of playing with them is to focus on each expansion, by just using about half the cards from the base game, and the rest from the expansion. That way each expansion shines to the maximum, with a lot of encounters being directly relevant to the theme of the game. Mixing them all just dilutes the experience and makes it a worse game than just plain base game.

19. Keyflower

The rating for Keyflower could change quite a bit (up or down) simply because I haven’t played it enough. My plays of it have been great, and I’m looking forward to playing it a lot more.

The mix of bidding on tiles and using them at the same time is brilliant. Knowing some of the winter tiles ahead of times let’s you plan more strategically for the end of game scoring. At least for the first few plays, this is probably best with 3 players, otherwise it’s too many villages to keep track of.

18. Kanban: Automotive Revolution

This is the only entry in this list for a 2014 game, and it wasn’t even part of the original list when I first put it together. However, a couple more plays quickly shot this up the ranks. I wouldn’t be surprised to see this higher up in next year’s list.

Kanban is a complex game, and on the surface like the kind of games I’ve been getting away from recently. I don’t like games that are complex for complexity sake (like Madeira), but Kanban pulls it all together and makes it work very well.

There are a lot of things going on on the board, but each department is quite simple. The complexity comes in how to put together all the actions in the most efficient way. I’m four plays into this game and still learning new games.

Also, each play is quite different given the meeting goals and end of game bonus. Variability between games is a theme that will appear over and over in this list.

17. Notre Dame

One of my favorite Feld games (Castles of Burgundy barely missed the list). It plays in about an hour, the card drafting adds a nice limitation on your actions and keeps you very interested in what your neighbors are doing.

The extra cards in the mini-expansion make the game much better since it’s more varied from game to game.

16. The Palaces of Carrara

I have to admit that my first play of this was underwhelming, but repeated plays quickly turned it around, and eventually pushed it all the way to #16.

Variability in this game is huge and no two games are alike. You have to re-evaluate everything depending on the scoring cards that come up for each game. Sometimes you build wide, sometimes you go tall, sometimes you focus on some buildings, or cities, or… The resource market is very simple and interesting, and the bits are great too.

Word of advice: Ignore the basic game. Sure, play with the basic goals once, but with the rest of the advanced rules (not moving the wheel, upgrading buildings, 8-point buildings, etc). I didn’t and I almost missed out on this gem of a game.

15. In the Year of the Dragon

This is my other Feld on the top 20, and not surprisingly, it’s from the same period as Notre Dame. It’s still pretty focused, and quite different than most other games: You see all the disasters ahead of time, it’s a matter of planning around them, maximizing scoring opportunities, and fighting everybody else who is trying to do the same thing. Deliciously tight and it plays great with 5 (and also great at lower player counts).

14. Race for the Galaxy

Yet another game that is hugely variable from play to play. Here you have some control (through Exploring) but ultimately it comes down to what you find and how you use it.

This is one of the rare games that has simultaneous action selection that I actually like, probably because other people’s choices can only benefit you, and never negate your whole turn (unless you tried to get really cute).

I haven’t even started playing with the expansions, or with any of the alternate lines. Looking forward to many more plays of this.

13. Santiago

Santiago was a total surprise. Yes, I had read very good comments from my Geekbuddies, but I really wasn’t expecting it would be that good.

Santiago embodies everything I’m looking for in games these days: - Strong player interaction (you’re competing but still need to rely on other people) - Short playtimes - Simple ruleset

The result is an amazing game that some people describe as vicious, but I choose to think as self-interested. You’re in it to win, and other players will have to make it worth your while if they want you to do something in particular. It’s great that the game can be won through smart bidding, being a hard negotiator as the Overseer, or just by saving up lots of money and being opportunistic.

12. The Great Zimbabwe

This is a very unique race game (the same way that Race for the Galaxy is a race) where you can change the ending line in exchange for some advantages. It features the kind of player interaction I like, where you’re competing with people, but at the same time using each other’s resources in a positive way.

Each game is very different depending on the map layout and which gods come out. The only downside is how suddenly the ending can arrive, and how it can be spotted a turn or two in advance and there’s nothing that can be done to stop it.

My favorite player count for this is probably 3 or 4.

11. Troyes

I never made a top-20 list last year, but I think this would have ended up much higher on last year’s edition. Unfortunately I only played it twice on 2014, so I’m not surprised it slid down a bit. Maybe next year it’ll go back up.

I love dice as inputs, coupled with dice-manipulation mechanics, and this game is all about that.

Extremely varied gameplay is again present here: Only a few cards come up in each game, that those completely determine how the game is played (along with the event cards). No two games are alike, although that sometimes can be a bad thing (I introduced a friend to this game and we ended up with punishing events and cards that didn’t combine very well with each other).

I was a bit disappointed with the expansion, so those of you looking for it, I’d say skip it and save all that money.

10. Ra

By now this list has quite a few games with bidding mechanisms (and there will be more further down). Ra is the purest of them all in that it’s only about bidding.

It’s another one of those games with a very simple ruleset that I love (with a slightly complicated end-game scoring, but it’s easy enough with the player mats).

The thing that makes Ra stand out is that the bidding is very “chunky”. You have four tiles and you can bid one of them. That’s all. Sure, it makes valuing things easier than Medici, but that’s not a bad thing.

9. Lord of the Rings: The Card Game

It’s not surprisingly that an LCG is very variable, but I’m always amazed at how completely different each scenario makes this game feel. And by that I don’t mean that one scenario is more combat heavy and another one more questing heavy. It’s that the new rules and events for each scenario change the game completely, and at the same time make it so extremely thematic. For example, in Journey Down the Anduin, at one point you get on a boat and you’re not attacked by the enemies in the engaging area, but you draw an extra card during the encounter phase (because you’re traveling so quickly).

I haven’t tried this solo yet (which I hear is brutally hard), but it’s great fun with two players. Some scenarios are extremely hard though (I’m looking at you, The Hobbit!), and most of them encourage you to tweak your decks in different ways.

8. Glen More

I love the jump-ahead mechanic in this game. There’s always a great tension between getting something before other people, and still getting the other tiles you want along the way. The end-of-game penalty for having larger towns makes it even more interesting.

The tile-laying part of the game is very interesting too, because you have some restrictions (roads and river) and only the tiles next to a tile you lay down are activated, so you need to plan very carefully.

All in all, it combines short play time, and extremely meaningful and difficult decisions. Oh and it plays great with two as well.

7. Hansa Teutonica

For the longest time I kept reading about Hansa Teutonica as the poster child of boring, dry euros, as well as having a reputation for being extremely aggressive.

I don’t mind my euros dry, but I typically don’t like them aggressive. However, it turns out this was love at first sight for me. From the first play, I enjoyed every minute of this game. I didn’t see the play as aggressive, but more of a blocking play that you get a benefit when someone moves you out of the way.

Unlike many other games in this list, this doesn’t have much (any?) variability from game to game, but because there are so many ways to approach it, it can still feel quite different.

I’m looking forward to many more plays and trying some of the expansion maps as well.

6. Pandemic

My highest-rated cooperative game. There may be some nostalgia in here, but I would pick this one above all other coops.

Cooperative games have strong random elements, and sometimes things are stacked in such a way that no matter what you do, you will fail. That’s the nature of the game. Pandemic has that problem as well, but play time is short enough that it doesn’t matter. You pick up the cards and do it again.

I love that the infection is driven by a deck of cards because you have a lot of information about what is likely to come up soon (since the discard pile is shuffled and put on top after every epidemic). That’s something that was sorely missing from Pandemic The Cure (the dice version of Pandemic).

Finally, in case things start feeling too samey, the expansions add a lot of variability (I didn’t like my first play of In The Lab, but I suspect it has a very steep learning curve).

5. Kingdom Builder

This game is the definition of variability from game to game. The rules themselves are just a couple of mechanics on how to put settlements on the board, and the scoring is mostly down to the cards that are drawn at the beginning of each game.

This was another one of those games that the first play felt a bit underwhelming. I thought the game was playing me, rather than me the game. It wasn’t until the second game that I realized it was about positioning myself to accomplish what I want to do, rather than trying to select different terrain cards. It’s also really interesting how you have very different goals early in the game (minimize terrain adjacencies so you can move to other places on the board), than you do late in the game (touch as many terrain types as possible so you can put your pieces anywhere you want).

Yes, you can get some ridiculously broken power/scoring combos, but again, games are quick. Have a laugh and play it again.

4. Homesteaders

Short, tense, and every decision is extremely meaningful. There are only 10 rounds to the whole game, and you won’t even get a building in each round. You can go hugely in debt, and yet (hopefully) everything kind of comes together at the end.

It certainly fits with me liking bidding games, but it might initially stand out as a game with very little apparent variability from game to game. Rahdo even dismissed it because it didn’t vary enough. Let me tell you something: This game is so tight, that minor changes will throw off your long-term plans in no time. I recently started a game where I had decided I was going to go for the Church and in-game points, but the way the bidding tiles came out, and the way Amy was bidding against me, I was forced to change plans and go for a resource/dudes strategy. So much for lack of variability!

It plays fantastic with any player count. The only downside is that a lot of buildings are available right from the start, so it’s a bit overwhelming for new players. Well worth a small time investment to learn this fantastic game though.

3. Age of Steam

I’ve tried a lot of train/route-building games, and Age of Steam comes out clearly on top (I need to play more Steam to decide between the two though).`

The auctions for player order/bonus are extremely interesting an unforgiving. The roles themselves are very well thought out so even if you don’t get first place, you can still snatch first build or first move.

The route building and delivery aspect are always very fun and satisfying, the only downside is the randomness of where new goods appear on the board (yes, you see where they’ll appear, but you don’t know when, which makes all the difference in the world). Players can also find themselves in a downward spiral that takes them out of the game completely, but that’s part of the experience of the game (although I agree that a 2-3 hour game where you’re out of contention for half of it is not very fun, experience or no experience, so don’t let that happen).

Finally, the variety of maps for Age of Steam out there make this game infinitely replayable. It’s not jut a different distribution of mountains and rivers, it’s that most maps radically change a few rules and transform the experience into something completely different.

2. Dominion

Speaking of variability, here’s Donald X’s other hugely variable game. Dominion was the first deck builder. I’ve played a bunch since then, and I keep coming back to this one: it plays quickly, works great with any player count (but best with just 2 or 3), and it’s always completely different.

Just like Troyes, some games are going to be more amazing than others depending on the combinations that come up, but it’s always really fun looking at that new set of cards and figuring out a path to victory.

1. Brass and Age of Industry

I’m giving my #1 game of all time award to both Age of Industry and Brass. It might feel like I’m cheating, but I really consider Brass as a particular Age of Industry map with some funky rules thrown in. The fact is, both games feel very similar (with card drawing being the main difference).

I might give Brass the edge as a best game, but Age of Industry wins because of the different maps available and the different experiences it provides. I go back and forth between which one I like better, but I have them both rated as 10 (my only two 10 ratings on BGG).

They have the exact kind of player interaction that I love, where players are jockeying for position, using each other resources, and yet trying to out-do each other. I love that player order is determined by how much you spend in a turn, which gives you very interesting control over your turn.

I also consider Brass to be game that can be best played online: each turn is fairly long and meaty, and there are at most 16 turns in a 4-player game (most likely you’ll have a couple of them back to back, so more like 14). As opposed to games that are 100+ turns long, and in each turn you’re just clicking OK to something. I’m always up for a game of Brass online, so if anyone is interested, let me know me and let’s play.

This list first appeared on BGG

I try to log every board game play, so I can see that I logged 674 different plays during the course of the year, and 246 of those were games new to me. I didn’t invent a time-expanding machine, so the time for that has come straight from TV, movies, and video games. After realizing that we were spending just about every evening playing board games, my wife an I cancelled our Netflix subscription and never looked back. I also only played a handful of PC and iOS games during the whole year.

Another good portion of those games were due to going to local game days, and to two board game conventions: Origins and BGGCon.

Having played that many games, I figured it would be interesting to write a best-of list for the year. For this one, I wanted to focus exclusive on games that came out on 2013. I played 43 games that came out on 2013, and of those, these are my favorites.

5. Nations

I absolutely love Civilization and Civ-like games. Unfortunately, for the most part, those work best on a computer due to the long play time, and the large amount of parts, pieces, and interconnecting systems. I’ve tried playing some board game versions (Through the Ages, Clash of Cultures) and while they weren’t bad, I would probably rather play the computer version of Civilization.

Nations is finally the first civilization board game that works really well for me. It has a reasonable play time (under 2 hours for 4 players), interesting player interaction without direct conflict, and a great power curve. One of the key things design aspects that allows this game to work so well is that they did away with the map (just like Through the Ages), so players can focus exclusively on advancing their civilization rather than exploring the map.

4. Bora Bora

It’s no secret that I like a lot of Stefan Feld games. I particularly like some of his earlier games (In The Year of the Dragon, Notre Dame), but I also like some of his recent ones (Castles of Burgundy). Some critics dismiss a lot of his designs as “point salads”, but I do like my veggies! Seriously, there are badly done point salads, where the things you do have little significance beyond getting you some points (Trajan fell in that category for me), and well-done points salads like Bora Bora, where you’re showered with points, but you’re also opening up new paths and possibilities with each new action. Bora Bora hits all the right buttons for me: interweaving of effects on the board, different strategies, and the sense of building something. It’s a tad long with 4 players, but it plays great with 2 or 3.

3. Spyrium

While Bora Bora is a game that hits you with its colorful game board, fancy player boards, and loads of chits, Spyrium stands in stark contrast and it manages to do with very few components: A deck of cards, a few tokens, and a minimal (almost unnecessary) central board. Spyrium is a amazing piece of design because it manages to have a very rich and deep gameplay with minimalistic components.

In spite of having so few components, Spyrium has a huge amount of variability between games. Apart from the variations from the actions other players take, there’s the order in which cards will come up, the order of events for each round, and even the tokens that go on the cards themselves (which is an extra layer of variability for those cards since it changes their cost or effect).

The key mechanic of adding or removing workers for money or the card is very innovative and fresh. It’s also a game that plays well, but very differently, at different player counts. So playing with 4 people after learning the game with 2 will cause you to re-learn everything you thought you knew about it.

2. Prosperity

Prosperity was co-designed by Reiner Knizia and Sebastian Bleasdale. While Dr. Knizia is the big name there, I’m a particular fan of Bleasdale’s earlier design, Keyflower. In spite of that, I was afraid this game was going to be a too-abstract tile-laying game not to my tastes.

Boy was I wrong! Prosperity was my surprise game of the show at BGG Con (and I played a lot of games at the con). It hits the perfect level of abstractness that you feel you’re in charge of a city, but without degenerating in a sea of details that bog the game down. It manages to hit that magical middle-weight game category, that packs a lot of very interesting decisions in a playing time of an hour or less.

1. Bruxelles 1893

Unlike other simpler games, Bruxelles 1893 presents the player with a myriad of choices. It falls a bit in the camp of a Feldian point-salad game, but where each action and choice is very interconnected to the rest of the actions and the state in the board. You want to do action A, but if you do action B before, then A will be a lot more efficient. But of course, if you do C, then B will be even more powerful. Those kind of tough choices elevate this game to the top of my 2014 games.

Bruxelles 1893 is by new designer Etienne Espreman, but with some involvement by Sébastien Dujardin, who designed my favorite board game, Troyes.

Maybe even more interesting than looking at the list itself, is to look at what those games have in common. I have fairly varied tastes in games, but all the games in this list have some strong common elements:

• Tough, interesting choices
• Themes that appeal to me (no generic fantasy or zombies)
• They play well with 2 players
• Some player interaction, but no direct player confrontation and screwage
• Play times under 2 hours

Some of my other favorite games for this year that didn’t quite make it in the top 5 were:

• Freedom: The Underground Railroad: A cooperative game (a weakness of mine) with a very strong history theme.
• Nauticus: By wonder duo Michael Kiesling and Wolfgang Kramer. I wasn’t impressed with their other 2013 release, Coal Baron, but Nauticus is a perfect middle-weight game with possibilities for clever play.
• Coup: Technically this is a 2012 release, but it became much more widely available in 2013. It’s an amazing, quick, social game of bluffing and deduction. . The love child of Resistance and Love Letter.
• La Boca: A unique spacial game that forces players to collaborate to build something they can only see from their side. You really have to experience to understand it.
• CV: A quirky dice-rolling game with delightful art where you build the life of your character. It’s mostly luck-driven, but it ends up with very memorable stories.

I didn’t play every board game released on 2013 (or even close), so take my list with a grain of salt. As a reference, these are the other games I played that didn’t make the cut: Amerigo, Bang! The Dice Game, Blueprints, Bremerhaven, Bruges, Canterbury, Caverna: The Cave Farmers, Cinque Terre, Coal Baron, Concordia, Cornish Smuggler, Francis Drake, Island Fortress, Legacy: The Testament of Duke de Crecy, Letnisko, Lewis & Clark, The Lord of the Rings Dice Building Game, The Lost Dutchman, Lost Legends, Mount Everest, Náufragos, NOIR: Deductive Mystery Game, Plunder, Quantum, Quarantine, Rialto, Rococo, Russian Railroads, Shadows over the Empire, Stone & Relic, Sushi Go!, Time ’n’ Space, Via Appia, Viticulture.

We don’t talk much about the role of luck in video games, probably because it’s hidden away under the black box of the computer simulation, but just like with board games, it can have have a large impact in the type of experience the video game provides.

Thinking about luck in these terms was crucial for the game I’m working on (still unannounced!). We made some crucial decisions thinking about how luck was part of the game and kind what kind of experience it created for the player. I’m hoping this post helps people with similar design challenges.

This post should apply to any kind of game in general (board or video games). Next time, I’ll be focusing especially on luck in video games using this as a launching point for a deeper look. Also, I’m limiting the definition of luck to random effects built into the game system itself, and not due just to player interaction.

No-Luck Games

In games with no luck, players rely completely on their skill to win. In that way, they’re closer to sports. Games become an intense, straight competition, pitting players’ brains against each other. Right there it shows how luck (or in this case, the absence of luck) creates a very specific feel to a game.

Good examples of games without any luck are classics such as Chess or Go. There are also plenty of modern board games with no luck, like Puerto Rico, Caylus (they both have a minimal amount of luck in the initial tile order), or Hive.

It’s interesting that a lot of abstract games tend to have no luck, and the more thematic a game gets, the more they seem to rely on luck.

Are You Feeling Lucky?

Having some amount of luck in a game can be very beneficial for most kinds of board games. It accomplishes many things:

1. Keeps things varied from game to game
2. Keeps players feeling they have a chance to win even if they’re not currently ahead
3. Removes pressure from winning players (“If someone beats me, it’s because they had a lucky streak”)
4. Makes players who didn’t win feel they stand a chance next time they play (“next time I’ll catch a break and I can win!”)

Points 2, 3, and 4 all encourage more people to play the game and feel they’re competitive at it, even if they didn’t win (and even if they’re not really competitive). One of the best examples of this is poker: Everybody feels they can do great at poker, if only they get good cards. In reality, this is not true in the long term, but poker introduces plenty of luck that it really is true in the short term.

A consequence of all those points is that having some amount of luck allows players of different skills to participate in the same game and enjoy it equally. For games that rely on having multiple people looking to play it, it can be a big factor.

Types of luck

For games that choose to add some luck element, there’s a whole range of amounts and types of lucks they can use for different effects. Unfortunately, it’s also possible to mix the wrong type of luck with a given game feature and create a frustrating experience instead of an enjoyable one.

• Post-action luck. This is luck introduced after the player has made a decision and executed an action. It can be in the form of flipping a coin to see if you unlock a chest, or rolling a dice to see if your armies invade a territory.
• Pre-action luck. Pre-action luck consists of the random events that happen before the player performs an action. The player is able to take them into account and make a decision based on them.
• Hidden information. Hidden information is the third kind of luck. I was a bit hesitant to include it as its own category first, but it seemed different enough from the other two to warrant being listed on its own. Hidden information refers to things that are known only to some players and will affect other players or the game scoring.

Post-action luck

OK, I’m going to say it: I’m not a fan of post-action luck. The player has already made its action and the outcome is random (even if it’s based on a probability curve the player is aware of, like rolling 3 six-sided dice). Since it doesn’t add to the choices the player has, it’s mostly uninteresting. This is the kind of luck that can add a bit of spice to an otherwise boring game, it doesn’t do much to make the game more interesting.

When used incorrectly, this kind of luck is extremely frustrating. The player can feel they chose the “best” action, but they rolled double 1s and their move backfired on them. Sure, there was some tension knowing that could happen, but was it really fun? Maybe the first time or two, but probably not long term.

While I typically really don’t like this kind of luck in my games, there are some situations in which even I will add it can add some interest to the game.

The first case is when the player can choose to perform one action or another, being aware of the different probability curves for both actions. For example, you can roll a single die and deal that damage to an enemy, or you can roll two dice, but if you roll two 1s, your character gets hit instead. In a situation like that, even though it’s still post-action luck, the player was presented with a meaningful decision ahead of time and had to weight the risks and rewards of both options.

The second case where post-action luck can work is when the action is repeated many times over the course of a game. That way, the outcome of each individual action in themselves is not game-breaking, and all the actions will eventually add up to the average over the course of the game. Luck in this case introduces a bit of noise and slight excitement without affecting things much.

This is a good situation to combine with the ability for players to slowly change their probability curves over the course of the game. That way, they can increase their chances of success for an action as the game progresses, presenting the player with a way to feel more powerful. This is often used in RPGs and video games.

Having some kind of post-action luck that affects the outcome of an action can also give players hope that they can do something, even if those chances are small. Otherwise, without any luck involved, they would see the situation is hopeless and lose interest in the game. At the same time, having that luck element makes predicting every possible outcome nearly impossible, so it encourages players to make a decision without spending a long time figuring out an ideal outcome.

Finally, another situation where post-action luck isn’t always a bad thing is in very short games. I love King of Tokyo even though it’s a complete dice fest with lots of post-action luck. Even if you get some really bad dice rolls, a game maybe only lasts 10-15 minutes, so it didn’t feel like a complete waste of time. On the other hand, losing a 4-hour game to a dice roll can be extremely infuriating.

The dark side of post-action luck is the human addition to random rewards, which is the reason why gambling or slot machines are so popular. Games can exploit that human quirk to their advantage and hook players in a game that would otherwise not be very interesting or fun.

A very meta post-action luck is buying “booster packs” of collectable card games (like Magic The Gathering). Purchasing the cards is the action, and the luck happens when you open it and see which random cards were in the pack. As most ex-Magic The Gathering players can attest, this can be extremely addictive.

Pre-action luck

This type of luck can add just as much randomness as post-action luck, but creates a very different feel for the game. Since the random event happens before the player action, even if you didn’t get the ideal outcome you were hoping for, you can choose to do the best action given your situation.

To illustrate the difference, consider power-ups in a first-person shooter. You open the door to one room and there’s a mysterious gift package power-up. You have no idea what it is, you pick it up and… it turns out it was health. Maybe that’s great because you were low in health. Or maybe you were maxed out and it was useless. That’s post-action luck.

Alternatively, imagine you open that door and you see 3 power-ups side by side. You see what they’re going to give you (health, ammo, or a new weapon). As soon as you take one, the others go away. Maybe neither one of them is exactly the ideal, but you can make a decision and pick the best one for your situation. That’s pre-action luck.

In board games, Stefan Feld is the master of pre-action luck. A lot of his games involve some kind of luck mechanism that limits your actions. For example, in The Castles of Burgundy or Bora Bora, you roll dice, and the numbers on those dice determine which actions you can take.

Without going that far, just about any games that involves drawing cards from a deck and having a “hand” of cards uses pre-action luck. The cards you’re dealt are the pre-action luck, and then you have to do the best you can with those cards.

An extreme type of pre-action luck is initial game layout. That happens only a single time during the game, and before players make any actions, so it has the potential to affect the full course of the game. Even players who are adamantly opposed to luck in games, are often willing to accept game setup randomness because it can be fully taken into account during the game without any surprises.

Pre-action luck isn’t as common in games as post-action luck, but it could be used just about anywhere that post-action luck is used. Consider the classic situation of a character attack some monsters and rolling a set of dice that determine whether he hits and how much damage it does. We could change that into pre-action luck by having players roll the dice (either all at once or separately), and having the dice restrict the options of what they can do. For example, low rolls on one dice could indicate that they can only do an attack close to the ground, while high rolls means they can attack flying enemies. Then the player can choose which of those actions to take, or maybe he can instead take a defensive stance or run away.

The main downside of pre-action luck is that it can extend every player action. The more it’s used, and the more possible choices it presents to the player, the longer the game might take, so it’s best to save it for times where the decisions really matter. If not, either post-action luck or no luck at all, might be better choices.

Hidden information

The most common example in board games is hidden end of game bonuses. For example, in Shipyard players get a set of goals that will score them points at the end of the game. There are two reasons for these goals: By giving each player different goals, it encourages players to focus on different aspects of the game instead of fighting over the same set of “optimal” actions. It also encourages players to pay attention to what other players are doing, and potentially try to anticipate or even block other players from getting too far ahead in their goals.

An even more interesting case is the game Troyes (one of my favorites!). Not only does each player get a set of end-of-game goals to get extra points, but all players, not just the player holding them, will be scored based on those goals. That makes paying attention to other players and trying to guess what they’re doing even more important.

At the extreme end of hidden information there are games like Discworld: Ankh-Morpork, in which each player gets a hidden winning condition. Players go about doing their actions until someone announces at the beginning of their turn that they have won the game, and they reveal their hidden victory condition card.

The higher the importance of the hidden information, the more casual and random the game becomes (and so, the shorter the game should be ideally).

I started writing this thinking it would be a quick entry about luck in games, but now it’s grown into something pretty large, and I didn’t even get a chance to touch on luck in video games. I’ll look into that next time.

Related reading These are a couple of great writeups/presentations on the same topic:

• Types of Luck by Antti Karjalainen
• Luck, Skill, and Hidden Information by Robert Gutschera

I’ve always loved board games, but being an only child, it was always challenging to find people to play with. My dad was often willing to play, and I would often pull out some of my favorites when friends came over. But it never felt like I could get enough.

It was no big surprise then, that as soon as I saw games on a computer I fell in love heads over heels. My first thought was “Wow! I can play games with the computer without having to depend on other people being here”. Imagine that! It didn’t matter that the games were uniformly horrible, hardly responsive, chunky, and monochrome. It was nirvana for me.

My second thought after seeing a video game was “I want to make those!” And that’s what I’ve been doing pretty much constantly since that day almost 30 years ago.

During that time board games took a back seat. My love was still there, but I was too busy with video games to consider them as serious entertainment. At times I’m sure I even thought that video games were a step up from board games; the next logical step in evolution.

That phase of my life lasted until grad school. That’s when a good friend of mine introduced me to this new little game that had just come out a couple of years before called Magic The Gathering. You’d think that grad school wouldn’t be the best time to get hooked on a game like that, and you’d be right. It was an amazing ride for several years (from Fallen Empires until Exodus for those of you keeping track). I realized how much I loved the physical and social aspect of board and card games, and how it was mostly missing from video games. At the time I was playing a lot of Warcraft and Quake 1 online, but the experience was nothing like playing Magic across the table from another person.

After my Magic phase, I tried some of the flourishing “euro” games that were just coming out at the time, but I was mostly absorbed back in video games again. Steam and iOS were taking up all my gaming time for a few years.

Then, last year, everything changed. Miguel came to visit with a car trunk full of board games. Literally! Not only did I have an absolute blast playing board games all week, but so did Amy, my wife. We were instantly hooked. Now all my gaming time is dedicated to board games, and I have hardly touched some kind of video game in the last year.

Why I Love Board Games

My love of board games comes from many different reasons.

Physical aspect. I love the convenience of digital distribution of books, music, and video games. But at the same time, I’m tired of manipulating pixels on the screen for all my activities (both work and entrainment). Touch screens are a small step in the right direction, but they still feel very artificial. When it comes down to it, nothing beats the feel of some wood pieces as you move them around the table. There are already too many screens and digital bits in my life.

Social aspect. Video games are mostly solitary experiences. Even multiplayer games, you’re usually not facing the people you play with, unless it’s with one of my favorite patterns: local multiplayer games, which are not extremely popular (and it’s hard to have the right people to play with around). On the other hand, most board games are inherently social. Not only I get to spend lots of extra time with my wife and friends, but I also get to meet lots of new people at conventions and game days.

Variety of experiences. I didn’t quite realize what a huge variety of experiences board games offer. Just a few games off the top of my head that offer a huge range of different experiences: Pandemic, Castles of Burgundy, Escape, Puerto Rico, Mr. Jack, Arkham Horror, or Dixit. And that’s only games in my collection, which is limited to my own tastes and doesn’t include war games or much Ameritrash. Video games probably offer an even wider range of experiences because a computer can be so versatile, but the board game landscape feels more varied and healthy. That might be due to video games being mass market, while board games are still trying to get there. I suspect that’s a topic for a future post.

Learning new games. I love learning new games. You could even say I’m addicted to it. I love to see how things fit together, how the game plays out, and trying to figure out a good strategy. Generally, I’m much happier playing 20 new games 5 times than playing a single game 100 times. With video games, since the game is the running the simulation, not me, there’s a lot less to learn. There’s also something about tutorials that are an absolute chore and I dread learning a game through them.

Replaying games. At the same time, I love replaying games I already know how to play. This is something that often doesn’t happen with video games. A lot of them are intended to make more progress every time you play, culminating in some climax at the end, after 10-20 hours of play. After that, you can play it again, but I usually move on to some other game. One recent notable exception is one of my all-time favorite games: The Binding of Isaac, which forces you to replay it over and over.

They’re for me. I feel I’m squarely in the target audience for a good portion of board games being released. I’m eagerly anticipating releases later on this year, and I can’t wait to read about the new Essen announcements. Most video games, on the other hand, with a few notable exceptions, leave me completely cold appealing mostly to adolescent power fantasies. A quick glance at the top sellers on the App Store or even most console titles, and it’s clear that I don’t belong there.

Fun New World

I love the experience of taking a field I didn’t know much about, and diving right in. I’ve been fully immersing myself in the world of board games. Every day I play one or more board games, and I go to a full-day gaming events once or twice a month. Sometimes it feels like drinking from the proverbial firehose. I’m constantly learning about new gameplay mechanics, the art of finely balancing a game, game evolution and trends, or even the works of specific designers and how they changed over time.

During a period like this, I tend to grow creatively a huge amount. Being exposed to something new causes all sorts of new associations result in a constant stream of new ideas. Some of them are related directly to board games, but a lot of them spill over into video games and even other parts of my life.

So what’s next for me? I’m not giving up making video games any time soon, don’t worry. I love making games in any format, and my goal is to continue making a living making games for different platforms. Board games are like another platform, but one with some strong challenges and constraints.

Of the video games I’m working on, one of them has very heavy board game influences in its design. I have no doubt that it would be completely different had I not rediscovered board games recently. We’re hoping to announce the game and talk more about it soon.

In addition to that, I’m, of course, dabbling on the side with board game ideas. Unfortunately there’s virtually no money in physical board games, so it’s just purely a hobby, but it’s a really fun one. And who knows, I might be able to make a digital version of one of those board games if I end up with a good design.

In the meanwhile, I expect to write quite a bit here about board games, and especially of the cross over between board and video games and what we can learn from each of them to make better games. You can also find me regularly on BGG.

However, the question I was asked over and over was why am I making games? What are my answers to that question? As I said before, there are no right or wrong answers. Everybody needs to find their own answers, so in that respect, my answers don’t really matter.

On the other hand, since what I write here is purely personal, and a lot of people are curious about it, I figured I would give it a shot and answers those questions publicly.

Another thing I also want to stress is that these are not even permanent answers for me. These are my answers for the projects I’m working on right now. Whenever it’s time to start a new project, I will ask myself this question again and probably come up with different answers. That, I think, it’s key to be able to focus on some aspects at the expense of others, instead of always compromising and trying to achieve a bit of everything.

Why I Am Not Making Games

You thought I was going to get right away to the reasons why I was making games? No, it’s not that easy. I need to start with the reasons I thought I was making games, but turns out I discarded (for this time around).

Innovation/awards

I’ve always felt an urge to innovate and create something original. Something that would be worthy of getting some awards and recognition. Even back in the Power of Two Games days, Charles and I (half) joked that if we won an IGF award, we would eat a whole cheesecake from E Street. Unfortunately (or fortunately, depending from your point of view), that never happened, and the cheesecake remained safe behind the glass. Flower Garden was pretty innovative (even to this day, there aren’t many similar games–if you can even call it a game). Casey’s Contraptions had some original aspects, but its main strength was in polish of execution.

For this time around, I decided that this was not going to be one of the main reasons behind me making games though. I’m sure I’ll come back to this one very soon though.

Money

It’s hard not to think about money when one of your main target platforms is iOS. I know they’re mostly the exception, but looking at the huge financial success of some indie developers, or the revenue of some of the games on the Top Grossing chart, it’s only natural to want a slice of that pie.

Since free-to-play is one of the most profitable approaches on the App Store, I originally started out some of my designs by trying to do “free-to-play done right”. I was going to show the world how you can make a free-to-play game, treat players as human beings, not play any psychological tricks, not make the game suck on purpose, and still make a boatload of money.

Along the way, two things happened:

1. It’s a lot harder to design a game that does all those things than to write it out in a neat, little sentence like I just did.
2. Chris called me on it. He asked the really pointed question of “if you don’t need the money, why focus your design around that idea when there are so many interesting design aspects out there to explore?” He was totally right! At this moment, I don’t have the financial pressure to ship a game, so why limit my game artificially like this?

Those two things, combined with the fact that I hate most games in the Top Grossing chart of the App Store, means that I should not even try making a game aimed at that audience.

Since money eventually runs out, I will have to consider money more seriously in future projects. Even so, I will probably go the route of making more personal, niche games that appeal to a small percentage of people, rather than try to go after a pure mass-market approach.

Expressing something

I fall in the camp of people who believe that games are (or can be) art. That’s especially true of indie games, which carry the strong creative touch of an individual or a small team. Starting from something you want to express. For this one time, I was going to put that aside and focus on other aspects.

Why I Am Making Games

After removing all those reasons (and lots more that I didn’t talk about), why am I making games? Or rather, why am I making the games I’m currently making? It turns out, the reasons for these game are mostly selfish.

Enjoy the development

I wanted to make a game where I would enjoy the actual development as much as possible. This is a lot more constrictive than it may seem at first glance, and it means many things:

• I wanted to avoid games with large components that are uninteresting to design and implement (in-game stores, credits, managing users, etc).
• I need to be exploring new concepts (to me) and trying new things to really enjoy the development process, so the game needs to be substantially different from other games that I made in the past: no growing virtual plants, or creating physics puzzles to share with friends.
• The focus of the game should be on the design itself, not some technology. Even though most of my past experience comes from programming, I’m only interested in design these days. Programming is just a means to quickly convert my ideas into something the computer can do, but I’m not interested in programming for the sake of programming (as anyone who looks at my code can attest to).

Create something I’ll be proud of afterwards

This is an easy one. It doesn’t matter how much money the game makes, but I wanted it to be something I can be proud of afterwards. Interestingly, the “innovation/awards” point is a subset of this reason, but this point is more general, and I can be proud of the games I make for many reasons.

You would think that this point should go without saying, and nobody should be making games they’re not proud of. Unfortunately, just a glance at the App Store will quickly convince anyone that’s not the case.

One more thing…

Indie game development can be a pretty solitaire affair, but that’s not something that bothered me too much (maybe growing up an only child had something to do with that). I never needed someone else to keep me motivated in or committed to a project.

I am currently working on not one, but two different projects at once. Soon I’ll be able to start talking about them in detail. Both of those projects are collaborations with other people. After working on them for a few months, I have come to realize that working with the right people can be really invigorating: Bouncing ideas back and forth and see them grow into something bigger than they started, or just pushing and motivating each other has become completely invaluable for me. I’m hoping to be able to continue working and collaborating with amazing people in my future projects.

Why are you making games? No, I don’t mean “why are you making games?”. Also, I don’t mean “why are you making games?”. And I certainly don’t mean “why (on earth) are you making games?”. I mean the question in the purest, most abstract sense. Just “why are you making games?”.

This was a key question for me last year, I spent a lot of time thinking about it, and I thought other people might benefit from asking themselves the same question. To see where I’m coming from, let’s do a quick flash back.

It’s late 2011. Miguel and I finished working on Casey’s Contraptions and my daughter was born. I took a couple months completely off from work (new dads will understand why), and then, right at the turn of the new year, I decided to start working again. The plan was to prototype a few of the juiciest ideas, pick one that really stood out, and make a new game in a few months. How hard could it be, right?

Right?

It turns out it was a lot harder than I ever imagined. I was confident in my list of hundreds of game ideas that I had been adding to over the years. I picked one idea, made a quick prototype in a day or two, and decided that I wasn’t as excited about that idea as I thought. So I picked another one, and another prototype that didn’t go anywhere. Then another, then another, and then another. This went on for months.

By the time I realized that my prototype icons weren’t fitting in a single page on my development iPad, I started worrying that something was wrong. Prototyping is great, it allows you to find the truly great ideas and all of that, but I wasn’t any closer to a game idea I wanted to make than when I started.

At this point, I even started prototyping game ideas I knew I was not going to make, just to remove any pressure on me, as well as to get it out of my system how much I hated those game genres. It was an interesting effort, but, as you can imagine, it didn’t lead to any shipped games.

I did, however, help me start figuring out what was going on. I realized that the prototypes I was creating were dissatisfying because my intentions were unclear. I wanted too many conflicting things out of this next game, and that was pulling the game in all kinds of opposite directions.

I wanted to make a unique a different game, yet at the same time I wanted it to reach a huge audience. I wanted to ride the wave of financially successful iOS games, but I didn’t want to sell my soul with freemium-based games, and instead, I was going to make a “good” freemium game. I wanted to make something innovative, but I wanted to do it in just 6-9 months. I wanted it all, and I wanted it yesterday. Bad combination.

That’s when I sat down and asked myself point blank “why am I making games?”. Or at least the more concrete version “why am I making my next game?”. That’s a surprisingly hard question to answer really honestly. I had to peel away a bunch of layers until I found the answers to my question.

My specific answers don’t matter because there isn’t a right and wrong answer. Any answer is good as long as it’s honest, from the bottom of your heart. There are as many valid answers as game developers out there. Some people make games because they want to be famous, others because they want to make money, others because it’s the only thing they know how to do, others because they want to win awards, others because they want to reach as many people as possible, others because they want to tell stories, etc, etc.

It’s actually amazing it took me this long to ask and confront this question. When you’re working on a game company things are different, and you don’t need to do that level of introspection. But once you go indie, it becomes really important. Even though I’ve now been indie for 6 years, I’ve been going in automatic pilot all this time, kind of skirting around this issue and winging it. I see a lot of indie developers out there doing the same thing, with greater or lesser success, but still conflicted internally about what they’re doing.

Once you have the answer to your question, you can use it to guide you in any big decisions related to game development, and even in life in general. Which projects to tackle, or even what features to add to your game, will often be easy to decide if you keep your goals in mind.

For me, within two weeks of answering the question, I had settled on not just one, but two different projects that I was super-excited about. Both projects are still moving along and I hope to start sharing information about them soon.

So I encourage anybody reading this to make some quiet time in the next few days, and pose yourself the question “why am I making games”. Once you find your answer, not only will it help you with future decisions, but it will make you more satisfied with them.

Right now I have a couple of game ideas in hand that I’m pretty excited about. I prototyped one of those, but I quickly realized it might be a game better suited for desktops rather than iOS, so I decided to write my next prototype for that game on the Mac. Maybe another day I’ll write a post about prototypes, but what I want to write about today is my experience moving over my prototyping code to the Mac.

I’ve been working on MacOS since 2007, when I switched because of iPhone development. For the most part, I’ve been loving it as an OS, but I never actually targeted it as a platform in my development. I figured it wouldn’t be very hard at all to compile my simple C/C++ + OpenGL code that I use as the basis for all my iOS games on MacOS. After all, that code started on Windows and DirectX, and porting it to iOS was pretty quick and easy. On top of that, I can use Xcode 4 for Mac, just like iOS (I’m not sure that counts as a plus though). How card can it possibly be?

The answer is, not hard at all, as long as you’re aware of a couple gotchas along the way.

Xcode project

I figured I would start from scratch, so I created a new Xcode project targeting MacOS. First surprise was that, unlike iOS, there wasn’t a template to create an OpenGL application. Instead, I need to create an empty Cocoa application and set things up by myself. No big deal.

I used the samples from Apple to get me started. I could have grabbed those and started adding stuff, but for some reason they have all the logic built into the OpenGL views, and I like to keep my views as simple as possible: they hold the surface to render, and they collect input and other window events. Done. So I had to do some moving around of code and responsibilities, which at the same time allowed me to make it more like the structure I used on iOS.

The AppDelegate does most of the initialization and creation on applicationDidFinishLaunching, and the OpenGL view gets created through the minimal nib file (with just a menu, window, and a view). AppDelegate is also responsible for setting up the display link callback, and then all the rest of the execution happens in my own code, away from those classes.

That was all pretty smooth. The only weird gotcha I ran into is that, unlike with previous Xcode 4 projects on iOS, I was no able to add any files to it that were not under the project root. They would turn red and not be available in the project. I tried adding them in every possible way and always got the same result. So, in the end, I gave in and just moved my External dir under the same directory as the Xcode project. Some people on Twitter had the same experience, while others didn’t, so there’s something fishy going on in there.

I have all my game static libraries (I refuse to call it an “engine”) as a separate Xcode project. I was able to add the project to the workspace and just change the target to be MacOS, and everything worked without a hitch.

OpenGL

I knew I was going to have to make some changes to the graphics code because all my iOS code uses OpenGL ES 2.0, and the Mac supports OpenGL “non ES” version. But I never did any graphics programming on the Mac, so I had no idea what to expect.

I decided to start with this sample code from Apple. After all, it seemed to be exactly what I was looking for: A simple project that sets up OpenGL both on iOS and MacOS. Score!

Once I started digging into it, I realized that it wasn’t going to be as straightforward as I realized. It turns out there were lots and lots of differences between OpenGL ES 2.0 and the OpenGL version on the Mac. Some were easy to catch, like different syntax for the shaders (“varying” became “in”, “uniform” was “out”, etc). Some were a pain to track down (needed to bind some Vertex Array Objects, and I was forced to use Vertex Buffers).

The documentation wasn’t clear on this point at all, but fortunately I got some good tips from Twitter. It turns out that the sample I used as a starting point created an OpenGL 3.2 context, and OpenGL 3.2 is very different from OpenGL ES 2.0.

	NSOpenGLPixelFormatAttribute attrs[] =
	{
		NSOpenGLPFADoubleBuffer,
		NSOpenGLPFADepthSize, 24,
		NSOpenGLPFAOpenGLProfile,
		NSOpenGLProfileVersion3_2Core,
		0
	};

	NSOpenGLPixelFormat* pf = [[NSOpenGLPixelFormat alloc] initWithAttributes:attrs];
	NSOpenGLContext* context = [[NSOpenGLContext alloc] initWithFormat:pf shareContext:nil];
	int swapInt = 1;
	[context setValues:&swapInt forParameter:NSOpenGLCPSwapInterval];

It turns out that leaving out the last attribute (NSOpenGLPFAOpenGLProfile) creates an OpenGL 2.1 profile, which is very similar to OpenGL ES 2.0 (GLSL 1.2 is the shader version that OpenGL 2.1 uses and that’s also very similar to GLSL ES). With that change, all the OpenGL code worked perfectly, with the only exception of the precision modifiers in shaders. Fortunately, I we can wrap those up in a simple #ifdef like this

#ifdef GL_ES
precision lowp float;
#endif

Unit tests

Apart from input, which will require a total re-work, the last major thing I had left was running unit tests after each build. I’ve seen references to Xcode’s unit testing functionality, but I was hoping to reuse UnitTest++ and run it like I do on iOS projects. It turns out it was even easier because you can run them natively without the extra step of the simulator like on iOS projects.

All I had to do was create a new phase in the Debug scheme to run a script. The “script” is just the name of the executable that was just build with the -unittest parameter added to it. I suspect that there’s probably a “better” way than hardcoding the .app/Contents/MacOS/ string in the middle to find the binary itself, so if someone knows the correct variable that includes that, let me know.

The only other thing that came up is that apparently Xcode adds the parameter -NSDocumentRevisionsDebugMode when you debug it from the IDE. No big deal, unless you have your program exit with an error if you pass an unrecognized parameter. I just made it accept and ignore that parameter, and everything worked like a charm.

Conclusion

It took a bit longer than I expected, but I finally have my code base running on MacOS. Iteration time is faster because it doesn’t even launch the simulator, and I suspect graphics performance will also be a lot faster (the iOS simulator does all the graphics processing in software), so I might end up doing more prototyping on the Mac unless the touch input is an integral part of the game.

I’ve been receiving a lot of questions about how it happened, how’s affecting me, and what my plans for the future are. So here’s my attempt to answer some of those questions.

The Game

It might come as a surprise to a lot of people, but selling Casey’s Contraptions was never in our minds. Miguel and I put all our energy into making a great game and infusing it with lots of personality. Half way through development we realized we had something special in hand, and when Apple selected it as the iPad game of the week and the reviews and comments came pouring in, we knew the game had connected with a lot of people. We were thrilled.

We were in for a big surprise when sometime after the iPad release, Rovio approached us and offered to buy Casey’s Contraptions. We were understandably very attached to the game we had just released, but Rovio eventually made us an offer we couldn’t refuse.

Rovio is one of the biggest players in the mobile market, and sometimes people get tired of seeing the top 10 charts full of Angry Birds. I’ve always had a lot of respect for them: Angry Birds wasn’t just a fluke, it’s a super approachable game, perfectly suited for mobile phones, and crafted to perfection. They deserve all the success they got. I don’t think I would have agreed to sell Casey’s Contraptions to most of the other big game companies.

As the co-creator of Casey’s Contraptions, I’m super excited to see it reach a much wider audience we could have ever dreamed of reaching ourselves. Every developer’s dream is to see people playing your games, and Rovio can make that happen with their huge marketing power and influence. Just look at all the media sites covering the news for the Amazing Alex announcement (most of those sites never talked about Casey’s Contraptions at launch :-)).

Miguel and I chose not to be involved in future development of Casey’s Contraptions/Amazing Alex. As an indie developer, your time is the most limited and precious resource. Every game you release takes some of that time away even after it launches: new content, higher-resolution graphics, fixing problems with the latest OS version, porting to other platforms, etc. This was a great opportunity to pass the baton to Rovio and be able to focus 100% on new creative projects.

Yes, it’s hard to let go. Not like I have experienced that yet, but it probably feels like sending your kid to college, knowing that he’s starting a new life without you. At least we can rest easier knowing Casey is in good hands. Whenever Amazing Alex becomes a worldwide hit, we can proudly point to that and say we started that game.

Personal life

On a personal level, my life hasn’t changed at all. Flower Garden has been doing a good job at keeping my family fed for the last few years, so getting more money doesn’t have a huge impact. This unexpected deal bumped up those numbers in the bank, but hasn’t otherwise changed much for me. No Ferraris in the garage, or even, for those who know me, no fancy new road bikes (still riding my trusty 12-year old Trek 5200). The only splurge was buying a 30" Apple Cinema Display.

The deal has, however, definitely affected my future projects. At one level, it means I don’t have to worry about funding for my next few games, which is great. Both Flower Garden and Casey’s Contraptions were done on a minimal budget. Now I can afford to spend more money in areas where it can make a difference, and be able to offer people money up front instead of just some percent of royalties.

One thing I never want to do is have full time employees though. Full time employees bring constant burn rate, and start adding a sense of pressure I do not want. One of the best things about being indie is the flexibility that it brings. Collaborations with other indies have worked really well so far, and I’ll continue to do that as much as possible for future projects.

The other way the deal has affected my future projects is that there will be more pressure to follow up Casey’s Contraptions with a great game. Even though Casey’s Contraptions is the same it was a few months ago, the fact that it was bought by Rovio will make a lot of people take notice and compare it to any of my future games.

I’ve already been struggling with those effects for several months. I’ve been making and discarding prototype after prototype. More than half of those would have made for fine games, but I have no desire to release a “just OK” game for the sake of releasing something. I’ve also decided that I’m not even going to consider potential for financial success for my next game. It’s not going to be aimed at the mass market, it’s not going to be full of zombies, and it’s certainly not going to be a typical freemium game.

Whatever my next game is, it’s going to be a more personal, more niche game. Something that I can be really excited about working for however many months it takes, and something I can be really proud of after launch.

Bye, bye Casey! We’ll miss you, but we can’t wait to see what you do out there!

Having a separate office not only gives me the distraction- and interruption-free environment I need, but also gives me some much-needed separation between home and work. My office is about a mile away from home, so that makes for a very nice 15-minute walking commute (with the Macbook Pro in my backpack). It’s also located right in the Carlsbad village, so it’s a block or two away from tons of cool places to have lunch, grab a coffee, or even go to the beach for a quick break!

Here are a few pictures from the new office. Things aren’t completely set up yet (that bare wall is waiting for some Flower Garden and Casey’s Contraptions posters that should be arriving today).

The thing I like the most about this office is the light. It’s a corner office and it’s aiming north, so that means great, indirect light all day long. Oh, and the windows can be opened to let some fresh air in. That might seem like an odd detail to mention to some of you, but you’d be surprised how many office buildings in the US have windows that can’t be opened. Really.

Both desks are adjustable, standing desks. The main desk in the pictures is the large frame GeekDesk one. I used to have the small frame because I didn’t have the room at home, but I love the extra space, especially with the 30" display! I’ve kept the small desk to the side, that way I can invite other people to come work from here for the day (indie work day!) or even (gasp) hire someone if the right time comes.

The rest is pretty standard: whiteboard (I prefer use one much larger, but that’s what I had around), bouncy ball (gotta have a bouncy ball!), and couch (pretty important to unwind, or just to take a break from standing all day).

So far, I’m very happy with this setup. Working from here is definitely invigorating, and the separation between work and home helps a lot in both locations. Maybe once our daughter is older and going to school I’ll consider working from home again, but this is perfect for now.

What does your work setup look like? Post some pictures and let me know. I love seeing other devs work setups.

There were even a couple of cases during Casey’s Contraptions that we wanted a particular effect, and couldn’t get it quite right, but it would have been trivial to whip up a shader if we had been using OpenGL ES 2.0. In the end, we had to resort to texture combiners (yuck), and it wasn’t exactly what we had in mind.

OpenGL ES 2.0 requires that you write shaders for any different kind of “thing” you render, whether it’s a simple 2D quad, or a fancy skinned character. I wanted to provide at least a basic 2D rendering shader in my graphics library, but the workflow was a bit awkward.

First of all, shaders on iOS are compiled at runtime [1], so we need to access the shader source and compile it. Initially I was thinking of having the shader source files as text in the graphics library, but for the game to access those files, it would have to explicitly add them to its resources so they can be copied during the build phase. That’s error prone (I would always forget to do that with every new project) and just plain ugly.

Another possibility would be to include the source text as a string in the library. That would simplify things, but it would mean that:

1. You would lose shader syntax highlighting.
2. You would need to have the annoying \ at the end of every line.

After asking on Twitter and getting some good suggestions, I decided the easiest way would be to use xxd. It’s a shell tool that comes preinstalled in all Macs (and Linux) that takes a file and converts it into a C array that can be included in a program directly.

I liked xxd even better than the old standby of bin2c because it already comes installed, and it generates the whole file, not just an array that then needs to be included from another file. Yes, it would be easy enough to modify, but why bother when you have a perfect tool for the job already?

Now, I can set up a custom rule in Xcode 4 like this:

Update: The array generated by xxd isn’t null-terminated, so we need to add a zero to the array. The easiest way is to use sed. Thanks to SixEcho Studios for the heads up. Also, I updated it to use the varaibles INPUT_FILE_DIR and INPUT_FILE_NAME thanks to Miguel’s comments.

Now, whenever the shader source changes, it generates a header file like this:

unsigned char PassThrough_fsh[] = {
  0x0a, 0x0a, 0x76, 0x61, 0x72, 0x79, 0x69, 0x6e, 0x67, 0x20, 0x6c, 0x6f,
  0x77, 0x70, 0x20, 0x76, 0x65, 0x63, 0x34, 0x20, 0x44, 0x65, 0x73, 0x74,
  0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6c, 0x6f, 0x72,
  0x3b, 0x0a, 0x0a, 0x76, 0x6f, 0x69, 0x64, 0x20, 0x6d, 0x61, 0x69, 0x6e,
  0x28, 0x29, 0x0a, 0x7b, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x67, 0x6c, 0x5f,
  0x46, 0x72, 0x61, 0x67, 0x43, 0x6f, 0x6c, 0x6f, 0x72, 0x20, 0x3d, 0x20,
  0x44, 0x65, 0x73, 0x74, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43,
  0x6f, 0x6c, 0x6f, 0x72, 0x3b, 0x0a, 0x7d
,0x00};
unsigned int PassThrough_fsh_len = 91;

To compile that shader in the game, all I have to add is two lines:

#include 
//...
const int fs = ShaderUtils::CompileShader(PassThrough_fsh, GL_FRAGMENT_SHADER);

Done! Clean, simple, and minimal cruft.

Update: I haven’t tried it yet because I haven’t had the need for any expensive shaders, but I hear really good things about Aras Pranckevičius GLSL optimizer. In that case, I would add it before the xxd step. You would still use the shader the same way, but it would be optimized and perform faster.

[1] That’s a huge mistake and I hope Apple fixes that ASAP. Loading times for any amount of non-trivial shaders adds up really quickly. That’s the last thing you want on a mobile device! The OpenGL ES 2.0 standard provides an optional way to load binary shaders, and all that Apple has to do is provide a tool that will compile shaders for the iOS hardware. To be completely future-proof, you they can even mandate that if the binary load fails, you still need to compile from source.

Fortunately, display technology caught up over the years and I’ve been happy with the LCD monitors I have since then.

Until now.

I’ve been looking into large LCD monitors to set up in my new office with my Macbook Pro, and I find myself pushing into the limit of technology once again. The results: I went through 3 monitors of about $1K each. Two fell completely short, and one was perfect.

The Shiny: 27" Apple Thunderbolt Display

The top contenders starting out were the 27" Apple Thunderbolt Display, and the 27" Dell U2711. I read reports that they both use the same display technology, and the differences between the two were relatively minor. After some conflicting reviews (that should have been the first warning), I decided to go with the Apple Display.

This display seemed perfect for me. It looks gorgeous, like an oversized iPad. The image quality is fantastic, with text being sharp and clear and having a great resolution. Since I’m only connecting a Macbook Pro to it, the connections are perfect: The Thunderbolt cable is great because it deals with display, USB, ethernet, and even audio in one fell swoop. The display also has a power cable for the Macbook Pro, so it’s like a perfect docking station.

It really was almost perfect. Almost.

The glare was simply horrible and I could see myself reflected on the screen, even with a white background. The shiny display was almost a perfect mirror. It might be the ideal display if you like to work in a dungeon, but I like light, the more the better. Even though my office has no direct sunlight, just the indirect light coming in was enough to see everything brightly reflected on it.

One possible way to work with the Apple Display

What’s worse, if one day I had to rearrange things or move to a different office and I had a window behind me, it would go from distracting to impossible to see anything.

Another annoying aspect of that monitor is that the stand height wasn’t adjustable. I like to look at a monitor straight while my arms are about a 90-100 degree angle, so the monitor ended up being several inches too low. I can do the old phone book trick, but it seemed ridiculous having to do that on such a beautiful display.

As much as I liked that display, it just wasn’t for me.

The Ugly: Dell U2711

In spite of some comments complaining about “over aggressive” anti-glare coating, I decided to give the Dell U2711 a try. I had used Dell monitors before and I was never bothered by the screen. And after all, the antiglare coating could fix the problem the Apple display had. Apart from the antiglare, it had some attractive options: Multi-card reader, several video inputs, and better color options. It could be the perfect screen.

It turns out it was far from that. It was the opposite of the Apple display in many ways. Connecting it to the Macbook Pro was a hassle since it shipped with a Displayport cable, but not a Mini Displayport one. I tried using my Mini Displayport to DVI adaptor, but it turns out it was single-link and you need a dual link to drive the full 2560 x 1440 resolution.

Once I finally got it up and running (with a Mini Displayport to Displayport cable), the resulting image was plain ugly. The colors were off, but more seriously, the antiglare coating really was horrible. It made the screen look like there was a fine coating of dust on it, or like it was film with a very obvious grain to it. I had to strain my eyes to focus on the text because the graininess kept throwing me off.

I suppose I could have gotten used to it, but since I use the Macbook Pro display next to it, going back and forth was torture. One moment I would see this clean, sharp, beautiful display, the next moment I would see this horrible grainy mess.

To make sure I wasn’t hallucinating, I took a picture of both displays with my iPhone displaying the same content. The differences are obvious (click on them to see them full size).

Macbook Pro with antiglare

Dell U2711

In case that wasn’t enough, the colors were severely off. I played around for an hour with the custom settings and some advice from the internet, but I just managed to make them a bit better. I still liked the colors on the Apple Display (or the Macbook Pro) way better.

The Good: 30" Apple Cinema Display

At this point I thought I was out of luck, and that I was going to have to settle for either one of those two monitors (I would take the 27" Apple one), or something smaller but without those problems.

Fortunately, several people on Twitter encouraged me to try out the 30" Apple Cinema Display. I never saw paid attention to one, but apparently they had fantastic image quality and a good antiglare coating. That sounded promising!

One of the pleasures of being a small company is that once I decide to do something, I can do it right away. There’s no paperwork, approval, ordering through certain channels, or any of that overhead. Since Apple doesn’t make that display anymore, I hopped on Craigslist, called a guy up, and picked it up a few hours later. Done!

And boy was I glad I did! That monitor is everything that was cracked up to be and more! Since it’s an older display, it doesn’t have Mini Displayport cables, so I had to use a Mini Displayport to Dual DVI adaptor. The image quality is indeed, outstanding, and the colors are quite good (although maybe a touch redder than the laptop).

I actually love that it’s a 16:10 ratio instead of the more common 16:9 in most monitors today. That gives results in a resolution of 2560 x 1600, which gives me some much-needed vertical space when writing code (which is 80-90% of my day).

As a totally minor, but nice side effect, the brightness controls on the keyboard control the screen brightness (which is kind of surprising considering it’s connected over the dual DVI adaptor, but there’s also a USB cable going into the display, so maybe it’s communicating through that channel).

The only drawback is that the back only has two USB ports (I wish it had 4 or more). It does have two Firewire ports as well, although I don’t have any devices that use those.

For completeness sake, I took a picture of the new monitor with the same image I used as a test earlier. It shows how much cleaner the image is than the Dell.

30" Apple Cinema Display

Conclusion

From left to right: The shiny, the ugly, and the good.

If you’re picky like me about the quality of your displays, and you’re looking for a large monitor, I can’t recommend the 30" Apple Cinema Display enough. It’s a shame that Apple stopped manufacturing it, so check Cragislist or Ebay for it. If you work in a dark place, maybe the 27" Thunderbolt one will work as well. Whatever you do, stay far, far away from the Dell one.

We can only hope that Apple will make a revision of the 27" Thunderbolt Display soon with a high-quality antiglare coating like the laptops. In the meanwhile, I’m thankful both Apple and Dell have good return policies.

That future is a reality today for some games and apps (Netflix, Kindle), and I’m convinced that players will expect that in most games in the next year or so. So obviously, the next bit of new iOS tech I decided to try was iCloud. I would love to turn Flower Garden into that kind of seamless experience, independently of the device you use to access it.

As a quick spoiler, it turns out I won’t be able to make Flower Garden quite so seamless without a lot of extra work. But I learned a lot along the way and I should be able to take a small step in that direction.

Design considerations

iCloud promises to be the “available from everywhere” storage solution that will be a key component towards the scenario of playing on any device at any time. Unfortunately, it’s just a component of that whole scenario and still requires quite a bit of work on the part of the developer.

Since I’m just retrofitting Flower Garden to work with iCloud, I wanted a simple way to simply replicate the game state on iCloud. Then, whenever you play it from any device, you get the latest state and everything works as expected.

The big problem with that approach is that you can’t always count on having an active (or fast enough) internet connection. iPod Touches, iPads, or even iPhone in a plane or a cell tower black spot will make it so your device has no way to communicate with iCloud. So what happens if the player plays without an internet connection, then comes home, grabs another device, plays for a bit with internet connection, and finally launches the original device, this time with internet connection. You’ll end up with two conflicting game states. Not fun!

I can think of two different strategies to deal with this situation:

• Require internet connection to play. Some games do this today. It can be quite frustrating not being able to play your favorite game in the plane, but it does solve the game state conflict problem because they can always communicate with iCloud, so they usually provide seamless multi-device play.
• Detect changes on both game states and intelligently solve them. Sounds good in theory, but it’s a pain in practice. Unless you have a very simple game state (a percentage completion for example), it means not only do you have to record the state, but you have to record the events that led to that state, and be ready to examine them, compare them, and marge them in the case of a conflict. And the worst part is that even if you put lots of care into it, there will be cases where the merge is not ideal and the player will feel like he lost something in the merge.

For Flower Garden, neither solution is particularly attractive. It’s the proverbial rock and a hard place. So for the moment I’ve decided to implement just a small step in that direction: Sync things that only progress in one direction, without any danger of conflict. That includes purchased IAPs and unlocked seeds. Later on, once that’s working solidly without any problems, I can consider adding counts like Fertilizer, Color Dust, or Green Thumb points. That will be a bit trickier because there’s still chance for conflict since the count can go up and down, but it’s possible to change each amount into two different values that always go up: Earned amount, and spent amount. That way I can always grab the maximum value that I find either local or on iCloud and things should work smoothly. (Update: What I wrote there about separating them in two different values is totally wrong, and any DB programmer worth his salt would be laughing at that. Separating them in two different values won’t help at all. That’s what I get for doing brain-dump posts on the fly :-)

Apple provides two APIs to store data in iCloud: key-value storage and file storage.

Key Value storage

Initially, this seems like a great way to go. It’s a very simple API, and it’s very familiar to most iOS programmers because it’s just like NSUserDefaults.

Don’t be fooled into thinking you need to store data in strings or one field at the time. You can stuff whatever binary chunk of data you want with NSData, so it’s quite versatile.

It does have three pretty big drawbacks thought:

• Size. You can only store up to 64KB of data. That’s definitely a big deal if you’re planning on storing large data files. Even for Flower Garden, each pot is about 30KB, so I wouldn’t be able to save the full garden state this way.

• Syncing. The Apple docs say that “Keys and values are transferred to and from iCloud at periodic intervals.” Ouch! What does that mean? Clearly this is intended for non-crucial data (like settings), so the potential delay isn’t a big deal. In my tests, I found that data was often not available when starting the app, but would become available a few seconds later.

• No way to check state. When starting the app, there’s no way to find out if the values you’re reading are up to date. Combined with the slow syncing, it makes it less than idea for important data. On the flip side, you can be notified when the data changes, but that means having to deal with changes while the game is running (which I was hoping not to do).

File storage

File storage might be a better option then since it doesn’t have any of those drawbacks: there’s no size limit, data is synced much more aggressively, and you can check if the data is up-to-date, and wait until it is otherwise.

What’s not to like about file storage then? The cumbersome and intrusive API that Apple created around it. If you read the docs, they make it sound like you need to inherit from UIDocument and load all your data through that class. I like to keep things simple and portable, so I’d really rather not introduce UIDocument into the file-loading process if I don’t have to.

It turns out that Apple’s docs on the iCloud SDK are quite sparse and lacking details. I was able to put together a demo from those docs and some experimentation, so hopefully this will be useful to other devs as well.

The device has an iCloud storage daemon running, monitoring specific directories for changes. You get the directory assigned to your app by calling URLForUbiquityContainerIdentifier.

To add a new file, the docs recommend first creating the file somewhere else, and then calling NSFileManager:setUbiquitous:itemAtURL:destinationURL:error: to move it to iCloud. Interestingly, I accidentally skipped this step and just wrote to the iCloud directory directly, and the files were stored correctly anyway. Might as well leave it just in case that behavior changes later on, or there’s some side effect I didn’t notice.

After that, every time you write to the file, the iCloud daemon will detect the changes and push them out to iCloud. This is an important point because a) You don’t have explicit control to say “start send it now” or even “This file is ready to be sent to iCloud” (that would be my choice), and b) I don’t know what it does with partial updates, so I would be very careful about writing to those files and make sure it’s an atomic operation (save somewhere else, and them move the file in one operation).

To get the latest version of the files on iCloud, you can check whether they’re fully downloaded or not by calling NSURL:getResourceValue:&isDownloaded forKey:NSURLUbiquitousItemIsDownloadedKey error:. If they are up to date, you can move on, otherwise, you can initiate a download of the files by calling NSFileManager:startDownloadingUbiquitousItemAtURL:error:.

One weird thing is that the startDownloading function doesn’t have a callback (that I can see). So I had to set up a timer to check periodically if the files are synced.

Since all the syncing happens at startup, there’s no complexity involved with data changing while the app is running. If that’s a case you need to handle, you might be better off using UIDocument since it at least detects conflicts. If you do it this way, the latest one will overwrite any past changes.

Also, working this way, it seemed easier to keep the files in Application Support (or Documents), and only move them to/from iCloud when I wanted to. That has the advantage of iCloud not changing things from under you while the game is running, and the fact that, if for some reason the files in iCloud are corrupt, you always have good local files to fall back to.

Demo

iCloudTest.zip

The demo iCloudTest saves data both with the key value pair, and the file storage system directly like I described above. If you run it from different devices, you’ll see that it’s amazing how quickly file data gets propagated, but key-value data takes a few more seconds.

The demo project also works in the simulator (there’s no iCloud support, so it just uses local files), and it even works with iOS4 (also using local files, and it avoids using any iOS5 symbols while detecting iCloud support).

Sharing Data Between Multiple Apps

Once you’re at this point, sharing data between multiple apps is really straightforward.

First you need to make sure the app IDs of both apps start with the seam Team ID. For some unknown reason lost in the midsts of time, Flower Garden and Flower Garden Free use different prefixes, so that option is out for me. Hopefully most people having been using the Team ID only.

Then you need to decide which type of data you want to share. If you want to share files, you need to add the app ID of the other app to your iCloud entitlement, and then as for the correct app ID (including prefix!) when you call URLForUbiquityContainerIdentifier. If you just want the app’s own iCloud data, you can pass NULL as the parameter.

As far as I can tell, each application can only have one set of key-value data. So sharing it between two apps means changing one app to use the app ID of the other app in the com.snappytouch.icloudtest field of the iCloud entitlements file. As long as both apps use the same Team ID prefix, it should work fine without having to do anything else.

One of the first features I looked into adding was multisample antialiasing (MSAA) support for OpenGL, which was originally introduced in iOS 4.0. The geometry generated for the petals in Flower Garden is fairly high contrast, and since it’s not like the textures were carefully created and laid out by an artist, the result is pretty bad aliasing around the edges. Perfect candidate for multisampling!

I based everything on the Apple documentation on multisampling. It was very straightforward and it works for both off-screen and view-based render targets. It was also extremely helpful to keep both the ability to use regular and multisampled render targets. That way I can easily run performance and visual comparisons, and, if necessary, I can disable multisampling on a particular device.

The main gotcha was making sure I used the GL_RGBA8_OES color format on the multisampling color buffer (otherwise it won’t work, and all you’ll get are 1282 “invalid operation” OpenGL errors). Also, if you’re in the same boat as Flower Garden, which uses OpenGL ES 1.1 and the fixed function pipeline, you’ll have to add _OES to just about every constant in the documentation.

Visual results

I will let the screenshots speak for themselves.

In this screenshot you can see the rendering of the pots using off-screen render targets.

Pretty impressive improvement, even if I say so myself! The improvement is even more striking in-game because the animation of the flowers moving in the wind doesn’t have jaggies popping in and out.

The improvements are more noticeable on the iPad because the pixels are bigger, and, not surprisingly, less so on retina-display iPhones. But even on the retina displays, jaggies are less noticeable during the flower wind-swaying animations.

Performance

So, no doubt: They look pretty. But how about performance? Under the hood, the rasterizer is generating four samples for every pixel, and then combining them at the end in a separate step. So you’ll get more of a performance hit with complex pixel shaders. Edit: My bad. I spaced out and I forgot that MSAA only evaluates the pixel shader once, so performance doesn’t depend on pixel shader complexity. Instead, the performance hit probably comes from the extra writes (four per pixel) to the render target, and should be fairly similar between games.

It turns out that Flower Garden is still using OpenGL ES 1.1, which is implemented using shaders at the driver level. Fortunately, even though I’m using some texture combiner operations and several input textures, those pixel shaders aren’t all that complex.

These are the frames per second I recorded with one particular flower pot on the different devices I have for testing. Notice that the lowest device I have listed is the iPhone 3GS. That’s because I have also stopped supporting arm6 CPUs to keep things simpler (and the market share is minimal).

Update: I was only running the game loop at 30 fps, so the initial performance numbers I had listed are pretty meaningless. Here are the correct numbers running at a max of 60 fps.

Of the devices I tested, MSAA didn’t slow down things much on the iPhone 3GS, and the iPhone 4S was maxed out at 60 fps both ways. The iPad 1 was a different story, and performance crashed from 60 fps to 18 fps. Like Rory pointed in the comments, looking at the actual time instead of the fps gives a better insight. Using MSAA adds 38.9 ms to each frame! The iPhone 4 also suffered from a big performance hit due to all the extra pixels in the retina display. I don’t care how pretty the flowers are, that’s just not acceptable (and no, I’m not adding user-tweakable graphics settings like PC games).

I should also add that Flower Garden is hugely CPU bound. It generates all that geometry every frame on the CPU, and that’s a lot of vector transforms. So if your game is GPU bound, you’re likely to see higher performance hits.

I was initially very surprised to see that the performance on the simulator tanked big time (as in, going from 30 fps to 5 fps). This was news to me, but apparently the iOS simulator runs completely in software, so the quadrupling of pixels brings it to its knees. I’m really, really, impressed at how smoothly it usually runs for being in software though. It had me fooled thinking it was using OpenGL under the hood!

As far as memory goes, after an animated discussion on Twitter, it seems that the iPhone does create full buffers for the multisample frame buffers, so there is a significant increase in memory usage. It’s not something easy to track because it all happens at the driver level, but it’s something to be aware of. Because of this, I might consider turning multisampling off on retina displays, since the improvement is not mind-blowing (and the extra memory is very significant because of the amount of pixels). I’ll also probably turn it off in the simulator so I can achieve a decent frame rate during development.

Conclusions

Multisampling only required adding a few lines of code and resulted in an impressive visual improvement and minimal performance impact in some devices. It’s a no-brainer on the 3GS and iPad 2. I’m wishing I had implemented it earlier!

Pre-iOS5, if you added a view with [self.view addSubview:myView], myView didn’t get any notifications. That was weird, but we learned to live with it. Tens (hundreds?) of thousands of developers worked around that and wrote gobs and gobs of code that worked with and counted on that behavior.

Then along comes the iOS5 SDK and someone decided that it would be good to “fix” that behavior, but doing it silently. By silently I mean no compile-time errors or warnings, but apparently it wasn’t deemed worthy of including in the iOS5 release notes either. I’m sure they thought they were doing us a favor. After all, if someone had a problem with it, it’s because they hadn’t structured their code correctly.

So now every view you add/remove will get those notifications. Which is fine, except if you had decided to call those by hand yourself. In that case, now they get executed twice, so you may be allocating things twice, making multiple network calls, and just causing general mayhem.

I also noticed that if you call presentModalViewController with a view controller, all the views underneath the modal view controller will get the viewWillDisappear/viewDidDisappear event. In my case this causes total chaos because those events trigger the freeing of some views that the modal view controller will need.

But wait, it gets even better! From what I’ve been able to tell, if your app compiled with the iOS5 SDK runs on iOS4, then you don’t get those events! That means that now you need to be ready to both deal with receiving them and not receiving them at the same time. Is that useful to anyone (other than devs making iOS5+ apps)?

Fortunately, existing apps compiled with the iOS4 SDK are consistent running both on iOS4 and iOS5 (no view events automatically generated). Phew! Dodged that bullet.

It seems I complain a lot lately, but this seems like another very justified complaint. Wasn’t there a way for Apple to add that behavior to new functions without changing the old ones? For example, they could have introduced UIView:addSubview:withNotifications: and eventually deprecated the old ones. Yes, that means that they would have had to duplicate 10-20 functions related to adding/removing views, but they wouldn’t have broken half the iOS code out there.

This is probably my fault for relying on viewDidAppear events and calling them myself. I started working around this problem by being super-careful about how and when I call viewDidAppear, but as soon as I realized that iOS4 does it differently, I threw in the towel and went for a different approach. Now, I just implement my own function in the different view controllers and call that instead of viewDidAppear. That way Apple can change those events all they want and it won’t affect me anymore.

Of course, all of this only matters if you’re making heavy use of view controllers (which unfortunately I am doing in Flower Garden). I’ve already said that my future game projects are going to be UIKit-free, so I won’t have to deal with this kind of problems again.

Brewing

A new game for me starts as an idea somewhere, sometime, that got jotted down into my “game ideas” personal wiki page. I have that page accessible 24 hours a day on my computer, iPhone, or iPad. Only while I’m running/cycling or in the shower am I away from that page (and unfortunately, that’s the time when most ideas seem to spark). I make a point of not censoring any ideas: If I thought that something would be neat (not just gameplay, but a setting, a visual, a mechanic, or anything), I jot it down.

Over time, I accumulate quite a few ideas. Every so often I review them and I might expand on some and flesh out sub-ideas. Or they might spark different ideas of their own and I jot them down too. I never delete any of them, because I’m consciously trying to not censor anything yet. This is purely brainstorming mode. I’ve even sent emails to friends about possible game ideas straight out of this list, crappy ideas and all.

During this time I’ll rearrange the list. I’ll move more likely ideas up, or ones that I’m more excited about. That has the effect of a kind of bubble sort, so the better ideas somehow rise to the top (except for the brilliant ones hidden in the depths somewhere).

This list is particularly useful when I’m in the middle of a project and I have what seem brilliant game ideas. Do I put the project aside to do this great idea instead? No. Instead, I add it to the list with all the others. If I’m really excited about it, I’ll flesh it out as much as I can, but it needs to wait its turn. As you can imagine, after a few days, the idea doesn’t seem so shiny anymore, so it was a good thing it didn’t derail the current project.

Struggling

Eventually the time comes when I need to pick a new project, and this is where the fun and the pain start.

You would think a good approach might be to read the list and start evaluating ideas. In a way, that’s what I do, but before I evaluate ideas, I need to some frame of reference to decide what’s a worthwhile idea and what isn’t.

In the past my criteria for considering a project involved the intersection of three requirements:

• The game must be interesting for me to work on. I want to learn something new and be excited about what I do. Not interested in cloning something or making a derivative game.
• A game I can realistically implement with the resources at my disposal. I’m not an artist, so that usually means not having a content-heavy game, and relying on code as much as possible (like the procedural geometry in Flower Garden–no artist modeled those flowers by hand).
• A game that has potential to sell well on the target platforms.

Unfortunately, meeting all those three requirements at the same time isn’t easy, especially given the astounding number of games already on iOS.

Before I go any further, I need to step back and ask myself a very important question: Why do I want to make this next game? This is a question we indies have the luxury of asking (and answering). I think most big (and small) studios are too busy staying afloat to be able to ask anything like that (besides, the answer is almost always “make more money” for them).

It turns out the indie life is treating me very well, so making lots money isn’t one of the main reasons to make this next game. That means I can safely remove that requirement from my previous list, which grants me a lot more freedom.

I’m probably going to spend the next 6 to 10 months working on this project, so it needs to be worthwhile. Does it need to be innovative and break new ground in a way no one has seen before? Does it need to make hard-core FPS players cry to be worth considering? Does it need to be radical, pixelated, mash up 5 different genres, an win the IGF and respect of my peers?

It turns out none of those are the reasons that drive me to make games. In the end, when I look deep down, the reason I want to make games is for the pleasure of taking a vision from the initial idea to something people can play. It’s the creativity involved that drives me. I imagine it’s the same reason people are driven to write or paint. If along the way, some of those games manage to be innovative, make money, or win an IGF award, that’d be fantastic, but in any case, the development process is its own reward.

Given all those factors, I can go down the list of games and evaluate each one. Does it have potential to meet those requirements and be a satisfying project? This might not come as a surprise for those of you involved in creative activities, but this is hands-down, the most difficult time of development for me. As long as I don’t have a project picked, my mind is constantly going over this. Anything I read, see, or hear is filtered and analyzed thinking of how it would fit in a game. During this time I’m often moody, volatile, and prone to depression if this goes on for too long.

Even though it seems I’ll never going to be able to come up with an idea worth doing, eventually something comes along, and the next phase starts.

Prototyping

I’ve already talked about prototyping at length before. I take the idea I’m considering and I try to answer the key questions in the shortest possible time. This is the time to shoot down any bad ideas, or prove why they’re not feasible or just boring.

The prototyping phase is a big high for me. It often involves manic activity and I can get a prototype done in a day or two from the initial excitement on the idea. The bad part is that most prototypes prove not to be that great, and they go back to the drawer of game ideas, and I’m left scrambling for another idea.

Struggle. Prototype. Discard. Repeat.

I usually repeat this cycle multiple times. Each time around I get more anxious, and the lows and highs are a bit more extreme.

Fortunately, at least so far, eventually I find a prototype that seems worth doing. Something I can see spending the next X months of my life doing. I usually run it by a few friends whose opinion I value highly, and if they’re excited about it too, then it becomes an internal green light and I move on to the implementation phase.

Tips

Apart from asking yourself why your making a game, here are a couple of things worth keeping in mind when picking a new project. They’re not new things and I’ve heard them before in one form or another, but they’re worth reiterating:

• Don’t compete with big companies. They can throw a lot more resources, and, most importantly, lots of marketing behind their titles. Don’t try to take Zynga head on. Do your own thing.
• Don’t chase trends. Take risks. Be different. You’re indie and have low overheard. Take advantage of that and do things the big boys would never dare risk $50 million on.
• Keep the scope of the project small and focused. It will be bigger than you envision anyway. By being small, you can come up with new ideas faster than big companies.

How about you? What’s your process for deciding to work on something? Do you struggle until you pick the project? Do you stick with an idea, or do you change and restart?

I was having a discussion today about free games with other developers and I thought I would post here some random thoughts and open it up for discussion.

Free-to-play (or freemium–even if they aren’t exactly the same, I’m bundling them all under the same category for this discussion), have have a fairly bad reputation, and they’ve been under fire recently from developers. It’s true that a lot of those games have been rather poor from a game design point of view, while raking in loads of money from players who are apparently happy to play them.

It’s important to separate the financial model (free with other ways for players to spend money in-game), and the quality of those early games or the intentions behind them. I am convinced that free games is the future of mass-market games (it’s already pretty much the present, so that’s not much of a stretch).

There’s no doubt that the financial model of game affects its design. Compare arcade games, retail console games, and subscription-based games just as an example. Free-to-play has a huge impact on the design as well.

Free-to-play games are in their infancy. Not only are they a relatively recent happening, but they were also wildly successful, which encouraged a lot of copying and not much innovation. So they’re stuck in a type of design that results in a local maximum of profit, while providing a not very satisfying experience for a lot of players. As game developers, we need to find out how to make great games while using the free-to-play model.

I’ve been going around this quite a bit recently, because I’m in the stages of deciding what my next game is going to be. The reality of the App Store are pushing me towards free-to-play, but I’m not interested in making a Farm/Store/Pet game.

These are my random thoughts on what we can charge for in a free game and how it affects game design:

• Reduce delays. This is very effective, but feels cheap and somewhat manipulative (yes, this is coming from the guy who did Flower Garden… but that was before IAPs and the whole point of the game was the nurturing/waiting part). It also falls in the category of the question I often ask myself: If I remove this from the game, will it be better or worse? The answer is (almost) always “better” by removing those delays.
• In-game currency. This is seems like a better approach as long as there’s no competitive multiplayer. However, it does wreck havoc with the game balance. Either it becomes too easy and not fun for those who paid, or boring and grindy for those who didn’t. Still, especially on mobile, it’s not a totally bad way to go. You’re letting people make a choice how they want their experience.
• Extra content. That seems to be the traditional, developer-approved way to go. PC and console games have been doing that with DLCs for a long time. The main problem is that not many players want that content, the amount of content you can sell is limited, and it often requires a lot of extra effort to generate.
• Extra choices. This includes different characters, clothes, weapons, etc. I see this as the sweet spot between the last two options. What you buy doesn’t completely throw off the game curve, but it’s also not just new levels or missions. Combine that with letting players earn credits to get those choices (by grinding if they want to, but it’s all optional) and it seems like a good way to go. You can also go the way of League of Legends (which I have yet to play!) and you can rotate in that extra content for limited amounts of time, so players get a taste and they have the option of buying them permanently.
• Cheats. By cheats I mean more lives, rewind/replay, invulnerability, etc. It feels like a throwback to the arcade days. Some players will be put off by it (either by the fact it exists, or by the fact that they finished the game too quickly with the help of those cheats), but it can work in the right game. You’d probably want to do something about high scores, like putting players who used those cheats in a different leaderboard.

What are your thoughts on this? What are some other ways that players can pay for in free games and still allows us to make a great game?

I have to say, I’m not a huge fan of the wobble movement, but it does indicate that icons are “loose” and movable. And since most players are already familiar with that metaphor, and the overhead view of Flower Garden resembles the launchboard quite a bit (not coincidence), I might as well go with it.

Each pot is a UIButton with a custom image (rendered from OpenGL into an off-screen target). So when I enter edit mode, I need to wobble them like the launchboard. Easier said than done.

I searched online and asked on Twitter, but surprisingly, nobody had done a perfect, reverse-engineering of the wobble movement. Most mentions just do a z rotation on a sine wave, which is not that close of the launchboard animation.

I found that to get closer to the real thing, you need to do that rotation around a random point offset from the center. That’s better, but it’s still not perfect. There’s an additional extra twitch in the original Apple animation. I’d love it if someone already reverse-engineered it and has a better model for that animation.

In the meanwhile, I figured I would share the core I’m using. It’s all very straightforward, except for the part about rotating a UIView around a different point other than the center. I initially thought I could just concatenate transforms, but no matter how you do it, UIKit always interprets the transform around the center.

To be more precise, it interprets the rotation around the anchor point, and that’s what you need to modify. The anchor point is defined in units that are a percentage of the layer’s size, so by default it’s 0.5, 0.5. When you move the anchor point, the layer itself will move, so you need to fix it up by moving the layer’s position by the same amount in the opposite direction.

This is what the final code looks like:

void StartPotWobble(UIButton* button, Random& random)
{
	const Range posOffset(0.10f, 0.2f);
	const float amplitude = random.GetFloat(1,1.5f);
	const Vec2 startCenter(Sign(random.GetFloat(-1,1))*random.GetFloat(posOffset), Sign(random.GetFloat(-1,1))*random.GetFloat(posOffset));
	const Vec2 endCenter(Sign(random.GetFloat(-1,1))*random.GetFloat(posOffset), Sign(random.GetFloat(-1,1))*random.GetFloat(posOffset));

	CGPoint center = button.center;	
	button.layer.anchorPoint = CGPointMake(0.5f + startCenter.x, 0.5f + startCenter.y);
	CGRect bounds = button.bounds;
	button.layer.position = CGPointMake(button.layer.position.x + bounds.size.width*startCenter.x, 
									button.layer.position.y + bounds.size.height*startCenter.y);
	button.transform = CGAffineTransformMakeRotation(-amplitude*DegToRad);
		
	[UIView beginAnimations:nil context:NULL];
	[UIView setAnimationRepeatAutoreverses:YES];
	[UIView setAnimationRepeatCount:FLT_MAX];
	[UIView setAnimationDuration:0.12];
	[UIView setAnimationDelay:random.GetFloat(0,0.09f)];
		button.transform = CGAffineTransformMakeRotation(+amplitude*DegToRad);
	[UIView commitAnimations];
}
	
	
void StopPotWobble(UIButton* button)
{
	[button cancelAllAnimationsRecursively];
	button.transform = CGAffineTransformIdentity;
	CGPoint anchor = button.layer.anchorPoint;
	CGPoint offset = CGPointMake((0.5f - anchor.x)*button.width, (0.5f - anchor.y)*button.height);
	button.layer.anchorPoint = CGPointMake(0.5f, 0.5f);
	button.layer.position = CGPointMake(button.layer.position.x + offset.x, button.layer.position.y + offset.y);
}

Ah yes, and I couldn’t find a way to loop the animation infinitely (without using the new UIView animation syntax), so FLT_MAX will have to do :-) Any cleaner way?

If someone has an animation that more closely resembles the iPhone launchboard, I’d love to hear about it. Let me know and I’ll update this post.

To put all the whining and complaining that’s about to come in perspective, I want to build some character references first. I’ve been programming for 27 years, 13 of those professionally in the games industry, although the non-professional years I probably did just as much coding. I have used everything from emacs + make files to the gamut in IDEs: Turbo C++, Visual Studio, KDevelop, and Xcode 3 among others.

Contrary to popular belief, I’m not that picky when it comes to a development environment. Or at least, it doesn’t feel that way to me. I want the basics, that’s all. Doesn’t everybody want this?

• Have 2-3 source code windows opened side by side.
• Build and see the output.
• Do everything with keyboard shortcuts.

There are lots of other things I want (project configuration, debugging, run tests), but those are the ones I’m doing 99% of the time. I never had a problem doing any of those things with past development environments.

The hit list

Having more tools and options is great. I love having the option to do advanced operations on my code, but I want to decide when to use those tools, not being bombarded by them. That, in a single sentence, is why Xcode 4, out of the box, is completely unusable for me.

Attempting to do something as simple as type code and build it is extremely frustrating. Xcode will react to me doing that by flashing all sorts of stuff on screen, bring up sidebars with messages, change which source files are displayed, draw squiggles all over my code, and flash alerts overlays. All of that, while not showing me the build output.

Working on Xcode is an extremely noisy (visually) and distracting experience. I understand it’s trying help me, but it’s failing miserably at it. I’m trying to concentrate on what the code does, and I feel I have a parrot yelling stuff from my shoulder and flapping in front of the screen in some misguided attempt to help me code better.

That might work for some people, but not for me. I’m the guy who turns off “spell check as I type” everywhere because I don’t want to be taken out of flow while I’m typing (I’ll do a spell checking pass later on, thank you). I don’t have notifications on my mail or Twitter; I check them explicitly when I want to. Now take that kind of annoyances and multiply it times 10 and you start getting an idea of what working on Xcode 4 is like. Xcode 4 feels like it’s squarely designed for the texting/ADD generation.

Fortunately, most of it can be remedied and turned into a half-way sane environment.

Here’s how to tame the noise in Xcode 4:

• Autocomplete is one of the worst offenders. Turn it off! Good autocomplete is awesome, but I’d rather ask for it (with a keystroke) than have it bombard me constantly. Not just that, but it’s so badly implemented that it will a) put shadows on top of the text I’m typing b) replace the text I’m typing on the fly so I can’t even see what I typed (see screenshot below). Off with it!

• Having the whole screen flash with squiggles on and off while I’m typing isn’t helping in any way. Xcode is trying to “highlight instances of selected symbol”, which is a really useful tool, but not while I’m typing/thinking! Not just that, but half the time it will add extra squiggles to the screen to make sure I can’t even read the symbols themselves. So off with that one too.

  

  Here’s one very frustrating thing: When I turn off the option to automatically highlight instances of the selected symbol from the settings, I lose the ability to do that at all. Yes, you heard that right: I can’t (as far as I can tell) press a key and have the IDE highlight all the instances on demand. No sir, that’s not the way it was intended.

• Continuing with the theme of noise and distractions, having a sidebar popup and start showing me errors with my code as I’m typing is also definitely NOT helping. Fortunately we can turn off “Show live issues” in the general settings.

• One thing that Xcode 4 appeared to have gotten right is the idea of showing multiple source files at the same time. Now many IDEs do that by default these days, so that was a plus. Unfortunately, it’s part of the “smart” assistant. As a general rule, I end up turning off anything with the word “smart” in it, and this is no exception. The assistant tries to present “relevant” files to the one I’m working on. No, no, no, no! *I* want to decide what to display and what to work on. Just get out of the way and let me do that easily. You can turn off the view to “Manual”, which helps a huge amount.

Just those changes make Xcode 4 into something that at least I can type into and not get sick at my stomach. We’re making progress.

Now I can finally build and… everything goes wrong. Xcode doesn’t show me the build output and it decides to flash a “build completed” screen overlay in case I looked away in the two seconds it took to build. Seriously? Maybe my attention was drawn into some of the flashing squiggles and I forgot it was building.

Fortunately for my sanity, it’s possible to fix both those things. The alert is easy: Go to behaviors, “Build succeeds” and turn off “Show bezel alert” (make sure it’s off for all the others while you’re there).

The other one is a bit trickier, but it’s certainly possible. Set up the “Build starts” behavior to open a named tab and go to the current log. Like this:

Now when you build your project, you’ll get to see something sane like this. It might initially be as a tab, but you can drag that out and make it into a separate window of its own so it doesn’t obscure the source code window:

You should be able to navigate through any issues found during the build with Cmd + ‘, or, if you like the fancy issue display, press Cmd + 4. Things are so much better when you initiate them on demand rather than when they’re forced on you!

Finally, a couple minor UI annoyances that are easily fixed:

• The toolbar is positively huge. I feel crammed in the high-res 15" MBP (1680 x 1050), so I can’t imagine how anyone works with something smaller. In any case, hiding the toolbar helps tremendously. As a bonus, the setting is per window, so your source code windows get the extra real estate, but the build output window keeps the toolbar so you have easy access to changing the target platform.
• Line wrap is on by default. Seriously? Off.

The suck list

If that was everything, I’d be happy. It means that even though Xcode 4 ships with retarded defaults, it can be whipped into shape into something usable. Unfortunately, there are a two major things I haven’t found a workaround for:

• Can’t build a single file. Someone at Apple, in their infinite wisdom, decided that building a single file was obsolete with the awesome new feature of “show live features”. Except that “show live features” sucks and has to be turned off. Apparently you either take it their way, or the highway, because there isn’t an alternative. As with the case of highlighting symbol instances, there isn’t a command I can use to “show live features” on demand.

  The alternative is doing a full build. Most of the time that’s fine, but whenever you’re doing one of those refactorings that breaks half the codebase, then you’re really screwed. The only way I can stay focused in a case like that is by cleaning up one file at the time, but this makes it impossible as you keep getting errors from all over the codebase. And as you fix one, the first error in the next build might be in a different file. Very annoying to say the least, especially because this was working and they removed it on purpose. I never file radar bugs (long story), but I made an exception just for this.

• Build configurations and schemes. Did you notice that when you built the project earlier there was no option to choose debug vs. release or some other configuration? That’s another one of those decisions that make you wonder if Apple uses Xcode to manage their own projects (and if they do, what kind of developers or projects they have).

  They turned something simple like project configurations that nobody ever complained about, into something byzantine worthy of some of the best efforts from Redmond. This gets the award for “most complicated feature nobody needed or wanted”. It seems now we have an extra layer of indirection. So you act on a scheme, which then decides which configuration to build and what to do. That’s why there are “Build for analysis” (debug), “Build for running” (release), “Build for testing” (WTF?), “Build for archiving” (distribution?).

  After spending some time with it, I’m still confused how I would go about running the release configuration, or how to mass-edit all the schemes at once (every target seems to get a different scheme).

  The whole schemes thing leaves me completely speechless, and that vein on the side of my forehead is pulsating just from thinking about it, so let’s just move on.

The good

After you take the time to fix Xcode 4 into something usable, and (somehow) put up with the new broken features, is there something to like?

I had to dig deep, but I found a couple of things that I like better than Xcode 3:

• Much better project settings UI: Multiple panes, showing multiple values for different configurations, etc. Way better than Xcode 3.
• Better semantic analysis. During the build process, Xcode 4 was able to correctly flag some problems in my code that Xcode 3 never warned me about. Mostly to do with constness of parent-scope variable when using blocks.
• Better keyboard shortcut support for opening files in different windows (assistant pressing the Option key). I never found a way to open a file in a specific window in Xcode 3 without having to use the mouse.

And that’s pretty much the whole good list that I was able to find :-|

Conclusion

Xcode has been progressively evolving from version 1 until version 3.2. I have no doubt that’s the reason for some of the quirks in the later versions. Xcode 4 seems to be a total re-design and re-implementation, and suffers from the classic second system effect. It’s change for the sake of change, most of the time scrapping useful features and not offering useful alternatives.

I’m sure I only scrapped the surface of the horrors of Xcode 4 in the day I spent with it. I didn’t look very closely at the debugger or the Interface Builder (saving those for another day). I keep hearing about developers with stability problems with Xcode, although I didn’t run into any crashes (but I hardly did any real work with it).

Overall, my recommendation is that, if you’re comfortably working on Xcode 3, stay as far as possible from Xcode 4. We’ll all be forced to move there sooner or later, but in the meanwhile you can continue being productive. Maybe by the time they discontinue Xcode 3, Xcode 4 will have improved a bit.

These are mostly quick notes to myself so I remember what to change when I upgrade my main machine, but I thought other developers hesitant to upgrade to Lion might find it useful as well.

Things that had to go

I found Snow Leopard (and Leopard before it) to be very palatable out of the box. I don’t remember any things I absolutely had to do. Lion, however, had some questionable defaults. Or at least, ones that I just can’t take.

After making these changes, Lion was quite usable:

• Turn off “natural” scroll direction. The fact that before you can run Lion for the first time it forces you to scroll that way should be a tip off that people aren’t ready for this one yet. Maybe in a few years it will make sense to have it as the default, but now it’s just against every single computer out there (including most Macs).

• Autocorrect. This is not an iPhone. It *has* a perfectly fine keyboard. Thank you.

• Setting up Spaces. Oh, excuse me, Mission Control (sigh). This is the one I was really afraid of. I absolutely rely on having about 8 spaces (2 monitors in each) open at all times and swapping with a keystroke. That’s something that I’ve been using for many years (first in Linux, then hacked onto Windows XP, and finally in Leopard/Snow Leopard). I just absolutely have to have that.

  Update: There are still a couple annoying things with the new setup. You can’t see the number of the desktop you’re currently on, like you did with earlier versions of OS X. Daniel Collin pointed me to an interesting thread on StackOverflow where people are discussing how to get access to desktop numbers. Give it a few more weeks. Also, when you restart the computer, Lion is smart enough to restore all the apps to their last state… except that they’re all opened in the first desktop, not in the one they were when the system restarted. How difficult would that have been? Here’s hoping it’s fixed in a patchupdate soon.

  The good news is that it’s possible to set up Lion that way. The bad news is that by default, it’s quite far from that. Bring up Mission Control and create by hand as many desktops as you want. Then go to System Settings | Keyboard | Keyboard Shortcuts and turn on one for each desktop. And for the love of god, please go to Mission Control settings and turn off “Automatically rearrange spaces based on most recent use”. Clearly Apple doesn’t think people want to go to a space directly, and instead prefer to browse them one at the time.

The next few changes just made my experience better, but they’re not absolutely musts:

• This might sound dumb, but Finder didn’t have a good way to easily browse my files. Instead, it has this weird “All My Files” view that jumbles together all your junk there. I have no idea who thought that was a good idea. Maybe Apple is trying to wean users from browsing around the file system. In any case, you can drag your home directory to the sidebar in Finder and that makes it much easier to browse. While you’re at it, turn off the “Hide extensions in files” dumb setting.
• Having used a Mac for several years, I’m much more sensitive to ugly UI elements. That’s why it’s so weird that, by default, Lion has the sidebar font in all programs just very slightly larger than the regular system font. Fortunately I wasn’t the only one bothered by that and it’s an easy fix.

That’s about it. Lion is not very nice and usable (i.e. looks like Snow Leopard). You have that weird “full screen” button on the top right of all windows, but that’s much better ignored (which funny that, was the feature Apple really pushed in their PR for Lion). I have no idea why anyone would want to use that other than the usual word processor that blocks out everything else. If I want to have crippled multitasking, I’ll just use the iPad.

Good things about Lion

So far, if all I’m going to do is disable new features so it looks like Snow Leopard, is there there any reason to upgrade to Lion? I surprised myself, but the answer is yes (although they’re all very, very minor).

• Resize a window in any corner and cursor feedback (finally, only 20 years late to the party).
• Perfect contact and calendar syncing with Google. I can’t tell you how much I appreciate that. I use Google as the backend for all my mail, calendar, and contacts. With Snow Leopard you could sync them, but you had to do some digging and mess around with the sync program. With Lion it works out of the box. Totally awesome. That alone makes it worth the upgrade.
• Much improved Mail UI: Better layout, better shortcuts, better conversation view, and, my favorite by far, finally an Archive command for Gmail. Archive doesn’t exactly do the same as archiving in Gmail, but it’s close enough. Finally I can have two different keyboard shortcuts: One for trashing an email and another one for archiving it.
• The jury is still out on how Lion restores the application state when you launch an app. I’ve been pleasantly surprised a couple of times, and annoyed a couple other times that I had to close a bunch of useless tabs. Maybe there’s a shortcut like Shift + Cmd + Q that quits and app without saving state? That’d be handy (OK, that one is not it :-) Oh wow, it’s Option + Cmd + Q! I love it when something makes sense like that and you can guess it. OK, so that makes this feature very cool.
• Slightly improved Safari (better download management). I really don’t see a reason to use Firefox or Chrome when you can use the preinstalled Safari. But then again, I can’t stand web UIs and I use native clients for lots of things.

Must-have apps

It’s actually quite refreshing to use the bare minimum number of extra apps beyond what comes with the OS. But there are some apps I just have to have in order to do anything. The main two are 1Password and Alfred. I can’t recommend them highly enough. They completely change how I work and make me ten times as productive (you’ve probably noticed by now that I’m not a mouse person).

After those two awesome super-apps, is the second tear of very useful ones: JustNotes, Things, and TextWrangler.

Probably Dropbox should be there somewhere, but for some reason, I still haven’t felt the need to use it heavily. I know some people put almost everything on Dropbox and can use across computers. Maybe it’s because I only have one that I haven’t felt the need for it too much. 1Password and Things both sync from the iPhone so it was really handy restoring all my data. I probably could have done the same thing with Dropbox.

Tomorrow I’ll post similar quick notes about Xcode 4. Those might not be quite as positive though.

I use the URL shortener in Flower Garden to send bouquets through SMS. Space is limited in a text message, so the message just contains some text explaining what is it and the URL pointing to the bouquet image. (Yes, I would much rather send them through MMS, but Apple isn’t exposing that yet to developers).

In this case, the full URL is http://flowers.snappytouch.com/sms.php?id=949618b4b3c6f3d76e32b45446e238a0 which gets thankfully shortened to http://goo.gl/IV5cq.

Sending bouquets through SMS has been in Flower Garden for several months, but it was using bit.ly before, which is probably the most popular URL shortener out there. I like their web interface and their super-easy to use API, but unfortunately it seems that I hit some mysterious API limit during the Mother’s Day Flower Garden promotion. That limit isn’t public anywhere, and as far as I can tell, I can’t even see it myself through the web interface or through an API query.

Finding out that I reached the API limit was quite shocking, because sending bouquets through SMS isn’t a particularly popular feature. Unfortunately I don’t have good analytics hooked up to that step, but I can’t imagine there were more than a few thousand per day.

They were very nice and contacted me instead of shutting down my account since it was just a spike. They also tried to sell me their “Enterprise” account, but $995/month is a tad too expensive for me. By about $990 probably, so I had to look for other options.

After a very quick research, goo.gl was the perfect alternative. Not only is it very fast (and backed up by the giant Google no less), but they have an API limit of 1,000,000 queries/day. If I ever blow that budget, I’ll be able to afford the $995/month without a problem :-)

All URL shorteners are very easy to use. You need an API key, and figure out the exact format of the HTTP message you send and the response you get.

goo.gl

Goo.gl has a great Getting Started Guide that tells you everything you need to know. Get your private API key from here, and you’re ready to rock.

Drop this in your app and start shortening away. You’ll notice I used a synchronous HTTP request, which is usually a big no-no. Here I felt it was justified since the user is blocked waiting for the SMS to be prepared and sent. Besides, goo.gl is very, very fast, so it’s never even noticeable.

const NSString* GooGlAPIURL = @"https://www.googleapis.com/urlshortener/v1/url?key=YOUR_API_KEY_HERE";

NSString* ShortenURLWithGooGl(NSString* longURL)
{
	NSURL* apiUrl = [NSURL URLWithString:GooGlAPIURL];
	
	NSMutableURLRequest* req = [[NSMutableURLRequest alloc] initWithURL:apiUrl];
	[req setHTTPMethod:@"POST"];
	[req setTimeoutInterval:Timeout];
	[req setValue:@"application/json" forHTTPHeaderField:@"Content-Type"];
	
	NSString* body = [[NSString alloc] initWithFormat:@"{\"longUrl\": \"%@\"}", longURL];
	[req setHTTPBody:[body dataUsingEncoding:NSUTF8StringEncoding]];
	[body release];
	
	NSError* error = [[NSError alloc] init];
	NSHTTPURLResponse* urlResponse = nil;
	NSData* data = [NSURLConnection sendSynchronousRequest:req returningResponse:&urlResponse error:&error];
	[error release];
	if (data == NULL || ([urlResponse statusCode] < 200 && [urlResponse statusCode] >= 300))
		return NULL;

	NSString* response = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];
	NSDictionary* responseDict = [response JSONValue];
	NSString* shortURL = [[responseDict objectForKey:@"id"] retain];
	[response release];
	return shortURL;
}

I’m using a JSON framework to parse the answer, but it’s so simple I probably wouldn’t even have to. I only used it because it’s already part of the project because of Facebook Connect.

bit.ly

Even though it’s not my favorite shortener, I’m adding it here for completeness (and because I had the code already written).

The one thing that bit.ly has going for it is that it’s even easier to use than goo.gl. No JSON involved, and you don’t even need to send a body with your request. As usual, get your API key by signing up with bit.ly.

const NSString* BITLYAPIURL = @"http://api.bit.ly/v3/shorten?login=%@&apiKey=%@&format=txt&";

NSString* ShortenURLWithBitLy(NSString* longURL)
{
	NSString* urlWithoutParams = [NSString stringWithFormat:BITLYAPIURL, LoginName, APIKey];	
	CFStringRef encodedParamCF = CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault,
															 (CFStringRef) longURL, 
															 nil, (CFStringRef) @"&+", kCFStringEncodingUTF8); 
	NSString* encodedURL = (NSString*)encodedParamCF;
	NSString* parameters = [NSString stringWithFormat:@"longUrl=%@", [encodedURL stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding]];
	NSString* finalURL = [urlWithoutParams stringByAppendingString:parameters];
	
	NSURL* url = [NSURL URLWithString:finalURL];
	
	NSMutableURLRequest* req = [[NSMutableURLRequest alloc] initWithURL:url];
	[req setTimeoutInterval:Timeout];
	
	NSHTTPURLResponse* urlResponse = nil;  
	NSData* data = [NSURLConnection sendSynchronousRequest:req returningResponse:&urlResponse error:NULL];
	if (data == NULL || ([urlResponse statusCode] < 200 && [urlResponse statusCode] >= 300))
		return NULL;

	NSString* response = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];
	return response;
}

That’s it. You should be able to drop in either one of those snippets in your project and spend your time working on the things that really matter in your games.

Recording particular events as part of the analytics is only part of it. The most important part is how that data is presented to the developer. Having tables with millions of entries does me no good, and as a busy indie developer, I can’t afford to spend hours writing scripts to analyze it. I want something that allows me to easily visualize the data and makes sense out of it at a glance.

One possible snagging point about analytics is that Apple was cracking down on some applications with analytics enabled a while back. Specifically, the iOS developer agreement states:

3.3.9 You and Your Applications may not collect user or device data without prior user consent, and then only to provide a service or function that is directly relevant to the use of the Application, or to serve advertising. You may not use analytics software in Your Application to collect and send device data to a third party.

Where does that leave us? Reading it carefully, it seems that the restrictions are limited to “user or device data”. I’m interpreting that to mean things like UDIDs and emails, not anonymous player usage data (how long did it take to reach level 5, how long are play sessions, etc), so I think we’re clear there.

The puzzling thing is that a lot (all?) of the third party analytics libraries do report device information, like what kind of hardware or iOS version is running. That is extremely important information that developers really benefit from knowing, but it seems to go against point 3.3.9. Maybe “device data” only applies to information about that specific device? (as in, the UDID that is now going away). I hope so.

Not having analytics isn’t really an option. Unless you make a game that you plan to throw on the App Store, never touch again, and hope for the best, you’re flying blind without analytics.

What are some of the options we have then?

Home brewed

If you read this blog regularly, you probably know that I’m a low-level, do-it-myself kind of guy, with a deep mistrust and suspicion of middleware. So you would think that I would want to write my own analytics package. After all, how hard can it be? Collect the data you want and ping your server with it. If you get fancy, you can even use a scalable server back end like AWS or GAE. Done.

Not so fast. To do that well, it’s a lot more involved than that. You want to batch when you send out the information, and you might want to distinguish between WiFi and 3G connection (to avoid causing extra data usage for players on a limited data plan).

That in itself is not even that bad. The real pain comes in visualizing that data, and that’s where you can easily sink in days or weeks, and you would still not have something as good as some of the other alternatives.

The other drawback is that if you have some successful applications, you may be generating Terabytes of data per day. Think about the storage and bandwidth costs for that. Yes, I know that sounds insane, but Playfish reported generating that much analytics data at a GDC 2011 talk (video for paid GDC Vault members).

Flurry

Flurry Analytics appears to be the most common analytics package out there among my indie iOS dev friends. It’s free and it’s easy to integrate. The visualization page is pretty good, and it even offers some fancy features beyond session length and events, such as flow through the application.

So what’s not to like? I was never able to make heads or tails of the application flow. When you get at that level, you start needing to spend some serious time making sense of data, which is what I don’t have as an indie. The web page to visualize the data is written in Flash, so for those of you using an iPad to check it, it’s not a good option. Update: Flurry apparently added a Flash-free web page since I last looked at it a few months back. Thanks Charilaos Kalogirou for the tip.

The killer deal for me was bloat. Adding the Flurry analytics library to an app increased the executable size by 500KB. I’m sorry, but that’s completely unacceptable for me. Memory is tight, and half a MB is very significant. I would rather add another large texture or another music track. And if you think about it, why does it need 500KB to buffer and send some events? That’s simply ridiculous.

Google Analytics

I never actually tried out Google Analytics. They have an iOS SDK and it integrates quite well into the web page Google Analytics environment. The main drawback I heard from other developers is that it’s designed more for web pages rather than for apps and games, so it wasn’t a perfect fit.

Anyone who used it want to expand on this in the comments?

Localytics

Localytics is a relative newcomer to the iOS analytics field, but it was love at first sight for me.

It has the same ease of integration of Flurry, and it provides very similar functionality. Localytics, however, is completely open source, so instead of a black box library, you get the source code and you can add it directly to your game. How much does it increase executable size? 4KB! You have to wonder what the other 496KB were for in the Flurry library.

As a bonus, their visualization web page works great on an iPad, although it can be a bit slow for very large data sets sometimes.

One of their biggest selling points is that they report the analytics in real time, but I really don’t care one way or another. Waiting 12 or 24 hours to see the analytics doesn’t bother me one bit.

Unlike Flurry, you can only add strings as parameters to events. That works fine if I have a set of discrete options. For example, when someone sends a bouquet in Flower Garden, I can send a “bouquet sent” event with a parameters that is “email”, “facebook”, or “sms”. As a result, I can see a nice pie chart with the breakdown of how people are sending bouquets. Very useful stuff!

But how about things that don’t have discrete options? For example, in Casey’s Contraptions, we wanted to see how long players take to solve each level. It turns out you can’t have a number as a parameter, but you can easily get around that by discretizing it yourself, which in the end, is easier to visualize. So when we send the LevelXXFinished event, we look at how long the player took to finish it, and we break it down into ranges: under 30s, 30s-1min, 1min-2min, etc.

This is what the report looks like for one of Casey’s Contraptions levels:

It looks like a fairly balanced level with the majority of the people spending under 3 minutes to solve it.

Random tips

A couple random things we learned along the way about analytics:

• Less is more. Start with just a few events and go from there. If you have tons of data, you might never have the time to look at it.
• Use analytics during playtesting. One thing is what people tell you, and another thing is what they really do. Since most of our playtesting is done remotely and we can’t observe as people play (which is invaluable), we can at least gather some hard data about it.
• Turn off analytics reporting in debug mode. Trust me on that one.

How about you? What’s your favorite analytics package and why?

[1] For example, Flower Garden has been on the App Store for almost two and a half years, and it has changed radically in that time!

Casey’s Contraptions is an iOS game created by the two of us, Noel Llopis and Miguel à ngel Friginal. Noel, an industry veteran for over a decade, turned indie over four years ago and found success with microtransaction-based Flower Garden on iOS. Miguel worked as a graphic designer in the advertising industry for years before becoming a web developer. Casey’s Contraptions is his first published video game, although his first paper role-playing game came out almost 20 years ago. We met through Twitter several years ago, and then finally in person at a 360iDev conference. Even thought we didn’t plan it that way, we ended up working together during a game jam, and that set us in the path to collaborate in a future project.

We knew we wanted to target iOS for our next project because we love the platform from a user and a developer point of view, and because it’s a platform where it’s possible for indies to succeed financially. Beyond that, starting a new game is never easy. Even though we have page after page of possible ideas, settling on a specific game idea is always very hard. We wanted something that met three requirements: The game had to be creative in nature as opposed to using destruction as the main gameplay element, it had to be something we were excited about, and it had to be something with the potential to sell reasonably well on the Apple App Store. Easier said than done!

We prototyped game idea after game idea, and even though a lot of them were not bad, none were a complete standout. Eventually, after seven or eight different prototypes, we settled on the concept of creating physics-based, Rube Goldberg-like contraptions to solve different puzzles. At its core, the game has similar mechanics to some classic games like The Incredible Machine, but with an emphasis on exploration of creative solutions rather than finding the one right answer to each puzzle, designed around sharing, and built from the ground up for a touch interface on iOS.

We structured this project as a pure 50-50 partnership, and without any external funding or publishers. Miguel quit his full time job (as a web programmer no less!) to do the art for Casey’s Contraptions, and Noel took care of the programming. Design was a fully collaborative activity, where we both contributed equal amounts to everything from the rules, the feel of the game, or level creation. Everything else we split up based on our strengths and expertise: web site, server back end, PR, etc.

This is not a true postmortem, but more of a post-launch analysis. Games today, and especially iOS and Facebook games, are becoming more of a service rather than a product. Development and launch are only the beginning of the story. If all goes well, we’ll have a lot more to say about Casey’s Contraptions in a few months.

What went right

Strong theme and style

Casey’s Contraptions started life without Casey! The initial prototype focused only on the creation of contraptions and the physics simulation behind it. It showed a lot of potential, but it was missing something to make it stand out. It needed more personality.

After some brainstorming, we quickly zeroed-in the idea of a smart 8 year old building the contraptions. Not only did that add the much-needed personality to the game, but it also focused the rest of the development. Instead of doing a physics game with generic pulleys and levers and other industrial-looking items, we naturally went with toys and household items Casey could have access to: His sister’s doll, paper planes, or his RC truck.

The choice of toys as game items not only influenced the level design, but also the goal of the levels. At the beginning we were thinking of doing levels to accomplish tasks Casey would have to deal with in real life, such as putting toys away or popping a balloon. After a while, the idea of “playtime†levels came up, where the setting and the goal were completely imaginary (rescue explorers from a jungle in their hot air balloon) just as Casey would have imagined during his playtime sessions.

The character of Casey also helped anchor the art direction. We were aiming for something that would appeal to a very wide range of people, from the very young to adult casual gamers. Although Miguel looked into everything from a most modern Cartoon Network style to manga, to crazier Warner Bros., we ended up settling with a mix of Calvin & Hobbes and something out of a 50s Hanna-Barbera show, with strong outlines and solid colors, to make people of all ages feel comfortable with the visuals. We were able to carry this distinct style to all the items, locations, and user interface in the game, which gave it a very consistent and unique look.

Social features

Creating something is fun, but creating something and being able to share it with people is twice as much fun. We wanted to make Casey’s Contraptions a very social experience from the start. Not something that you played through and put away, but something you could share with friends along the way.

Since Casey’s Contraptions uses a full physics simulation at its core, it’s possible to come up with very unique, chaotic, and often unexpected solutions to most puzzles. Weeks after launch, we’re still amazed at how some people are solving some levels, coming up with item combinations we never thought about during development. Since these solutions are very unique and fun to watch, they were a perfect candidate to share with friends.

With that in mind, we designed the game with sharing from the start. After you complete each level, you can share your solution with all your friends just with the tap of a button (and, in the latest update, the default is to autosubmit a solution if you improved your score). Your friends’ solutions are equally accessible, as cropped thumbnails in the level completion screen, and tapping on any of them will bring up a full-screen view that you can even replay and watch the full solution.

In addition to sharing solutions, we also included a level editor to allow players to create their own contraptions from scratch. This was the same level editor we used to create all the levels in the game. Nothing like eating your own dog food to make something solid and usable. Initially, players were able to share these levels through email, and, in the first update, we also added the ability to share them publicly through a web site (http://shared.caseyscontraptions.com), effectively giving players access to hundreds of new levels.

As it’s the case with most games that include level editors, we were aware that only a small minority of players would take the time to create their own contraptions. But it’s also those players that are really devoted to the game, and take it upon themselves to spread the word about your game and champion it to all their friends.

Iterative development

For Casey’s Contraptions we used a very stripped down and relaxed form of iterative development. We had short iterations and in each iteration we aimed to fully implement what we considered the most important features at the time. We didn’t do any real estimating of tasks (other than, “yeah, we think we can do that in about two weeks†), and we didn’t set a hard-limit on the iteration (they varied naturally between one and a half and three weeks). The most important concept is that at the beginning of each iteration we would make decisions about what to work on next, and those decisions were made with all the knowledge leading up to that point.

For example, we didn’t start the project by coming up with the full list of items we were going to have in the game. Instead, we had a list on the wiki of possible items (to which we would add more whenever we thought of a new one during development), and we only decided which new items to add to the game at the beginning of each iteration. That allowed us to make good decisions based on what we had learned so far: “Most objects we have so far fall down. We need more items that add forces upwards†, or “The magnet is fun, but we need more metal items to make it more useful†.

This mentality applied to everything: From level creation, to menus, features, etc. Looking back, we can say there’s no way we would have made the same decisions early in the project than we did as we went along.

Strong launch

In less than 24 hours after launch, Casey’s Contraptions worked its way up to the top 10 paid apps in the US and in over 20 different countries. A day later, it reached the #2 overall spot in the US and had great initial weekend sales.

This strong launch wasn’t just pure luck. It was something we planned months in advance and worked hard to achieve (although we did need a dash of good luck). We wanted to build awareness and buzz around the game, but given the short development cycle for iOS games, we would have to do it in a shortened scale.

We started out by announcing the game more than six months before release (25% of the way into development). During the following months, we continued talking about the game on Twitter and our blogs, often showing work in progress or outtakes. The next big milestone was showing the game around at GDC. Not only did we get a lot of other developers to play it and give us invaluable feedback on it, but we also met with some of the game press, and that resulted in some very nice previews afterwards.

The final push came as soon as we submitted the game to Apple for review. We decided to set a fixed release date three weeks after the submission, which would give us enough time to do all the PR work: creating a video, putting together a media packet, contacting media outlets, etc. In the weeks leading up to the launch, we also stepped up our blogging of different interesting aspects of the game.

As a result of everything we had done up to that point, we were very lucky that Casey’s Contraptions attracted Apple’s attention, and they featured it prominently on launch day as iPad Game of the Week worldwide. We had given the press enough time with the game, so a lot of very positive game reviews came in right around launch day, helping get the word out for the game.

Even though we had been originally thinking of pricing the game at $4.99, we decided to shoot for volume instead and priced it at $2.99. That turned out to be the right decision and immediately put us on the top 10. Sales on iOS charts follow a sharp, exponential drop off, so being in the top 5 represents a huge increase in sales over just a few positions down the chart.

Looking at the App Store today, it’s apparent that it’s becoming harder for small, quick games to be really successful and top the charts. With over half a million different apps on the App Store, and hundreds of games released every day, you really need to stand out from the rest to be noticed. Most of the games that manage to do that are ones that required significant time and effort investment and have good production values. The App Store gold rush is over.

Enough development time

From start until launch, Casey’s Contraptions was in development for 8 months. That seems like a long time by iOS standards, although more and more successful iOS games are starting to take that long.

Our initial plan was to ship the game by Christmas. It wasn’t based on any rigorous estimating, just an off-the-cuff estimate. It just “felt†like we could be done by then. Obviously we were wrong.

Taking the amount of time that we did was a very positive thing. We didn’t really waste much time, it’s just that the game needed that amount of time to mature and get to where it is today. If we had chosen to ship it earlier, the final product would have suffered significantly.

One of the hardest things in a game like Casey’s Contraptions is making interesting levels. Having this amount of development time, with a working game editor since the very beginning, allowed us to make a lot of levels along the way. Looking back at a lot of the early levels we made, they were laughably hard and not that interesting. After several months, were able to develop a sense about what made good levels and what the appropriate difficulty was.

Having enough time allowed us to make fairly fundamental changes to the game design when something wasn’t working. For instance, initially each level had three different goals you could achieve. Depending on the number of goals you accomplished, you earned a bronze, silver, or gold medal. It quickly became apparent that players were extremely confused by the three goals and weren’t able to keep them straight.

We changed levels to have a single goal, but since we still wanted to have some replayability beyond “solving†a level, we added optional stars you could collect by touching them with any item in the game. The first star would be very easy to get, the second one a bit harder, and the third one would require some serious thinking. That was a huge improvement, but then we discovered that most players expected to get all three stars in their first playthrough, and would stubbornly keep playing the same level until they got them all or quit in frustration. That prompted us to change the stars yet again. This time the difficulty of getting them was tied to the overall level difficulty, so players could easily get all three stars in the early levels. We’re very happy with the final design, and we would not have gotten there if the project had been rushed.

We also gave ourselves sufficient time at the end for polish and style. Polish isn’t going to make the game design any better, but it’s going to contribute a huge amount to first impressions. Every animation, sound, and particle effect becomes really important in those crucial first few seconds with the game. On a mobile platform, polish becomes even more crucial. If your game doesn’t immediately engage the player, there are lots of other things they can shift their attention to.

What went wrong

No simultaneous iPhone launch.

The initial prototype of Casey’s Contraptions was running on an iPhone. While it showed a lot of promise and it was fun to assemble contraptions even that early on, it was clearly begging for more screen space.

The iPad was the obvious platform of choice. Its large screen can display very nicely detailed graphics, and allows for very natural, direct manipulation of items. It was a perfect fit for Casey’s Contraptions.

Even so, while there are a lot of iPads out there (14 million), the iPhone and iPod Touch are the undisputed kings of the App Store (about 185 million devices). Especially for a game that relies on the social component, getting a critical mass of users playing at the same time, sharing solutions, and sending levels is very important. We definitely stormed up the iPad charts, but that still left the majority of iOS users not being able to purchase our game.

Why didn’t we wait until we had the iPhone version to launch? No particularly good reason other than we were itching to get the game out. We also had no idea what kind of impact a strong launch would have on our servers, so the idea of an iPad-first launch seemed like the way to go. In hindsight, we would have been better off waiting to launch both versions at the same time (or almost the same time, maybe a week or two apart at most).

To make up for this, we’re planning on making the iPhone release a second launch of sorts: We’ll make the iPhone version coincide with a new game update that includes a lot of new content (for free for people who already bought it), and we’ll try to repeat the same strong launch we had on the iPad. The idea is to get everybody who’s already bought the game playing again, along with all the new iPhone players and create that critical mass.

Butting heads too much

We make the perfect two person team: We have complementary specialties, but we also have a lot of overlapping skills. We also seem to always approach things from opposite ends: Aesthetics vs. usability, performance vs. gameplay, simplicity vs. interest, uniqueness vs. familiarity, or tea vs. coffee. That is actually a really good thing and the success of Casey’s Contraptions was in no small part due to our combination of personalities and skills.

There is, however, such a thing as too much of a good thing. We are both very stubborn and it will take a lot to convince us to see things in a different way. There were some times during development that we spent more time debating one point than actually implementing it.

The fact that we’re working remotely didn’t really affect most of our day to day development, but it definitely made hashing out these situations significantly harder, dragging them out for far longer than they should have. This was also the first time we were working together on a significant project, so it made resolving those situations more difficult. Now that we’ve gone through this first project, we’ll hopefully be better equipped to handle similar situations in the future.

Unnecessary rework

Rework is a necessary part of a creative process. You’re unlikely to get everything right in the first draft of a text or a music composition. That’s even more so the case for game development because there are so many parts interacting with each other. Not only is it hard to predict the exact final outcome, but even if you could, you often don’t know exactly what you want until you’ve seen one version of it.

Just about every screen and every item in the game went through several revisions of art, behavior, sounds, or layout. There’s no doubt that every revision made them better. However, we had a few parts of the game that we had to revise a few too many times. This was mostly in some of the UI screens, like the now infamous “level completed†screen, or the look and positioning of the in-game menu, which we must have gone through at least 6 or 7 complete redesigns.

We believe that design doesn’t just flow one way. If you want the best final product, you can’t just decide on the functionality of some user interface (or any part of the game for that matter), implement it, and then give it a pretty face with some graphics. Both the implementation and the graphic design will actually feed back into the functionality of the UI. Once you go around this cycle a few times, you can zero in on a very strong design, both from a functional and a design point of view.

The problem comes when you go around that loop too many times, or when the loop doesn’t converge into a particular design, but keeps shifting around. That was often caused because we were fuzzy on some of the details, or when we were forgetting about some particular feature that had to be reworked into the design. Some other times feedback from our testers made us realize that a screen wasn’t clear enough as we had designed it and caused us to rework it. As tempting as it was to push through and call it good enough since it was already completely done, it was always the right decision to go back and re-work things as needed.

For example, at the beginning of each level Casey explains what goal you need to achieve. This screen seemed straightforward enough, except that the version we had early on was blocking the view of the level while it was up. Our testers complained it was hard to remember what you had to do without seeing the items it was referring to at the same time, so the final design shows Casey’s dialog along the bottom of the screen, and the goal items are even circled with a marker.

We can alleviate this problem by iterating on particular pieces of UI or the game in smaller cycles, without taking each one to completion. Instead of starting by completely implementing a screen, or completely creating a perfect mock up with all the graphic elements, we need to start by laying out a screen with simple boxes and buttons and implement the basic functionality. Then we can make a first pass at a real layout and some graphical element, implement some of the new functionality and animations suggested by that, and continue iterating until it’s done. We got much better about this in the last third of development, and it’s something we want to carry forward to future projects.

Not enough unit-testing

Noel is a big fan of unit testing and test-driven development (TDD). Those are some techniques we used in past projects to very good effect and something we definitely wanted to carry into Casey’s Contraptions as well.

The goal was never to have 100% unit-test coverage or write every single line of code through TDD, but to write only the tests that would benefit the project. That usually meant some code that a lot of other code relied on (the game item management), or something complicated (toolbox interaction), or something prone to breaking (rope manipulation).

In the end, we slipped to the side of not having as many unit tests as we would have liked. Some things like object attachments kept repeatedly giving me headaches during development because of the complicated, untested code. By the time we noticed those problems and wanted to start adding tests, it was too late because some of that code relied on non-unit test friendly APIs like UIKit or Box2d.

We should have taken the time when those problems started appearing to start writing some tests and slowly refactor the code as we fixed the bugs and added new features. Instead, since we seemed to be constantly “just a few months away from shipping†, we decided to skip that and definitely paid the price later on. We even shipped with a few off edge cases that we knew were buggy but we didn’t dare fix weeks before submission.

We’re already working on updates and and an iPhone version of Casey’s Contraptions, so we’ll continue dealing with that code for quite a while to come. We’ll slowly introduce unit tests in areas of the code that we need to revisit during this time.

Fixed price model

The pricing model is something we went back and forth about several times during development. In spite the long-term success of Flower Garden and other free games with microtransactions on the App Store, we chose to go with a fixed-price model for Casey’s Contraptions.

We thought our audience would appreciate a traditional fixed-price model better than a microtransaction-based one. We also figured it was a model that was working well for other similar iPad games such as Angry Birds or Cut The Rope, so it made sense to follow their lead on that.

Unfortunately it seems that might have been the wrong decision from a financial point of view. After a really strong initial launch, Casey’s Contraptions dropped down the charts very rapidly after just a few weeks. Since revenue follows a very sharp exponential drop off, even being in the top 100 iPad games means very little revenue per day, and points to a very thin “tail†to the sales curve.

In spite of the bad reputation microtransaction games have in traditional game development circles, they’re a lot harder to develop than single-purchase, fixed-price games. Not only do you need to implement very robust server features, but you need to finely balance the in-game economy, the pace of rewards, and the cost of new purchases. Our optimistic estimate was that it would add at least a full month to the development time (and given how our estimates were, it probably meant two months for real).

A possible alternative would be to have a traditionally-priced game, but offer new levels or locations as extra purchases. This would have been a lot simpler to implement, but it probably wouldn’t have made much difference in revenue. Usually, only about between 2% and 5% of the players buy any extra content, so for microtransactions to really pay off, they need to be unlimited (in-game currency, fertilizer, etc), or have a very large user base. With a small, fixed amount of possible things to buy, we would need to rely on having lots of players to make much of a difference.

We’re hoping releasing the iPhone version will generate lots of renewed interest in Casey’s Contraptions, spur lots of sales on both platforms, and hopefully increase the long-term chart staying power. If that’s not the case, we can always consider the possibility of experimenting by changing the pricing model in the future.

Conclusion

We’re extremely happy with the development of Casey’s Contraptions and with the initial launch. We managed to create a unique game around creativity that was very well received critically and from a sales point of view.

The first free update should be available by the time you read this. It adds the ability to share contraptions with everybody through the web site, as well as browing public contraptions, which should increase the popularity of that feature quite a bit. It also adds some of the most-requested features, such as multiple player profiles for all the parents playing Casey’s Contraptions with their children out there.

Unlike a traditional retail game though, the story is far from over. What we do in the next few months will have a very significant impact on the long-term success of the game.

Facts

• Web site: http://www.caseyscontraptions.com
• Release date: May 19, 2011 (iPad). Summer 2011 (iPhone)
• Development time: 8 months
• Team size: 2 (full time)
• Development cost: Cost of living for 8 months + a tad over $1000
• Open source code: Box2d, UnitTest++
• Primary tools: Xcode, svn, Versions, Trac, TexturePacker, Adobe Illustrator, Audacity
• Lines of code: 46,518
• Raw asset size: 510 MB
• Total app size: 12.2 MB
• Subversion commits: 2442
• Trac tickets closed: 683
• Gallons of tea brewed: 77
• Shots of espresso consumed: around 1500

This article was initially published on Gamasutra in June 22, 2011. I reprinted it here for completeness, to keep it with all the other Casey’s Contraptions articles.

The main idea of the talk was the importance of “flexible pricing”: Letting players spend as much money as they want to get more enjoyment out of the game and customize their play experience. Flexible pricing can result in revenues much higher than the traditional fixed price approach (derived from manufactured goods). It’s also orthogonal to the concepts of freemium and social games, and we’re only scratching the surface in ways to effectively implement it in games.

Here’s the official conference abstract:

Did the biggest fans of your game spend more money on it than someone who played it just a few times? If not, you’re leaving a huge percentage of money on the table. This session will talk about how digital distribution has made fixed prices obsolete and how you can take advantage of flexible pricing through in-app purchases in your games. We’ll also talk about the consequences (good and bad) of making a game free to play. The session will present hard data from iOS platforms, but applies to all digital distribution platforms.

Answering questions preemptively: I definitely don’t think that allowing players to spend the amount of money they want is “evil” in any way. And someone during the comment came up with a great point: Isn’t it more “evil” to have players spend $60 on a game, just to find out 10 minutes afterwards that they don’t like it or it doesn’t run very well on their system/network?

I was going to record the audio, but unfortunately I forgot to turn on Audacity before the talk. Note to self: Start recording when I set up the laptop, even if it’s 10 minutes ahead of time.

One Price Does Not Fit All

View more presentations from Noel Llopis

[Slideshare mobile] [Slides in pdf format]

Are we slacking? Sipping piña coladas by the beach? Did we grow complacent with the successful launch of Casey’s Contraptions on the iPad?

The reason it’s taking so long is that we’re not putting out a rushed version of Casey’s Contraptions on the iPhone. Instead, we’re taking the time to re-work every screen to fit the smaller screen. Even if we could just cram everything into the screen, fingers don’t get any smaller on the iPhone, so the target areas need to be just as easy to touch.

This goes beyond just every UI screen. Quick, name a popular iPhone game that can’t be played with just your thumbs. I’m drawing a blank, and that’s for a good reason. The gold standard for iPhone games is whether it can be played with two thumbs. If it can’t, then it’s probably a bad design. So even though the levels are the same, we’re having to make major changes to how objects are manipulated.

In other words, good things take time.

Which reminds me of this wonderful quote from the Mythical Man Month:

Good cooking takes time. If you are made to wait, it is to serve you better, and to please you. Menu of Restaurant Antoine, New Orleans

All The Little Details

It’s more than just doing a good job though. There’s often a surprisingly large amount of work involved implementing small parts of a game. Things that, when you do them right, go completely unnoticed, but when you don’t, are jarring and irritating. Games are full of examples of things like that, from AI to menu screens.

In this case, I’ve spent a whole day and a half getting the behavior of the toolbox area working correctly on the iPhone. On the iPad, the toolbox area is the bottom wooden box that holds the items you can use in the game.

On the iPhone, since we’re dealing with a different screen aspect ratio, we decided to spend all our screen real estate on the game area, and keep the toolbox hidden. (And yes, since you asked, you can zoom and pan on the game area). Whenever you want to use the toolbox, you press the button and it slides out. Easy isn’t it? How could that possibly take a day and a half?

That’s one of those cases where the devil is in the details.

• Apart from pushing the button, the player can also tap it and slide in and out. When that happens, we need to detect what you mean with that gesture and finish opening/closing the toolbox after you release the touch.
• Whenever the player grabs an item out of the toolbox, we need to close the toolbox so it’s not in the way.
• Whenever the player grabs an item in game and drag it near the toolbox, it needs to open so you can drop it there.
• But dragging an item towards the edge of the screen indicates that you want to move it outsize the zoomed area, so we don’t always want to open the toolbox. Only when the player drags it near the button that opens the toolbox.
• But perhaps it wasn’t possible to scroll down any further (either because we’re zoomed out all the way, or we’ve moved the camera as far down as it goes), in which case dragging any object anywhere near the bottom of the screen indicates we should open the toolbox.
• Of course, it’s possible that the player just grabbed an object out of the toolbox, the toolbox closed automatically, and he didn’t move the touch near the bottom of the screen. In that case, we want to be aware of that and not open the toolbox.
• Dragging an item towards the toolbox, and then away from it should close it since it means the player changed his mind and doesn’t want to get rid of it yet.

Take all those cases, and make them work with and without zoom. And there are even a few more weird combinations we need to deal with, but you get the idea. If that sounds like crazy, well, it is kind of crazy. Fortunately it was something I was able to implement writing unit tests, so it’s pretty easy to modify and update as we need to change things.

For the curious, this is what just a couple of the unit tests look like:

struct F
{
	F()
	{
		camera.pixelSpaceWidth = GameParams::WorldWidthInPixels;
		camera.centerSS = Vec2(0.5f*GameParams::WorldWidthInPixels, 0.5f*GameParams::WorldHeightInPixels);
	}

	Camera camera;
	ActionQueue actionQueue;
};

TEST_FIXTURE(F, ItemOnBottomRightCornerDoesNotOpenToolbox)
{
	camera.zoomFactor = 2.0f;
	camera.centerSS = ScreenCenter;
	CameraUtils::Update(0.02f, camera, true, Vec2(ScreenCenter.x + 0.5f*ScreenCenter.x, ScreenCenter.y - 0.5f*ScreenCenter.y), false, actionQueue);
	CHECK(camera.triggeredToolboxOpen == false);
	CHECK_EQUAL(0, actionQueue.count);
}

TEST_FIXTURE(F, ItemOnBottomLeftCornerOpensToolbox)
{
	camera.centerSS = ScreenCenter;
	CameraUtils::Update(0.02f, camera, true, Vec2(10, 10), false, actionQueue);
	CHECK(camera.triggeredToolboxOpen == true);
	CHECK_EQUAL(1, actionQueue.count);
	CHECK_EQUAL(ActionType::OpenToolbox, actionQueue.actions[0].type);
}

TEST_FIXTURE(F, ItemOnBottomRightCornerOpensToolboxIfCantScroll)
{
	camera.centerSS = ScreenCenter;
	CameraUtils::Update(0.02f, camera, true, Vec2(GameParams::WorldWidthInPixels, 0), false, actionQueue);
	CHECK(camera.triggeredToolboxOpen == true);
	CHECK_EQUAL(1, actionQueue.count);
	CHECK_EQUAL(ActionType::OpenToolbox, actionQueue.actions[0].type);
}

TEST_FIXTURE(F, ItemOnBottomLeftCornerWithZoomOpensToolbox)
{
	camera.zoomFactor = 2.0f;
	camera.centerSS = ScreenCenter;
	CameraUtils::Update(0.02f, camera, true, Vec2(ScreenCenter.x - 0.5f*ScreenCenter.x, ScreenCenter.y - 0.5f*ScreenCenter.y), false, actionQueue);
	CHECK(camera.triggeredToolboxOpen == true);
	CHECK_EQUAL(1, actionQueue.count);
	CHECK_EQUAL(ActionType::OpenToolbox, actionQueue.actions[0].type);
}

When it’s all working, if we did our job correctly, all that players will think about is using their toolbox items and none of those weird cases will cross their mind. Things will just work as expected.

One More Thing

Here’s another example of another task with “hidden” work to behave as expected. When moving object in place in Casey’s Contraptions, and releasing the touch, how many times did the item move slightly because your finger grazed the screen as you released it? I hope the answer is never (or not very often). That’s not coincidence. We have some logic in place to detect those really annoying, but very common touch movements as people lift their fingers from the screen. Without it in place, it was pretty frustrating placing pieces accurately. With it, nobody notices because it works as expected.

So next time you find yourself using a smooth user interface, take a second to consider all the work that went into making it as transparent as possible and thank those poor developers.

Next week we’ll have a post showing some of the specific changes we had to make for the iPhone and comparing screenshots side by side (let’s see if I can get Miguel to write that one).

We’re convinced that the update will be well worth the wait. To tie you over, I leave you with a video from our friends at Bolt Creative.

As programmers, we’re constantly reusing code. Sometimes it’s in the form of low-level OS function calls, or game middleware, and sometimes code that our teammates wrote. At the same time, unless you’re writing top-level game script code, chances are that the code you’re writing will be used by people as well.

I’m purposefully avoiding labeling reusable code as “libraries” or “middleware”. Those are just two of the many forms in which we end up reusing code. Copying some source files onto your project, calling an API function, or instantiating a class written by someone else in your team, are all different forms of code reuse.

Getting Started

Without a doubt, the most difficult job writing reusable code, is making sure it solves a problem correctly and meets everybody’s needs. That’s not a easy task when we’re writing code for ourselves, but it’s much more difficult when we’re targeting other programmers. Too many libraries get it only half right, and they solve some problems at the expense of introducing a bunch of new ones. Or they force the user to jump through all sorts of hoops to get the desired result in the end. We need a clear understand of the exact problem we’re trying to solve with our code.

Libraries often fall in the trap of presenting an implementation-centric interface. That is, their interface is based on the implementation details of the library, rather than how the users are going to use it in their programs.

The best way I’ve found to address both those shortcomings is to start by implementing code that solves the problem for one person. Just one. Forget about multiple users and code reusability for now. If you don’t have immediate and constant access to that one person, then you need to play that role and create a game or application that is as close as possible to what one of your users is going to be developing. By using this approach, I find that the interface to the reused code is much more natural, and it’s based on the experience of having solved the problem at least once. Otherwise, you run the risk of creating an interface that is not a good fit and forcing everything to conform to it in unnatural ways.

Once you have implemented a solution, take a moment to think before you dive into doing any more work. Many times you’ll find there is no reason to abstract it any further since your code is only going to be used in one place. If that’s the case, step away from the code and go do something more productive. You can always come back later whenever there is a real need to reuse it later in the project or in a future game.

There are exceptions to this approach of implementing a solution first, and abstracting it later. Some problems are very simple, or very well understood, so we might be able to jump in and implement the reusable solution directly (an optimized search algorithm, or a compression function for example). It’s also possible that you’ve implemented a similar system several times before, and you know exactly at what kind of level to expose the interface and how things should look like. In that case, it’s perfectly valid to draw on your past experience. Just try to avoid the second-system effect: The tendency to follow up a successful, simple first system, by an overly-complex system with all the ideas that didn’t make it into the first one. Setting Goals

Most successful reusable code was created with specific goals about how it was meant to be used. Sometimes those goals are explicitly stated, most of the time they are implied in the code design.

Some of the most common goals are flexibility, protection, simplicity, robustness, or performance. You can obviously not meet all those goals at once, and even if you could, you probably shouldn’t try. That would be a tremendous waste of time and resources. Stop thinking in the abstract and think about your one user. What does he or she need? What is the most important goal for them?

Many APIs and middleware packages are designed with protection as one of the primary goals: They don’t want the user to accidentally do anything wrong. In itself is not a bad goal, and it can often be implemented by having clean, unambiguous interfaces, clearly-named types and functions, and strongly-typed data types. Unfortunately, a design with protection as a main goal can often result in encumbered interfaces, verbose code, slow performance, and inflexible code. There is nothing more frustrating than wanting to do something that is explicitly being protected against, and having to work around the interface.

If your target users are professional game developers, give them the benefit of the doubt and don’t try to overprotect all your code. Save that for the scripting API exposed to junior designers and released to the customers with the game. If you’re concerned about programmers using your code correctly and not making mistakes, provide good sample code, tests, and documentation. If that’s not enough, and you feel that everybody would benefit from some level of protection, try to keep it to a minimum and maybe even provide lower-level functions that bypass it for power users.

Whatever the primary goal, the libraries and APIs I prefer to work with, help me get whatever I need done, while getting out of the way as much as possible.

Architecture

There are many different ways to architect code that is intended for reuse. The best approach will depend on the particular code: How complex is it? How much of it is there? What are its goals?

Unless your goals are to make quick, throwaway applications, I strongly recommend against a framework type of architecture (see Figure 1.) A framework is a system in which you add a few bits of functionality to customize your program into an existing system. They may sound like a clean and easy way to use complex code, but they’re inevitably very restrictive, and they make it very difficult, if not impossible, to do things with them beyond what they were intended to.

A more flexible approach is a layered architecture (see Figure 2.) Each layer is relatively simple and provides a well-defined set of functionality. Higher-level layers build on top of lower-level layers to create more complex or more specific functionality.

Keep in mind that it is not necessary, or even desirable, to have higher-level layers completely abstract out and hide the lower-level ones. By letting layers be fully transparent, they allow you to mix and match at what level you want to access the code. This can be very important, especially towards the end of a game when fixing some bugs or trying to squeeze some more performance out of the engine.

For example, one layer can expose functionality to create and manipulate pathfinding networks and nodes, another one can implement searches and other queries on those networks, while a third, higher-level layer, can expose functions to reason on the state of the network.

A toolkit architecture (see Figure 3), is the most flexible of all. It provides small, well-defined modules or functions with very few dependencies on other modules. This allows users to pull in whatever modules they need into their game to meet their needs and nothing else. Users can also start by reusing some modules, and but replace them down the line when they want to go beyond the existing functionality. Because of this, toolkit architectures are particularly well suited to game development.

Object Oriented

I tend to avoid complex class hierarchies in most of my code, but that’s especially important in the case of reusable code. Class hierarchies are very rigid, and impose a particular structure on their users.

If you want to remain object-oriented, a better approach is to emphasize composition of objects instead of inheritance. That allows users to much more easily pull in the functionality they need, and create their own objects based on their own constraints.

You can even go a step further and provide purely procedural interfaces. Plain static functions that operate on data types. Interfaces based on static functions are often much easier to understand and grasp than interfaces that involve classes and inheritance. They are also much more convenient for users to wrap and use in many different ways.

Remember what you learned about object-oriented design and having private data? Forget about it, and keep everything accessible. You may think you’re doing the user a favor by making some variables private and reducing the complexity of the interface. That’s partly true, but eventually, your users will want to have access to some of those variables that you took pains to hide. And if they really want to, they will get to them, even if it means direct addressing into an object or vtable.

It’s true that large codebases can be intimidating, and exposing all the internal details along with the regular interface would make it unwieldy for new users. An effective approach is to separate the public interface from what is intended for internal use only, but still make it available through some other means. For example, private data and functions could be wrapped in a different namespace, or simply in a different set of headers. Anything that clearly sets them apart, and doesn’t clutter the initial look at the interface, but that allows experienced developers to get to them and get their hands dirty. Extensibility

As soon as you make your code available to a wide range of developers, you’ll find that people want to use it in progressively more complex and bizarre situations. Your code might have completely solved the case of your first couple of users, and since it’s layered and modular, it can meet a lot of different requirements. But eventually, some people will start taking it to extremes you hadn’t imagined and it falls short for them. What to do?

You could start adding more options and more modules and more callbacks to your code. That way programmers can hook up into almost any part of the code and replace it with their own. The problem with that approach is that you’ve taken something that was relatively simple and made it into an large, ugly, fully-customizable, behemoth that tries to keep everybody happy. That sounds like a lot of common APIs we know and hate. Most successful products try to completely meet the need of some people rather than meet everybody’s needs part way. Otherwise you inconvenience 95% of your users for the benefit of 5% of them.

A better approach is to let those 5% users fend for themselves, but give them the means to do it. How so? With source code. Without source code, developers feel caged and constrained. They know they can’t look behind the interfaces, let alone modify anything in case something goes wrong (and we all know something will go wrong). The more code there is, and the more a project relies on it, the more important it is to have access to the full source code. Many teams will refuse to use some libraries or middleware unless the full source code is available. As soon as you make source code available to your users, you immediately put them more at ease because they feel more in control, and you allow those with special requirements to make whatever modifications they need to do.

Even those developers without a need to modify the code, they will be able to browse the code and see how certain functions are implemented. Not only will it make people much more likely to use your code, but they will probably also fix your bugs and suggest performance improvements, so it’s a win-win situation for everybody.

For extra bonus points, the source code should be accompanied by a set of tests. The more comprehensive the better (unit tests, functional tests, etc). Hopefully you created all those tests while you were developing the code, so distributing it along with the source code shouldn’t be any extra effort, but it will make a huge difference to your users. It will give them much more confidence modifying the code to suit their needs and still see that all the tests are passing.

Upgrades

As soon as you release some code and you have your first user, the question comes up of how to deal with new versions. There are many ways you can go about it, depending on how often you’ll release new versions, and how important it is to maintain backwards compatibility.

On one extreme, you can change the code and the interface to fit new features, changes in architecture, or any other reason. Whenever you release a new version, users will have to choose to remain with their current version or upgrade to the latest and make whatever changes are necessary. This is a common approach in open source projects and internal company code.

On the other extreme, once you release a version, you stick to that interface whether it’s a good idea or not. This can be good for users because they can get new versions without any extra work on their part, but it can be very constraining. It can make new features impossible to add, and it can prevent performance optimizations. This approach is more common on code that will become the foundation of many programs, like OS libraries and low-level APIs.

A good compromise is to keep interfaces the same during minor versions, and only change them whenever a major version is released. That way other developers only have to put in the time to upgrade to a major release if they really need the new features at that time.

Another approach is not to change existing functions or classes, but to introduce new ones and slowly deprecate the old ones over time. That way code continues to work, but users can take advantage of the new functions. After a few versions, you can completely drop off deprecated functionality, at which point most people will have upgraded already. If you do this, make sure to label deprecated functions with a #pragma warning or some other way. That way, developers know it will be phased out and they can start thinking about upgrading to the new interface.

The easier you make the transition to the new version, the better for your users, and the more likely they will be continue using your code. For example, you can provide scripts that parse their source code and upgrade it to match the new interfaces. That can be a bit risky, but it can be really worthwhile if you have a lot of required changes that are relatively mechanical (renamed functions, changed parameter order, etc).

Is there any point to all this talk of interfaces if you made your source code available? Yes, very much so. Most developers will look at the source code to know how things work under the hood, but they probably won’t modify it. Even if they do, they know that’s something they do at their own risk, so they’ll be more than willing to make a few changes whenever a new version is released. Everybody else will still definitely benefit from a relatively stable interface.

Conclusion

Writing reusable code starts with solving a problem and solving it well. The rest should all fall from there, and you can pick whichever method is more appropriate for your particular code and how you want to share it.

This article was originally printed in the March 2009 issue of Game Developer.

Over the years I’ve mellowed out quite a bit. Now I recognize my obsession with writing endless tools and technology was more of an escape from doing the really hard part of game development. In comparison to making all the decisions involved in the design and implementation of the game itself, writing the perfect resource streaming system sounds like a really good time. It’s amazing how early it started too: I still remember spending weeks writing a Basic-to-assembly translator in 1986, before I had even heard the word “compiler”.

Now I just want to make games.

I’ll gladly use code other people have written as long as they fit my needs. For example, I’m very happy with Box2d in Casey’s Contraptions. Was Box2d perfect? Far from it, but it was good enough and I’m very happy with the tradeoff.

So how come that Casey’s Contraptions, apart from Box2d, is all custom code, written in C and OpenGL? Why didn’t I use some of the great tools and libraries available like Cocos2d or Unity?

In the land of AAA console games, developers routinely pull off the performance card: Only our code will allow us to make something that is so tailored to the hardware that it will make the game stand out above all other games. But for Casey’s Contraptions that’s not much of an issue: Any of the middleware options would be able to handle that amount of performance on an iPad just fine.

It wasn’t a lack of features either. We’re not doing anything particularly fancy, and I’m sure it would be possible to implement it in almost any other framework.

So if it’s not performance of features, why not?

Creating a game is a craft. It’s a mix of art and technique, and the craftsman or artisan has a vision and slowly probes, guides, and implements the final product. As with most crafts, the tools you use are extremely important and will directly affect the final product. If I’m going to change my tools and my techniques, the payoff I get needs to be huge to make up for the change. It’s as if I give up a set of hand woodworking tools for a power tool. Maybe it will be faster when I’m cutting in a straight line, but it will be very different and will completely change what I create.

As a game developer, my main tool is the language I use (C with a dash of C++). Things like editors or IDEs are secondary, and I can easily adapt to different ones without much of a problem (like when going from Visual Studio to Xcode). The way I like to work involves fast iterations, unit tests, very explicit memory layout, and processing data at a global, rather than local (entity) level.

I feel that by going to Unity or Cocos2d, I would have to completely give that up. A lot of middleware are frameworks instead of toolkits. That means I’m restricted to implementing bits of code that are embedded in a larger system and called by the rest of the framework. I give up the ability to architect the game in the way I want. Maybe it wouldn’t be a big deal if I liked a component or inheritance-based, entity-centered approach like a lot of frameworks, but I don’t.

Will I get enough benefits from using one of those frameworks to make up for the loss of productivity and techniques I like to use? Doubtful. All of a sudden I can’t just use fwrite on a memory block for serialization, I have to parse data files to load game state, undo/redo isn’t as simple as copying a block of memory, rendering becomes much more generic and less tied to the game, I can’t easily do test-driven development, iterate quickly, etc.

Sometimes, shifting workflows and techniques is inevitable. You may be used to solving collisions in a particular way, but when you move to a full physics engine, you need to process things differently. The payoff you get from using a pre-made physics engine is huge, so it’s worth it in my book.

For 2D games, there isn’t that much that I need: cross-platform support, simple math functions, ways to load textures, play sounds, and render simple things. Throw in some kind of simple UI display and layout tool, and that’s it. The rest I can handle. Actually, that’s not true, let me rephrase that: The rest I prefer to do myself, because it’s always highly dependent on the game.

What I really want is a toolkit-based middleware that does exactly those things without imposing any workflow or architecture on me. It saves me from doing the boring bits, gets out of the way as much as possible, and lets me work the way I want to. That’s already what I have with my current code, minus the UI library/tool part, which I’m still stuck using mostly UIKit. The idea of using Unity just so I can have cross-platform UI seems total overkill.

Even in the case of middleware that fits my style, I still need to be weary of the quality of the code. The last thing I want to do is trade time implementing my own code for time debugging someone else’s crappy code. Also, having learned my lesson years ago, I’ve sworn that I will never use a middleware/library that I don’t have source code access to. Nothing more frustrating than having to reverse-engineer UIKit classes to work around their bugs just because I don’t have the source code.

I’m not even going to go in the debate about middleware lock-in, future portability, building tech vs. relying on it, and financial aspects. All those are important, but secondary to how using some middleware will affect my everyday development.

If my next game were a networked, 3D game involving moving some kind of character or vehicle around a world, and I had nothing to start from, then the pain of adopting a foreign workflow and architecture might be worth it. Might. Even though it might be a wash on the code side, the tools and pipeline might be the thing that sways me over to adopt it.

In the end, as an indie developer, I chose to make games because it’s something I enjoy doing. I get up every morning excited about it, and I enjoy the development process. Whichever platform, tool, middleware or process I end up choosing, it needs to be enjoyable. The day I’m not having fun making my own games, I shouldn’t be an indie anymore.

What do you think? Am I being overly paranoid? Is there a good solution out there that’s multiplatform and will double my productivity once I get used to it? Has anyone written their own code for a 2D game and later switched to Cocos2d/Unity and liked it?

Casey’s Contraptions went on sale on Thursday, and in a few hours it entered the top 100 charts all around the world. We were super-fortunate the Apple picked Casey’s Contraptions as their iPad Game of the Week worldwide and that gave it enough momentum to reach #2 on the top apps chart!

The response so far has been overwhelmingly positive. Not just the press, but the comments on forums, Twitter, Facebook, and even on the App Store. I keep reading stories of parents playing the game along with their 3 and 4 year olds (as well as sneaking out later and playing it by themselves). I’m really glad it resonates with people and so many people are able to enjoy it!

Here are just a few of the main reviews Casey’s Contraptions has received in the few days since launch:

• TUAW I thought it was the most impressive iOS game I’d ever seen.
• 148Apps Editor’s Choice. I don’t think I’ve played a better iOS puzzle game this year.
• Pocket Gamer Gold Award […] one of the most polished iOS games you can currently lay your hands on
• TouchArcade. I can tell you this – Casey’s Contraptions is a joy to play.
• AppleNApps 5/5 Casey’s Contraptions is without a doubt one of the best games ever made for the iPad.
• TheAPPera 10/10 Casey’s Contraptions is the best puzzle game on the iPad right now.

It has been an amazing release. I’ll write a post soon on what we did for marketing to create this launch. The only thing I could have wished for was that the story had been picked up by one of the super popular web sites like Gizmodo or Kotaku. It’s not every day that two indies can get together and storm the App Store charts!

We have lots of updates planned, so the story doesn’t end here. This is just the beginning! We’ll keep posting on our progress and some initial sales info as soon as the dust settles down.

For the rest of you still twiddling your thumbs, eagerly awaiting for midnight, here’s some insight on what went on in the level creation for Casey’s Contraptions.

Built-in editor

From the beginning of the project, the idea was to have a level editor built in to the game. The level editor was the very first thing I implemented in the game. Before there were menus, or levels, or anything else, the game was a level editor without any goals.

That’s what Miguel and I used to create all the levels we shipped with. Nothing like eating your own dog food to make something solid and usable. This is the final level editor that players can use to create their own contraptions from scratch and send them to friends.

The only difference is that user-created levels only have the goal of getting the 3 stars in the level, whereas game levels have a separate goal. This was mostly because of the UI work required to allow the user to set different goals affecting multiple objects. It would have been way too complicated, although we’re not ruling out the possibility of extending it in the future. Miguel and I had to edit level files by hand to add the extra goal information.

Levels. Evolved.

One thing that worked really well in Casey’s Contraptions is that we had a working level editor from day one. The first prototype was a tiny level editor! Of course, we’ve been refining it since then, but the core was there.

After a month or two of development, we implement some goals and the ability to play through the levels created. That means we’ve been able to make, play, and test levels for 6 months before shipping.

It was having that amount of time to create levels that allowed the level creation to mature and let us discover what went into a fun level, to refine the difficulty, and create much more interesting levels in the end. Some of that was influenced by what mechanics were fun and which ones weren’t (placing something in a pixel-perfect position).

The different game items also had a huge influence over the level design. Clearly their functionality is going to affect level design hugely, but the surprise was that the character of the items also influenced level design quite a bit.

For example, early on, most of the levels were about solving everyday tasks Casey had to deal with: put toys away, knock down a ball from the roof, pop a balloon, etc. But as soon as we started adding some of the more colorful items, like the doll, the piggy bank, or the RC truck, our levels shifted into being mini-stories: The doll is jumping from a building and you need to catch her, the piggie is being taken away on the truck, etc. That’s when we realized that we could make really fun levels based on playtime stories, not just real situations. Probably about a third of the shipping levels are playtime levels.

Creation process

Initially, we just created levels without thinking too much about it. Used whatever items we wanted and created something that seemed fun. We were learning a lot by doing that: Playing with item interactions, seeing what was possible and what wasn’t, which items we were gravitating towards and which ones were no fun to play with. We weren’t doing it consciously, but what we really doing was exploring the possibilities of the “level space”.

As you can imagine, most of what we created early on went out of the window pretty quickly. Most of the levels were insanely difficult, and a lot of them were simply no fun at all. We were also creating the levels to challenge each other to solve them, and while that was really fun, a lot of those levels were devilishly difficult. We’ve been dialing back the difficulty level ever since then.

The actual process for creating a level wasn’t very involved. Either one of us would go ahead and create a first pass at a level. Sometimes I would sit down and consciously decide to create a new level (especially if it was a level designed to teach about a new item), but more often than not, an idea would come up while doing something unrelated in the game.

Once we had this first pass, we would send it over to the other person and have them either poke holes on the design (if the level can be solved trivially by just dropping a ball somewhere for example), or tweak it to tighten it and make it more fun. Later on, we would revisit levels based on tester feedback or us becoming more experienced.

I estimate that the average time to create a level, from the first item added to the time it was added to the game, was about half an hour. Some of them were much faster, and some much slower though. And for yet some others, we struggled with them for a whole day and finally dropped the idea completely.

What makes a good level

As we quickly learned, making a cool-looking contraption and removing a few pieces does not a good level make.

The best levels always have multiple solutions. Otherwise, it becomes a game of “guess what the designer had in mind”. There are plenty of games like that out there (and I hate them all when I feel that it turns into that). So even if we started with a complete contraption, we would always make sure there were at least two different ways of accomplishing the goals.

The other thing to avoid in a level is the possibility of trivial solutions. If a level can be accomplished by placing a single item that drops and causes the goal to complete, that’s not very fun. It was a tough balance between leaving enough freedom to create your own solutions, but making it so there were no really “cheap” and boring ones.

The stars were tricky. Each level has thee stars you can get, but they’re completely optional. Initially I wanted our levels to be easy to solve, but each star was progressively more difficult to get. Getting the third star required some serious thinking. My reasoning was that people would solve the levels first, get comfortable with the game, and then come back and get three stars in everything. Boy, was I wrong! It was clear right away that most people wanted (no, expected!) to get all three stars in their first pass through the game. So we changed most of the levels so getting the stars is not hugely difficult, especially in the early levels.

Letting the testers loose on the game was an extremely valuable experience. Not only did they catch a fair share of bugs, but they also had a fresh perspective on the game. It was amazing seeing them solve levels in totally different ways than we had anticipated. It was extremely rewarding to see people come up with solutions and even interactions I had never considered even though I had written all the code.

Here’s a good example (Spoiler alert. Skip ahead to the next section if you don’t want to learn multiple solution to one level).

Here’s one level I designed called “Angry Doll” (any references to popular iOS games must be purely coincidence, by the way :-)

This was a level intended to highlight the use of the slingshot, but you already had the doll and one slingshot in place (and the doll misses the pig with its initial flying kick). One possible solution I had in mind was to use the remaining balls and slingshots to alter the course of the doll and knock the pig down.

There are a few different ways you can do that, which made it an interesting level. However, I was totally unprepared for some of the solutions the testers came up with.

This solution uses multiple slingshots on the doll, which changes its course and adds a lot more force to it. It flies straight for the pig and smashes it on impact. Not just that, but for extra style points, the tennis balls go flying out on a totally chaotic pattern, and they get all the stars!!

This other solution might be my favorite. It completely bypasses even the intermediate goal (alter the doll’s trajectory) and instead attaches a slingshot directly on the piggie bank and smashes it against the wall (getting a star along the way). Genius!

Item sequence

There was one additional constraint to making levels that we didn’t start dealing with until fairly late in the project: Item sequence. Initially, only a few, simple items are available to solve the goals in each level. As you play your way through the game, we introduce new items slowly, making sure their properties are well understood before introducing a new one.

The first time you complete a level with a new item, you’re “awarded” that item, and you can start using it in your own contraptions in the level editor. You can even see the stickers of the items you’ve earned so far on the cover of the “My Contraptions” book (I was playing Psychonauts at the time, so I suspect that might have influenced that design decision a bit).

Having a set item sequence meant we had to be very careful which items were available in which level. We kept a spreadsheet with all the levels, and which items were introduced when. Every new item we introduce follows the following steps:

• When an item first appears, it’s already placed in the level. That means the player gets to see how that item behaves.
• The next level, we give that item to the player so he or she can place it in the level to solve the contraption.
• Another level or two making use of that item. That reinforces the behavior of the item and makes the player comfortable with it before moving on.

We were shooting for fewer levels, but it’s not hard to see why we ended up with 72 levels in the first version of the game.

To be really sure we were respecting the right item sequence, we even wrote a script that parsed all the level files in order and spit out where each item was used. It even made sure that every item followed the rules above (first placed in the world, then available to place by the player). This will continue being extremely useful as we add more items and levels in future updates.

Make your own

There you go. That should give you an idea of what was involved in creating the levels for Casey’s Contraptions. In the near future we’re planning on holding contraption-creation contests and we hope to highlight some player-created contraptions. Go ahead and buy the game and start practicing to make your own contraptions. Remember: The crazier the better!

For those of you who have been living under a rock and haven’t seen the Casey’s Contraptions video, go watch it now. I’ll wait. Or even better, here it is. You don’t even have to leave this page:

The core interaction loop of Casey’s Contraptions gameplay is placing some items, pressing the Play button, seeing the simulation, and repeating based on what you learned from seeing the simulation. We designed this loop to be very tight and without any penalty: Running/stopping the simulation is instant, there’s no limit on the number of times you run it, and you can even stop the simulation by tapping anywhere on the screen. Even if you painted yourself in a wall by creating a solution that needs a very specific placement of items on the screen, making small changes is very quick and painless.

There’s a very important, underlaying assumption in that loop: Running the same simulation multiple times will result in the same behavior. Imagine how frustrating it would be to create a complex chain reaction, just to find out it only works every other time you run it. That would truly deserve a ONE STAR WANT MY MONEY BACK!!

Determinism

That’s referred to as the game being deterministic: Given the same inputs, it produces the same outputs. It’s not the first time I’ve had to deal with that. Actually, it seems that I’ve had to deal with that in one form or another for all my games in the last 10 years (except for Flower Garden). Most recently, it was a key component of gameplay for the unreleased game we were working on at Power of Two Games.

To make a game deterministic, you need to remove the obvious sources of “accidentally different” inputs. That’s usually random number generators, and making sure the initial state is truly the same. It’s too easy to have some leftover state from the previous simulation that throws things off a little bit.

Another potential source of problems are uninitialized variables. If at some point the simulation depends on an uninitialized variable, different runs will cause different results based on whatever value happened to be in memory at that time.

Once you fix all of those little things, that should be it, right? Same input should create the same output. Not really. We’re missing the most crucial input: The timer.

Casey’s Contraptions runs at 60 fps on an iPad one, but the timestep isn’t set to a fixed 16.667ms. Instead, I use a high-resolution timer to measure how much time has really elapsed since last frame, and I advance the simulation by that much.

The problem is that the timer doesn’t always return the exact same amount. It’s not super-precise, and it can be slightly affected by other things going on in the device. It won’t be off by more than 1/10th of a ms, but that’s enough to cause different results and start diverging the simulation.

Fixed timestep

One way to get around this problem is to hardcode the timestep to always be 16.667ms, independently of what it really was. That would fix the determinism problem, but it would add its own drawbacks. If the simulation can’t keep up with 60 fps, the game will appear to slow down, which is not the effect we want. Casey’s Contraptions includes a level editor to make your own free-form contraptions, and even though we have a cap in the maximum number of items you can add, it’s probably possible to add enough of them to start slowing down an iPad 1.

Recording timesteps

A tempting solution (and one I’m ashamed I even briefly tried), is to record the exact timestep during the first run of the simulation. Then, in subsequent runs when nothing has changed, we use the initially recorded stream of timesteps instead of the real ones from the clock.

Apart from being a clumsy-looking solution (I hate having two modes for doing the same thing), the solution only works in the case of replaying the exact same input. This approach falls apart if you make a change to an item that doesn’t affect the simulation until after several seconds have passed. At that point, you’re running a new simulation with new timesteps and everything can diverge before the affected object is changed. Bad, idea. Bad.

Fixed simulation timesteps

This is the correct solution and it’s what we’re doing in Casey’s Contraptions.

You probably already know that physics simulations are a lot more stable if you take small steps, always of the same size. So you really never want to advance your physics simulation by your frame time. Instead, you want to have a small and fixed simulation step (we’re running at 120Hz, but 200 or even 300 are not unheard of). Then, you run the physics simulation as many times as you can fit into your larger timestep. It’s also important to keep around the amount of leftover time you haven’t simulated, to apply it to the next frame.

The simulation loop code looks something like this:

	const float timestep = 1.0f/120.0f;
	accumulator += dt;
	while (accumulator >= timestep)
	{
		// Do physics simulation by timestep
		accumulator -= timestep;
	}

Once we have this loop in place, our real frame time doesn’t matter anymore. The physics simulation will either run for a full step (or multiple steps), or it won’t. There’s no half-way states. So it doesn’t matter if in the first run of the simulation the loop gets executed 3 times in the first frame and 2 times in the second, and in the next simulation is 2 times in the first and 3 in the second. The state of the world will be the same.

Glenn Fiedler writes about this kind of simulation loop in much more detail in this excellent article.

Jitterbug

There’s still a slight problem with the above loop. Nothing horrible; the simulation is deterministic at this point (assuming everything else is taken care of correctly). But you might notice some annoying jittering as things move around.

That’s because how we’re sampling the simulation. We might render a frame after 3 steps of the physics simulation, but another one after 2. So movement isn’t going to be very smooth.

To get around it, we need to interpolate the state of the world to match the time at which the frame is rendered. To do that, we need to simulate one timestep in the future, and then interpolate between the previous one and the future one by the percentage amount left in the accumulator.

Something like this:

	const float timestep = 1.0f/120.0f;
	accumulator += dt;
	while (accumulator >= timestep)
	{
		GameState lastState = gameState;
		// Do physics simulation by timestep
		accumulator -= timestep;
	}

	const float t = accumulator/timestep;
	GamePhysicsUtils::LerpState(interpolatedState, prevState, gameState, t);

The interpolation is just the positions and rotations of every item in the world. That’s only used for rendering, so no need to worry about velocities or forces (unless those are affecting rendering in a very obvious way too).

This is when it comes really handy to have the game state as a contiguous, relocatable block of memory. You can really easily copy states around without having to worry about allocating any memory or fixing any pointers. Yes, I really like my simple, pre-allocated data.

One added benefit of this approach: If you do it right, you can have extreme slow-motion in your game and everything will be right. Even if you only run one physics simulation loop every 10 rendering frames, the interpolation will take care of making sure everything is super smooth.

Odds and ends

A couple gotchas to watch out for (that’s code speak for “I didn’t think of this and it bit me hard”).

If you have any game logic that affects the physical state of your game objects, it needs to be executed in the inner, simulation loop. Anything that deals with visual or audio stuff can go in the regular update loop and only be executed once per frame.

A good example of this was the balloon code. Every frame, I iterate through all existing balloons, and apply some upward force based on buoyancy and other factors. Initially that was done once per frame, after the physics simulation, but that’s obviously wrong because it will leave balloons (and everything else) in a different state depending on your frame delta time. I moved it to the inner simulation loop and everything became deterministic again.

This second gotcha sounds silly, but it was a big deal: Make sure you deal with the situation where you have different numbers of items in the previous game state than in the next game state. For example, if a balloon is destroyed, between the two states, the item count will be different. In my case I don’t free any memory, but I was initially making the assumption that I could interpolate based on the index of the items on both states. But since I removed the balloon and pushed all the items behind it forward, the interpolation was totally wrong.

I solved that problem by marking items as invalid, and removing them only after the inner simulation loop was complete. That allowed me to still interpolate in a very straightforward way and not have to worry about missing items.

Taking it further

Not only are we relying on this determinism while you play a level of Casey’s Contraptions, but also to let you replay solutions from your friends. We simply load their initial layout and run the simulation locally.

[Pause]

Ah, yes, astute reader, you’re right to be puzzled about that. How do we make the simulation deterministic across platforms and versions? Good question. Stay tuned.

TL;DR

Great book for someone starting out on iOS development. You would be at a severe disadvantage if you don’t know about most practices described in the book. Single resource for lots of good practices you’d have to pick up from blogs or Twitter otherwise.

In More Detail…

Don’t be fooled by the title. The Business of iPhone and iPad App Development by Dave Wooldridge and Michael Schneider isn’t one of your boss’ stuffy business book. This is a practical, hands on, guide to making a successful iOS app. It assumes you already have an idea and know how to develop it, but it guides you through the steps of focusing the app, designing it so it can be profitable, and releasing it with the best possible chance of becoming a good seller.

You’ve probably heard a dozen stories of developers who create a great app, submit it to the App Store, and then wonder why they only sold a dozen copies. This is the book they need to go along with that great app.

The book roughly follows the development timeline of an app, from the initial concept, design, implementation, testing, and release. At each of the stages, it covers any aspects that can have a significant impact in the sales success of the app. Even though you can read the book cover to cover, the chapters are very well defined, so it’s easy to jump directly to the part that interests you the most.

With Casey’s Contraptions almost ready to submit to Apple, I read with particular interest the chapters on creating prerelease buzz and increasing awareness of the app. Lots of good advice there.

Don’t expect anything groundbreaking though. If you’ve been following this blog for a while, or you’ve been hanging out on Twitter with all the iOS developers, you will know most of what the book has to offer. It might still be worth it for a few pearls of wisdom here and there to fill some blind spot. However, the book should be required reading for any new iOS developer. Easily the best $20 you can spend as far as impact in the final app sales.

I usually have no patience for technical books with filler chapters and sections. This book is very good about getting to the point, although it has a few sections that feel a bit out of place in that they’re quite basic and technical (like details of generating provisioning profiles, did we really need that in this book?). All in all, that’s a pretty minor point and easy to get around them.

Delayed Release

Probably my main criticism is that the book doesn’t mention one release technique that I consider to be a requirement for any major launch these days: The delayed launch. As the authors mention several times in the last few chapters, once you submit your app, you have no control over when Apple approves it, so you have to play some guessing games.

Instead, you can delay the release of the app once it has been approved, and set it to a known, fixed date in the future (say, a week from approval). At that point, you can really kick in your PR in high gear, contact media outlets, and, most importantly, send them promo codes for your app, even though it’s not available for sale yet.

The goal is to have all the PR hit on launch day or shortly after. The more you can make that happen, the more successful any PR efforts will be, and the bigger the initial launch (and hopefully the following sales) will be.

All in all, The Business of iPhone and iPad App Development is an easy recommendation for the new iOS developer. Go read it right now before you even think of shipping another app.

What can we tell by looking at those games? I see two clear categories: Games with a strong, established IP (Street Fighter, Sonic), or independent games with a huge amount of polish and style.

It’s All About Polish

The love and care developers put into those games shows the moment you start them up. Look at the textures in Tiny Wings, the sound effects in Angry Birds, the feedback animations in Words With Friends, or all the little details in Cut The Rope. All the tiny particle effects, transitions, sounds, and general squish and responsiveness. Every single one of those games is oozing with its own style and contributes to a very enjoyable first (and repeat) experience.

And that is the main point of this post: To make a successful game on the App Store, the main thing you need is love. You can skimp on features, on content, on marketing, on a web site, or even on gameplay balance. All those are things that you can add or improve after shipping, but polish and style are responsible for that crucial first impression. Miss the chance to hit the player with all you’ve got the first time they play your game, and that might be a lost sale (and a lost advocate of your game since word of mouth is such a strong force on the App Store).

Not convinced about the difference polish and style makes?

This game is essentially Tiny Wings with just a little bit of polish (and it does have *some* polish).

And this is Tiny Wings.

I rest my case.

Ship As Soon As Possible?

Traditional game development (especially for consoles) usually goes along these lines: Plan everything, create a game with everything that you want/can fit, ship it once it’s done, and hope not to touch it again. On the other hand, in this day of web/Facebook/mobile development the favored approach is to release a product as soon as possible, and then iterate from there.

My preference in the last few years has been more along the shipping as soon as possible lines (even if I haven’t always been successful at it). But then I paused and really thought about why and what I would accomplish by shipping early. These are the main reasons I could think about:

• Getting to market first. This is a big one in the web world (and maybe even in the hardware world). Even if your product is imperfect, or its UI is less than ideal, getting that initial critical mass of users could be what tips the balance in your favor.
• Canceling the project early. Maybe it takes as long as the first version of a product to realize there isn’t demand for it. So it was better to have spent 6 months instead of 3 years before canceling the project.
• Focussing your efforts. An impending ship date will make wonders to keep people on track and focused on what’s important to ship a game.
• Become profitable as soon as possible. Even if you make the same amount of money and spend the same amount of time working on a project, if you start bringing in money at the 6 month mark rather than at the 3 year mark, you’ll be profitable earlier. And as any RTS fan will tell you, getting extra resources early in the game can put you at a huge advantage.
• Changing the product based on early user feedback. Otherwise you might spend years working on a product that people don’t really want, or they would prefer something slightly different.

How do those reasons apply to iOS games?

• As an iOS game, your biggest moment is launch. That’s when you can get most momentum and get the word of mouth ball rolling. First impressions matter a lot and a lot of people will make snap decisions about your game in the first few seconds. If it’s not looking its best, it doesn’t matter if it came earlier than another game. Besides, games, for the most part, aren’t providing as much of a service as they’re a form of entertainment. Barring brand new genres or whole new platforms, getting to market first doesn’t mean much. And even if you’re making a brand new genre, chances are it’s unique so the clones won’t start showing up until after you launch and becomes popular (unless you announce way in advance).
• Stopping development on an unsuccessful game earlier rather than later is always a good thing.
• Likewise, having a milestone around the corner does wonders for focussing your efforts. That was one of the big benefits we got from submitting Casey’s Contraptions to the IGF.
• The last one is tricky. It might seem like a benefit for iOS games as well, but I’m going to argue it isn’t. Don’t get me wrong, I think that testing and user feedback is very valuable. But games ultimately are a form of art[1] and you are the creator. In the end, you need to decide what your game and your vision are like. Feedback will help with usability and balancing issues, but not with what the game is fundamentally. Stick to your vision.

My Approach

Looking at those lists, it makes it clear to me that iOS game development is not all about getting a product out of the door as soon as possible. There’s no need to create a finished product for your first release. Instead, save every feature and content you can for free updates or even future in-app purchases.

I’m convinced that polish and style are one of the most important things an iOS game can have. It’s not the first time I say that. So get the product out as soon as you can, but do not, under any circumstances, cut any polish from your game. Plan on spending a good month or longer after your game is “done” polishing it. That time will definitely be well spent and will increase the value of your game more than any other month you spent developing it.

What I’m suggesting here is actually quite different from what Chris Hecker talked about at last year’s GDC. Even though we’re both saying “take your time and do your game right”, he’s emphasizing the completeness of the game (in the sense of exploring all its potential), while I’m emphasizing the presentation. I think the main reason our messages are so different is the platform we’re developing for. On a platform like iOS, I really think you can explore the full potential of a game after it ships without any real drawbacks.

Right now we’re in the polish phase in Casey’s Contraptions. The game has been “done” for a while, in the sense that we have all the items, lots of levels, you can play through all the puzzles, make your own contraptions, etc. Even though it already has a lot of style and polish, it definitely needs that extra layer of shine to make it really stand out and bring it to the quality of those games in the top 10 list. We can only hope that Casey’s Contraptions joins them after we launch!

Interested in keeping up with Casey’s Contraptions as we’re gearing up for launch? Join the Facebook page.

[1] Feel free to substitute whatever word you want there that isn’t offensive to you: entertainment, interactive media, etc. Games are something *you* create from your imagination, so art seems like the best term to me.

2010 was the year I was hoping to ship Casey’s Contraptions, but that didn’t happen. However, it’s great to see Pocket Gamer took notice of the ongoing work on Flower Garden. Every update feels like a small product release on its own, and there were many updates in 2010!

Flower Garden has come a long way since its initial release on April 2009 (almost two years ago!). There is so much more to the game now than the initial release! These are just some of the most important new features since then:

• Multiple seed packs
• Multiple garden locations
• Day/night cycles
• Fertilizer
• Green Thumb Point awards
• Game Center achievements and leaderboards
• Sending bouquets through Facebook (and SMS in this next update)

Flower Garden also recently reached the milestone of 5 million bouquets sent. That’s a lot of flowers! Thanks to all the Flower Garden fans for making this possible!

One of the most important steps on the way to becoming a profitable independent iOS developer is diversifying your revenue stream.  While business lingo like that makes me throw up a little, all it really means is discovering all the ways you can earn money using the platform.  New developers sometimes pigeonhole themselves into a single App Store strategy: “Sell as many copies at 99 cents as possible”.  More savvy developers mix  multiple strategies: “Paid apps”, “In-App Purchases”, and “Advertising”.  I want to make sure all developers know about another additional option often overlooked: LinkShare.

LinkShare is a company which pairs publishers with retailers who pay said developers for driving clicks to their sites that result in sales.  How does this apply to iOS developers?  Luckily Apple (specifically iTunes) is one of the retailers which uses LinkShare.  A good FAQ page for the iTunes affiliate program can be found here, but the basic gist of it that you earn a 5% commission on items bought on the App Store from your affiliate links.  As an iOS developer you are probably already using links to your apps (and perhaps others) in lots of places, including “More Games” screens, twitter, newsletters, banner ads, or your web site.  Replacing all these existing (and future) links to the App Store with your affiliate links is a great start.  Retro Dreamer even wrote a nice quick guide to creating links that work seamlessly in iOS (there are some pros and cons to different link formats).

Now you might think the chances of someone actually buying an app you link to are relatively low, but that’s where things get interesting.  If you read the fine print it turns out affiliates get paid 5% of any purchases made within a 72 hours after following your link.  Lets say Joe clicks on a link to say, Pocket Frogs (our latest free game) which included your affiliate id, which even doesn’t result in a paid purchase even if the app is downloaded.  But perhaps Joe ends up buying Angry Birds ($0.99) an hour later earning you $0.05, or Real Racing 2 ($4.99) that night earning you $0.25, or just maybe the Beatles Box Set ($149.00) the next morning earning you $7.45!  The cool thing about the iTunes affiliate program is that it gives the affiliate 5% of any and all purchases made through iTunes within 72 hours including ring tones, songs, apps, in-app purchases, movies, tv shows, or rentals.

This of course means you can still earn revenue from linking to free apps,  which can end up being a powerful thing.  For example, in Pocket Frogs we run a promotion every week where we offer an in game item for downloading a certain free app (with a LinkShare link of course).  This not only keeps players checking back, but lets us promote apps we like (like Flower Garden) or even our own.  Like most other revenue sources LinkShare isn’t going to make you a whole lot of money if there aren’t that many people clicking your links, but it will certainly grow along with the number of users you have.  While the majority of revenue generated from Pocket Frogs (which fluctuates between 150k and 200k daily active users) comes from the IAP included in the game, it also generates a healthy amount of revenue from LinkShare (in conjunction with some links inside other apps) as seen below.

The great thing about LinkShare is that it gives you a lot of freedom on how you use it. It doesn’t use up any bandwidth or take up CPU cycles, and it doesn’t require you to shoehorn 3rd party code into your app. It is as invisible or invasive as you want it to be. So whether you’re a new iOS developer just starting out or an experienced dev, you owe it to yourself to take a look at using LinkShare if you’re not using it already.

Fast-forward another year, and this morning I woke up to a very pleasant and unexpected surprise: Flower Garden was featured on the App Store under Apps For Valentine’s Day!

Flower Garden is still going strong, but I wasn’t expecting that at all. Thank you, Apple! Not only that, but this feature also appears on the device App Store. Flower Garden was featured twice before by Apple, but never before on a spot that appeared on the device. So that’s a first for Flower Garden!

To make things more interesting, I had been planning on doing a bit of promotion around Valentine’s Day like last year. So a few days ago I released a new update, and included another in-app purchase for the most asked-for feature: More pots in another garden space.

Finally, to round things off, I planned on doing a similar promotion to what I did last year around Mother’s Day, and I set Flower Garden to be free from today until Valentine’s Day to encourage even more people to try it. To get the word out of the price drop, I got some promotion going from Pocket Frogs and a few other apps encouraging users try out the now free Flower Garden. I’m also hoping a few media outlets cover the sale to get the word out as much as possible.

As of this moment, Flower Garden is in the top 100 apps in the US and in the top 50 games, so the combination of everything seems to be working. We’ll see how things develop over this coming week. Until then, it’s going to be an exciting ride.

For the latest news on Flower Garden, join the Facebook page by clicking “Like”:

This Week

This week Miguel and I are wrapping up our current iteration with the main focus of the first five minutes of gameplay. The reason we’re doing this now is that we spent quite a bit of time on the user interaction experience and we tried to nail that before adding more items and more levels. All along, we’ve been testing the game with unsuspecting victims and we quietly watched over their shoulders as they fumbled with the game without any instructions or tutorials.

I really believe we made huge improvements. We designed the whole game with touch interface from the ground up, and it really shows. I think it’s a very direct and intuitive interface, but even so, there’s only so much you can do without any instructions.

So, the focus of this last iteration is to concentrate on what a new player will see in the first five minutes of gameplay. That way, we can continue testing the game on new players and get a much better feel for how they learn to interact with the game and what works and what doesn’t.

Iterations

For every iteration (roughly about 2 weeks each) we focus on one main area, based on what we feel needs to be addressed the most at that point. To give you an idea, these have been the focus of some of the past iterations:

• Proof-of-concept prototype
• New, non-physical items
• Game screen flow and level progression
• Final user interaction
• New items and locations

One thing that we’ve been doing is trying to take each goal to completion. This is an idea from Scrum and Agile development that I really like. If you implement an idea to the 90% state, apart from the fact that the remaining 10% is going to take another 90% of the time, you probably don’t have a good idea how it will really be once it’s completed. That makes future planning and re-prioritizing more difficult.

Obviously there’s such a thing as going overboard. Trying to get every single, tiny effect and animation to fully complete before moving on would be crippling. I try to think about the impact that feature, even if it’s very small, will have in the final game. The exact effect when you pick up a star isn’t crucial (assuming you have some sounds and some effects already), but the animations for the item selection and manipulation contribute a lot more to the feel of the game, so they’re more important.

In some of my past projects I took a different approach and built the game bottom up. At the very end all the little niceties and polish touches went in, which made the game radically different. Now I’ve come to admit that all those little touches contribute a lot to the final feel of the game and should be considered all along and not as an afterthought at the end.

Last year I wrote about the basics of Data-Oriented Design (see the September 2009 issue of Game Developer). In the time since that article, Data-Oriented Design has gained a lot of traction in game development and many of teams are thinking in terms of data for some of the more performance-critical systems.

As a quick recap, the main goal of Data-Oriented Design is achieving high-performance on modern hardware platforms. Specifically, that means making good use of memory accesses, multiple cores, and removing any unnecessary code. A side effect of Data-Oriented Design is that code becomes more modular and easier to test.

Data-Oriented Design concentrates on the input data available and the output data that needs to be generated. Code is not something to focus on (like traditional Computer Science), but is something that is written to transform the input data into the output data in an efficient way. In modern hardware, that often means applying the same code to large, contiguous blocks of homogeneous memory.

Applying Data-Oriented Design

It’s pretty easy to apply these ideas to a self-contained system that already works over mostly-homogeneous data. Most particle systems in games are probably designed that way because one of their main goals is to be very efficient and handle a large number of particles at high framerates. Sound processing is another system that is naturally implemented thinking about data first and foremost.

So, what’s stopping us from applying it to all the performance-sensitive systems in a game code base? Mostly just the way we think about the code. We need to be ready to really look at the data and be willing to split up the code into different phases. Let’s take a high-level example and see how the code would have to be restructured when optimizing for data access.

Listing 1 shows pseudocode for what could be a typical update function for a generic game AI. To make things worse, that function might even be virtual and different types of entities might implement it in different ways. Let’s ignore that for now and concentrate on what it does. In particular, the pseudocode highlights that, as part of the entity update, it does many conditional ray casting queries, and also updates some state based on the results of those queries. In other words, we’re confronted with the typical tree-traversal code structure so common in Object-Oriented Programming.

void AIEntity::Update(float dt)
{
    DoSomeProcessing();
    if (someCondition && Raycast(world))
       DoSomething();
    if (someOtherCondition && BunchOfRayCasts(world))
       DoSomethingElse();
    UpdateSomeOtherStuff();
}

Listing 1

Ray casts against the world are a very common operation for game entities. That’s how they “see” what’s around them and that’s what allows them to react correctly to their surroundings. Unfortunately, ray casting is a very heavy weight operation, and it involves potentially accessing many different areas in memory: a spatial data structure, other entity representations, polygons in a collision mesh, etc.

Additionally, the entity update function would be very hard to paralellize on multiple cores. It’s unclear how much data is read or written in that function, and some of that data (like the world data structure) might be particularly hard and expensive to protect from updates from multiple threads.

If we re-organize things a bit, we can significantly improve performance and paralellization.

Break Up And Batch

Without seeing any of the details of what’s going on inside the entity update, we can see the raycasts sticking out like a sore thumb in the middle. A raycast operation is fairly independent of anything else related to the entity, it’s heavyweight, and there could be many of them, so it’s a perfect candidate to break up into a separate step.

Listing 2 shows how the broken up entity update code would look like. The update is now split in two different passes: The first pass does some of the updating that can be done independently of any ray casts, and decides which, if any, raycasts need to be performed sometime this frame.

void AIEntity::InitialUpdate(float dt, RayCastQueries& queries)
{
    DoSomeProcessing();
    if (someCondition)
        AddRayCastQuery(queries);
    if (someOtherCondition)
        AddBunchOfRayCasts(queries);
}

void AIEntity::FinalUpdate(const RayCastResults& results)
{
    UpdateSomeOtherStuff(results);
}

Listing 2

The game code in charge of updating the game processes all AI entities in batches (Listing 3). So instead of calling InitialUpdate(), solve ray casts, and FinalUpdate() for each entity, it iterates over all the AI entities calling InitialUpdate() and adds all the raycast query requests to the output data. Once it has collected all the raycast queries, it can process them all at once and store their results. Finally, it does one last pass and calls FinalUpdate() with the raycast results on each entity.

RayCastQueries queries;
for (int i=0; i<entityCount; ++i)
    entities[i].InitialUpdate(dt, queries);

// Other update that might need raycasts

RayCastResults results;
for (int i=0; i<queries.count; ++i)
    PerformRayCast(queries[i], results);

for (int i=0; i<entityCount; ++i)
    entities[i].FinalUpdate(results);

Listing 3

By removing the raycast calls from within the entity update function, we’ve shortened the call tree significantly. The functions are more self-contained, easier to understand, and probably much more efficient because of better cache utilization. You can also see how it would be a lot easier to parallelize things now by sending all raycasts to one core while another core is busy updating something unrelated (or maybe by spreading all raycasts across multiple cores, depending on your level of granularity).

Note that after calling InitialUpdate() on all entities, we could do some processing on other game objects that might also need raycast queries and collect them all. That way, we can batch all the raycasts at compute them all at once. For years, we’ve been drilled by graphics hardware manufacturers how we should batch our render calls and avoid drawing individual polygons. This is the same way: By batching all raycasts in a single call, we have the potential to achieve much higher performance.

Splitting Things Up

Have we really gained much by reorganizing the code this way? We’re doing two full passes over the AI entities, so wouldn’t that be worse from a memory point of view? Ultimately you need to measure it and compare the two. In modern hardware platforms, I would expect performance to be better because, even though we’re traversing through the entities twice, we’re using the code cache much better and we’re accessing them sequentially (which allows us to pre-fetch the next one too).

If this is the only change we make to the entity update, and the rest of the code is the usual deep tree traversal code, we might not have gained much because we’re still blowing the cache limits with every update. We might need to apply the same design principles to the rest of the update function to start seeing performance improvements. But at the very least, even with this small change, we have made it easier to parallelize.

One thing we’ve gained now is the ability to modify our data to fit the way we’re using it, and that’s the key to big performance gains. For example, after seeing how the entity is updated in two separate passes, you might notice that only some of the data that was stored in the entity object is touched from the first update, and the second pass accesses more specific data.

At that point we can split up the entity class into two different sets of data. One of the most difficult things at this point is naming these sets data in some meaningful way. They’re not representing real objects or real-world concepts anymore, but different aspects of a concept, broken down purely by how the data is processed. So what before was an AIEntity, can now become a EntityInfo (containing things like position, orientation, and some high-level data) and AIState (with the current goals, orders, paths to follow, enemies targeted, etc).

The overall update function now deals with EntityInfo structures in the first pass, and AIState structures in the second pass, making it much more cache friendly and efficient.

Realistically, both the first and second passes will have to access some common data (for example the entity current state: fleeing, engaged, exploring, idle, etc). If it’s only a small amount of data, the best solution might be to simply duplicate that data on both structures (going against all “common wisdom” in Computer Science). If the common data is larger or is read-write, it might make more sense to give it separate data structure of its own.

At this point, a different kind of complexity is introduced: Keeping track of all the relationships from the different structures. This can be particularly challenging while debugging because some of the data belonging to the same logical entity isn’t stored in the same structure and it’s harder to explore in a debugger. Even so, making good use of indices and handles makes this problem much more manageable (see Managing Data Relationships in the September 2008 issue of Game Developer).

Conditional Execution

So far things are pretty simple because we’re assuming that every AI entity needs both updates and some ray casts. That’s not very realistic because entities are probably very bursty: sometimes they need a lot of ray casts, and sometimes they’re idle or following orders and don’t need any for a while. We can deal with this situation by adding a conditional execution to the second update function.

The easiest way to conditionally execute the update would be to add an extra output parameter to the FirstUpdate() function indicating whether the entity needs a second update or not. The same information could be derived form the calling code depending on whether there were any raycasts queries added. Then, in the second pass, we only update those entities that appear in the list of entities requiring a second update.

The biggest drawback of this approach is that the second update went from traversing memory linearly to skipping over entities, potentially affecting cache performance. So what we thought was going to be a performance optimization ended up making things slower. Unless we’re gaining a significant performance improvement, it’s often better to simply do the work for all entities whether they need it or not. However, if on average less than 10 or 20 percent of the entities need a ray cast, then it might be worth avoiding doing the second update on all the other entities and paying the conditional execution penalty.

If the number of entities to be updated in the second pass is fairly small, another approach would be to copy all necessary data from the first pass into a new temporary buffer. The second pass can then process that data sequentially without any performance penalties and it would completely offset the performance hit of copying the data.

Finally, another alternative, especially if the conditional execution remains fairly similar from frame to frame, is to relocate entities that need raycasting together. That way the copying is minimal (swapping an entity to a new location in the array whenever it needs a raycast), and we still get the benefit of the sequential second update. For this to work all your entities need to be fully relocatable, which means working with handles or some other indirection, or updating all the references to the entities that swapped places.

Different Modes

What if the entity can be in several, totally different modes of execution? Even if it’s the same type of entity, traversing through them linearly calling the update function could end up using completely different code for each of them, so it will result in poor code cache performance.

There are several approached we can take in a situation like that:

• If the different execution modes also are tied to different parts of the entity data, we could treat them as if they were completely different entities and break each of their data components apart. That way, we can iterate through each type separately and get all the performance benefits.
• If the data is mostly the same, and it’s just the code that changes, we could keep all the entities in the same memory block, but rearrange them so that entities in the same mode are next to each other. Again, if you can relocate your data, this is very easy and efficient (it only requires swapping a few entities whenever the state changes).
• Leave it alone! Ultimately, Data-Oriented Design is about thinking about the data and how it affects your program. It doesn’t mean you always have to optimize every aspect of it, especially if the gains aren’t significant enough to warrant the added complexity.

The Future

Is thinking about a program in terms of data and doing these kind of optimizations a good use of our time? Is this all going to go away in the near future as hardware improves? As far as we can tell right now, the answer is a definite no. Efficient memory access with a single CPU is a very complicated problem, and matters get much worse as we add more cores. Also, the amount of transistors in CPUs (which is a rough measure of power) continues to increase much faster than memory access time. That tells us that, barring new technological breakthroughs, we’re going to be dealing with this problem for a long time. This is a problem we need to deal with right now and build our technology around it.

There are some things I’d like to see in the future to make Data-Oriented Design easier. We can all dream up of a new language that will magically allow for great memory access and easy paralellization, but replacing C/C++ and all existing libraries is always going to be a really hard sell. Historically, the best advances in game technology have been incremental, not throwing away existing languages, tools, and libraries (that’s why we’re still stuck with C++ today).

Here are two things that could be done right now and work with our existing codebases. I know a lot of developers are working on similar systems in their projects, but it would be great to have a common implementation released publicly so we can all build on top of them.

Language

Even though a functional language might be ideal, either created from scratch or reusing an existing one, we could temporarily extend C to fit our needs. I would like to see a set of C extensions where functions have clearly defined inputs and outputs, and code inside a function is not allowed to access any global state or call any code outside that function (other than local helper functions defined in the same scope). This could be done as a preprocessor or a modified C compiler, so it remains very compatible with existing libraries and code.

void FirstEntityUpdate(input Entities* entities, input int entityCount, output RayCastQueries* queries, output int queryCount);

Dependencies between functions would be expressed by tying the outputs of some functions to the input of other functions. This could be done in code or through the use of GUI tools that help developers manage data relationships visually. That way we can construct a dependency diagram of all the functions involved in every frame.

Scheduler

Once we have the dependencies for every function, we can create a directed acyclic graph (DAG) from it, which would give us a global view of how data is processed every frame. At that point, instead of running functions manually, we can leave that job in the hands of a scheduler.

The scheduler has full information about all the functions as well as the number of available cores (and information from the previous frame execution if we want to use that as well). It can determine the critical path through the DAG and optimize the scheduling of the tasks so the critical path is always being worked on. If temporary memory buffers are a limitation for our platform, the scheduler can take that into account and trade some performance time for a reduced memory footprint.

Just like the language, the scheduler would be a very generic component, and could be made public. Developers could use it as a starting point, build on top of it, and add their own rules for their specific games and platforms.

Even if we’re not ready to create those reusable components, every developer involved in creating high-performance games should be thinking about data in their games right now. Data is only going to get more important in the future as the next generation of consoles and computers rolls in.

This article was originally printed in the September 2010 issue of Game Developer.

For me this marked my fourth year as an indie developer. It’s not a huge milestone by any standard, but the fact that I’ve been going at it for this long and I’m not broke yet means it’s a sustainable business. Actually, as you know if you followed some of my numbers posts, this was the first year that income from my apps finally became (reasonably) profitable. Now, *that* is something worth celebrating!

Doing this kind of small-scale game development is exactly what I would do if I didn’t have to worry about money at all, so this is truly living the dream for me.

Goals

All is not sunshine and roses in Noel indie-land though. I accomplished some of my goals for this year, but missed others by a mile and a half:

I released a bunch of Flower Garden updates and ran a bunch of promotions that kept bringing revenue up. That went very well. Check.

Miguel and I also submitted Casey’s Contraptions to the IGF (with three hours to spare the day of the deadline). I’m very proud of that one. Even if we’re not selected as a finalist, the experience of submitting it and finally participating in the IGF was fantastic. Check.

Games shipped. Hmm…. That’s the total fail. The only new game I shipped this year was Lorax Garden, a very short project in collaboration with Oceanhouse Media, mixing their Dr. Seuss license with the flower technology I had already developed. I didn’t even manage to ship Casey’s before Christmas like I was hoping to do. I could make excuses but there’s no point. Real life happens and things go slower than planned (that topic is already brewing for a future post).

My goal for next year is to ship three new games, but always of the quality that I can be proud of. I’ll have to be very careful about design and scope to achieve that (I’m hoping that hanging out with Ian and Dave will rub off some of that genius they have for making awesome games on a crazy-short schedule).

Games From Within

I started my blog, Games From Within, exactly 7 years ago today. I guess that counts as my longest-running project. Some years it was sorely neglected with just a few posts, but this year I spent a significant amount of time on it, thanks in large part to #idevblogaday.

These were the most popular posts for this year (based on page views, which RSS readers make much harder to track):

• Making A Living (Barely) On The iPhone App Store (aka The Numbers Post)
• Data-Oriented Design (Or Why You Might Be Shooting Yourself in The Foot With OOP)
• The Const Nazi (This one is a total surprise that it made it this high up!)
• Making A Living (Comfortably) On The App Store (aka The Numbers Post #2)
• Prototyping: You’re (Probably) Doing It Wrong
• Google App Engine As Back End For iPhone Apps
• Increase Your App Ratings On The App Store
• GDC 2010: The Best of Both Worlds: Using UIKit with OpenGL

2011 should be an exciting year for me personally and for all indie game developers out there. Happy new year everybody!

New beginning

When my job position got terminated last summer, I decided I would take a break from working for big companies, and try to make a game on my own at home. I’ve been in the video game industry since the late nineties, I’ve written code for a few released PC and console AAA titles and probably as many canceled ones, and I wanted to do something different at least for a few months. Go back to the roots of game development, when one or two guys could create something in their garage or bedroom, without spending years and millions of dollars on it.

Platform choice

I’m a C# fan, I won’t deny it, I even explained why back in 2007. I also toyed with XNA each time a new version was released, and I like how straightforward it is to use. When the beta version of the tools to develop for the Windows Phone 7 (WP7 for short) became available, just at the right time for me, it seemed like the obvious choice for what I wanted to do. And developing for a brand new platform sounded exciting as well, even if it was perhaps a riskier bet than targeting well established markets. For those who don’t know much about WP7 development, I just want to highlight a few key points:

• Applications are written in Visual Studio using C#, and either XNA or Silverlight. This may come in the future, but at this point you are not allowed to mix them, and use the controls of Silverlight with the faster rendering of XNA, for example.
• In App Purchases are not supported yet.
• Programmable shaders are not available for WP7 (they obviously are for the PC and XBox360) in the current version of XNA. Instead, you have 5 predefined shaders with features that can be toggled on and off (such as fog or per pixel lighting). These shaders should cover most of the usual needs, but if you want something very specific that they don’t support, you’re out of luck (or have to fall back to multiple passes and alpha blending).
• Paid applications have a trial mode, that allows users to evaluate them before making a purchase. Unlike on XBox Live Indie Games (XBLIG), where the game just abruptly stops after 8 minutes, developers have a complete control over what they want to show in trial mode (for example, the first few levels of a game, with no time limit, or without the ability to save, etc). In theory, this means you shouldn’t have to make a free version of your game in addition to the paid one, but I’ll come back to this later.
• The certification requirement document is only 27 pages long, table of content and change history included.

The project

I wrote a few ideas on paper, but they all looked too ambitious for a first project. One thing in particular that I wanted to avoid was to require a lot of content, since I didn’t even know how I was going to find an artist yet. During a bike ride, I thought about using photos, and that’s how I decided to make a jigsaw puzzle game. Not super original, I know, there are several of those on the iPhone and iPad, but there was none on the WP7 since it wasn’t released yet :), and it looked like a good project of the right scope to keep things under control and learn the platform. For example, I had never worked with a touch screen device, and it was pretty obvious from the start that I was going to spend quite some time iterating on the user interface part of the game.  The game is called Jigsaw Guru, and you can go to http://www.fairyengine.com to see more screenshots and a video.

What went right

I actually already mentioned some of the things that went right in my opinion, but here is the full list:

• Working on a short project, without too much content to integrate, instead of picking another more ambitious and time consuming idea, was the right choice. My goal was to have Jigsaw Guru on the marketplace the day of the WP7 launch, I submitted it on 10/18 and it got approved on 10/23 (the update I submitted in December passed even faster than that), two weeks before the US launch and two days after the other territories started getting their phones. At times I still had more items on my to-do list than I would have liked, and I made a good push in the last 3 weeks to get everything wrapped up in time, but it never got too crazy.
• Knowing the tools beforehand, and using C# and XNA, definitely made me feel productive. The phone emulator was surprisingly stable in the beta, and the XNA forums were very helpful when I wasn’t sure about something. It’s also worth noting that the website used to submit bug reports allows everybody to see what other people have written, and vote on issues you think are important and should be fixed first.
• Garbage collection is the one thing that can hurt your framerate really bad when using XNA on XBox360 and WP7. But it’s not that hard to avoid: when I had a look at my memory usage late in the project, I only found two spots where I was creating some amount of garbage each frame (both related to strings). Basically what I do is each time the player goes from one screen to another, I load everything I need for the new screen, and I force a garbage collection. This gets rid of temporary objects that may have been created during the loading process, and objects from the previous screen that are not used anymore. After that, I don’t allocate anything, and the garbage collector never has to do any work.
• Finally: testing. Very few people tested my game while I was developing it, but my friend Alexis alone kept giving me more feedback than I thought I could handle. He never said anything was good just to make me happy, he reported every little detail he didn’t like, even and especially the ones I was aware of but never had time to fix (and maybe thought nobody would notice). Even if I sometimes felt he was nitpicking, this was exactly what I needed.

What went wrong

Developing for a new platform has its share of drawbacks, it’s just how it is. The final version of XNA 4.0 only came out mid-September, and before that I was working with the beta version. As mentioned before, it was pretty solid already, but there was no redistribution of the runtime. Which means you had to install all the tools to be able to just run the game, and you could only install them under Vista or Windows 7 because the phone emulator requires DirectX10, and some of my potential testers were still using Windows XP. So, even if what I was sending was a PC version of the game, they couldn’t run it on XP just because the phone emulator, which they didn’t need, requires a more recent operating system. To work around this problem, I made an XNA 3.1 PC version of Jigsaw Guru, but maintaining two versions side by side for several weeks and merging back and forth between them was not fun.

Coding for a device that wasn’t released yet also meant I could not test what I was doing on real hardware. I unsuccessfully tried to get a phone prototype, and basically the only way I found to run my game on a WP7 at the end of September was to go to a Microsoft event. Even there, I thought there would be phones locked to the tables, that developers could connect to their laptop and use for the whole day, but I was wrong: there was no such thing, and the 4-5 persons who already had a working application to test like me had to do it on the phone of the event’s organizer, when he wasn’t using it for something else. I was able to run my application twice for about 5 minutes each time, and fortunately everything seemed to be working fine, which is a testament of the reliability of the emulator.

When the game began to be really playable, with most of the screens that are in the final version, I started looking for an artist and a musician. I did it way too late, and I was lucky to find Aimee and Tim through the XNA forums, and that the time estimates they gave me just matched my own programming deadline. So, in the end everything worked out well, but even if I wanted to wait and be sure the UI elements were final before asking somebody to redo them, I believe I also waited just because I was more comfortable doing my programming tasks first, and that was a mistake.

Marketing was a difficult problem: I knew I needed to do some, but how? When Jigsaw Guru got approved, the WP7 wasn’t out yet in the US, and had just been released in a few countries of Europe and Asia, as well as Australia and New Zealand. Basically, nobody had that phone yet, and I couldn’t find any website reviewing its games. Even when I had a second look for the US launch, I only saw a couple of sites testing 2 or 3 of the official XBox Live games (that’s how they’re called even if they run on a phone), but nothing related to indie games. So, what ended up happening is I did nothing, and just hoped that because there weren’t too many applications on the marketplace yet (which is the reason for being a launch title, right?), people interested in jigsaw puzzle games would fine mine. Of course I was wrong, and the fact that the search in the Zune software didn’t seem to care about the keywords you enter when submitting your app, or couldn’t find my game when searching for ‘jigsaw’, didn’t help (this seems to be fixed now). I don’t know how other developers releasing WP7 games around the same time as me handled their marketing, but I would be curious to hear it, that question is still a mystery to me.

Downloads and sales

This could have been another paragraph in the “what went wrong” category, but I think this subject deserves its own section. Download and sale numbers for WP7 apps became available on Wednesday 12/08, and I don’t think anybody was thrilled. Don’t get me wrong: WP7 is a very young platform, there aren’t tens of millions of phones in circulation, sale expectations have to be low for now, and can certainly not be compared with those of competing platforms. No, what really surprised me is not the number of sales, but the number of downloads.

On the “top selling” tab of the Zune software (which is more likely based on downloads, like the Bing visual search), Jigsaw Guru has always been in the middle of the paid games from the puzzle & trivia category, the biggest category by far (about 45% of all games). Today for example, it’s 174th out of 363. How many downloads do you think this means on average? About 0.65 per day. This totally blows my mind, but is in line with a graph a developer made with the numbers from his 4 apps, that shows the relationship between the marketplace rank and the number of downloads.

One potential problem is it’s not obvious enough that lots of paid games support the free trial mode, and some players don’t know they can download them for free. But really, it all comes down to most players only checking the XBox Live and the free sections, and rarely visiting the paid one. Which is why although I said earlier that in theory, and thanks to the trial mode, you shouldn’t need to make a free version of your paid game, in practice it’s obvious this doesn’t work, and you have to do it. My friend Noel, who is an experienced iPhone developer, had actually told me I should release my game as a free, ad supported app; unfortunately, the ad SDK’s for XNA didn’t exist at launch and are still in their infancy, the one from Microsoft got released this month and only serves ads in the US, the other ones are either not finished or not stable yet.

So, what’s next? I’m submitting Jigsaw Guru Free (with the Microsoft ad SDK) this week, because as a developer, I still want people to play my game and hopefully have a good time with it. The two projects I started working on after Jigsaw Guru might follow the same path and be free apps from the start, whether it’s on the WP7 or another platform.

This was my third 360iDev Game Jam, and it gets better all the time. It’s great to see that it has become a 360iDev tradition, and that the number of people participating is going up every time. The last couple of times we had one invited guest to participate remotely (and preside over everybody else in the big video screen), but this time we opened it up so anybody, anywhere in the world could join us and participate in the updates and discussions through the web site (big thanks for Mike Berg for all the excellent work on the web site!).

What’s The Point

Some people don’t understand what the point of the game jam is. Other people see the value in it, but disagree with what other people see. The point of a game jam is the same as a jamming music session: To create something while surrounded by other developers and feed off each other’s energy and enthusiasm.

In addition to the jamming aspect of it, different people have different goals, and they’re all just as good and valuable:

• Trying a new game idea
• Learning a new API or technique
• Making a finished product
• Starting something new
• Being totally experimental
• Stretching their comfort zone

There were even people using the game jam as a means to make progress in their own game or app they had already started. It’s a bit far from the original intent, but why not? It’s the jamming part that is the most important.

I was glad to see that most people decided to work the theme (“changing the world”) in the game somehow. I definitely find that having some constraints helps me focus and be more creative at the same time.

One of the most attractive aspects of a game jam for me is that it’s a very focused, but very short effort. Yes, it sounds epic: “A full night of pizza, coffee, and coding…” but it’s only 8-10 hours. That means the cost of “failure” is minimal. It’s about a work day. That’s it. So that means it’s possible to try new, risky, experimental things, and, most importantly, be OK if they don’t work out. You don’t learn by succeeding at everything.

Swapping Roles

The last two game jams, I experimented with different kinds of game designs (heavy use of multi touch and limited visibility). This time around I’m in the middle of a new project (Casey’s Contraptions), and Miguel and I did about 5-6 prototypes earlier this year, so I wasn’t itching to do another experimental gameplay prototype.

So instead, Miguel and I paired up again, but with a twist: He would do all the programming and I would do all the art. How’s that for crazy? Actually, he’s in a lot better shape because he’s a good programmer in addition to being a great artist. Me, on the other hand, I can barely find my way around in Photoshop to copy and paste images from Google Images, so this was definitely going to be way out of my comfort zone.

As you can expect, we didn’t make as much progress as we had hoped. On the other hand, I never had more fun or learned more new things at a game jam before! It helped a lot that I wasn’t just flailing around with Photoshop, but that Miguel was there giving me pointers and showing me what the right way of doing things was. I went from not knowing that there was such a thing as a path tool, to becoming relatively proficient with it over the course of the night. It was like drinking a potion of +5 to Photoshop skills.

Apart from learning a lot, I also developed an even deeper appreciation and admiration of game artists. I knew it wasn’t easy stuff and that you needed a lot of talent. What I wasn’t quite fully appreciating is how technically involved art creation is! It’s very different from traditional painting and drawing, and it’s very highly technical. In a way, it’s almost like 3D modeling in how it requires mastery of a very complex tool and you need to work on very small parts for a long time.

Here’s a screenshot of the game showing all the assets I created during the jam:

Lessons Learned

Some random, unsorted, lessons learned from this jam:

• Come ready with an empty project you can start working on. The jam is not the time when you want to start stripping out old code. I learned that one in my first game jam, but didn’t come prepared with an iPad project (Hint: the iPhone -> iPad automatic conversion sucks–does anything automatic not suck?).
• Everything takes longer than you think. If you think you’ll just finish the game by morning, it’s probably too big. Choose something smaller.
• Learning stuff during the jam is great. Just adjust expectations about what you’ll create (we knew this going in, but still caught us by surprise).
• Take a moment to interact with the people around you. We’re all in a hurry to make something awesome, but take some time to talk to other developers. It’s well worth it, and makes the long night more bearable (and energizes you more).
• Pizza and coffee is a killer combination. I suspect I might never have to go to sleep if I keep the two in balance ;-b
• When wifi sucks, it’s hard to take the time to post updates or read other people’s updates.
• Hotel wifi always sucks.
• The jam is not a popularity contest. Sure, it’s great to show it off the next day, but make sure you create what you want for yourself and not based on what will demo best the next day.

If you haven’t done a game jam, you should. I strongly suggest collaborating with at least one other person, and doing it live with a bunch of other developers. The energy is incredible and it will be an experience you’ll learn a lot from and will remember for a long time.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter. This will be my last post for iDevBlogADay for a while (need to give those people in the massive waiting list a chance!), but I’ll definitely continue posting regularly.

Session description

The common-sense approach to make money on the App Store used to be to do anything to get on the top charts. In-app purchases changed all of that. Good in-app purchases can make your app profitable without being anywhere on the charts, and are the best hope for the independent developer. Come to this session to learn why IAPs can be so effective and how to leverage them effectively: what makes a good IAP, how to increase your user involvement, how to present IAPs in an attractive way, what things attract users, and what things turn them away. We’ll go through lots of detailed real-world data from Flower Garden and other games with strong IAPs.

Presentation slides: [Slideshare] [pdf]

Today I want to share my latest experiment, and how it was a total and complete failure: Adding the option to gift IAP items from within the game.

Promo Codes

Before I can talk about how I implemented the “Gift This” feature, we need to talk about promo codes. Since Flower Garden has so many in-app purchases, I figured it would be very handy to have the ability to give some of them away to people for any reason: They mistakenly downloaded the wrong thing, they bought it in the free version and want to upgrade to the full one, or they supposedly paid for one but they never got it. Whatever the reason, having my own promo codes for in-app purchases was one of the best decisions I could have made. I would highly recommend it if you have IAPs, especially consumables.

The implementation of promo codes was totally straightforward. I have two tables in the Google App Engine: One for active promo codes and one for redeemed ones. The active promo code includes the code itself, the IAP item it refers to, the amount, and whether it’s limited to one user or not (if it’s limited to one user, the code goes away as soon as it’s redeemed, otherwise, any amount of users can redeem it).

Here’s an example of a code I just added (yes, feel free to redeem it in the Flower Shop):

Whenever a promo code is redeemed, I enter that data in the other table. That way not only do I have a log of what codes where redeemed and when, but I can also check and prevent the same device from redeeming the same code multiple times.

Another important reason to keep a redeemed promo code table is that I want promo codes to be as valid as purchasing the IAP directly. That means that if you ever attempt to purchase an item, I check first to see if you’ve redeemed a promo code for it, and if so, you can re-download it for free. Same thing when you do a restore purchases (although I believe I haven’t gotten around to implementing that part yet :-)

Here’s my plea to Apple: Please, please, please, give us an “iTunes Account ID” along with the IAP data. Right now the best we can do is associate a purchase with a device (which is not good enough), or have a whole registration system for users (which is a pain and more time consuming for users). They’re already doing this with a Game Center ID, so why not with an iTunes Account?

Gift This

Once the promo code system was in place and field tested for a couple of months, I finally implemented the Gift This functionality. In the Flower Shop, users have the option to buy an item for themselves, or for someone else.

The first time you use the Gift This feature it explains how it works: You purchase the item and then you send it to someone else through email.

Under the hood, it purchases the IAP, contacts the server to generate a new promo code for that item, and then creates an email with the promo code and even a custom URL. Whenever someone receives a gift email, they can just click on the custom link and they immediately receive the item (assuming they have Flower Garden installed, of course).

One interesting consequence is that to implement this, you need to create one new IAP item for every item you want to gift (especially if they’re not consumable). Otherwise, someone couldn’t gift an item they had already purchased, or they couldn’t gift it more than once. This can add quite a few extra IAPs in your list!

In the case of Flower Garden, I started with the easiest case, and I only implemented gifting for fertilizer purchases (because they’re consumable, so I don’t have to keep track of who receives them and restoring them).

Total Failure

I really had great hopes for the Gift This feature. I had already envisioned writing a blog post showing IAP revenue going up 20-30% because of that feature. Not quite! It was pretty much a complete flop. The only positive thing I can say about it is that it didn’t actually lower regular sales.

See for yourself. In a period of about a month and a half, all the fertilizer gifting in the free and paid versions of Flower Garden amounted to a whopping $191!

Definitely not worth the 3-4 days it took me to implement it (and the opportunity cost of not being able to add some other feature or IAP).

Compare it to fertilizer sales during that period of over $7,000:

(That’s the spike of the new feature plus the Pocket Frogs cross promotion if you were wondering about it).

I’d love to know how the Gift This feature on the App Store is working out for Apple. I’m sure it’s doing better than my attempt at it, but I’m going to bet it’s still a very small percentage compared to regular sales.

Here comes the important question: Why was it a failure? Do people don’t like to gift? Was it presented badly? Did most people not know it existed?

There’s no way to know for sure, but my current guess is that people don’t like to gift something they don’t already own. Psychologically, there are several too many steps involved: Oh, I want to gift something, look for it in the store, purchase it, and send the email. Not a very fulfilling experience.

On the other hand, gifting something you already own is much more appealing. You have it in front of you, you’re proud of it and it looks great. You tap on a button and send it to someone. That is a lot more satisfying. So in the future I’ll take that approach and allow people to gift things they already own, even if they had to previously pay for it in some way.

What do you think? Do you have a better theory? How could it have been improved?

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

At one point I was half-seriously toying with the idea of creating a Twitter account that would show our latest commits to Subversion. Apart from the 140 character limit, that falls in the TMI category.

So instead, I thought it might be interesting to give a weekly update with what Miguel and I have been up to each week. Hopefully it will give you a glimpse at what’s going on behind the scenes. Any feedback in what you want to see more or less of is totally welcome.

Week Of Oct 29

Both Miguel and I felt this was a very slow week. I’m not sure exactly why, but I suspect post-IGF submission syndrome. We’re feeling the mini-crunch we did leading up to it, and now we have lots of not-so-fun tasks to do on our planes. Still, I think we’ve been slowly picking up speed. Next week should be a fully productive one.

Sound Effects

We added audio for in-game sound effects. Up until now we only had background music and no sound effects at all. Yes, that means we submitted to the IGF without sounds. Yikes! We had to cut corners somewhere. Fortunately, since the IGF takes updates, the next update will include some initial sound effects.

We’re using OpenAL for audio playback, and I was pleasantly surprised to see how simple and efficient it is. What a great change from using AVAudioPlayer!

We’re currently trying to define the style of the sound effects. It’s a strange mix of slightly cartoony, but not too much. Hopefully we’ll zero in on that next week.

Sharing Of Levels

Sharing of levels. We can finally share levels through email. It makes sending levels to each other much easier, and it should come in handy for the next round of testing. Of course, we still can’t reference an attached file from an email using the Apple API, so we’re having to submit levels and a screenshot to the server and store it there.

Sandbox editor

There’s more to creating a level than just placing objects. To create a playable level we need a title, a specific background, some goals, and a set of available items in the toolbox. Since we want people to share fully-featured levels, we’re building this UI right into the game, and it’s what we’re going to use to create all the levels ourselves. We can’t wait to see what people create, but, I have to admit, it’s somewhat of a pain to implement :-). We still have a few days of work in this one.

Right now it’s all using UIKit on top of OpenGL. I’m still waiting for a great, cross-platform GUI library + tools that gives us the basic features from UIKit + Interface Builder but it’s fully cross-platform.

Also, while Miguel isn’t looking, I’ll share with you what we have so far for the level editor sidebar.

In case you missed it, Casey has a Twitter account. And apparently he likes Angry Birds quite a bit! :-)

Also, don’t forget to join the Casey’s Contraptions Facebook page.

I’m going to say this very clearly so it doesn’t get lost in the middle of a paragraph.

360iDev is the best conference you can go to if you’re doing any kind of iOS development.

There. I’ve said it. And no, they’re not paying me any to say that.

I’m clearly not the only one who feels that way either. Just earlier this week Mike Berg wrote a post about how awesome 360iDev is, and we didn’t even compare notes. Great minds think alike apparently.

Yes, Apple puts the big show for WWDC. It’s a unique experience: the keynote, the crowds, the unveiling of the latest technologies, the sessions, the labs… But in the end, it’s a big show from Apple to woo its developers. You’re getting the official message through very polished presentations. Which is fine, but it feels a bit… too polished. Too streamlined. Too overproduced.

Talk to developers who’ve been to WWDC multiple times, and you’ll quickly find out that the parts they like best are the labs (access to Apple engineers) and the networking (some with Apple, but mostly with other attendees). That’s why keeping track of the parties during WWDC is almost a full-time job!

For Developers, By Developers

360iDev on the other hand is a conference for developers by developers. You don’t get fed the official party line. Instead, you get to hear how some API really worked (or didn’t) in the trenches, how developers had to work around bugs, or, why not, how some technology was a dream to work with. Nothing like hearing it straight from the horse’s mouth.

Strong Game Development Track

There are usually three simultaneous tracks at 360iDev: Business, Sights and Sounds, and Development Tricks. As you can expect, sights and Sounds is usually entirely devoted to games, and there’s plenty of game-related info in the other tracks as well.

I’m going to be totally honest here: The quality of the sessions varies a lot from one to the other, and they can be somewhat hit or miss. When the presentations are awesome, they’re really awesome. And on the average, I’d say they’re very good. That’s the flip side of not having a super-rehearsed, super-polished presentations like WWDC.

Hacker Vibe

You walk into WWDC, and you get a very strong corporate feeling (trying to be developer friendly). The moment you walk into 360iDev, it has a palpable hacker vibe [1]. The people presenting might not have the most polished slides, but they can do some amazing things on the iPhone. There are even presentations on the internals of the iPhone and what’s going on under the hood, something you’ll never get from Apple!

Game Jam

As a perfect example of the hacker mentality, the Game Jam has become a regular feature at 360iDev. On the last night, developers get together in a big rooms, and either flying solo or grouping into teams, they create a game prototype in a few hours. The next day at lunch, we have a big gathering and get to demo the games created the previous night to everybody. What a perfect (and exhausting!) last day to the conference!

It’s Nimble And Agile

This might not seem like a big deal to some, but it’s very important: 360iDev happens twice a year. Technology conferences that happened once a year might have been fine 10-15 years ago, but as the pace of technological advance continues to accelerate, once a year doesn’t cut it anymore. Especially if you have to submit talk proposals 6-8 months in advance, they’re old news by the time the conference rolls around.

360iDev is much more agile than that. It happens twice a year, and you only need to submit a general overview a few months in advance. Given the content of a lot of the talks, they probably come together just weeks (if not days) before the conference itself. That’s part of the reason for the uneven quality of the talks, but it’s a price worth paying.

Networking

360iDev is a small conference. I don’t know the official numbers, but I think there are usually around 200-300 attendees. You’ll be seeing the same faces all three days, especially the ones that share your same interests and end up going to the same sessions as you do. Even if you’re a total introvert, you’ll end up meeting a bunch of new, very interesting people, and creating lots of new possibilities for your future.

Even better, the speakers are part of that small number of attendees, and they get to hang out with everybody else. There aren’t special VIP parties, or secret off-site invitation-only parties (if they are, they’re so secret I missed them). Everybody hangs out during the sessions, at lunch, and the evening festivities. So if there’s someone in the speaker list you particularly want to meet, this is your chance.

It’s not just other developers either. The iPhone media often comes to the conference as well, so you might get a chance to talk to people from TouchArcade or TUAW.

360iDev three full days of sessions, one day of tutorials, one night of game jam, three evenings or parties, and one conference full of awesome. And that for a fraction of price of the big conferences. How can you go wrong? [2]

[1] I mean hacker in the good sense of the word, not in the “cracker”, malicious one!

[2] If this doesn’t convince Gavin and Craig to come to 360iDev, I don’t know what will. Go buy the awesome Linkoidz so they’ll be forced to attend :-)

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

We’ve all had things nagging us in the back of our minds. They’re nothing we have to worry about this very instant, just something we need to do sometime in the future. Maybe that’s changing those worn tires in the car, or making an appointment with the dentist about that tooth that has been bugging you on and off.

Dynamic memory allocation is something that falls in that category for most programmers. We all know we can’t just go on allocating memory willy-nilly whenever we need it, yet we put off dealing with it until the end of the project. By that time, deadlines are piling on the pressure and it’s usually too late to make significant changes. With a little bit of forethought and pre-planning, we can avoid those problems and be confident our game is not going to run out of memory in the most inopportune moment.

On-Demand Dynamic Memory Allocation

The easiest way to get started with memory management is to allocate memory dynamically whenever you need it. This is the approach many software engineering books consider as ideal and it’s often encouraged in Computer Science classes.

It’s certainly an easy approach to use. Need a new animation instance when the player is landing on a ledge? Allocate it. Need a new sound when we reach the goal? Just allocate another one!

On-demand dynamic memory allocation can help to keep memory usage to a minimum, because, you only allocate the memory that you need and no more. In practice it’s not quite as neat and tidy because there can be a surprisingly large amount of overhead per allocation, which adds up if programmers become really allocation-happy.

It’s also a good way to shoot yourself in the foot.

Games don’t live inside a Computer Science textbook, so we have to deal with real world limitations, which make this approach cumbersome, clunky, and potentially disastrous. What can go wrong with on-demand dynamic memory allocation? Almost everything!

Limited Memory

Games, or any software for that matter, run on machines with limited amounts of memory. As long as you know what that limit is, and you keep extremely careful track of your memory usage, you can remain under the limit. However, since the game is allocating memory any time it needs it, there will most likely come a time when the game tries to allocate a new block but there is no memory left. What can you do then? Nothing easy I’m afraid. You can try to free an older memory block and allocate the new one there, or you can try to make your game tolerant to running out of memory. Both those solutions are very complex and difficult to implement correctly.

Even setting memory budgets and sticking to them can be very difficult. How can a designer know that a given particle system isn’t going to run out of memory? Are these AI units going to create too many pathfinding objects and crash the game? Hard to say until we run the game in all possible combinations. And even then, how do you know it isn’t going to crash five minutes later? Or ten? It’s almost impossible to know for certain.

If you insist in using this approach, at the very least, you should tag all memory allocations, so you have an idea of how memory is being used. You can either tag each allocation based on what system initiated it (physics, textures, animation, sound, AI, etc) or even on the filename where it originated, which has the advantage that it can be automated and should still give you a good picture of the overall memory usage.

Memory Fragmentation

Even if you take lots of pain not to go over your the available memory, you might still run into trouble because of memory fragmentation. You might have enough memory for a new allocation, but in the form of many small memory blocks instead of a large contiguous one. Unless you provide your own memory allocation mechanism, fragmentation is something that is very hard to track on your own, so you can’t even be ready for it until the allocation fails.

Virtual Memory

Virtual memory could solve all those problems. In theory, if you run out of real memory, the operating system swaps out some older, unused pages to disk and makes room for the new memory you requested. In practice, it’s just a bad caching scheme because it can be triggered at the worst possible moment, and it doesn’t know about what data it’s swapping out or how your game uses it.

Games, unlike most other software, have a “soft realtime” requirement: The game needs to keep updating at an acceptable interactive rate, which is somewhere around 15 or more frames per second. That means that gamers are going to make a trip to the store to return your game if it pauses for a couple of seconds every few minutes to “make some room” for new memory. So relying on virtual memory isn’t a particularly attractive solution.

Additionally, lots of games run in platforms with fixed amounts of RAM and no virtual memory. So when memory runs out, things won’t get slow and chuggy, they’ll crash hard. When the memory is gone, it’s really gone.

Performance Problems

There are some performance issues that are relatively easy to track down and fix. Usually ones that occur every frame and are happening in a single spot: some expensive operation, a O(n3) algorithm, etc. Then there are performance problems introduced by dynamic memory allocations, which can be really hard to track down.

Standard malloc returns memory pretty quickly, and usually doesn’t ever register on the the profiler. Every so often though, whenever the game has been running for a while and memory is pretty fragmented, it can spike up and cause a significant delay for just a frame. Trying to track down those spikes has caused more than one programmer to age prematurely. You can avoid some of those problems by using your own memory manager, but don’t attempt to write a generic one yourself from scratch. Instead start with some of the ones listed in the references.

Malloc spikes are not the only source of performance problems. Allocating many small blocks of memory can lead to bad cache coherence when the game code access them sequentially. This problem usually manifests itself as a general slowdown that can’t be narrowed down in the profiler. With today’s hardware of slow memory systems and deep caches, good memory access patterns are more important than ever.

Keeping Track Of Memory

Another source of problems with dynamic memory allocation are bugs in the logic that keeps track of the allocated memory blocks. If we forget to free some of them, our program will have memory leaks and has the potential to run out of memory.

The flip side of memory leaks are invalid memory access. If we free a memory block and later we access it as if it were allocated, we’ll either get a memory access exception, or we’ll manage to corrupt our own game.

Some techniques, such as reference counting and garbage collection can help keep track of memory allocations, but introduce their own complexities and overhead.

Introducing Pre-allocation

On the opposite corner of the boxing ring is the purely pre-allocated game. It excels at everything that the dynamically-allocated game is weak at, but it has a few weaknesses of its own. All in all, it’s probably a much safer approach for most games though.

The idea behind a pre-allocation memory strategy is to allocate everything once and never have to do any dynamic allocations in the middle of the game. Usually you grab as big a block of memory as you can, and then you carve it out to suit your game’s needs.

Some advantages are very clear: no performance penalties, knowing exactly how your memory is used, never running out of memory, and no memory fragmentation to worry about. There are some other more subtle advantages, such as being able to put data in contiguous areas of memory to get best cache coherency, or having blazingly-fast load times by loading a baked image of a level directly into memory.

The main drawback of pre-allocation is that is more complex to implement than the dynamic allocation approach and it takes some planning ahead.

Know Your Data

For preallocation to work, you need to know ahead of time how much of every type of data you will need in the game. That can be a daunting proposition, especially to those used to a more dynamic approach. However, with a good data baking system (see last month’s Inner Product column), you can get a global view of each level and figure out how big things need to be.

There is one important design philosophy that needs to be adopted for preallocation to work: Everything in the game has to be bounded. That shouldn’t feel too restrictive; after all, the memory in your target platform is bounded, as well as every single resource. That means that everything that can create new objects, including high-level game constructs, should operate on a fixed number of them. This might seem like an implementation detail, but it often bubbles up to what’s exposed to game designers. A common example is an enemy spawner. Instead of designing a spawner with an emission rate, it should have a fixed number of enemies it can spawn (and potentially reuse them after they’re dead).

Potentially Wasted Space

If you allocate enough data for the worst case in your game, that can lead to a lot of unused data most of the time. That’s only an issue if that unused data is preventing you from adding more content to the game. We might initially balk at the idea of having 2000 preallocated enemies when we’re only going to see 10 of them at once. But when you realize that each of those enemies is only taking 256 bytes and the total overhead is 500 KB, which can be easily accommodated in most modern platforms today.

Preallocation doesn’t have to be as draconian as it sounds though. You could relax this approach and commit to having each level preallocated and never having dynamic memory allocations while the game is running. That still allows you to dynamically allocate the memory needed for each level and keep wasted space to a minimum. Or you can take it even further and preallocate the contents of memory blocks that are streamed in memory. That way each block can be divided in the best way for that content and wasted space is kept to a minimum.

Reuse, RecycleM

If you don’t want to preallocate every single object you’ll ever use, then you can create a smaller set, and reuse them as needed. This can be a bit tricky though. First of all, it needs to be very much specific to the type of object that is reused. So particles are easy to reuse (just drop the oldest one, or the ones not in view), but might be harder with enemy units or active projectiles. It’s going to take some game knowledge of those objects to decide which ones to reuse and how to do it.

It also means that systems need to be prepared to either fail an allocation (if your current set of objects is full and you don’t want to reuse an existing one), or they need to cope with an object disappearing from one frame to another. That’s a relatively easy problem to solve by using handles or other weak references instead of direct pointers.

Then there’s the issue that reusing an object isn’t as simple as constructing a new one. You really need to make sure that when you reuse it, there’s nothing left from the object it replaced. This is easy when your objects are just plain data in a table, but can be more complicated when they’re complex C++ classes tied together with pointers. In any case, you can’t apply the Resource Acquisition Is Initialization (RAII) pattern, but it doesn’t seem to be a pattern very well suited for games, and it’s a small price to pay for the simplicity that preallocation provides.

Specialized Heaps

Truth be told, a pure pre-allocated approach can be hard to pull off, especially with highly dynamic environments or games with user-created content. Specialized heaps is a combination of dynamic memory allocation and pre-allocation that takes the best of both worlds.

The idea behind specialized heaps is that the heaps themselves are pre-allocated, but they allow some form of specialized dynamic allocation within them. That way you avoid the problems of running out of memory, or memory fragmentation globally, but you still can perform some sort of dynamic allocation when needed.

One type of specialized heaps is based on the object type. If you can guarantee that all objects allocated in that heap are going to be of the same size, or at least a multiple of a certain size, memory management becomes much easier and less error prone, and removes a lot of the complexity of a general memory manager.

My favorite approach for games is to create specialized heaps based on the lifetime of the objects allocated in them. These heaps use sequential allocators, always allocating memory from the beginning of a memory block. When the lifetime of the objects is up, the heap is reset and allocations can start from the beginning again. The use of a simple sequential allocator bypasses all the insidious problems of general memory management: fragmentation, compaction, leaks, etc. See the code in http://gdmag.com/resources/code.htm for an implementation of a SequentialAllocator class.

The heap types most often used in games are:

• Level heap. Here you allocate all the assets and data for the level at load time. When the level is unloaded, all objects are destroyed at once. If your game makes heavy use of streaming, this can be a streaming block instead of a full level.
• Frame heap. Any temporary objects that only need to last a frame or less get allocated here, and destroyed at the end of the frame.
• Stack heap. This one is a bit different from the others. Like the other heaps, it uses a sequential allocator and objects are allocated from the beginning, but instead of destroying all objects at once, it only destroys objects up to the marker that is popped fro the stack.

What About Tools?

You can take everything I’ve written here, and (almost) completely ignore it for tools. I fall in the camp of the programmers who consider the runtime as a totally separate beast from the tools. That means that the runtime can be lean and mean and minimalistic, but I can relax and use whatever technique makes me more productive when writing tools. That means you can allocate memory any time you want, you can use complex libraries like STL and Boost, etc. Most tools are going to run on a beefy PC and a few extra allocations here and there won’t make any difference.

Be careful with performance-sensitive tools though. Tools that build assets or compute complex lighting calculations might be a bottleneck in the build process. In that case, performance becomes crucial again and you might want to be a bit more careful about memory layout and cache coherency.

On the other hand, if the tool you’re writing is not performance sensitive, you should ask yourself if it really needs to be written in C++. Maybe C# or Python are better languages if all you’re doing is transforming XML files or verifying that a file format is correct. Trading performance for ease of development is almost always a win with development tools.

Next time you reach out for a malloc inside your main loop, think about how it can fail. Maybe you can pre-allocate that memory and stop worrying about what’s going to happen the day you prepare the release candidate.

This article was originally printed in the February 2009 issue of Game Developer.

This is a bit of a different project than some of my past ones. This one is a collaboration with Miguel Ángel Friginal from Mystery Coconut. I’m doing the programming, Miguel is doing all the art, and we’re both contributing equally to the design and everything else. It has been great having some awesome art to go with the game, but also to collaborate with someone really closely on the game.

Casey’s Contraptions was one of those ideas for a game that I kept wanting to make for quite a while, and now it was finally the right time. It meets the three main requirements that I’m looking for in a game project:

• Something original
• Has potential to sell well
• It involves a creative activity (instead of something violent or destructive)

The idea of games based on mechanical contraptions is not new, but there are surprisingly few games based on it. We are hoping to bring a lot new to the table: Casey himself, unlockable items, interface built from the ground up for multi touch, modern physics simulation, social features, sharing of solutions, and even creation and sharing of new levels. We are really excited to be working on this project and we can’t wait until it’s released.

Development

Casey’s Contraptions started as a prototype back in the summer. After a day or two messing with physics engines and creating some objects, I knew there was something there, so I spent a two more weeks creating an initial version. It wasn’t much more than a tech proof-of-concept, but it was clear that there was a game there (even with my horrible stand-in clip art assets).

I sent that built to a couple of friends for initial feedback. It was laughably early, but that’s the time when it’s possible to really make radical changes to the design. I knew the people I was sending it to a) were used to seeing games at early stages, and b) were not afraid to tell me if something sucked. Actually, I told them to skip the nice parts and just focus on everything that they didn’t like. Not surprisingly, that initial feedback was crucial, and really shaped how Casey’s Contraptions evolved since then.

Shortly after that, Miguel joined me full time on the game and we dove right into it. For our development, we used a super light-weight agile approach: We have high-level “user stories”, and two week iterations. Iterations are somewhat flexible (plus minus a few days) and we don’t strictly estimate the tasks, just take on as many user stories as we think we can do in that time. The important parts are to always be focused on the most important stories, and to take them to completion each time.

Miguel lives in Seattle and I’m in Carlsbad, so we do all of our work remotely. We use Subversion hosted remotely, and we’re in constant communication through iChat and email. That allows us to iterate on a piece of art, an item behavior, or a menu item multiple times very quickly. It’s not as good as sitting side by side, but I haven’t felt like working remotely has gotten in the way at all.

After a busy month and a half, we got to where you see it today, and we submitted the game to the Independent Games Festival (IGF).

Independent Games Festival

Some people have asked me why I wanted to submit it to the IGF. I have to be honest and admit that I had never really considered not submitting it.

I imagine everybody reading this blog knows about the IGF already. It’s the closest thing to the Movie Academy Awards that we have for independent games. There are many reasons why you’d want to submit your game. The amount of press and prestige associated with winning or even being nominated as a finalist is huge. The prize money is a nice touch, but it’s not really enough to make much of a difference in the game itself (I’m talking about the Mobile Category prize). Of course, there are many other fantastic games that are competing at the IGF (a total of almost 400 games!), so it’s hard to count on becoming a finalist.

One very real and concrete reason to enter the IGF was to have a very well-defined milestone. It wasn’t that different from one of our iterations, except that we had a real customer (the IGF judges) and a non-flexible deadline. That made us really focus our efforts and put in some extra effort in the last couple of weeks to get it in shape for the IGF. Looking back at the game just a couple of weeks ago, it’s amazing how far it came in that time.

For me personally, the biggest reason to enter the IGF is because it’s something I’ve always wanted to do. I’ve followed the IGF since the first one in 1999 (which was, coincidentally, the first GDC I attended). Every time I walked through the booths or watched the award ceremony, I wanted to be part of it. So finally, this was my chance to do it. It was a great experience going through the process. Now the next life goal will be to actually make it as a finalist.

Submitting Casey’s Contraptions had another, unexpected side effect: It tipped our hand and forced us to announce the game sooner than we were planning on doing. It wasn’t until a few days before the submission that we realized the list of IGF games would go public right away. Originally we were planning on announcing the game at 360iDev in mid November, but this forced us to move the schedule up somewhat. Hopefully that will be a good thing and will allow us to build some good buzz in the upcoming months all the way until the release. Keep an eye out for a gameplay video and some hands-on previews in the next few weeks.

If you want to keep up to date with Casey’s Contraptions development, join the Facebook group, follow Miguel and me on Twitter, or keep an eye on the Touch Arcade thread.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

But that was because of the kind of games I was making, which were very light on content, with mostly procedurally generated assets (the consequence of working by myself and being much better at programming than at Photoshop).

That game that Miguel and I are working on right now is a lot heavier on assets. It has locations, and levels, and the whole shebang. And that’s where XCode starts falling short.

Explicit Resources

Before, I was adding all my game assets to the Resources folder in XCode. That adds the file to the “Copy Bundle Resources” step. And as you expect, when you do a build, it checks the date of the existing file, and only copies it if the source file is newer than the destination file.

Personally, I really like this approach. I like to be explicit about what gets included in a game, and I don’t mind at all having to add files manually to the project.

Unfortunately, it has one major flaw: It collapses all assets at the root of the application directory, ignoring the directory structure where they came from. I have no idea what the rational is for this “feature”, but someone needs to be taken out to the public town plaza and whipped for that. Actually, make that a double-whipping session if the reason was “convenience”.

The reason this becomes a big deal now is that we have per-level resources. To keep things sane, we decided to use a directory hierarchy, so Levels/Level00 contains all the files necessary for that level. Same thing with Level01, etc. The problem comes that both those levels have similarly named files: Background.jpg Layout.bin, etc.

Any guesses what happens if you add to XCode two files with the same filename in different paths? Yup. One of them overrides the other. Not a single warning either. Let’s make that a triple-serving of whipping, please.

I briefly considered prefixing all the files with the level name (Level00_Background.jpg), but if later I decide to move Level00 to Level05 that’s a lot of files to rename, so I would end up having to write scripts, or create a separate file with the level ordering, or just generally waste my time doing something that should have been taken care of by the tool.

Folder References

Even though I had read they had their share of problems, I decided to look a folder references (at Miguel’s prompting mostly).

When you add some resources to XCode, you have an option to check “Create Folder References for any added folders”. That option automatically adds any files in those folders without having to explicitly add them to XCode. So you could add the Levels folder, and then any files you create there will be copied with the game.

I’m not a big fan of assets copied automatically, but as a side effect, that step preserves the directory hierarchy each of those files was in. So any files copied this way can be accessed from within the game by using their full directory structure.

I have to ask again: Why are directory structures preserved here but not with explicit resources added to the project? The mind boggles.

But hey, at least it works, right? Not exactly. There are a couple of gotchas.

The big one I had read in multiple places, is that XCode doesn’t detect any changes to files inside the referenced folder. So you can be making all sorts of changes, building the game, and not seeing anything different. The recommended solution was to add an extra step to the build process that would start by touching the reference folder, forcing a full copy of all assets.

I tested that, I’m glad to report that at least in XCode 3.2.4, that’s not the case. If you modify any file inside a referenced folder, the file will get copied over correctly during the build process without the need of extra steps.

The bad news is that all the files in the referenced folder will be copied. Why oh why??? They clearly know which file changed, why do they feel the need to copy all of the other files? No idea. This is not a big deal early on, but as you start to accumulate dozens and hundreds of megabytes of assets, build times start increasing quite a bit, especially on the device itself.

This is what the copy command looks like for referenced folders:

CpResource build/Debug-iphonesimulator/Test.app/Levels Levels
cd /Users/noel/Development/Test/trunk/Test
setenv PATH "/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin:/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin"
/Developer/Library/PrivateFrameworks/DevToolsCore.framework/Resources/pbxcp -exclude .DS_Store -exclude CVS -exclude .svn -Testve-src-symlinks /Users/noel/Development/Test/trunk/Test/Levels /Users/noel/Development/Test/trunk/Test/build/Debug-iphonesimulator/Test.app

It’s nice touch that it automatically excludes .svn directories though. I was wondering why they use CpResource instead of plain, old cp, but I guess that’s to be able to -exclude specific files. Fair enough.

However, what CpResource apparently doesn’t do is to process any of the resources in ways that were processed before by XCode. For example, a png file would have been processed by premultiplying it and byte swapping it so loading it in the iPhone would be slightly more efficient. CpResource just does a regular copy and leaves it alone. So if you were relying on that behavior, you need to do it explicitly yourself in your asset baking step.

What I Really Want

For now, I’m using folder references for the levels, and explicit references for everything else. That way I keep the data size to a minimum but I get to have the directory hierarchy. Not ideal, but at least it works.

This is what I would really like thought:

1. Easiest: Explicit assets with paths. I really want to just add resources to XCode and have it preserve the directory structure. It’s not that hard. If XCode were open source, I would have made that change a long time ago. Can we at least have this as an option?
2. Second easiest: Folder references that only copy the changed resources. That would also be OK in my book, and I can’t believe it would be much harder to implement either.
3. Best: A remote file system hosted on the Mac during debug build. All file references go out to the host machine and get loaded on the fly. This would allow for fastest build times and loading times would not be that different from a fragmented drive on an old device probably. I know some of you already have something like this. Has anybody made one open source (preferably minimalistic and standalone)? I’d love to check it out.

Of course, all of this has probably changed already with XCode 4, but I’m deathly afraid of installing it while working on a production game. Has anybody tried it yet? Have they fixed anything, or is it even more broken?

To wrap things up, and since Miguel is spilling the beans on Twitter, I’ll share a few assets from our current game. Now back to the game because we’re submitting it to the Independent Games Festival on Monday. Next Thursday I’ll talk about the IGF. Wish us luck!

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Google App Engine Costs

At the time I wrote that post, the bandwidth of Flower Garden (and Flower Garden Free) was just under the free bandwidth allowance, so I didn’t have to pay anything. However, this last week, after riding the Pocket Frogs rocket, Flower Garden Free shot up to the #56 overall app in the US. And boy, did that make a difference in server usage!

That screenshot shows the Google App Engine dashboard with the number of requests per second for a 4 day period. Can you guess when the Pocket Frogs cross-promotion started? It went from an average of 1.5 requests per second, to a peak of about 16. That’s a factor of 10 right there.

Let’s see what that means in terms of cost. With the Google App Engine, you get charged for different resources you consume: CPU usage, incoming bandwidth, outgoing bandwidth, data storage, and recipients emailed. Of those, the only one that Flower Garden makes any dent on is outgoing bandwidth (mostly the HTML pages and images in the Flower Shop).

For each of those resources, you get a free amount every day. In the case of outgoing bandwidth, we get 1GB before we have to start paying anything. Afterwards, it’s $0.12 per GB. The peak day for Flower Garden Free, when it was all the way at #56, it consumed 7.8 GB. Now that it’s back down, it’s using up at around 2-3 GB of outgoing data every day.

That means, that the busiest day I was charged $0.82 in bandwidth, but the profits for that day were over $1500! I can only hope for days just as busy!

As a reference, these are my Google App Engine charges for the busy week during the promotion. $3.28 for the week? Bring it on! :-)

Minimizing Bandwidth

When I first saw the first day’s bandwidth, I was pretty surprised it was taking up that much. I hadn’t tried to minimize it first, but 8 GB seemed like a significant amount.

The first thing I did was to forward any requests to the “More Games” and “News” section back to the Dreamhost account. After all, if that server goes down, it doesn’t really matter, and they can take up significant bandwidth with all the images.

To forward something, I couldn’t just do it from the configuration file. Instead, I set a handler for the moregames directory like this:

- url: /moregames/.*
  script: redirect.py

And the redirect script is very simple:

import os
import sys
import logging
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class RedirectMoreGames(webapp.RequestHandler):
	def get(self, params):
		url = "http://flowers.snappytouch.com/moregames/Snappy_Touch_More_Games/More_Games.html"
		self.redirect(url, permanent=True)
	
application = webapp.WSGIApplication(
							[(r'/(.*)', RedirectMoreGames),
							 ],
							debug=True)

def main():
	run_wsgi_app(application)

if __name__ == "__main__":
	main()

Another thing you can do is to turn on explicit caching of files in the app.yaml file for your Google App Engine app. I don’t know how much that helps with data requested from within an iPhone app, but it can’t hurt.

Finally, I made some changes to Flower Garden. I never had bandwidth minimization in mind, so for example, I was happily requesting updated news and other data at the beginning of every session. I wised up a bit and now it only requests that data at most once every few hours or days.

In case you’re ever getting close to using up your quotas, I learned that you can query them at runtime. I haven’t done anything about it yet, but I suppose I could start forwarding image fetches to a backup server whenever I get close to the limit. But for that, I’ll have to sell many more units of Flower Garden!

Google App Engine Shortcomings

Overall, I’m very happy with the Google App Engine. But it’s not all a bed of roses over here. I think the Google App Engine excels at scaling under heavy load, but surprisingly, it doesn’t have anything close to 100% uptime. I’m going to say it probably doesn’t even have 95% uptime! With such a massive system and all the redundancy underneath, I’m surprised they can’t have better uptime. Supposedly their status page makes you think everything is perfect, but it’s really far from it.

Apart from downtime, people are also reporting severe latency issues sometimes, and I’ve seen some errors like that in my log every so often. Chalk it up to being a beta I suppose. I imagine it’s only going to get better with time.

I would still use the Google App Engine for any new game I’ll release in the future, whether it has big or small backend needs. If nothing else, the ease of development and testing beats every alternative for me.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Most often, the advice stops there. So you get your initial sales spike and then sales go way down. What do you do then? Usually, developers release new features and updates. That’s great, but how do you get people to notice it. You need to establish some form of communication with your players.

Update Messages

The simplest form of communication is through the “What’s new” section in the update. You can use that section not just to list what features you added and what bugs you fixed, but also to let your players know about other things: plans for the future, other games to try, or even the URL for your Facebook group.

Credit goes to Igor for bringing up this idea and pointing out the URLs are even clickable in this field (but they aren’t in the app description).

Of course, if you’re like me and you update your apps only once in a while, this technique isn’t as effective. Right now my iPhone tells me I have 67 new updates available. I’m clearly not going to be reading through the release notes of each one.

In-Game News

A more direct way of reaching out to your players is to have some sort of in-game news system. At any point you can update a file in your web server with any news, and it is displayed in the game. It can be implemented in many different ways depending on “on your face” you want to be about it: a pop up that comes up when the user runs the game, a ticker that runs constantly across the bottom of the screen, or, what I did for Flower Garden, a news icon with a badge indicating how many unread items there are, that brings up the news page when you tap it.

Make sure players are able to click on URLs in your news messages so you can direct them to different web pages easily. More on that on a bit.

Emails

Update messages are only good for players who update their apps (ahem, ahem), and in-game news for active players who’re currently launching your apps. For maximum effect, you can send an email newsletter and that way you can also reach users who played the game at some point in the past, but aren’t currently playing it now. They’re the ones probably most interested in new updates and features, and chances are you can rekindle their interest in the game.

To do this, you should encourage users to register for your mailing list, or collect their emails with their consent in some other way. This was a technique I started using back in December of last year with great success.

I’m currently using Your Mailing List Provider as a means to delivering thousands and thousands of emails [1]. By the way, don’t miss a chance to join the Flower Garden mailing list :-)

Facebook

Similar to the mailing list approach, Facebook groups can be a very effective form of communication. An additional benefit is that friends of your players might see them participating in the page and might make them try out your game.

Case Study: Pocket Frogs Cross-Promotion

All that is fine in theory. How does it work in practice? I have been using all four forms of communication for a while, and I’m definitely seeing good bumps of sales and downloads with each update and each major communication.

Last week, Ian and Dave from Nimblebit and I, decided to set up a cross promotion between Pocket Frogs and Flower Garden.

I updated the in-game news and send out an email newsletter coinciding with the latest Flower Garden update telling the players that if they downloaded Pocket Frogs from within Flower Garden, they would be awarded 5 doses of fertilizer. Nimblebit awarded players a flower if they downloaded Flower Garden Free from within Pocket Frogs.

When you have over a million downloads in a week like Pocket Frog did, that kind of player communication is the equivalent of a nuclear cannon. The effects of the cross-promotion were obvious the instant the news went live:

As you can see, Flower Garden Free made it all the way to the number 56 in the iPhone Top Apps chart in the US! The effect even spread to the paid version of Flower Garden:

Pocket Frogs at the time was hovering at around #9 on the charts, so it was difficult to have much of an impact on that position without major numbers, but we suspect it might have hovered there a little longer because of the extra downloads from Flower Garden.

Here is what the downloads for Flower Garden Free looked like for the last month. The Pocket Frogs cross-promotion is quite noticeable:

All those downloads also translated into in-app sales through the Flower Shop. Here are the revenues for that time period:

One consequence I wasn’t expecting, but in retrospect I’m not that surprised about, is that the ratings for Flower Garden Free dropped by a whole star (from 4 to 3), with a large percentage of 1-star reviews. That’s because a lot of people who wouldn’t have downloaded Flower Garden otherwise did it anyway, didn’t like it, and deleted it right away.

To wrap things up on a better note, there was yet another side effect of the cross promotion. The Pocket Frogs link was using my LinkShare referral code. Sending all those users to the App Store to download a free game link resulted in about $200 in referral profit for the week.

Conclusion

Communicating with your players is more than just profitable: It’s crucial to the sustained success of your games. Make sure you try to engage with them in every way you can, keep them up to date with developments in your game, and don’t hesitate to run the occasional cross-promotion, especially with other games that are a good match for your target audience.

[1] If you decide to use them and wouldn’t mind using my referral code (WQHVUF), I can get a small percentage back.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Initially I dismissed it for me because I’m reasonably active: I either run or bike 5 or 6 days per week, and the rest of the days I go for a walk around the neighborhood. It was the combination of increased studies on the effects of sitting, some people in my social circle finally making the jump and raving about it, and me developing some problems in a hamstring that finally made me consider it more seriously. And trust me, if you’re a cyclist or a runner, the last thing you want to have is hamstring problems.

It turns out that my 5-mile morning runs weren’t making me immune to the dangers of sitting. It’s not how much exercise you get per day, but how long do you sit on a chair continuously. And no, buying an fancy, expensive chair might help with your back, but it’s not going to do one bit of good with all the other problems.

One of the many advantages of working from home is that I can try weird things that would be much more difficult in a regular workplace. I also have a track history of liking to experiment on myself [1], so it didn’t take much convincing to give this a try.

After some initial research, it appeared that the way to go was an adjustable-height desk. That way you can work standing, but you still have the flexibility to sit down whenever you need to. I also found out that apparently this is not all that uncommon in Europe. The only drawback is that adjustable standing desks are not easily available here in the US, and the ones that are out there are aimed at offices and big companies, with matching eye-popping price tags. Even though Flower Garden continues to do well, I wasn’t quite ready to plop down several thousand dollars on something I might end up hating.

So I decided to start cheap and work my way up from the bottom. First I had to decide if I even liked this whole working-while-standing thing. I wasn’t even sure I would be able to type! I thought about raising my desk with cinder blocks, but I would have to raise a lot and would make it very unstable. So instead, I created four stacks of books on top of my desk and put a board on top. Then I was able to put the keyboard, computer, and monitor on the board and give that a try.

Let me tell you: It was weird.

At first I didn’t know how to type. My fingers were constantly off to the side. It turns out my initial height was too low. You really want to have the keyboard at a height that puts a 90 degree bend on your elbows. That’s what they say about the sitting position, but somehow I can manage to have less. However, standing, I really needed that height.

Even once I adjust the height, the first day was kind of rough. After an hour, I was definitely feeling it in my feet. I think I did half a day the first day and I felt totally exhausted. All that running wasn’t helping that much standing at my desk apparently.

Fortunately things got better very quickly. In a few days, I was able to work for hours without much problem. I would find myself constantly shifting my weight between my feet and moving around a little bit. I was definitely liking that whole standing thing.

At this point I decided that the stack of books was too annoying, but I wasn’t quite ready for an expensive desk. So I picked up a Fredrik Ikea desk from Craigslist for $50. The nice thing about this desk is that it can be assembled so the top of the desk is almost at any height. You can’t change it on the fly, but at least it serves as a standing desk.

Finally I was able to work standing and be comfortable at the same time now that I had space for more than a mug along with the keyboard. The extra shelves on the desk were also very handy (one over the table top and one underneath).

After spending a couple of weeks with the desk, I decided that I definitely liked standing and it was something I wanted to continue doing long term. I felt more focused while standing, and my productivity was up. An unexpected side benefit was that when someone else comes into the room, they can walk right up to the desk and we can look at the monitor together, or look at some papers, much more easily than if it was a sitting desk.

However, it was also clear that I needed to switch things up a bit. Spending the whole day standing continued to be pretty tough on my feet, and it even made it so I didn’t want to work any longer than I had to. I really had to combine the standing with some hours of sitting down, which is probably a healthier thing to do anyway.

At this point I had two options: I could go full out and get a motorized adjustable desk, or I could get an adjustable draft chair which are tall enough to sit comfortably at a standing-height desk. Since I already had a nice office chair, and the Ikea desk was a bit wobbly set up like that, I decided to spend the money on the motorized desk.

Again, after a bunch of research, it seemed that one of the best options was to order a (very appropriately named), GeekDesk online. They have them in two sizes, Classic and Mini. It turns out I wanted something more in the middle (around 60" wide), so I ended up ordering the Mini frame and a desk top from Ikea for $80. It wasn’t cheap, but it was way cheaper than the alternatives and all the reviews and experiences I read were very positive.

It arrived very quickly and it was a breeze to assemble. I did have a scary moment that it looked like one of the wheels didn’t fit in the metal opening, but I eventually managed to coerce it. Hopefully this kind of manufacturing problems aren’t common.

I can only describe the final setup with one word: Awesome. The desk isn’t wobbly at all, and it takes just a few seconds to change heights with the push of a button. Not just that, but I can fine tune the height at any moment in tiny increments. And in the sitting position, I can put it low enough to make typing very comfortable (most desks are set up too high to type really comfortably for me).

My daily work routine has settled into working standing in the mornings, sitting for an hour or so after lunch, standing all afternoon, and then sitting in the evening I have have to do some work. With the wonderful weather here in Southern California, I’ll often take my sitting work time outside in the back patio in the shade. So all in all, I probably spend one or two hours sitting down at the desk.

Yes, it was a significant amount of money, but it’s something I’m using every day and it improves both my productivity and my health. Definitely worth it.

Standing desks have one downside though. This situation happens embarrassingly often. Good thing I work home alone!

[1] A few years ago I gave polyphasic sleep a try. It was actually really great for gaining extra hours, but eventually I gave it up because I was constantly out of sync with the rest of the world. I’d do it again if I were spending months alone in a research station in the Antarctica.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Lag In Games

I’m defining lag as the time elapsed between the moment the player performs an input action (press a button, touch the screen, move his finger), until the game provides some feedback for that input (movement, flash behind a button, sound effect).

Mick West wrote a great article on the causes of lag in games, followed up by another one in how to measure it. I’m going to apply some of that to the lag I was experiencing in my game.

Lag can be introduced in games by many different factors:

• Delay between gathering input and delivering it to the game. Delay updating the simulation to reflect new inputs.- Delay rendering simulation state.
• Delay displaying the latest rendered state on screen.

The new game runs on the iPad and involves moving objects around the screen with your finger. To make sure it wasn’t anything weird with the rest of the game code, I wrote a quick (and ugly!) program that draws a square with OpenGL that follows your finger on the screen. When you run the sample, the same lag becomes immediately obvious.

The iPad is a much larger device than the iPhone, and it encourages a physical metaphor even more. As soon as you attempt to move an “object” on screen, the lag kills that sense of physicality. Instead of moving an object around with your finger, you’re dragging it around with a rubber band. It moved the player from applying direct action on screen, to being removed and disassociated with the actions on screen.

Loop Structure

The place to start looking for lag is in my main loop. The main loop looks something like this:

	ProcessInput();
	UpdateSimulation();
	RenderWorld();
	PresentRenderBuffer();

So I was reading the input correctly before the simulation. Nothing weird there.

Touch input is delivered to the code as events from the OS. Whenever I received those events (outside of the main loop), I queue them, and then process them all whenever the main loop starts in ProcessInput().

The loop runs at 60Hz, so the lag here is at most 16.7 ms (if you’re running at 30Hz, then you’re looking at a delay up to 33.3ms). Unfortunately, the lag I was seeing in the game was way more than one frame, so there was to be something else.

Rendering

For some reason, I thought that iDevices were triple buffered. I ran some tests and fortunately it looks like it’s regular double buffering. That means that if I render a frame and call presentRenderBuffer(), the results of that render will be visible on screen at the next vertical sync interval. I’m sure there’s a slight lag with the iPad screen, but I’m willing to be is close to negligible when we’re talking about milliseconds, so we’ll call that zero.

Main Loop Calls

The game uses CADisplayLink with interval of 1, so the main loop is called once every 16.7 ms (give or take a fraction of ms). I thought that perhaps CADisplayLink wasn’t playing well with touch events, so I tried switching to NSTimer, and even to my old thread-driven main loop, but none of it seemed to make any difference. Lag was alive and well as always.

That the simulation and rendering in the game are very fast, probably just a few ms. That means the rest of the system has plenty of time to process events. If I had a full main loop, maybe one of the two other approaches would have made a difference.

It looks like the lag source had to be further upstream.

Input Processing

On the dashboard, press and hold on an icon, now move it around the screen. That’s the same kind of lag we have in the sample program! That’s not encouraging.

A touch screen works as a big matrix of touch sensors. The Apple OS processes that input grid and tries to make sense out of it by figuring out where the touches are. The iOS functions eventually process that grid, and send our programs the familiar touchesBegan, touchesMoved, etc events. That’s not easy task by any means. It’s certainly not like processing mouse input, which is discrete and very clearly defined. For example, you can put your whole palm down on the screen. Where are the touches exactly?

TouchesBegan is actually a reasonably easy one. That’s why you see very little lag associated with that one. Sensors go from having no touch values, to going over a certain threshold. I’m sure that as soon as one or two of them cross that threshold, the OS identifies that as a touch and sends up the began event.

TouchesMoved is a lot more problematic. What constitutes a touch moving? You need to detect the area in the sensor grid that is activated, and you need to detect a pattern of movement and find out a new center for it. In order to do that, you’ll need several samples and a fair amount of CPU cycles to perform some kind of signal processing on the inputs. That extra CPU usage is probably the reason why some games get choppier as soon as you touch the screen.

Measuring Lag

Measuring lag in a game is tricky. You usually can’t measure it from within the code, so you need to resort to external means like Mick did in his tests.

I decided to do something similar. I pulled out my digital video camera, and started recording my finger moving on the screen. The quality leaves much to be desired, but it’s good enough for the job. I can see how far my finger gets from the center of the square, but that’s not enough information to quantify the lag. How fast is my finger moving exactly? Fortunately, that’s something I can answer in code, so I added that information to the screen [1]. Now, for a given frame, I can see both how far the finger is from the center of the square and how fast it’s going.

The square is 100 pixels wide. When I move my finger at about 500 pixels per second, the center of my finger is on the edge of the square. That makes a rough 100 ms total delay from input until it’s rendered. That’s a whopping 6 full frames at 60 fps!

What Can We Do About It

As iOS developers, there isn’t much we can do. Make sure your loops are set up correctly to avoid an extra frame delay. Make sure you provide feedback as soon as you can and don’t delay it any longer than you have to. Other than that, there’s nothing much we can do.

I’ve been saying this for a while, but I’m a big fan of layers as long as you can get to the underlying layers when you need to. Here’s a perfect case where it would be fantastic if Apple gave us raw access to the touch matrix input. Apart from being able to process the input faster (because I know what kind of input to expect for the game), can you imagine the possibilities that would open up? Input wouldn’t be limited to touch events, and we could even sense how “hard” the user is pushing, or the shape of the push.

At the very least, it would be very useful if we had the option to allocate extra CPU cycles to input recognition. I’m not doing much in my game while this is going on, so I’d happily give the input recognition code 95% of the frame time if it means it can give me those events in half the time.

I’m hoping that in a not very far distant, iDevices come with multiple cores, and maybe one of those cores is dedicated to the OS and to do input recognition without affecting the game. Or maybe, since that’s such a specialized, data-intensive task, some custom hardware could do the job much faster.

Until then, we’ll just have to deal with massive input lag.

How about you? Do you have some technique that can reduce the touch event lag?

LagTest source code. Released under the MIT License, yadda, yadda, yadda…

[1] I actually shrank the OpenGL view to make sure the label wasn’t on top if it because I was getting choppier input than usual. Even moving it there caused some choppiness. This is exactly what I saw last year with OpenGL performance dropping when a label is updated every frame!

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Flower Garden was using a shared Dreamhost server to upload bouquets, host in-game news, and, most importantly, to serve the Flower Shop (which involves hosting the static shop files, redeeming promo codes, recording transactions, verifying receipts with Apple’s server, and delivering purchased content). All the Flower Shop functionality was implement in PHP and using a mySQL database.

After some stressful sorting through the logs, it looked like the server was running out of memory and killing PHP processes. Dreamhost’s technical support didn’t help much and just claimed that I was simply something that happened and they would be happy to sell me a virtual private server. I was surprised to hear it was using up a lot of memory, and certainly nothing I could see with the tools available to me showed that. I certainly hadn’t done any changes in a while and traffic was constant. Doing some Googling at the time brought up lots of other angry Dreamhost users in a similar situation, so I suspected server configuration problems.

Whatever the case, I couldn’t let it be that way and I had to fix it somehow. I briefly considered a virtual private server, or even a dedicated host in some highly recommended providers. But in the end, the simplicity, scalability, and affordability of the Google App Engine won me over.

Now that Flower Garden has been using the Google App Engine for several weeks, I still think it’s fantastic. I wish someone had whacked me on the head when I started writing server code and forced me to use it. Hopefully this post will be more pleasant than a hit to the head and will still have a similar (good) effect.

Google App Engine Overview

Even though I was pretty much completely new to the Google App Engine (I had looked into it briefly before the launch of Flower Garden but dismissed it because they couldn’t support the amount of email traffic I needed), it took me two days to port over everything. Actually flipping the switch from the old system to the new one took a while longer, but that required more courage than work. More on that later.

This is not a Google App Engine tutorial. Instead, it’s going to be an overview of what it can do and why you should consider it as a backend for your iPhone app instead of using a shared or even dedicated server.

The first shock (at least for me) when looking into the Google App Engine is that you can’t run PHP. You’re limited to Java or Python. It turns out Python is my go-to scripting language, so I was thrilled at the idea. PHP is fine, but Python is a whole class above.

Next, it took a bit of adjusting to the fact that the environment to run anything on the Google App Engine has to be carefully controlled. You need to create a configuration file indicating the entry points and their handlers. Fortunately, all of that is well covered in the extensive online documentation.

This is what the Flower Garden config file looks like, indicating both static files and several execution entry points:

application: st-flowershop
version: 1
runtime: python
api_version: 1

handlers:
- url: /catalog
  static_dir: catalog

- url: /emailassets
  static_dir: emailassets

- url: /news
  static_dir: news

- url: /moregames
  static_dir: moregames

- url: /.*
  script: purchase.py

If you hit one of the URLs listed as static_dir, you get those files directly (like the in-game news). Anything else is handled by the Python script purchase.py. Google even provides the webapp framework that easily allows you to connect different parts of the script to handle different requests and easily get to the input parameters.

Once you’re past that and you have the requisite “Hello World” up and running, then it’s all fun and games.

One of the great features of the Google App Engine is that it comes with a fully-featured, local environment. This environment is installed automatically with the Google App Engine SDK, and there’s nothing to configure. It’s certainly nothing like setting up Apache + mySQL + PHP in your dev station! That way, you can do all your work locally, and only update the servers when you’re confident everything is working.

Beyond this, the only other thing that is different from what you may be used to is the datastore. They provide a query language called GQL, which is not exactly SQL, but it’s very similar. Close enough that I had no trouble porting over my very simple queries anyway.

Adding data to a data store couldn’t be simpler. For example, to add a promo code, I need the following class definition:

class PromoCode(db.Model):
	code = db.StringProperty()
	productid = db.StringProperty()
	amount = db.IntegerProperty(default=1)
	singleuse = db.BooleanProperty(default=True)
	enabled = db.BooleanProperty(default=True)

And then I can add it this way:

	promocode = PromoCode()
	# fill it up here
	promocode.put()

How do I check if a promocode is valid? I can use the SQL-like syntax, or something even simpler:

	q = PromoCode.all()
	q.filter("code =", code)
	return q.get()

Once you know that, you can go to town with your Python programs.

Google App Engine Advantages

Scalability This was the big reason that forced me to move away from Dreamhost. I have no way of testing how scalable it really is, but I’m going to take Google’s word for it. They know a tiny little bit about scalability. And besides, Flower Garden is nothing compared to other apps (it serves about 1.5 requests per second on average).

Since I don’t know how things are implemented under the hood in the Google App Engine, I don’t have a good feel for performance best practices. So I’m hoping that my very simple queries (add to the end of a table, or see if a record is present) are just fine. If not, I can do some tuning down the line (and I won’t have to wait for any approval to make the code go live!).

Local environment The local environment is simply great. It makes developing a pleasure. You can add any test data you want to the local datastore through a web interface without affecting the live server. Probably my favorite feature is the console: Since the server is running locally, you can print out any debug info in the server code and you can see it live as you perform some action. That saved me hours of debugging compared to doing it on a remote server with php!

Server synching In the past, I’ve synced my server scripts through ftp and I kept meaning to write an automated script to do that. In this case, Google provides a script (and a GUI tool) to do the actual synching with the servers, which is great. It even waits a few seconds until it can verify that the new version is live on the server.

Python I’m not a big Java fan (although I used it a bit way back in the day when it first came out), but I do love Python. For a program with some complexity like this, it’s definitely a much better choice than PHP.

Real-time stats The Google App Engine web console displays real-time stats of access to your site. You see it expressed in requests per second and you can visualize it in the past few hours, days, or even month. You also have access to all the logs, and you can filter by type of log message (debug, info, warning, error).

Price This is one category where the Google App Engine definitely shines. Every day you’re given a free quota of bandwidth, disk usage, and CPU usage. If you stay under that quota you don’t pay anything. If you go over, you pay per unit as you go (at very reasonable prices). You can even set a maximum cap on daily spending so you don’t encounter any nasty surprises at the end of the month.

So far, Flower Garden has been hovering right at the edge of the outgoing bandwidth free quota (the rest of the metrics it doesn’t even come close). So even if traffic were to double, expenses would be very reasonable.

Development Tricks

Along the way, I figured out a couple of interesting tricks that helped me during development.

Bypass the App store for testing For me, few things are more annoying than doing development in the actual device. Iteration is slow and stepping with the debugger is mostly impossible (seems to depend on the mood of the debugger that day). So I try to do the bulk of the development on the simulator and just test on the device when things are ready.

Unfortunately, the device can’t access the App Store, which makes testing all the server functionality dealing with the App Store painful. What I did was to add a define that, when present, the program bypasses the App Store but still functions as usual.

Local server During development, you want to be using your local server. In my case, I set it up so that Debug versions of the code access the local server, but Release and Distribution ones access the live one.

There is one problem: The Google App Engine server, by default, binds itself to localhost, not to the IP address of your network card. That meant I was able to hit it from the simulator just fine accessing http://localhost:8080, but not from the device. In order to access the development server from the device, I had to explicitly tell the server to bind itself to another address like this:

dev_appserver.py --address 192.168.1.150 FlowerShop

Now I can access the server at 192.168.1.150:8080 from both the simulator and any device.

Switching servers Porting the code took just a couple of days. Making the switch on the live server was a lot scarier though. I couldn’t wait for a new update to be released because that seems to take over a week these days, so I forwarded requests from the old server to the new one.

I did it one system every day, avoiding weekends which is when there’s the most traffic. News was easy, and so was the web view with more games. But I ran into a snag with promo codes and Flower Shop purchases.

Initially, I was just redirecting requests in .htaccess this way:

Redirect permanent /Shop/catalog http://st-flowershop.appspot.com/catalog
Redirect 301 /Shop/catalog http://st-flowershop.appspot.com/catalog

As I learned the hard way, POST variables aren’t preserved in a redirect of that kind (I guess to avoid state being changed in multiple servers). I tried a bunch of things, but in the end, I wrote a quick php script that gathered all POST variables and re-posted them to the new request. A bit ugly, but it did the trick:

<?php
function PostRequest($url, $_data) {

    $data = array();    
    while(list($n,$v) = each($_data)){
        $data[] = "$n=" . urlencode($v);
    }    
    $data = implode('&', $data);
    // format --> test1=a&test2=b etc.

    $url = parse_url($url);
    if ($url['scheme'] != 'http') { 
        die('Only HTTP request are supported !');
    }

    $host = $url['host'];
    $path = $url['path'];

    $fp = fsockopen($host, 80);

    fputs($fp, "POST $path HTTP/1.1\r\n");
    fputs($fp, "Host: $host\r\n");
    fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
    fputs($fp, "Content-length: ". strlen($data) ."\r\n");
    fputs($fp, "Connection: close\r\n\r\n");
    fputs($fp, $data);

    $result = ''; 
    while(!feof($fp)) {
        $result .= fgets($fp, 128);
    }
    fclose($fp);

    $result = explode("\r\n\r\n", $result, 2);
    $header = isset($result[0]) ? $result[0] : '';
    $content = isset($result[1]) ? $result[1] : '';
    return $content;
}

print PostRequest("http://st-flowershop.appspot.com/purchase", $_POST);
?>

Conclusion

I hope this post is enough to at least make you interested in checking out the Google App Engine and help you get over a couple of the initial hurdles. I’ll definitely be using it in any future projects.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Yesterday things changed a bit. As a result of last week’s “growing” post, Manuel Freire is going to join me to help with Flower Garden development. That makes two of us banging on the same codebase, and from two different time zones, so we don’t get the benefit of being in the same closet as it was the case with Power of Two Games. Since I was in my get-things-done mindset, I figured I would just set up a new svn repository for the project, move over the Flower Garden data, give us both access to it, and move on.

But no, it couldn’t be that easy. Can you guess what the first words out of his mouth were when I asked him about version control? “Oh, I love Git!”

That was the last straw. I had to do a mini-research session on version control systems, so I spent a couple of hours looking into it. If we were going to move over to something, now would be the time to do it.

Git and Mercurial

Git and Mercurial both look great. I was debating which one to go with until I realized that they’re both two flavors of the same thing, so it comes down more to personal preferences and tastes. This is best description I found on the differences between Git and Mercurial. When it comes to computers, I’m totally a MacGyver guy (actually, that might be true when it comes to anything now that I think about it), so that made my decision easier.

The big feature everybody keeps talking about for distributed version systems is effortless branching. That’s great, but I really have no intention of branching much. I haven’t created a single branch in the last four years, and I don’t expect to start doing that now. Next.

The other big feature is working disconnected from the network. That’s something I could use, but considering I’m only offline a handful of times a year, it really isn’t enough of an incentive to switch to a whole new system.

Git sounds like a great tool for large, distributed, open-source projects with hundreds of contributors, but frankly, I couldn’t find anything else that was worth mentioning for a small project and a handful of people. I feel like someone is trying to sell me a Porsche when my beat-up Hyundai is still perfectly functional for driving to the grocery store once a week. Am I missing something obvious?

Hosting

Hosting the actual repository was part of the consideration. This is for a private project, so all open-source sites are out. Ideally I wanted to host it just like I do with Subversion in Dreamhost, but the instruction page on how to install Git and Mercurial are enough to put most people off. Clearly, there’s a steep learning curve there.

So I asked on Twitter for recommendations. I’ve learned that Git and Mercurial users are definitely very vocal and are always willing to help someone join their ranks. Within minutes I had all the suggestions listed below:

• Github (Git)
• Repositoryhosting.com (Git, Hg, SVN)
• Bitbucket (Hg)
• Klin (Hg)
• CodaSet (Git)
• Assembla (Git, SVN)
• Unfuddle (Git, SVN)
• Beanstalkapp.com (Git, SVN)
• SourceRepo (Git, Hg, SVN)
• Plain Dropbox (Git I guess)
• Self-hosting (gitosis for managing Git)

Prices varied a lot. From the $25/month/user of Kiln, to the $6/month of RepositoryHosting (gets you unlimited users and 2GB of storage). The Snappy Touch repository is already over 2GB, so it would end up being a bit more expensive than that, but not too bad.

Of those, Github was definitely the most recommended. I was starting to feel Git might not be the one for me, so I looked a bit more into RepositoryHosting because they had Subversion support. It turns out they also provide Trac, which is a great tool, although I already have that set up myself.

Wish List

Binary files

The one thing I really want in a version control system is good large binary file handling. I check in everything under version control, source code, assets, raw assets, and even built executables for each version. I want to be able to throw multiple GB psd files in the repository and have it work correctly (meaning fast, and using the least amount of space possible).

Perforce did an OK job with that. Git and Mercurial apparently are both horrible at it. So is Subversion, but at least it’s a tool I already know and I don’t have to spend time learning the ins and outs of how to optimize the Git database or how to make backups, or cull unused trees.

GUI

I love my command-line tools. I live in Terminal for a good part of the day, and having a real shell is one of the things that makes my life so much more pleasant under Mac OS than under Windows. But there are some things for which a GUI tool is a really useful addition, and version control is one of them.

On the Mac, I’ve been using Versions as a client for Subversion and it does everything I want. It’s fast, handles multiple repositories, lets me browse history, diff changes, etc. From my brief search and other people’s comments, there’s nothing quite like that for Git or Mercurial yet. That’s a pretty big, gaping hole.

Low-level access

Looking at all those hosting providers, I realized how much I want to have low-level access to the database. I want to be able to back it up myself, and run svnadmin when I want to. A lot of those hosting sites looked really pretty, but you were very limited in what you could do.

Coming To A Decision

If this were a thriller, you’d be disappointed. I’m afraid there are no plot twists and you can already guess the conclusion.

In the end, since neither Git nor Hg are built to address my biggest need (large binary files), I’ll stick with svn. It might be old, it might not be cool, but it serves my needs, I already know how to use it, I have the tools, I can admin it and fix a problem. I can concentrate on what matters instead: Writing games.

I decided to continue hosting it myself on Dreamhost. I can easily have one repository for every major project. However, by default, Dreamhost creates svn repositories using htaccess security and HTTP protocol. That’s OK, except that none of the actual data is encrypted as it would be if I were using ssh. I could use HTTPS, but then I would have to set up a certificate and pay for a fixed IP address, so instead I found an alternate way to have a secure connection.

All Subversion repositories live in a user account (svnuser). I create a new user group for every repository, and change all the files in the repository to belong to that group. Make sure you also set the SGID bit so any files created in that directory still belong to the group. Then I can create a new shell user for every collaborator, and add him to the groups of the repositories I’d like him to have access to. At that point, he’ll be able to access the repository as svh+ssh://username@hostname.com/home/svnuser/repository. All safe and secure.

Bonus SVN-Fu

Here’s something that I learned yesterday while I was moving repositories around. It’s probably common knowledge for seasoned SVN admins, but it was new to me.

I had a repository that included a bunch of my projects. What I wanted was to create a new repository that still had all the history, but only for the FlowerGarden part of the tree. I knew about svnadmin dump for transferring whole repositories, but I didn’t know there was a very simple way to only transfer part of it.

First you need to dump it as usual:

svnadmin dump repository > repos-dumpfile

Now, it turns out you can process the dump file before adding it back to another repository. So we can do:

svndumpfilter include FlowerGarden --drop-empty-revs --renumber-revs < repos-dumpfile > fg-dumpfile

Finally, you can add the resulting dump file into a fresh repository and have all the history for that project and only that project:

svnadmin create flowergarden
svnadmin load --ignore-uuid flowergarden < fg-dumpfile

Amazingly, for a repository that was over 2GB, that only took a few minutes. Go Subversion!

[1] Or reading. Or going for a walk. Heck, even sleeping would be a better use of my time than futzing around with a new tool.[2]

[2] And yes, I realize I sound like a grumpy old man. Getting there apparently. Now get off my lawn.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Markus suggested that about 20% of developers fall in that middle class, but my gut feeling, when it comes to iPhones and games, is that it’s more like 5-10%. But it’s just a made up number based on personal observation anyway. It would be very interesting to conduct some sort of survey (or analyze the App Store data), but I fear the results would get muddled up due to the differences between full time indies, hobbyists, and big companies.

Snappy Touch falls squarely in the developer’s middle class. I’ve been very lucky and Flower Garden’s sales have been remarkably stable, hovering at around $2,000 per week (and spiking up during promotions and new updates).

For the 90% of developers that don’t make their money back, their choices are limited to either stopping, or digging deeper in their pockets (or somebody else’s pockets) and try again. For the 0.1% that hit it out of the park, they bring in so much money they can pretty much choose to do anything they want without risking the company.

For us middle-class developers, things are tougher. We have two choices:

• The first course of action is plodding along doing what we’re doing, making a reasonable living and putting some money aside. We can build our personal and business nest egg, and then a bit more. And we can love every minute of it.
• Choice number two is to take any spare money and reinvest it in the company. And in the case of iPhone development, that can only mean getting more people involved creating the games.

The first choice is nice and safe. We can keep doing what we love, making a living from it, and even saving some money. Assuming the App Store doesn’t collapse overnight, we might be able to pull that off for a few more years. But it has a horrible hidden cost: The opportunity cost.

Most long-term, successful apps will require a fair amount of updates. New content keeps users interested, and they also expect support for new hardware (iPad, iPhone 4, etc). All the time I spend creating updates for Flower Garden is time I don’t spend making a new game. At the top of my list of hundreds of game ideas, I have four or five that I know will be successful, but the bottleneck from making them happen is my bandwidth. I can only do so much by myself.

That’s why I’ve decided that Snappy Touch needs to grow. Mind you, I’m not talking big corporation, I don’t ever want to even get to 20 people. But I would love to eventually be able to have a small team working on a new project and a few other developers maintaining and updating existing projects. I envision it happening mostly as distributed development and not in a traditional office setting.

The problem is how to start. Going from one to two people is probably the hardest step in growing a company. It’s a 100% increase! That’s probably another reason why successful startups often have three people involved from the start: Adding a fourth person is “only” a 33% increase in size, which seems more manageable.

Adding another person is also scary from a money point of view. It’s going from saving just a bit of money, to potentially spending it all so that maybe we can produce more games and make more money in the end. That’s a lesson I learned very clearly in Dope Wars: You need money to make money. To get crazy scores in that game, you had to take a huge loan out from the start (and then be really lucky). Except that in this case there’re no loans (I’m totally self-funded). And it’s also not a game, it’s real life.

Having said all of that, I’m going to turn this post into a recruiting tool (which is great because it self-selects the target audience to people who read this blog or follow me on Twitter).

Position Description

[Edit: Thanks for the overwhelming response! I already have enough candidates and the tricky part is selecting just one! I’ll post again whenever a similar opportunity opens up. Thanks!]

I’m looking for an programmer intern/part-time entry level position. Later on, if things work out, it could become a full-time position. I’m looking for someone who can dive into the Flower Garden source code and quickly be able to start maintaining it and adding features. I’ll definitely remain involved with the project, but I’ll be mostly setting the direction and working on the harder bits. I expect us to be in contact on a daily basis, and set up a quick iChat call once or twice a week (or if you’re local we can work half a day a week together).

Requirements

• Very familiar with iPhone development (you should have some apps under your belt).
• Very familiar with Objective C and the UIKit framework
• Good knowledge of C (and a tiny bit of C++)
• Available to work 10-20 hours per week. This is very flexible.
• Bonus points for knowing Python or having used the Google App Engine.
• I’d prefer someone who can work for several months (and maybe longer term).
• Local to San Diego would be great, but not a requirement as long as we can voice chat easily.

Compensation is hourly and based on experience, but remember this is an intern/entry level position. Possible bonuses based on performance and sales. Sounds like something you’d be interested in? Drop me a note and convince me you’re the right person for the job..

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Every so often, something happens that bumps up the priority of one of the items in my list. Maybe it’s another related blog post, or a game coming out with something relevant to what I wanted to write about. In this case it was a tweet from Paul Pridham [1]:

Customizing colors in a sprite or texture is very frequent in games, from changing player characters into blue and red teams, to creating color variations of an armor piece, to letting the player pick the exact shade for their pet’s fur color. Or, in the case of Flower Garden, to change the colors of the petals on the fly.

There are two requirements for this:

• We want to change colors dynamically.
• We only want to affect certain areas of the original texture.

That rules out creating texture variations ahead of time, although that might be a valid approach sometimes if you have lots of art resources, don’t mind increasing the download size, and you have a fixed number of variation to deal with. It also rules out modulating/blending the texture by a particular color because it would tint all the texture, and we want to limit the effect to particular areas (leave the player’s arms their normal color, but change their shirt color).

This is one of those funny cases that it was a lot easier to do many years ago, when we used palletized color modes. You could set all the custom color areas to a particular palette entry, and then update that entry on the fly. Ah, all the awesome tricks palettes opened up the door to! I still miss them to this day.

In modern hardware it’s also really easy to do with a shader, but Paul wanted to use it across any iPhone device, and the majority of them are still stuck on OpenGL ES 1.1, so fixed-function pipeline it is.

The simplest approach would be to just render the model twice: First pass renders the texture, and second pass renders the custom color bits (you can render them with a white texture modulated by the global color to get the right color). The main drawbacks are that you’re doubling the number of draw calls, and, with 3D objects, it gets a bit tricker because the second pass needs to use the glDepthFunc(GL_EQUAL) depth comparison function.

The better way to do this is using the texture combiners. Texture combiners allow us to perform a limited number of operations to control the final look of a pixel. We can add two textures, or multiply them, or even do a few more complex operations. The true power of the combiners is that they can be chained together, so the output from one feeds into the input of another, allowing us to create much more complex operations.

The iPhone 3G is limited two two texture combiner units [2], but even two combiners are good to create a good range of effects.

Let’s think about what we want to accomplish. We want to leave some parts of the texture completely alone and display the original pixel value. In some other parts of the texture, we want to replace the pixels there with a custom color. Actually even better, we probably want to multiply those pixels by a custom color. That way we can author the part of the texture that is going to change with grayscale details, and our color adds the tint to it.

Let’s express it mathematically. Let’s make a function M that is 1 for every pixel we want to color, and 0 for the ones where the original texture is supposed to be displayed. Our desired color is c and the texture it t. In that case, the final pixel color (p) is:

p = M*(c*t) + (1 - M)*t

We could express that with two combiners: The first one is a modulate (multiply) operation with c and t, and the second one an interpolation operation between the result of the previous combiner and the original texture, based on the function M.

Obviously M is just a mask texture. We can paint it white where we want to color the texture, and black elsewhere. We could even use the alpha channel of the original texture, but there’s one big thing to watch out for: If you have your texture as a png and process it through the default iPhone resource operations, the image will be premultiplied for you (whether you want it or not), so your color information will be set to zero everywhere that the alpha channel is zero. Oops. You’ll probably want to use the alpha channel to store transparency anyway, so we’ll keep the mask separate. If not, make sure you encode the image yourself (as raw or PVRT formats) so it’s not premultiplied ahead of time.

Are we ready transfer that formula to the texture combiners? Not quite. Apparently (and this was just trial and error, I haven’t seen it documented), the texture assigned to a combiner can only be the one at that stage. If you look at the second combiner, we would need to have the first texture as one of the parameters, in addition to the mask.

So instead, we can reorganize the function above like this:

p = c*(M*t) + (t - M*t)

What did we gain by that? The color is what’s going to change dynamically, but the mask and the texture always stay the same. We could precompute the M*t term by simply multiplying the texture and the mask. We can call that new term A. We can do the same thing with the (t - M*t) term, which just means turning black all the pixels in the texture where mask will go. That one will be B. The easiest way to “precompute” those values is just doing it in Photoshop and exporting it as a new png.

Our new formula is now:

p = c*A + B

Nice and simple! Now we can really add that to the texture combiners like this:

// c
glColor4f(m_customColor.r, m_customColor.g, m_customColor.b, 1);

glActiveTexture(GL_TEXTURE0);
// A = M*t (precomputed)
glBindTexture(GL_TEXTURE_2D, m_maskHandle);
glEnable(GL_TEXTURE_2D);
// c*A
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);

glActiveTexture(GL_TEXTURE1);
// B = t - M*t (precomputed)
glBindTexture(GL_TEXTURE_2D, m_textureHandle);
glEnable(GL_TEXTURE_2D);

// c*A + B
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_COMBINE);
glTexEnvi(GL_TEXTURE_ENV, GL_COMBINE_RGB, GL_ADD);
glTexEnvi(GL_TEXTURE_ENV, GL_SRC0_RGB, GL_PREVIOUS);
glTexEnvi(GL_TEXTURE_ENV, GL_SRC1_RGB, GL_TEXTURE);

One more thing to watch out for: Because we’re using two textures, you need to have two sets of texture coordinates. In this case, we want them to be the same, so we can just point them to the same set of data:

glClientActiveTexture(GL_TEXTURE0);
glTexCoordPointer(2, GL_FLOAT, sizeof(Vertex), &vertices[0].u);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glClientActiveTexture(GL_TEXTURE1);
glTexCoordPointer(2, GL_FLOAT, sizeof(Vertex), &vertices[0].u);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);

That’s it! You can see the results in the included project and play with the register combiners to achieve different operations.

At this point I was going to describe the texture combiner setup I use in Flower Garden to render the petals, but this post ended up taking longer than I had hoped for (I’m trying to shoot for an hour per post, but this has taken me already two hours between the code and the the post itself), so I’ll save that for another time.

• Demo source code. Released under the MIT license.

[1] Paul developed Sword of Fargoal, by far my favorite iPhone RPG. [2] The 3GS allows up to eight I believe.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

I want to have great experiences, not just long ones, and I’m willing to pay for them. Just like other media, each game has an ideal length to develop its ideas fully, and we shouldn’t fall to pressures to make then any longer than they should be. I don’t want endless grinding or backtracing through the level to meet a minimum time quota. Just like every book doesn’t have to be The Lord Of The Rings, there is room for both epic sagas like Dragon Age and Fallout and short experiences like Limbo or Braid. Maybe there’s even room for the equivalent of Italo Calvino’s shortest short story, The Dinosaur.

I would go as far to say that games should err on the side of being too short rather than too long. There’s a saying in Spanish that goes “Lo bueno, si breve, dos veces bueno”, which roughly translates into “Good things, if short, twice as good”. I’d rather be left wanting more than not being able to finish something. As a busy gamer, I don’t get to finish many games these days, and the ones that I do really stand out from the others and hold a special spot in my heart.

Besides, a lot of other games can’t even be measured in length. How long is Starcraft? Team Fortress? Civilization? Tilt To Live? They are as long or as short as you want. Should we be docking them for being too short? Should we force all games to have infinite replay value? No, the important thing is that they’re great experiences.

Forget length. Give me awesome.

Other Indie Voices On Game Length

• Alex Amsel of Tuna
• Alex Okafor of One Man Left
• Brett Douville of Bethesda Softworks
• Chris DeLeon of HobbyGameDev
• Chris Hecker of Spy Party
• Cliff Harris of Positech Games
• Dave Gilbert of Wadjet Eye Games
• Eitan Glinert of Fire Hose Games
• Greg Wohlwend of Intution Games
• Gavin Bowman of Retro Dreamer
• Jamie Fristrom of Torpex Games
• Jeffrey Rosen of Wolfire
• Jonathan Blow of Number None
• Markus Nigrin of The Pocket Cyclone
• Martin Pichlmair of Broken Rules
• Matt Gilgenbach of 24 Caret Games
• Peter Jones of Retro Affect
• Rob Jagnow of Lazy 8 Studios
• Ron Carmel of 2DBoy
• Scott Macmillan of Macguffin Games
• Steve Swink of Enemy Airship

Twitter tag: #gamelength

A good game prototype is something fast/cheap that allows you to answer a specific question about your game. The key points there are fast/cheap and specific question. It’s not a level of a game, it’s not a “vertical slice”, and it’s certainly not an engine for the game.

Chris Hecker and Chaim Gingold gave one of the best presentations on the subject of rapid prototyping. It was hugely influential for me, and it made me re-think the way I do prototypes. If you get a chance, find the audio for the presentation, it’s definitely worth it.

Mistake #1: Going With The First Idea

Every company I’ve ever worked at has done this mistake. The team hashes out a bunch of ideas, and somehow they pick one (or create it by committee). Maybe they’ll create a prototype to show something about the game, or maybe they’ll dive straight and start writing a design document and developing technology. If you’re lucky, or you have an extremely talented game director, the game that comes out of the other end might be fantastic. In most cases, it’s just a so-so idea and the team only realizes it when the first level comes together, years later, at around alpha time. At this point the choice is canning a project after spending millions of dollars, or patching it up to try to salvage something. Neither idea is particularly appealing.

Creating a prototype for a game you know you’ve already committed to is pointless. It’s nothing more than an exercise to keep management happy. Frankly, I even made that same mistake at Power of Two, when we prototyped the game idea we had in mind and immediately moved on into pre-production (and yes, later we realized we had to change things to make it more interesting).

What I do now is to force myself to prototype several of my top ideas before committing to any one project. I have a page (actually, a wiki page) with every game idea or thought I have. That page has way over a hundred entries, and every so often I cull and reorganize it, bringing up the most promising ideas towards the top. Whenever I’m in prototyping mode, I start grabbing them from the top and prototype them.

With a good prototype it’s easy to see if an idea is worthwhile. If it’s not, I discard it and move on to the next one. If it has potential but it’s just so-so, I either choose to continue just a bit longer (to ask another, better question) or I shelve it back in the list of potential game ideas. Maybe at some later time, things might click in or I might have a new inspiration and the game idea might become a lot stronger.

Also, often times, after doing one prototype and deciding against it, a new idea will come up. Usually a variation on the original prototype or something directly sparked from it, so I’ll find myself jumping to that idea instead of one of the ones I had saved in my list.

Eventually, one idea will click and you’ll know that’s “the one”. If you’re lucky (or unlucky) enough to have that happen with the first one you try, I still recommend doing a few more prototypes. If nothing else, you might be prototyping future projects, so it’s certainly not wasted time. For my current project, I went through eight prototypes before finding “the one” (several of them were a collaboration with Miguel). Eight to ten prototypes per project is roughly what I’m hearing from other indies with this approach.

Mistake #2: Not having a good question

A good prototype attempts to answer a question about the game. But not all questions are created equal. First of all, a question needs to be relevant and crucial to the project. “Can I have a pretty settings screen?” isn’t a particularly difficult question to answer, so it doesn’t deserve its own prototype. “Can I control little planes by drawing smooth lines on the screen?” is a much bigger unknown (before Flight Control anyway, today you can just download the game and immediately answer yes).

Also, a good question is concise and can be answered in a fairly unambiguous way. “Is this game awesome?” isn’t a good question because “awesome” is very vague. A better question might be “Can I come up with a tilt control scheme that is responsive and feels good?”. Feels good is a very subjective question, but it’s concrete enough that people can answer that pretty easily after playing your prototype for a bit.

Even though most questions are about game design, they can also be about any other aspect of the game. Maybe you’re doing something tricky with technology and you want to make sure it’s feasible. If not, there’s no point in even starting. It’s more uncommon to think of prototyping art, but it’s also a very valid approach: “Will this art style allow foreground objects to stand out enough?” “Will the lighting approach allow the player to see important features in enough detail?”. Often you can do these art “prototypes” directly in Photoshop or a 3D modeling package.

In the case of Flower Garden, the main unknown was the technology behind the procedural flowers. So I created a prototype to answer the question “Can I create compelling procedural flowers that grow in real-time and the user can interact with them?”. The prototype had several parts to answer that question: the geometry generation, the animation, the simulation, and the rendering. There isn’t anything else particularly ground-breaking in the rest of the Flower Garden code, so as soon as I was able to answer “yes” to that question, I green-lighted the project and started production on it.

For larger projects, you might have several major, outstanding questions, so you’ll need to do multiple prototypes. Unless they’re very closely related, I find it easier to keep them separate instead of building on top of the same prototype.

Without a good question, it’s too easy for a prototype to go on for a long time. You feel you’re making progress because new things are added, but you have no real sense of when to stop or when it’s done. You really have to focus on the question, ignore everything else, and be merciless in your approach.

Mistake #3: Taking too long

One of the key concepts in the definition of a prototype was that it has to be fast/cheap (which are two sides of the same coin). What’s fast enough? It depends on the length of the project itself. It’s not the same thing to do a prototype for a two-month iPhone game, than for a three-year console game. Also, a larger, more expensive project probably has more complex questions to answer with a prototype than a simple iPhone game.

In my case, I shoot for one-day prototypes. If you already have the tech to create games with, one day allows you to focus 100% on answering the question. By the end of the day, or even before, I have a pretty good idea how the game is going to work out. My shortest prototype ever was a 2-hour one. I thought it was going to be longer, but I managed to complete everything to answer the question in two hours (for the record, I didn’t can that idea, but I shelved it for a possible future project).

Game jams are a great way to get over the mental block of doing quick prototypes. There you are focused on making the game on a very short time, surrounded by people trying to do the same thing. I can’t think of a more fun way to prototype than that!

Also, think beyond programming. Is there a faster way you can answer the prototype question? Maybe you can use the modding capabilities of an existing game, or even do a mockup with paper moving pieces around. Don’t fall in the trap of thinking you have to code a prototype if something simpler and faster will do.

If you find that a day is not enough, take a step back and really ask yourself why you need more time. Were you getting side-tracked on things that were irrelevant to the prototype (menus, tech, art, etc)? Are you asking a question that the prototype can’t answer? Do you have the game idea clearly defined in your head?

Mistake #4: Building a system, not a game

When you’re making a prototype, if you ever find yourself working on something that isn’t directly moving your forward, stop right there. As programmers, we have a tendency to try to generalize our code, and make it elegant and be able to handle every situation. We find that an itch terribly hard not scratch, but we need to learn how. It took me many years to realize that it’s not about the code, it’s about the game you ship in the end.

Don’t write an elegant game component system, skip the editor completely and hardwire the state in code, avoid the data-driven, self-parsing, XML craziness, and just code the damned thing.

When you’re prototyping, it’s a race to answer the main prototype question. Everything is expendable. Don’t even worry about memory leaks, hardwired numbers, or how you load resources. Just get stuff on the screen as quickly as you can.

And don’t ever, ever, use the argument “if we take some extra time and do this the right way, we can reuse it in the game”. EVER.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Development tools and hardware have changed quite a bit from the times I was writing in straight Z80 assembly and saving the programs to a tape. But I’ve also changed and learned a lot during all those years developing games, and even though I’m writing games by myself again, I’m doing things very differently from how I did them back at the start.

One thing that I’ve always done is to question everything. Why should I do things a certain way? Why is that the “accepted” way of doing something? And not surprisingly, at each step of the way, I’ve changed my development style to match my situation (often in ways that went against the “common wisdom”).

When it comes to solo development, I rely heavily on the concept of goals and iterations at multiple levels:

• Immediate (< 1 minute): Prioritizing the ideas going through my mind. Writing tests. Writing code.
• Short Range (< few hours): Tasks that move the project forward in some way.
• Mid Range (< 2 weeks): “Stories” that define a self-contained, significant part of the project.
• Long Range (full project, several months): Ship date, beta testing, etc.

It turns out, I use a different set of tools to help me manage the items at each level of the development cycle.

Immediate

These are the actions I take and complete in less than a minute or two. Most of them are writing tests (with UnitTest++, of course), writing code to make those tests pass, refactor code, and check it into Subversion. I have my Subversion repository hosted on Dreamhost and I access it through SSH so it’s secure, accessible from anywhere with an internet connection (or 3G signal and HandyLight), offsite, and easy to backup. And because Subversion works great offline (unlike Perforce), it’s not a problem to work without connection to the repository for a while.

I also need to manage my minute-to-minute thoughts, write down ideas and reprioritize them when the time comes. When I’m “in the zone”, I get way more ideas than I can execute with my fingers: This function needs to be moved to a different file, I really should be compacting that data over there, who was stupid enough to name this file this way?, that variable shouldn’t be cached, etc. If I don’t write things down, I will either forget them, or I’ll stress for hours until I finally get around to doing them. I could also do things as I think about them, but then I would be chasing a rabbit down a neverending hole and wouldn’t get any work done (I’m sure anybody who’s gotten lost browsing web pages can identify with that).

I used to do this the old-fashioned way, simply with paper and pencil (like Bob described in his blog post). However, I found that physical paper and pencil was just too limiting: I can type a lot faster than I can write, switching to writing requires moving away from the keyboard, and, most importantly, I need to bring the notes with me everywhere and it’s very difficult to rearrange, sort, or coalesce them in any way.

So instead, my tool of choice these days is JustNotes. It’s perfect for jotting down thoughts in a matter of seconds without even interrupting my train of thought. I have JustNotes bound to a global key, so in the middle of typing a line of code, I can press that key, enter whatever I’m thinking about, press the key again, and finish the line of code. All in 4-5 seconds. Don’t laugh: The fact that I can do that in just a few seconds without moving my hands away from the keyboard means I can use it any time without much penalty. It’s amazing how many things I jot down that I wouldn’t do otherwise.

Short Range

To manage tasks up to a few hours in length, I use Trac. Trac is a fantastic issue-tracking tool: It’s free, it’s fast, it’s simple, and it’s configurable. In the past I’ve used anything from spreadsheets, to Bugzilla, to publisher-owned bug databases, and nothing comes close to Trac for my needs. It also scales very well to teams of more than one person (although it might not be good enough for hundreds of people).

Just like Subversion, I have Trac hosted externally, through my web hosting company. Sometimes it’s frustrating if I need to access it and the server is down, but again, the convenience of having it off site makes it well-worth it.

Any task that requires more than 10-15 minutes goes in Trac, and then I can easily prioritize tasks depending on their importance. Usually, if an item has been on my instant queue in JustNotes for about a day and I haven’t gotten around to it, it either gets deleted or it gets moved into a full item in Trac. The only way progress happens is by ticking off Trac tasks. In the end, my projects live and die by Trac.

Medium Range

User stories (to borrow terminology from Scrum/XP) are visible, relatively self-contained features of the final project. They’re made up of several tasks, and usually take a few days to a few weeks to implement. A group of user stories make up a full iteration (sprint) of the game, which is usually between one and two weeks long. Sometimes user stories are complex enough (add a replay feature visible on the web site) that the full iteration is just the one user story.

I keep track of these stories in Trac as well. Trac is both an issue-tracking system and a wiki, so the wiki part is perfect to keep these user stories. In addition to that, I label tasks as belonging o a particular iteration. That allows me to separate what needs to be done for this iteration, from other tasks that I added for the future. At the start of each iteration, I decide on the user stories and either generate new tasks or label existing tasks as due for this iteration.

The wiki in Trac is extremely valuable for all sorts of other things: game design ideas, general brainstorming, gathering reference material, etc.

Trac ends up being the perfect mid-range vision of my project.

Long Range

User stories and tasks in Trac aren’t enough to cover a project that is potentially 3-4 months long. I need something that helps me with the longer view, otherwise I find that things creep up on me without realizing it because I’m so focused on the short and mid-range items.

The best tool I’ve found so far is very low-tech: A printable month-per-page calendar covering the full length of the project. Right now I’m shooting for a November release of my current project, so I printed August, September, October, and November and pinned them to the corkboard on my office. It’s amazing the sense of urgency that seeing your ship date gives you. You realize that you only have a handful of weeks before shipping and makes it much easier to prioritize tasks (and chop off features or save them for an update).

I realize this long-term, calendar view isn’t very useful if you don’t have a set release date and you want to continue chipping at your game until it’s ready. But even if your release date is flexible, having this long-term view can help you keep budgets in perspective and manage them accordingly.

Finally, for an extra bit of motivation (or maybe this falls in the category of excessive pressure), I just started using a countdown widget for the Mac OS Dashboard. Just in case the calendar view wasn’t enough, here’s a countdown (down to the second) of the time left until release.

Speaking of which, I think it’s time I get back to work. Only 102 days left!

Fast-forward to now, and as more companies are jumping into the bandwagon of games that need “points” to make progress, we’re still bundles. Again, I chucked that up to legacy reasons and doing what worked with the standalone apps.

Discovering Bundles

It was at the last 360iDev in San Jose, that Mark Johnson said something that really stuck with me. I can still hear him say it with his fine British accent: “I think we might be underestimating how much people are willing to pay for in-app purchases”. Really?

As soon as I had a chance, I looked at the best-selling IAPs for some popular games. The screenshots below were taken today, not back when I looked at them, but the results are very much the same. I let you guess which games these IAPs came from.

I was very surprised with what I saw. The top-selling IAP was never a $0.99 one, and there were bundles of $49.99 or higher towards the top! That was crazy! I was indeed underestimating what players are willing to buy by only offering a measly $0.99 fertilizer bottle in Flower Garden!

Bundles In Flower Garden

As part of the next Flower Garden update, I decided to run a little experiment and add two more fertilizer options: A $2.99 one and a $5.99 one, each of them giving you a slightly better deal on fertilizer (20, 70, and 150 doses). That was still nothing compared to the price tags I was seeing in those other games, but I didn’t want to alienate users by slapping some ridiculously high bundle prices.

The results?

The most popular item by number of sales was still the single fertilizer bottle for $0.99. But a lot of people took advantage of the the other two bundles as well. This is how fertilizer sales for Flower Garden Free have been for the last two months:

But now, let’s look at that same period by plotting revenue (again, only Flower Garden Free, the full version is very similar but it wasn’t easy to combine the two to display them here):

Now the two bundles are a lot closer to the single bottle, especially the larger, $5.99 bundle.

More Than Meets The Eye

In the end, were bundles effective, or are people buying the same amount of fertilizer and leaving less money in the process? Unfortunately I can’t answer that question from a pure data point of view. Looking at fertilizer sales before and after I introduced the bundles is no good because the number of users increased dramatically at each update. I can’t even normalize them by the number of sales, it would have to be by the number of daily users, and unfortunately that’s not a statistic that I’m tracking.

However, I think we can argue two really good points about why bundles are great.

1. More choice

Having different levels of bundles give players more choice on how they want to purchase something. From what I’ve read about buyer psychology, people love having choices when buying something (just don’t give them too many choices!). They are more involved in the buying process, they evaluate it, and they feel better about the decision they eventually make. So that seems to indicate that more people might buy fertilizer if there are a few bundle options than if there’s only one.

2. Commitment

This is the biggie. Whenever a user purchases a $5.99 bundle (or a $49.99 one!), they became more committed to your game. You can also guarantee they will come back again to get their money’s worth from that purchase. Even if they had the intention of coming back to your game without the purchase, having spent that money is a nice reminder to do so. And having people come back to your game is what this is all about: They will explore more of the game, get hooked more, make more in-app purchases, show it to more of their friends, and send more bouquets to their family.

I have no doubt that I’ll be using bundles in the future. Players get a good deal, and you get committed players. It’s a win-win situation.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

Testing Code

The purpose of writing unit tests is to verify the code does what it’s supposed to do. How exactly do we go about checking that? It depends on what the code under test does. There are three main things we can test for when writing unit tests:

• Return values. This is the easiest thing to test. We call a function and verify that the return value is what we expect. It can be a simple boolean, or maybe it’s a number resulting from a complex calculation. Either way, it’s simple and easy to test. it doesn’t get any better than this.

• Modified data. Some functions will modify data as a result of being called (for example, filling out a vertex buffer with particle data). Testing this data can be straightforward as long as the outputs are clearly defined. If the function changes data in some global location, then it can be more complicated to test it or even find all the possible places that can be changed. Whenever possible, pass the address of the data to be modified as an input parameter to the functions. That will make them easier to understand and test.

• Object interaction. This is the hardest effect to test. Sometimes calling a function doesn’t return anything or modify any external data directly, and it instead interacts with other objects. We want to test that the interaction happened in the order we expected and with the parameters we expected.

Testing the first two cases is relatively simple, and there’s nothing you need to do beyond what a basic unit testing-framework provides. Call the function and verify values with a CHECK statement. Done. However, testing that an object “talks” with other objects in the correct way is much trickier. That’s what we’ll concentrate on for the rest of the article.

As a side note, when we talk about object interaction, it simply refers to parts of the code calling functions or sending messages to other parts of the code. It doesn’t necessarily imply real objects. Everything we cover here applies as well to plain functions calling other functions.

Before we go any further, let’s look at a simple example of object interaction. We have a game entity factory and we want to test that the function CreateGameEntity() finds the entity template in the dictionary and calls CreateMesh() once per each mesh.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    EntityDictionary dict;
    MeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
    // How do we test it called the correct functions?
}

We can write a test like the one above, but after we call the function CreateGameEntity(), how do we test the right functions were called in response? We can try testing for their results. For example, we could check that the returned entity has the correct number of meshes, but that relies on the mesh factory working correctly, which we’ve probably tested elsewhere, so we’re testing things multiple times. It also means that it needs to physically create some meshes, which can be time consuming or just need more resources than we want for a unit test. Remember that these are unit tests, so we really want to minimize the amount of code that is under test at any one time. Here we only want to test that the entity factory does the right thing, not that the dictionary or the mesh factory work.

Introducing Mocks

To test interactions between objects, we need something that sits between those objects and intercepts all the function calls we care about. At the same time, we want to make sure that the code under test doesn’t need to be changed just to be able to write tests, so this new object needs to look just like the objects the code expects to communicate with.

A mock object is an object that presents the same interface as some other object in the system, but whose only goal is to attach to the code under test and record function calls. This mock object can then be inspected by the test code to verify all the communication happened correctly.

TEST(CreateGameEntityCallsCreateMeshForEachMesh)
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

dict.meshCount = 3;

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);

    CHECK_EQUAL(1, dict.getEntityInfoCallCount);
    CHECK_EQUAL(gameEntityUid, dict.lastEntityUidPassed);
    CHECK_EQUAL(3, meshFactory.createMeshCallCount);
}

This code shows how a mock object helps us test our game entity factory. Notice how there are no real MeshFactory or EntityDictionary objects. Those have been removed from the test completely and replaced with mock versions. Because those mock objects implement the same interface as the objects they’re standing for, the GameEntityFactory doesn’t know that it’s being tested and goes about business as usual.

Here are the mock objects themselves:

struct MockEntityDictionary : public IEntityDictionary
{
    MockEntityDictionary() 
        : meshCount(0)
        , lastEntityUidPassed(0)
        , getEntityInfoCallCount(0)
    {}

    void GetEntityInfo(EntityInfo& info, int uid)
    {
        lastEntityUidPassed = uid;
        info.meshCount = meshCount;
        ++getEntityInfoCallCount;
    }

    int meshCount;
    int lastEntityUidPassed;
    int getEntityInfoCallCount;
};

struct MockMeshFactory : public IMeshFactory
{
    MockMeshFactory() : createMeshCallCount(0)
    {}

    Mesh* CreateMesh()
    {
        ++createMeshCallCount;
        return NULL;
    }
};

Notice that they do no real work; they’re just there for bookkeeping purposes. They count how many times functions are called, some parameters, and return whatever values you fed them ahead of time. The fact that we’re setting the meshCount in the dictionary to 3 is how we can then test that the mesh factory is called the correct number of times.

When developers talk about mock objects, they’ll often differentiate between mocks and fakes. Mocks are objects that stand in for a real object, and they are used to verify the interaction between objects. Fakes also stand in for real objects, but they’re there to remove dependencies or speed up tests. For example, you could have a fake object that stands in for the file system and provides data directly from memory, allowing tests to run very quickly and not depend on a particular file layout. All the techniques presented in this article apply both to mocks and fakes, it’s just how you use them that sets them apart from each other.

Mocking Frameworks

The basics of mocking objects are as simple as what we’ve seen. Armed with that knowledge, you can go ahead and test all the object interactions in your code. However, I bet that you’re going to get tired quickly from all that typing every time you create a new mock. The bigger and more complex the object is, the more tedious the operation becomes. That’s where a mocking framework comes in.

A mocking framework lets you create mock objects in a more automated way, with less typing. Different frameworks use different syntax, but at the core they all have two parts to them: A semi-automatic way of creating a mock object from an existing class or interface. A way to set up the mock expectations. Expectations are the results you expect to happen as a result of the test: functions called in that object, the order of those calls, or the parameters passed to them.

Once the mock object has been created and its expectations set, you perform the rest of the unit test as usual. If the mock object didn’t receive the correct calls the way you specified in the expectations, the unit test is marked as failed. Otherwise the test passes and everything is good.

GoogleMock

GoogleMock is the free C++ mocking framework provided by Google. It takes a very straightforward implementation approach and offers a set of macros to easily create mocks for your classes, and set up expectations. Because you need to create mocks by hand, there’s still a fair amount of typing involved to create each mock, although they provide a Python script that can generate mocks automatically from from C++ classes. It still relies on your classes inheriting from a virtual interface to hook up the mock object to your code.

This code shows the game entity factory test written with GoogleMock. Keep in mind that in addition to the test code, you still need to create the mock object through the macros provided in the framework.

TEST(CreateGameEntityCallsCreateMeshForEachMesh) 
{
    MockEntityDictionary dict;
    MockMeshFactory meshFactory;
    GameEntityFactory gameFactory(dict, meshFactory);

    EXPECT_CALL(dict, GetEntityInfo())
        .Times(1)
        .WillOnce(Return(EntityInfo(3));

    EXPECT_CALL(meshFactory, CreateMesh())
        .Times(3);

    Entity* entity = gameFactory.CreateGameEntity(gameEntityUid);
}

MockItNow

This open-source C++ mocking framework written by Rory Driscoll takes a totally different approach from GoogleMock. Instead of requiring that all your mockable classes inherit from a virtual interface, it uses compiler support to insert some code before each call. This code can then call the mock and return to the test directly, without ever calling the real object.

From a technical point of view, it’s a very slick method of hooking up the mocks, but the main advantage of this approach is that it doesn’t force a virtual interface on classes that don’t need it. It also minimizes typing compared to GoogleMock. The only downside is that it’s very platform-specific implementation, and the version available only supports Intel x86 processors, although it can be re-implemented for PowerPC architectures.

Problems With Mocks

There is no doubt that mocks are a very useful tool. They allow us to test object interactions in our unit tests without involving lots of different classes. In particular, mock frameworks make using mocks even simpler, saving typing and reducing the time we have to spend writing tests. What’s not to like about them?

The first problem with mocks is that they can add extra unnecessary complexity to the code, just for the sake of testing. In particular, I’m referring to the need to have a virtual interface that objects are are going to be mocked inherit from. This is a requirement if you’re writing mocks by hand or using GoogleMock (not so much with MockItNow), and the result is more complicated code: You need to instantiate the correct type, but then you pass around references to the interface type in your code. It’s just ugly and I really resent that using mocks is the only reason those interfaces are there. Obviously, if you need the interface and you’re adding a mock to it afterwards, then there’s no extra complexity added.

If the complexity and ugliness argument doesn’t sway you, try this one: Every unnecessary interface is going to result in an extra indirection through a vtable with the corresponding performance hit. Do you really want to fill up your game code with interfaces just for the sake of testing? Probably not.

But in my mind, there’s another, bigger disadvantage to using mock frameworks. One of the main benefits of unit tests is that they encourages a modular design, with small, independent objects, that can be easily used individually. In other words, unit tests tend to push design away from object interactions and more towards returning values directly or modifying external data.

A mocking framework can make creating mocks so easy, to the point that it doesn’t discourage programmers from creating a mock object any time they think of one. And when you have a good mocking framework, every object looks like a good mock candidate. At that point, your code design is going to start looking more like a tangled web of objects communicating in complex ways, rather than simple functions without any dependencies. You might have saved some typing time, but at what price!

When to Use Mock Frameworks

That doesn’t mean that you shouldn’t use a mocking framework though. A good mocking framework can be a lifesaving tool. Just be very, very careful how you use it.

The case when using a mocking framework is most justified when dealing with existing code that was not written in unit testing in mind. Code that is tangled together, and impossible to use in isolation. Sometimes that’s third-party libraries, and sometimes it’s even (yes, we can admit it) code that we wrote in the past, maybe under a tight deadline, or maybe before we cared much about unit tests. In any case, trying to write unit tests that interface with code not intended to be tested can be extremely challenging. So much so, that a lot of people give up on unit tests completely because they don’t see a way of writing unit tests without a lot of extra effort. A mocking framework can really help in that situation to isolate the new code you’re writing, from the legacy code that was not intended for testing.

Another situation when using a mocking framework is a big win is to use as training wheels to get started with unit tests in your codebase. There’s no need to wait until you start a brand new project with a brand new codebase (how often does that happen anyway?). Instead, you can start testing today and using a good mock framework to help isolate your new code from the existing one. Once you get the ball rolling and write new, testable code, you’ll probably find you don’t need it as much.

Apart from that, my recommendation is to keep your favorite mocking framework ready in your toolbox, but only take it out when you absolutely need it. Otherwise, it’s a bit like using a jackhammer to put a picture nail on the wall. Just because you can do it, it doesn’t mean it’s a good idea.

Keep in mind that these recommendations are aimed at using mock objects in C and C++. If you’re using other languages, especially more dynamic or modern ones, using mock objects is even simpler and without many of the drawbacks. In a lot of other languages, such as Lua, C#, or Python, your code doesn’t have to be modified in any way to insert a mock object. In that case you’re not introducing any extra complexity or performance penalties by using mocks, and none of the earlier objections apply. The only drawback left in that case is the tendency to create complex designs that are heavily interconnected, instead of simple, standalone pieces of code. Use caution and your best judgement and you’ll make the best use of mocks.

This article was originally printed in the June 2009 issue of Game Developer.

One of the things you potentially give up by keeping the game runtime to a minimum is an editor built in the game itself. But that’s one of those things that sounds a lot better than it really is. From a technical point of view, having an editor in the game usually complicates the code a huge amount. All of a sudden you need to deal with objects being created and destroyed randomly (instead of through clearly defined events in the game), and you have to deal with all sorts of crazy inputs and configurations.

The worse part though, is having to implement some sort of GUI editing system in every platform. Creating the GUI to run on top of the game is not easy, requiring that you create custom GUI code or try to use some of the OpenGL/DirectX libraries available. And even then, a complex in-game GUI might not be a big deal on a PC, but wait and try to use that interface on a PS3 or iPhone. After all, making games is already complicated and time-consuming enough to waste more time reinventing the widget wheel.

Remote game editing manages to keep a minimal runtime and allow you to quickly create native GUIs that run on a PC. It’s the best of both worlds, and although it’s not quite a perfect solution, it’s the best approach I know.

Debug Server

Miguel Ángel Friginal already covered the basics of the debug server, so I’m not going to get in details there.

The idea is that you run a very simple socket server on the game, listening in a particular port. This server implements the basic telnet protocol, which pretty much means that it’s a line-based, plain-text communication.

The main difference between my debug server and Miguel’s (other than mine is written in cross-platform C/C++ instead of ObjC), is that I’m not using Lua to execute commands. Using Lua for that purpose is a pretty great idea, but goes against the philosophy of keeping the runtime as lean and mean as possible.

Instead, I register variables with the server by hand. For each variable, I specify its memory address, it’s type, any restrictions (such as minimum and maximum values), and a “pretty name” to display in the client. Sounds like a lot of work, but it’s just one line with the help of a template:

registry.Add(Tweak(&plantOptions.renderGloss, "render/gloss", "Render gloss"));
registry.Add(Tweak(&BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3))

And yes, if I were to implement this today, I would probably get rid of the templates and make it all explicit instead (ah, the follies of youth :-)

TweakUtils::AddBool(registry, &plantOptions.renderGloss, "render/gloss", "Render gloss");
TweakUtils::AddFloat(registry, &BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3);

The debug server itself responds to three simple commands:

• list. Lists all the variables registered in the server.
• set varname value. Sets a value.
• print varname. Gets the value for that variable.

For example, whenever the server receives a set command, it parses the value, verifies that it’s within the acceptable range, and applies it to the variable at the given memory location.

Telnet Clients

Because we used the standard telnet protocol, we can start playing with it right away. Launch the game, telnet into the right port, and you can start typing away.

However, most telnet clients leave much to be desired for this. They all rely on the history and cursor manipulation being handled by the shell they assume you’re connected to. Here we aren’t connected to much of anything, but I’d like to be able to push up arrow and get my last command, and be able to move to the beginning of the line or the previous word like I would do in any text editor. The easiest solution I found for that was to use a telnet client prepared for that kind of thing: A MUD client! Just about any will do, but one that works well for me is Atlantis.

So far, we’ve implemented the equivalent of a FPS console, but working remotely. And because the code is fully portable, our game can be in just about any platform and we can always access it from our PC. Not just that, but we can even open multiple simultaneous connections to various development devices if you need to run them all at once.

Custom Clients

Game parameter tweaking is something that is OK through a text-based console, but really comes into its own when you add a GUI. That’s exactly what we did at Power of Two Games. We created a generic GUI tool (based on WinForms since we were on Windows at the time), that would connect to the server, ask for a list of variables, and generate a GUI on the fly to represent those variables. Since we knew type and name of each variable, it was really easy to construct the GUI elements on the fly: A slider with a text field for floats and ints, a checkbox for bools, four text fields for vectors, and even a color picker for variables of the type color.

It worked beautifully, and adjusting different values by moving sliders around was fantastic. We quickly ran into two problems through.

The first one is that we added so many different tweaks to the game, that it quickly became unmanageable to find each one we wanted to tweak. So, in the spirit of keeping things as simple as possible (and pushing the complexity onto the client), we decided that the / symbol in a name would separate group name and variable name. That way we could group all related variables together and make everything usable again.

The second problem was realizing that some variables were changing on the runtime without us knowing it on the client. That created weird situations when moving sliders around. We decided that any time a registed variable changes on the server, it should notify any connected clients. That worked fine, but, as you can imagine, it became prohibitively expensive very quickly. To get around that, we added a fourth command: monitor varname. This way clients need to explicitly register themselves to receive notifications whenever a variable changes, and the GUI client only did it for the variables currently displayed on the screen.

During this process, it was extremely useful to be able to display a log console to see what kind of traffic there was going back and forth. It helped me track down a few instances of bugs where changing a variable in the client would update it in the server, sending an update back to the client, which would send it again back to the server, getting stuck in an infinite loop.

You don’t need to stop at a totally generic tool like this either. You could create a more custom tool, like a level editor, that still communicates with the runtime through this channel.

Flower Garden Example

For Flower Garden, I knew I was going to need a lot of knobs to tweak all those plant DNA parameters, so I initially looked into more traditional GUI libraries that worked on OpenGL. The sad truth is that they all fell way short, even for development purposes. So I decided to grab what I had at hand: My trusty tweaking system from Power of Two Games.

I’m glad I did. It saved a lot of time and scaled pretty well to deal with the hundreds of parameters in an individual flower, as well as the miscellaneous tweaks for the game itself (rendering settings, infinite fertilizer, fast-forwarding time, etc).

Unfortunately, there was one very annoying thing: The tweaker GUI was written in .Net. Sure, it would take me a couple of days to re-write it in Cocoa (faster if I actually knew any Cocoa), but as an indie, I never feel I can take two days to do something tangential like that. So instead, I just launched it from VMWare Fusion running Windows XP and… it worked. Amazingly enough, I’m able to connect from the tweaker running in VMWare Fusion to the iPhone running in the simulator. Kind of mind boggling when you stop and think about it. It also connects directly to the iPhone hardware without a problem.

VMWare Fusion uses up a lot of memory, so I briefly looked into running the tweaker client in Mono. Unfortunately Mono for the Mac didn’t seem mature enough to handle it, and not only was the rendering of the GUI not refreshing correctly, but events were triggered in a slightly different order than in Windows, causing even more chaos with the variable updates.

Here’s a time-lapse video of the creation of a Flower Garden seed from the tweaker:

Drawbacks

As I mentioned earlier, I love this system and it’s better than anything else I’ve tried, but it’s not without its share of problems.

Tweaking data is great, but once you find that set of values that balances the level to perfection… then what? You write those numbers down and enter them in code or in the level data file? That gets old fast. Ideally you want a way to automatically write those values back. That’s easy if the tool itself is the editor, but if it’s just a generic tweaker, it’s a bit more difficult.

One thing that helped was adding a Save/Load feature to the tweaker GUI. It would simply write out a large text-based file with all the variables and their current values. Whenever you load one of those, it would attempt to apply those same values to the current registered variables. In the end, I ended up making the Flower Garden offline seed file format match with what the tweaker saved out, so that process went pretty smoothly.

Another problem is if you want lots of real-time (or close to real time) updates from the server. For example, you might want to monitor a bunch of data points and plot them on the client (fps, memory usage, number of collisions per frame, etc). Since those values change every frame, it can quickly overwhelm the simple text channel. For those cases, I created side binary socket channels that can simply send real-time data without any overhead.

Finally, the last drawback is that this tweaking system makes editing variables very easy, but calling functions is not quite as simple. For the most part, I’ve learned to live without function calls, but sometimes you really want to do it. You can extend the server to register function pointers and map those to buttons in the client GUI, but that will only work for functions without any parameters. What if you wanted to call any arbitrary function? At that point you might be better off integrating Lua in your server.

Future Directions

This is a topic I’ve been interested in for a long time, but the current implementation of the system I’m using was written 3-4 years ago. As you all know by now, my coding style and programming philosophy changes quite a bit over time. If I were to implement a system like this today, I would do it quite differently.

For all I said about keeping the server lean and minimal, it could be even more minimal. Right now the server is receiving text commands, parsing them, validating them, and interpreting them. Instead, I would push all that work on the client, and all the server would receive would be a memory address, and some data to blast at that location. All of that information would be sent in binary (not text) format over a socket channel, so it would be much more efficient too. The only drawback is that we would lose the ability to connect with a simple telnet client, but it would probably be worth it in the long run.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

The good news is that unit testing is a lot simpler than driving a standard shift, but there are a lot of small details you need to get right to do it successfully. Even after reading about unit testing and being convinced of its benefits, programmers are often not sure how to get started. This month’s column is not going to try to convince you of the many benefits of unit testing (I hope you are already convinced), but rather, describe some very concrete tips to help you get started right away.

Goals of Unit Testing

There are many different reasons to write unit tests:

• Correctness testing. Checking that the code behaves as designed.
• Boundary testing. Checking that the code behaves correctly in odd or boundary situations.
• Regression testing. Checking that the behavior of the code doesn’t change unintentionally over time.
• Performance testing. Checking that the program meets certain minimum performance or memory constraints.
• Platform testing. Checking that the code behaves the same across multiple platforms.
• Design. Tests provide a way to advance the code design and architecture. This is usually referred as Test-Driven Development (TDD).
• Full game or tool testing. Technically this is a functional test, not a unit test anymore because it involves the whole program instead of a small subset of the code, but a lot of the same techniques apply.

Some developers use unit tests only for one of the reasons listed above, while others use many kinds of tests for a variety of different reasons. It’s important to recognize that because there are so many different uses for unit tests, no single solution is going to fit everybody. The ideal setup for some of those situations is going to be slightly different than for others. The basics are the same for all of them though.

When working with unit tests, these are our main goals:

• Spend as little time as possible writing a new test.
• Be notified of failing tests, and see at a glance which ones failed and why.
• Trust our tests. Have them be consistent from run to run and robust in the face of bad code.

Testing Framework

Most of us have created one-off programs in the past to test some particularly complicated code. It’s usually a quick command line program that runs through a bunch of cases and asserts after each one that the results were correct. That’s the most bare-bones way of creating unit tests.

Unfortunately, it’s also a pain and it misses on most of the unit-testing goals described in the previous section: creating a new program just for that is a pain, we have to go out of our way to run the tests, and it usually gets out of date faster than the latest Internet meme. That is in part why a lot of programmers have an initial aversion to writing unit tests.

If you’re considering writing even a small unit test, you should use a unit-testing framework. A unit-testing framework removes all the busy work from writing unit tests and lets you spend your time on the logic of what to test. This doesn’t mean that the framework writes the tests for you. Be very wary of any tool that claims to do that! No, a unit-testing framework is simply a small library that provides all the glue for running unit tests and reporting the results. Sorry, you still need to use your brain and do (some of) the typing.

A quick search will reveal plenty of unit-testing frameworks to choose from for your language of choice, and most of them are free and open source so you can rely on them and modify them to suit your needs. For C/C++ and game development, I strongly recommend starting with UnitTest++. Charles Nicholson and I wrote that framework a few years ago specifically with games and consoles in mind. Many game teams have adopted it for their games and tools, and it has been used on lots of different game platforms including current and last generation consoles, Windows, Linux, and Mac PCs, and even on the iPhone. In most situations, it should be a straight drop-in to your project and you’re up and running.

If you end up using a different testing framework, or even if you roll your own, the techniques described here still apply, even if the syntax is slightly different.

Hello Tests

Writing your first test is easy as pie:

#include <UnitTest++.h>

TEST(MyFirstTest)
{
    int a = 4;
    CHECK(a == 4);
}

To run it you need to add the following line to your executable somewhere. We’ll talk more about the physical organization of tests in a moment.

int failedCount = UnitTest::RunAllTests();

Done! Easy, wasn’t it? When you compile and run the program you should see the following output:

Success: 1 test passed.
Test time: 0.00 seconds.

Let’s add a failing test:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK(a == 4);
}

Now we get:

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: a == 4
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

That’s great, but if we’re going to diagnose the problem, we probably need to know the value of the variable a, and all the test is telling us is that it’s not 4. So instead, we can change the CHECK statement to the following:

TEST(MyFailingTest)
{
    int a = 5;
    CHECK_EQUAL(4, a);
}

And now the output will be

/fullpath/filename.cpp:17: error: Failure in MyFailingTest: Expected 4 but was 5
FAILURE: 1 out of 2 tests failed (1 failures).
Test time: 0.00 seconds.

Much better. Now we get both the error information and the value of the variable under test. Virtually all unit-testing frameworks include different types of CHECK statements to get more information when testing floats, arrays, or other data types. You can even make your own CHECK statement for your own common data types such as colors or lists.

As a bonus, if you’re using an IDE, double-clicking on the test failure message should bring you automatically to the failing test statement.

When To Run

When to run unit tests will depend on what is being tested and how long it takes to do so. In general, the more frequently you run the tests, the better. The sooner you get feedback that something went wrong, the easier it will be to fix. Maybe even before it was checked in and it spread to the rest of the team. On the flip side, realistically, building and running a set of unit tests takes a certain amount of time, so it’s important to find the right balance between feedback frequency and time spent waiting for tests.

At the very least, all tests should run once a day, during the nightly build process in your build server (You have a build server, don’t you? If not, run over and read this column in the August 2008 issue). It doesn’t matter how long they take or how how many different projects you need to run. Just add them to the build script, and hook their output into the build results.

On the other extreme, you can build your tests every time you build the project and execute them as a postbuild step. That way, any time you make a change to a project, all tests will execute and you’ll see if anything went wrong. This is a great approach, but I wouldn’t recommend it if tests add more than a couple of seconds to the incremental build time, otherwise, they’ll be slowing you down more than they help.

For most developers, some approach in the middle will make most sense. For example, take a small, fast subset of tests that are more likely to break, and run those with every build. Whenever any code is checked in to version control, the build server can run those tests plus a few more, slower ones. And finally, at night, you can bring out the big guns and run those really long, really thorough ones that take a few hours to complete. You can separate those tests into different projects, or, if your framework supports them, into different test suites, which allow you to decide which sets of tests to execute at runtime. Reporting Results

If a unit test fails and nobody notices, is it really an error? Just running the tests isn’t good enough. We have to make sure that someone sees the failure it and fixes the problem.

Most unit-testing frameworks will let you customize how you want the failure errors to be reported. By default they will probably be sent to stdout, but you can easily customize the framework to send them to debug log streams, save to a file, or upload them to a server.

Even more important than the actual error messages is detecting whether there were any failures. After running all the tests, there is usually some way to detect how many tests failed. The program that was running the tests can detect any failures, print an error message, and exit with an error code. That error code will propagate to the build server and trigger a build failure. Hopefully by now alarm sounds are going off across the office and someone is on his way to fix it.

Project Organization

When people start down the unit test path, they often struggle to figure out how to physically lay out the unit tests. In the end, it really doesn’t matter too much as long as it makes sense to you, the final build doesn’t contain any tests, and they’re still easy to build and run.

My personal preference is to keep unit tests separate from the rest of the code. Usually I end up creating one file of tests for every cpp file. So FirstPersonCameraController.h and .cpp have a corresponding TestFirstPersonCameraController.cpp. Since I use this convention regularly throughout all of my code, I have a custom IDE macro to toggle between a file and its corresponding test file. I also put all the tests in a separate subdirectory to keep them as physically separate as possible.

I prefer to break up my code into several static libraries for each major subsystem: graphics, networking, physics, animations, etc. Each of those libraries has a set of unit tests, but instead of compiling them into the library, I create a separate project that creates a simple executable program. That project contains all the unit tests and links against the library itself, and in its main entry point it just calls the function to runs all unit tests and returns the number of failures. This keeps the tests separate from the library, but still very easy to build.

If all your code is organized into libraries, and your game is just a collection of libraries linked together, that’s all you need. Most games and tools, however, have a fair amount of code that you might want to test in the project itself. Since the game is an executable, you can’t easily link against it from a different project like we did before. In this case, I build the unit tests into the game itself, and I optionally call them whenever a particular command-line parameter like -runtests is present. Just make sure to #ifdef out all the tests in the final build.

Multiplatform Testing

Running the tests on the PC where you build the code is very straightforward. But unless you’re only creating games and tools for that platform, you will definitely want to run your tests on different platforms as well. Unit tests are an invaluable tool for catching slight platform inconsistencies caused by different compilers, architecture idiosyncrasies, or varying floating point rules.

Unfortunately, running unit tests on a different platform from your build machine is usually a bit more involved and not nearly as fast as doing it locally. You need to start by compiling the tests for the target platform. This is usually not a problem since you’re already building all your code for that platform, and hopefully your unit testing framework already supports it. Then you need to upload your executable with the tests and any data required to the target platform and run it there. Finally, you need to get the return code back to detect if there were any failures. This is surprisingly the trickiest part of the process with a lot of console development kits. If getting the exit code is not a possibility, you’ll need to get creative by parsing the output channel, or even waiting for a notification on a particular network port.

Some target platforms are more limited than others in both resources and C++ support. One of the features that makes UnitTest++ a good choice for games is that it requires minimal C++ features (no STL) and it can be trimmed down even further (no exceptions or streams).

For example, running unit tests on the PS3 SPUs was extremely useful, but it required stripping the framework down to the minimum amount of features. It was also tricky being able to fit the library code plus all the tests in the small amount of memory available. To get around that, we ended up changing the build rules for the SPUs so each test file created its own SPU executable (or module). We then wrote a simple main SPU program that would load each module separately, run its tests, keep track of all the stats, and finally report them.

Running a set of unit tests on the local machine can be an almost instant process, but running them on a remote machine is usually much slower, and can take up to 10 or 20 seconds just with the overhead of copying them and launching the program remotely. For this reason, you’ll want to run tests on other platforms less frequently.

No Leaking Allowed

Finally, if you’re going to have this all this unit testing code running on a regular basis, you might as well get as much information out of it as possible. I have found it invaluable to keep track of memory leaks around the unit test code.

You’ll have to hook into your own memory manager, or use the platform-specific memory tracking functions. The basic idea is to get a memory status before running the tests, and another one after all the tests execute. If there are any extra memory allocations, that’s probably a leak. In that case, you can report it as a failed build by returning the correct error code.

Watch out for static variables or singletons that allocate memory the first time they’re used. They might be reported as memory leaks even though it wasn’t what you were hoping to catch. In that case, you can explicitly initialize and destroy all singletons, or, even better, not use them at all, and keep your memory leak report clean.

You’re now armed with all you need to know to set up unit tests into your project and build pipeline. Grab a testing framework and get started today.

This article was originally printed in the May 2009 issue of Game Developer.

No Flash? Try the QuickTime video version.

Let me get one thing out of the way to stop all the trigger-happy, const-bashing, would-be-commenters: const doesn’t make any guarantees that values don’t change.

You can change a const variable by casting the constness away, or referencing it through a pointer, but you really had to go out of your way to do that. If it helped with that, const would solve (or improve) the memory aliasing problem like Hitler pointed out. It doesn’t, so const is pretty weak as far as promises go. It just says “I, the programmer, promise not to change this value on purpose (unless I’m truly desperate)”. Still, even a promise like that goes a long way helping with readability and maintenance.

With that out of the way, what exactly do I mean by using const everywhere?

Const non-value function parameters

Any reference or pointer function parameters that are pointing to data that will not be modified by the function should be declared as const. If you’re going to use const just for one thing, this is the one to use. It’s invaluable glancing at a function signature and seeing which parameters are inputs and which ones are outputs.

void Detach(PhysicsObject& physObj, int attachmentIndex, const HandleManager& mgr);

Marking those parameters as const also serves as a warning sign in case a programmer in the future tries to modify one of them. Imagine the disaster if the calling code assumes data never changes, but the function suddenly starts modifying that data! const won’t prevent that from happening, but will remind the programmer that he’s changing the “contract” and needs to revisit all calling code and check assumptions.

Const local variables

This is a very important use of const and one of the ones hardly anyone follows. If I declare a local (stack) variable and its value never changes after initialization, I always declare it const. That way, whenever I see that variable used later in the code, I know that its value hasn’t changed.

const Vec2 newPos = AttachmentUtils::ApplySnap(physObj, unsnappedPos);
const Vec2 deltaPos = newPos - physObj.center;
physObj.center = newPos;

This is one of the reasons why I did a 180 on the ternary C operator (?). I used to hate it and find it cryptic and unreadable, but now I find it compact and elegant and it fulfills my const fetish very well.

Imagine you have a function that is going to work in one of two objects and you need to compute the index to the object to work on. You could do it this way:

int index;
if (some condition)
	index = 0;
else
	index = 2;

DoSomethingWithIndex();

Not only does that take several lines not to do much, but index isn’t const (argh!). So every time I see index anywhere later on in that function, I’m going to have to spend the extra mental power to make sure nothing has changed (and, with my current coding style, I would assume it has changed).

Instead, we can simply do this:

const int index = (some condition) ? 0 : 2;
DoSomethingWithIndex();

Ahhhh… So much better!

Const member variables

This one doesn’t really apply to me anymore because I don’t use classes and member variables. But if you do, I strongly encourage you do mark every possible member function as const whenever you can.

The only downside is that sometimes you’ll have some internal bit of data that is really not changing the “logical” state of an object, but it’s still modifying a variable (usually some caching or logging data). In that case, you’ll have to resort to the mutable keyword.

Const value function parameters

Apparently I’m not a total Const Nazi because this is one possible use of const that I choose to skip (even though I tried it for a while because of Charles).

Marking a value function parameter as const doesn’t make any difference from the calling code point of view, but it serves the same purpose as marking local stack variables as const in the implementation of the function. You’re just saying “I’m not going to modify that parameter in this function” so it makes the code easier to understand.

I’m actually all for this, but the only reason I’m not doing it is because C/C++ makes it a pain. Marking parameters as const in the function declaration adds extra verbosity and doesn’t help the person browsing the functions at all. You could actually put the const only in the function definition and it will work, but at that point the declaration and the definition are different, so you can’t copy and paste them or use other automated tools or scripts.

The concept of const is one of the things I miss the most when programming other languages like C#. I don’t understand why they didn’t add it to the language. On something like Python or Perl I can understand because they’re supposed to be so free form, but C#? (Edit: How about that? Apparently C# has const. It was either added in the last few years or I completely missed it before). It also really bugs me that Objective C or the Apple API doesn’t make any use of const.

Frankly, if it were up to me, I would change the C/C++ language to make every variable const by default and adding the nonconst or changeable (or take over mutable) keyword for the ones you want to modify. It would make life much more pleasant.

But then again, that’s why the call me the Const Nazi.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

The Plan

Mother’s Day happens at the beginning of May (in the US and Canada anyway, I’m afraid I was too busy in April and I missed Mother’s Day in a lot of European countries). I figured it would be the perfect time to do another push.

If there’s something I learned from past experience, is that the more you manage to concentrate any kind of promotion, the more effective it will be. So in preparation for Mother’s Day, I created a new update (with iPad support), a couple of new in-app purchase items (new set of seeds and some fertilizer bundles), and sent out the usual announcement on the Flower Garden mailing list, Facebook page, and Twitter.

But in addition to all of that, I tried a new strategy: I gave Flower Garden away for free. Yes, completely for free.

The idea sounded really scary at first. After all, I would be giving away my baby for free. Would I lose a lot of money doing that? Would it depreciate the perceived value of Flower Garden? Would it annoy loyal users seeing an app they paid for given away? Fortunately it appears that the answer to those questions was no.

The reason I decided to give Flower Garden away for free was mostly to get it into more people’s hands. I was thinking I would lose some money initially, but then more than make up for it when I turned it back to paid because of all the extra users and word of mouth. There is already a free version with a limited number of pots and seeds, but people are hungry to download paid apps for free.

To add extra impact to this price change, I had Flower Garden featured as the free app for Mother’s Day weekend in Free App Calendar. Unlike other free app web sites, the folks at Free App Calendar are very developer friendly and are not out to take a cut of your profits or charge outrageous fees. It was an absolute pleasure dealing with them.

Mother’s Day

On Saturday May 8th, a few minutes past midnight the day before Mother’s Day, I switched Flower Garden over to free. Now I was committed!

Right away there was a lot of positive reaction around the announcement. Everybody on Twitter and Facebook were responding really well and spreading the word. Major sites like Touch Arcade and 148Apps covered the Mother’s Day promotion and got lots of extra eyeballs on the sale.

After the first day, the data was in: Flower Garden had been downloaded 12,500 times. That was great! As a reference, Flower Garden Free was usually downloaded between 800 and 1,000 times per day, so that was a 10x improvement.

On Sunday, Mother’s Day, things got even better. News had time to propagate more, and people were sending bouquets like crazy, so by the end of the day there had been an additional 26,000 downloads. That’s exactly what I was hoping for!

As a matter of fact, it was doing so great, that I decided to leave it for free as long as the number of downloads was significantly higher than what the free version was normally getting. After Mother’s Day downloads started going down, but they were still pretty strong the following Sunday. Here’s what the download numbers looked like for those 9 days:

The important question is how much revenue was there during that time? I was giving the app away for free but it had in-app purchases. Would they make up for it? The answer was a resounding yes!

Mother’s Day went on to become the biggest day in terms of revenue since Flower Garden was launched. Bigger even than Christmas or Valentine’s Day! Things started going down after that, but still at a very high level. The little bump towards the end of the week is a combination of the weekend (which always results in more sales), and the feature of Flower Garden on the App Store across most of Europe.

These numbers took a bit to sink in. It really shows that in-app purchases are definitely tied to the number of downloads. If you manage to give away twice as many copies, you’ll probably get close to twice as many in-app purchases. That effect is amplified if you have multiple in-app purchase items available.

It’s also interesting to notice that revenue didn’t follow the same drop-off curve as downloads. It wasn’t nearly as sharp. I suspect two things are going on in there:

• Some users downloaded Flower Garden during the sale weekend and weren’t interested in it at all. Downloading it was a knee-jerk reaction to any app that goes free, so that never translated into an in-app purchase. Users later in the week however, probably downloaded it because they received a bouquet or were interested in it, so they had a much higher likelihood of buying something through the Flower Shop.
• Fertilizer. Fertilizer is the only consumable item available for purchase in Flower Garden. Unlike a non-consumable item, the number of sales is not tied to the number of new users, but to the number of current, daily users. The more users launch your app every day, the higher the sales of consumable items. Some of the new users of Flower Garden went on to buy fertilizer later in the week, making revenue higher than you would expect from the download curve.

Flipping The Switch

The number of downloads on Sunday May 16th was slightly over 2,000. At that point I decided that it was close enough to the number of downloads Flower Garden Free was normally getting, so I flipped the switch back to paid. Things were going great, so messing with it was a pretty scary thing to do. Even scarier than it had been setting it free in the first place.

During that week, Flower Garden rose up on the charts. It reached #73 in the top games in the US and was charting very high in all the subcategories and on the iPad charts. As soon as I flipped the switch back to paid, it dropped out of sight from the charts. Fortunately, within a couple of days it came back to its position before that crazy week.

Most importantly, Flower Garden Free, which had dropped quite a bit during that promotion, immediately went back up to the top 100 in the Kids and Family subcategories like before.

As you can expect, as a result of giving it away for free, the ratings on the App Store went down quite a bit. While it was a paid application, the ratings were around 4 stars, but they dropped down to 2.5 stars after that week. It seems people love a free app, but are very quick to criticize it and give it a low rating (especially if it has in-app purchases).

Fortunately bad ratings can be easily fixed with a new update, and some encouragement to users to leave positive rating on the App Store. Now it’s back up to over 4.5 stars.

Aftermath

Now it’s two months later and the dust has had a chance to settle down. Apart from the very nice sales spike during the sale, was it worth it? Again, the answer is a definite yes.

Here’s the revenue since the start of the promotion:

As you can see, it quickly went down, but it settled at a reasonably high level. In fact, compare this two-month period (highlighted in blue) with the previous sales:

Before the promotion revenue was hovering around $1,500 per week, now it settled down to about $2,400 per week. An average day today is bigger than Christmas day! Very nice change!

I think the reason why it settled at a higher revenue level than before is because it got more exposure during that week. Lots of people sent bouquets, which introduced new users to Flower Garden. It’s the viral effect I was hoping for from the start, and although it never reached epidemic proportions, it has been enough to keep Flower Garden alive and well.

Adding iPad support as part of the latest update probably helped too. The iPad market is smaller than the iPhone one, but a lot of early adopters are eager to find good apps for their new toys. The smaller market size also allowed Flower Garden to appear in the iPad charts more easily, increasing exposure that way.

Here’s a breakdown of where revenue came from in the last month (I’m excluding the period where Flower Garden was free to get a more accurate view):

As you can see, consumable items (fertilizer) account for almost half the revenue. Consumable items are a factor of your current userbase, so getting a large influx of new users can result in a permanent revenue increase instead of just a sales spike. It also shows what a small percentage actual app sales are, which explains why even while Flower Garden was free, revenue was still up.

Conclusion

This was a wild ride again! It was definitely worth doing the promotion and it definitely brought home how powerful free can be. However, I’m trying to decide the pricing scheme for my next game, and even though free plus in-app purchases is very tempting, I’m not sure it’s the way to go.

What do you think? Are new games better off being free with in-app purchases, or can indie games be successful being paid (and still having in-app purchases)?

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

What becomes significantly more difficult is learning to effectively apply TDD to code with more dependencies. A question that I’m often asked from people trying to use TDD for the first time is “How can you possibly use TDD for high-level game code? It’s impossible!”.

When The Going Gets Tough

At that point, the temptation to give up on using TDD for high-level code becomes very strong. It seems that all the time spent writing mocks and hooking objects together is a waste of time and that you could be writing game code much faster without it. “Maybe it would be best to skip TDD, just for that part”.

Bad idea!

By giving up on TDD for high-level code, you’ll be missing out on the main benefit of TDD: designing better code. I’ve said this many times, but it bears repeating: TDD is not a testing technique. It’s a design technique. You get lots of benefits from applying TDD [2], but one of the main one is much more modular and dependency-free code.

That’s because TDD turns our weaknesses into advantages. We’re humans and we’re lazy. We don’t want to repeat ourselves constantly or write complex code. If we’re writing a test for some code we’ll write in the future, we’re going to create a very simple test. We’re probably going to want to put some object or data on the stack and make a function call and nothing else. We certainly don’t want to initialize the graphics system, create a new world, add some level data, start the physics simulation, and then call the function! So by being lazy, we end up writing very simple code with the least amount of dependencies.

What does that mean for our high-level code? The AI logic seems to need access to every single game system, at least the way we implemented it in the past. Using TDD to implement AI logic doesn’t mean writing tests to write the same code you would have written before. It means writing tests and then writing some code that makes those tests pass and nothing else. For example, we might realize that the AI doesn’t need to access the physics system, cast a ray in the world, find out what entities is interesting, and return that information. Instead, all it might need to do is output some data saying that it wants a ray query done at a later time (which in turn is the key to batching those ray casts and achieving high performance).

You soon realize that the code you write is very different (and better!) to what you would have written before. Once you get over that hump, TDDing high-level code is not about twisting your code with mocks, but becomes simpler code that takes simple input data and creates simple output data. By pushing through and forcing yourself to apply TDD, you made a breakthrough and reaped all the benefits.

That’s the main reason why TDD books push the idea that no code can be written without a test first. Otherwise, inexperienced TDD programmers would be too quick to quit and would miss out on the technique completely.

Not only that, but code that is not created with TDD tends to have the bad habit of spreading. When code was not designed through TDD, it means that other code that uses it will probably be difficult to TDD itself. So it’s a bit like const-correctness: You’re either in our out, and there’s very little room for half measures.

On the other hand, I would argue that using TDD on a math library is a bad idea. It’s essential to write good unit tests for a math library, but probably not to design it through TDD. Are you really going to implement a cross product differently just because you wrote tests before? The emphasis there has to be on correctness and performance, not on creating the interface or implementation through tests.

It’s A Tool To Help Development

A few days ago, I received an email from Caleb Gingles with a great question:

I’ve started a new game project and have been making an effort to strictly follow TDD from the beginning. Which has been a challenge, as much of the coding at the beginning of the project has consisted of operating system requirements and framework stuff. […] However, there are other situations that seem much harder to test. Like the main loop in a game. According to Kent Beck’s book, everything begins with testing, and no code is written unless a test requires it. But how can a test require the main loop framework in a game? The loop just… is. It exists to allow you to do certain other things, it doesn’t have much purpose in and of itself.

Applying what I just talked about, we could use TDD to create the main loop of a game. It definitely requires thinking about it differently and breaking preconceptions. It’s about writing code to pass the tests, not writing tests to fit the code we want to write.

So we could do something like this:

TEST(MainLoopExecutesCorrectNumberOfTimes)
{
    MainLoopState loopState;
    MainLoopUtils::Execute(loopState, 4);
    CHECK_EQUAL(4, loopState.frameCount);
}

All of a sudden we see a way to test something that before it seemed impossible to test. We can add tests to make sure certain functions are getting called (maybe we add those function pointers to the loopState), that the main loop exits under certain conditions, etc.

For some games with complex GUIs and different game modes, it might make a lot of sense to have a more flexible main loop and develop it completely through TDD. On the other hand, for a simple iPhone game that has a single main loop that never changes, there might be no point at all in using TDD. The loop code is going to end up looking more or less the same as it would without TDD, just more complex and generic. TDD didn’t help any in that case.

How about code that calls into OpenGL, or some other non-TDD-friendly library? Do we have to TDD that? Frankly, I wouldn’t bother. I did give it a good try at one point several years ago and I can say it is possible, but it’s a pain and you gain almost nothing from it. The main benefits is trusting that the library you’re using really does what it’s supposed to do, but it won’t affect much the design of your code. So what you’re really doing is adding tests to that library (which might or might not be of value).

My advice in a situation like that: Call the library from as few places as possible (I didn’t want to say make a thin wrapper because that implies isolation which is not the goal here), and just test that your code is called at the right time with the right data. The actual code that calls the library should be small and simple enough that you should feel OK not having any tests for it.

Even though it seems to go against what I said in the previous section, TDD is not all or nothing. It’s not a religion.

When you’re starting out, I encourage you to push yourself to think how you could apply TDD to situations you don’t think are possible. Once you’re experienced enough, you’ll know when to say enough is enough, and use TDD only when it actually benefits you. At that point you’ll also know how to write code in a way that plays well with TDD, even if the code itself was not developed through TDD.

In the end, TDD is a tool to help you develop better and faster. Don’t ever let it get in the way of that.

[1] That’s actually a horrible place to apply TDD. More on that in a second.

[2] Usable API, simple code, unit tests, documentation, safety net for refactoring among others.

Joey was kind enough to share the story behind Unearthed, some of the decisions leading up to the final game, and the initial reception and sales numbers. Thanks Joey!

Inspiration

Nine months ago, without warning, a gnome crept into my head. It was an idea inspired from a friend’s chance mention of an activity called geocaching, where participants use a GPS to hide and seek containers across the world. Within hours, I had envisioned a global cross between geocaching, the Amazing Race, and online travel agency representatives. The idea didn’t look much like the resultant product nine months later, but it formed the most critical ingredient to any venture: the obsession to see it completed.

I spent my evenings for a few short weeks hammering out a design, scoping and cutting features, and beginning a prototype, but it didn’t take long for me to admit to the unavoidable truth that such a project would take months working full time or probably 3 to 4 times as many months working nights and weekends. I had to sit and carefully consider the pros and cons of leaving my job to pursue something I desperately wanted to create.

Half-Hearted Dissent

For two weeks, I debated with my wife and close friends with startup experience. I drafted a list of pros and cons to quitting my job to pursue an iphone project and stared at it for hours. Those of you who have made the leap or considered it may recognize some of the points or have more of your own to add:

Advantages of quitting to work full time on an iphone idea

• Focus effort, complete project 3-4 times faster
• Gain valuable unique experience of running solo
• Avoid coding burnout, alienating loved ones, loss of sanity
• Resume padding

Advantages of keeping day job while working on iphone idea

• Steady income in a scary economy
• Not having to job search in a scary economy
• Peace of mind (in a scary economy)

You can see where a large portion of the dissenting argument forms its basis. After two weeks of listening to unanimous encouragement (bordering on persistent nagging) to quit my job, two weeks of trying my best to convince myself why I should not quit my job, I did what my gut knew from day one was going to be best for me. I marched to my boss’s room and quit my job. Ok, I had maybe two false starts where I turned around and went back to my seat.

The Project Unearthed

Over the next 8 months, scoping and an inclination to appeal to a casual segment evolved the project into a global paranormal investigation titled Unearthed. I tried my best to figure how to maximize the success of the product, and compiled a list of all the ways I could imagine increasing the product’s exposure:

• Banner ads
• Spamming blogs and websites to review my product
• Spamming friends on facebook
• Writing a blog
• Viral app features

I had trouble justifying the cost to pay for banner ads, and I had too much urgency to implement the product to devote time to a blog, so I focused on occasionally talking about the app on facebook, compiling a list of potential blog candidates for the day the app released, and devising viral features.

Apart from typical facebook posting and email-a-friend features, I decided to employ a “refer-an-agent” feature which would enable users to invite others to join their network, essentially a grouping of users that correlated with how effectively their app could process scanned creature data. The larger the network, the more credit users would receive for uploading data, and the better they would perform in leaderboards and achievements. The approach was an “everybody wins” style where anytime any user in the network gained a referral, every single person in the network would benefit from the growth of the network. The hope was that this feature would gain a viral quality.

Friends placed immense pressure to release the product as soon as humanly possible under the theory that doing so would reveal the 70% of my design that was wrong, so I divided Unearthed into three releases. The app itself has three game modes, each incrementally accessible as a given region “levels up” from users’ data uploads. I would have time to release the latter modes while the first mode of the game, the most casual mode where users look around where they stand and scan for anomalies, was being played.

In App Purchases and Ads

Considering the App Store was flooded with free apps and that I had no name in the industry to immediately convince users to immediately pay for my app, I concluded that the most likely approach for success was to release a free app with In App Purchases. This appeared to be the best way to get as many users to at least try the app and decide how much they wanted to spend to access more functionality. Initially, I offered the following items:

• All content, present and future ($5.99)
• Scanner upgrades level 2-3 ($1.99)
• Scanner upgrades level 4-5 ($1.99)
• Bounty Mode (coming soon) ($1.99)
• Blitz Mode (coming soon) ($1.99)

However, I hit a snag with Apple’s terms which did not permit me to even mention any features that were not yet implemented. The first item was intended to be a bulk pricing investment in the forthcoming completion of the app, and the ‘coming soon’ items were just client-side displays to tease of the future modes (not actually registered products in the IAP servers). Because I had to remove them, I revised my IAP content to the following:

• Scanner upgrades level 2 ($.99)
• Scanner upgrades level 3 ($.99)
• Scanner upgrades level 4 ($.99)
• Scanner upgrades level 5 ($.99)
• Scanner upgrades 2-5 ($2.99)

The app essentially gives users access to all features of the game, but with a lowly basic scanner that doesn’t boast the speed, range, and creature clearance levels of a fully boosted scanner.

I added Greystripe’s ad system as a means to offset the cost I might incur from using Google App Engine as my server solution. Considering the large amount of rank tracking required by my app, I estimated the revenue from ads to break even with the cost of using App Engine.

Release and Reception

The app was approved in the early afternoon of June 21st, a Monday. Excited, I quickly moved up the release date and it hit the stores later that afternoon. One week later, the sales are dismal, with about 400 total downloads and only a handful of purchases.

I considered possible contributors to this could be the following (in order of impact):

• Requiring 3GS
• Releasing 4 days before the iPhone 4 (dropping off New Releases)
• Requiring OS 3.1.3 (instead of 3.1)
• Releasing the app in the late afternoon (dropping off New Releases faster)

Without knowing the specifics on the sales of the 3GS versus the 3G and the iPod Touch, the fraction of 3GS users able to even see my app in the store might be a small fraction. Unnecessarily releasing under an OS that prevented users from installing unless they went back to iTunes to update their OS was likely a dealbreaker for browsing users (quickly remedied but days later). And considering how many of the 1.5 million users would be browsing the App Store for free apps Thursday, releasing a few days too early may have been the biggest avoidable mistake yet.

Moving Forward

As uninspiring as it was getting only a few hundred downloads and enough revenue for lunch, I had heard that word of mouth was a powerful vehicle, possibly the best vehicle, for an unknown developer, so my plan is to continue giving it more time and implement the GPS-enabled Bounty Mode which will take considerably less time, possibly a short 1-2 months. One week is certainly not a large enough measure of success, and already I’ve received a few responses from blogs willing to write reviews, so I’m hopeful of seeing the userbase grow and eagerly anticipating the next big server issue. In the coming weeks, there will definitely be no “sitting back while the cash rolls in”, as I recently had the blessing/curse of another inspiring must-create iPad app revelation and am sweating bullets designing and scoping while scheduling in work on Unearthed, all the while keeping a grave eye on the lifeline of my business.

Looking back at the struggling decision I made to quit my job, I’m reminded of some advice that was given to me beforehand. “Quitting your job will be the hardest decision, but it will make you feel great.” And like an enormous burden lifted from me, it did. And to this day, with earned lunch money in hand, it continues to.

This was one of the first columns I wrote. At first glance it might seem a completely introductory topic, not worth spending that much time on it. After all, all experience programmers know about pointers and indices, right? True, but I don’t think all programmers really take the time to think about the advantages and disadvantages of each approach, and how it affects architecture decisions, data organization, and memory traversal.

It should provide a good background for this coming Thursday’s #iDevBlogADay post on how to deal with heterogeneous objects in a data-oriented way._

From a 10,000-Foot view, all video games are just a sequence of bytes. Those bytes can be divided into code and data. Code is executed by the hardware and it performs operations on the data. This code is generated by the compiler and linker from the source code in our favorite computer language. Data is just about everything else. [1]

As programmers, we’re obsessed with code: beautiful algorithms, clean logic, and efficient execution. We spend most of our time thinking about it and make most decisions based on a code-centric view of the game.

Modern hardware architectures have turned things around. A data-centric approach can make much better use of hardware resources, and can produce code that is much simpler to implement, easier to test, and easier to understand. In the next few months, we’ll be looking at different aspects of game data and how everything affects the game. This month we start by looking at how to manage data relationships.

Data Relationships

Data is everything that is not code: meshes and textures, animations and skeletons, game entities and pathfinding networks, sounds and text, cut scene descriptions and dialog trees. Our lives would be made simpler if data simply lived in memory, each bit totally isolated from the rest, but that’s not the case. In a game, just about all the data is intertwined in some way. A model refers to the meshes it contains, a character needs to know about its skeleton and its animations, and a special effect points to textures and sounds.

How are those relationships between different parts of data described? There are many approaches we can use, each with its own set of advantages and drawbacks. There isn’t a one-size-fits-all solution. What’s important is choosing the right tool for the job.

Pointing The Way

In C++, regular pointers (as opposed to “smart pointers” which we’ll discuss later on) are the easiest and most straightforward way to refer to other data. Following a pointer is a very fast operation, and pointers are strongly typed, so it’s always clear what type of data they’re pointing to.

However, they have their share of shortcomings. The biggest drawback is that a pointer is just the memory address where the data happens to be located. We often have no control over that location, so pointer values usually change from run to run. This means if we attempt to save a game checkpoint which contains a pointer to other parts of the data, the pointer value will be incorrect when we restore it.

Pointers represent a many-to-one relationship. You can only follow a pointer one way, and it is possible to have many pointers pointing to the same piece of data (for example, many models pointing to the same texture). All of this means that it is not easy to relocate a piece of data that is referred to by pointers. Unless we do some extra bookkeeping, we have no way of knowing what pointers are pointing to the data we want to relocate. And if we move or delete that data, all those pointers won’t just be invalid, they’ll be dangling pointers. They will point to a place in memory that contains something else, but the program will still think it has the original data in it, causing horrible bugs that are no fun to debug.

One last drawback of pointers is that even though they’re easy to use, somewhere, somehow, they need to be set. Because the actual memory location addresses change from run to run, they can’t be computed offline as part of the data build. So we need to have some extra step in the runtime to set the pointers after loading the data so the code can use them. This is usually done either by explicit creation and linking of objects at runtime, by using other methods of identifying data, such as resource UIDs created from hashes, or through pointer fixup tables converting data offsets into real memory addresses. All of it adds some work and complexity to using pointers.

Given those characteristics, pointers are a good fit to model relationships to data that is never deleted or relocated, from data that does not need to be serialized. For example, a character loaded from disk can safely contain pointers to its meshes, skeletons, and animations if we know we’re never going to be moving them around.

Indexing

One way to get around the limitation of not being able to save and restore pointer values is to use offsets into a block of data. The problem with plain offsets is that the memory location pointed to by the offset then needs to be cast to the correct data type, which is cumbersome and prone to error.

The more common approach is to use indices into an array of data. Indices, in addition to being safe to save and restore, have the same advantage as pointers in that they’re very fast, with no extra indirections or possible cache misses.

Unfortunately, they still suffer from the same problem as pointers of being strictly a many-to-one relationship and making it difficult to relocate or delete the data pointed to by the index. Additionally, arrays can only be used to store data of the same type (or different types but of the same size with some extra trickery on our part), which might be too restrictive for some uses.

A good use of indices into an array are particle system descriptions. The game can create instances of particle systems by referring to their description by index into that array. On the other hand, the particle system instances themselves would not be a good candidate to refer to with indices because their lifetimes vary considerably and they will be constantly created and destroyed.

It’s tempting to try and extend this approach to holding pointers in the array instead of the actual data values. That way, we would be able to deal with different types of data. Unfortunately, storing pointers means that we have to go through an extra indirection to reach our data, which incurs a small performance hit. Although this performance hit is something that we’re going to have to live with for any system that allows us to relocate data, the important thing is to keep the performance hit as small as possible.

An even bigger problem is that, if the data is truly heterogeneous, we still need to cast it to the correct type before we use it. Unless all data referred to by the pointers inherits from a common base class that we can use to query for its derived type, we have no easy way to find out what type the data really is. On the positive side, now that we’ve added an indirection (index to pointer, pointer to data), we could relocate the data, update the pointer in the array, and all the indices would still be valid. We could even delete the data and null the pointer out to indicate it is gone. Unfortunately, what we can’t do is reuse a slot in the array since we don’t know if there’s any data out there using that particular index still referring to the old data.

Because of these drawbacks, indices into an array of pointers is usually not an effective way to keep references to data. It’s usually better to stick with indices into an array of data, or extend the idea a bit further into a handle system, which is much safer and more versatile.

Handle-Ing The Problem

Handles are small units of data (32 bits typically) that uniquely identify some other part of data. Unlike pointers, however, handles can be safely serialized and remain valid after they’re restored. They also have the advantages of being updatable to refer to data that has been relocated or deleted, and can be implemented with minimal performance overhead.

The handle is used as a key into a handle manager, which associates handles with their data. The simplest possible implementation of a handle manager is a list of handle-pointer pairs and every lookup simply traverses the list looking for the handle. This would work but it’s clearly very inefficient. Even sorting the handles and doing a binary search is slow and we can do much better than that.

Here’s an efficient implementation of a handle manager (released under the usual MIT license, so go to town with it). The handle manager is implemented as an array of pointers, and handles are indices into that array. However, to get around the drawbacks of plain indices, handles are enhanced in a couple of ways.

In order to make handles more useful than pointers, we’re going to use up different bits for different purposes. We have a full 32 bits to play with, so this is how we’re going to carve them out:

• The index field. These bits will make up the actual index into the handle manager, so going from a handle to the pointer is a very fast operation. We should make this field as large as we need to, depending on how many handles we plan on having active at once. 14 bits give us over 16,000 handles, which seems plenty for most applications. But if you really need more, you can always use up a couple more bits and get up to 65,000 handles.

• The counter field. This is the key to making this type of handle implementation work. We want to make sure we can delete handles and reuse their indices when we need to. But if some part of the game is holding on to a handle that gets deleted—and eventually that slot gets reused with a new handle—how can we detect that the old handle is invalid? The counter field is the answer. This field contains a number that goes up every time the index slot is reused. Whenever the handle manager tries to convert a handle into a pointer, it first checks that the counter field matches with the stored entry. Otherwise, it knows the handle is expired and returns null.

• The type field. This field indicates what type of data the pointer is pointing to. There are usually not that many different data types in the same handle manager, so 6–8 bits are usually enough. If you’re storing homogeneous data, or all your data inherits from a common base class, then you might not need a type field at all.

struct Handle
{
    Handle() : m_index(0), m_counter(0), m_type(0)
    {}

    Handle(uint32 index, uint32 counter, uint32 type)
        : m_index(index), m_counter(counter), m_type(type)
    {}

    inline operator uint32() const;
    
    uint32 m_index : 12;
    uint32 m_counter : 15;
    uint32 m_type : 5;
};

Handle::operator uint32() const
{
    return m_type << 27 | m_counter << 12 | m_index;
}

The workings of the handle manager itself are pretty simple. It contains an array of HandleEntry types, and each HandleEntry has a pointer to the data and a few other bookkeeping fields: freelist indices for efficient addition to the array, the counter field corresponding to each entry, and some flags indicating whether an entry is in use or it’s the end of the freelist.

struct HandleEntry
{
	HandleEntry();
	explicit HandleEntry(uint32 nextFreeIndex);
	
	uint32 m_nextFreeIndex : 12;
	uint32 m_counter : 15;
	uint32 m_active : 1;
	uint32 m_endOfList : 1;
	void* m_entry;
};

Accessing data from a handle is just a matter of getting the index from the handle, verifying that the counters in the handle and the handle manager entry are the same, and accessing the pointer. Just one level of indirection and very fast performance.

We can also easily relocate or invalidate existing handles just by updating the entry in the handle manager to point to a new location or to flag it as removed.

Handles are the perfect reference to data that can change locations or even be removed, from data that needs to be serialized. Game entities are usually very dynamic, and are created and destroyed frequently (such as enemies spawning and being destroyed, or projectiles). So any references to game entities would be a good fit for handles, especially if this reference is held from another game entity and its state needs to be saved and restored. Examples of these types of relationships are the object a player is currently holding, or the target an enemy AI has locked onto.

Getting Smarter

The term smart pointers encompasses many different classes that give pointer-like syntax to reference data, but offer some extra features on top of “raw” pointers.

A common type of smart pointer deals with object lifetime. Smart pointers keep track of how many references there are to a particular piece of data, and free it when nobody is using it. For the runtime of games, I prefer to have very explicit object lifetime management, so I’m not a big fan of this kind of pointers. They can be of great help in development for tools written in C++ though.

Another kind of smart pointers insert an indirection between the data holding the pointer and the data being pointed. This allows data to be relocated, like we could do with handles. However, implementations of these pointers are often non- serializable, so they can be quite limiting.

If you consider using smart pointers from some of the popular libraries (STL, Boost) in your game, you should be very careful about the impact they can have on your build times. Including a single header file from one of those libraries will often pull in numerous other header files. Additionally, smart pointers are often templated, so the compiler will do some extra work generating code for each data type you instantiated templates on. All in all, templated smart pointers can have a significant impact in build times unless they are managed very carefully.

It’s possible to implement a smart pointer that wraps handles, provides a syntax like a regular pointer, and it still consists of a handle underneath, which can be serialized without any problem. But is the extra complexity of that layer worth the syntax benefits it provides? It will depend on your team and what you’re used to, but it’s always an option if the team is more comfortable dealing with pointers instead of handles.

Conclusion

There are many different approaches to expressing data relationships. It’s important to remember that different data types are better suited to some approaches than others. Pick the right method for your data and make sure it’s clear which one you’re using.

In the next few months, we’ll continue talking about data, and maybe even convince you that putting some love into your data can pay off big time with your code and the game as a whole.

This article was originally printed in the September 2008 issue of Game Developer.

[1] I’m not too happy about the strong distinction I was making between code and data. Really, data is any byte in memory, and that includes code. Most of the time programs are going to be managing references to non-code data, but sometimes to other code as well: function pointers, compiled shaders, compiled scripts, etc. So just ignore that distinction and think of data in a more generic way.

Check out the new sidebar with all the #iDevBlogADay blogs. We’re also putting together a common RSS feed if you want to subscribe to that instead.

Writing is addictive, so don’t be surprised if this once-a-week minimum turns into multiple-times-a-week.

Every developer who’s been working on a team for a while is able to tell the author of a piece of code just by looking at it. Sometimes it’s even fun to do a forensic investigation and figure out not just the original author, but who else modified the source code afterwards.

What I find interesting is that I can do the same thing with my own code… as it changes over time. Every new language I learn, every book I read, every bit of code I see, every open-source project I browse, every pair-programming session, every conversation with a fellow developer leaves a mark behind. It slightly changes how I think of things, and realigns my values and priorities as a programmer. And those new values translate into different ways to write code, different architectures, and different coding styles.

It never happens overnight. I can’t recall a single case where I changed my values in a short period of time, causing dramatic changes to my coding style. Instead, it’s the accumulation of lots of little changes here and there that slowly shifts things around. It’s like the movement of the Earth’s magnetic pole: very slow, but changes radically over time (although maybe just a tad bit faster).

Why Talk About Coding Styles

Coding style in itself is purely a personal thing, and therefore, very uninteresting to talk about. However, in its current form, my coding style goes against the grain of most general modern “good practices”. A few weeks ago I released some sample source code and it caused a bit of a stir because it was so unconventional. That’s when I realized it might be worth talking about it after all (along with George bugging me about it), and especially the reasons why it is the way it is.

Before I even start, I want to stress that I’m not advocating this approach for everybody, and I’m certainly not saying it’s the perfect way to go. I know that in a couple of years from now, I’ll look back at the code I’m writing today and it will feel quaint and obsolete, just like the code I wrote during Power of Two Games looks today. All I’m saying is that this is the style that fits me best today.

Motivation

This is my current situation which shapes my thinking and coding style:

• All my code is written in C and C++ (except for a bit of ObjC and assembly).
• It’s all for real-time games on iPhone, PCs, or modern consoles, so performance and resource management are very important.
• I always try to write important code through Test-Driven Development.
• I’m the only programmer (and only designer).
• Build times in my codebase are very fast.

And above all, I love simplicity. I try to achieve simplicity by considering every bit of code and thinking whether it’s absolutely necessary. I get rid of anything that’s not essential, or that’s not benefitting the project by at least two or three times as much as it’s complicating it.

How I Write Today

So, what does my code look like these days? Something like this (this is taken from a prototype I wrote with Miguel of Mystery Coconut fame):

namespace DiverMode
{
    enum Enum
    {
        Normal,
        Shocked,
        Inmune,
    };
}

struct DiverState
{
    DiverState()
        : mode(DiverMode::Normal)
        , pos(0,0)
        , dir(0)
        , o2(1)
        , boostTime(0)
        , timeLeftInShock(0)
        , timeLeftImmune(0)
    {}

    DiverMode::Enum mode;
    Vec2 pos;
    float dir;
    float o2;

    float boostTime;
    float timeLeftInShock;
    float timeLeftImmune;
};

namespace DiverUtils
{
    void Update(float dt, const Vec2& tiltInput, GameState& state);
    void Shock(DiverState& diver);
    void StartSprint(DiverState& diver);
    void StopSprint(DiverState& diver);
}

The first thing that stands out is that I’m using a struct and putting related functions in a namespace. It may seem that’s just a convoluted way of writing a class with member functions, but there’s more to it than that.

By keeping the data in a struct instead of a class, I’m gaining several advantages:

• I’m showing all the data there is and how big it is. Nothing is hidden.
• I’m making it clear that it’s free of pointers and temporary variables.
• I’m allowing this data to be placed anywhere in memory.

The fact that the functions are part of a namespace is not really defensible; it’s pure personal preference. It would have been no different than if I had prefixed them with DriverUtils_ or anything else, I just think it looks clearner. I do prefer the functions to be separate and not member functions though. It makes it easier to organize functions that work on multiple bits of data at once. Otherwise you’re stuck deciding whether to make them members of one structure or another. It also makes it easier to break up data structures into separate structures later on and minimize the amount of changes to the code.

Probably one of the biggest influences on me starting down this path was the famous article by Scott Meyers How Non Member Functions Improve Encapsulation. I remember being shocked the first time I read it (after having read religiously Effective C++ and More Effective C++). That reasoning combined with all the other changes over the years, eventually led to my current approach.

Since everything is in a structure and everything is public, there’s very little built-in defenses against misuse and screw-ups. That’s fine because that’s not a priority for me. Right now I’m the only programmer, and if I work with someone else, I expect them to have a similar level of experience than me. Some codebases written with a defensive programming approach have an amazing amount of code (and therefore complexity) dedicated to babysitting programmers. No thanks. I do make extensive use of asserts and unit tests to allow me to quickly make large refactorings though.

Another thing to note that might not be immediately obvious from the example above is that all functions are very simple and shallow. They take a set of input parameters, and maybe an output parameter or just a return value. They simply transform the input data into the output data, without making extensive calls to other functions in turn. That’s one of the basic approaches of data-oriented design.

Because everything is laid out in memory in a very simple and clear way, it means that serialization is a piece of cake. I can fwrite and fread data and have instant, free serialization (you only need to do some extra work if you change formats and try to support older ones). Not only that, but it’s great for saving the game state in memory and restoring it later (which I’m using heavily in my current project). All it takes is this line of code:

oldGameState = currentGameState

This style is a dream come true for Test-Driven Development (TDD). No more worrying about mocks, and test injections, or anything like that. Give the function some input data, and see what the output is. Done! That simple.

One final aspect of this code that might be surprising to some is how concrete it is. This is not some generic game entity that hold some generic components, with connections defined in XML and bound together through templates. It’s a dumb, POD Diver structure. Diver as in the guy going swimming underwater. This prototype had fish as well, and there was a Fish structure, and a large array of sequential, homogeneous Fish data. The main loop wasn’t generic at all either: It was a sequence of UpdateDivers(), UpdateFish(), etc. Rendering was done in the same, explicit way, making it extra simple to minimize render calls and state changes. When you work with a system like this, you never, ever want to go back to a generic one where you have very little idea about the order in which things get updated or rendered.

Beyond The Sample

To be fair, this sample code is very, very simple. The update function for a reasonable game element is probably larger than a few lines of code and will need to do a significant amount of work (check path nodes, cast rays, respond to collisions, etc). In that case, if it makes sense, the data contained in the structure can be split up. Or maybe the first update function generates some different output data that gets fed into later functions. For example, we can update all the different game entities, and as an output, get a list of ray cast operations they want to perform, do them all in a later step, and then feed the results back to the entities either later this frame or next frame if we don’t mind the added latency.

There’s also the question of code reuse. It’s very easy to reuse some low level functions, but what happens when you want to apply the same operation to a Diver and to a Fish? Since they’re not using inheritance, you can’t use polymorphism. I’ll cover that in a later blog post, but the quick preview is that you extract any common data that both structs have and work on that data in a homogeneous way.

What do you think of this approach? In which ways do you think it falls short, and in which ways do you like it better than your current style?

First one bit of good news: That prompt-on-delete feature seems to be gone on iOS 4.0. That should make everybody happy. Even so, taking a pro-active position on this can only benefit developers.

Next a personal opinion without any real hard data to back it up: I don’t think App Store ratings matter very much. I see plenty of quality, 5-star apps that never go anywhere, as well as horrible apps with close to 1-star rating on the top 10. I’m sure good ratings help a bit, but not as much as some people may think.

One of the little features I snuck in the latest update of Flower Garden, was a prompt to rate (or re-rate) the app. This idea isn’t new, and I’ve seen quite a few apps doing it already. I think it makes a lot of sense and provides rating that is much more balanced and representative of how users feel about the app.

You have to be careful about how you prompt users though. This are the priorities I had in mind when I implemented the rating prompt in Flower Garden, roughly in order of importance:

1. Don’t be annoying!
2. Disrupt the user as little as possible
3. Make rating the app as easy as possible.
4. Prompt at every update.

Don’t be annoying

This is by far the most important priority. If your prompt for review is annoying, you might get more angry users and one-star reviews than you would have without it. Be respectful of the user and don’t nag.

For quite a few versions, I’ve had a review button in the “Feedback” tab in the settings screen. Not many people use it or even know about it because it was several taps away from the normal flow of Flower Garden. That’s an example of a feedback prompt that is too unobtrusive. Clearly, I needed something a bit more noticeable.

What I did was to add an actual alert view with the prompt. That alert comes up very infrequently, and gives users the option to either rate it right away, be reminded later, or just stop asking altogether. Many more users rated the app through this mechanism (and I haven’t gotten a single complaint about it being annoyed yet–keeping my fingers crossed).

Disrupt the user as little as possible

First of all, you should never ask the user to rate the first time they launch it. They haven’t had a chance to check it out yet! Give them a few days (or a few launches of the app). That will benefit you in two ways: users will have a better chance to give an honest rating, and users who didn’t like your app probably won’t be around in two days, so they’ll never see the prompt.

Second, choose a good time to ask. Please, don’t ask me as soon as I launch the app! Chances are I want to do something with it, so being pestered with a review prompt is disruptive, especially because adding a rating involves leaving the app and going to iTunes. At the same time, don’t ask me in the middle of a level or when I’m engrossed in game. A good moment to ask for a rating would be after completing a level or two, or, in the case of Flower Garden, after sending a bouquet.

For extra bonus points, catch the user on a “good” moment. Maybe only ask them to rate the app if they actually beat the level. They’ll probably leave a better review than if they just lost and are in a bad mood.

The exception I made was if a certain amount of time went by and the user never got to the point of being prompted (some people just don’t send bouquets in Flower Garden). In that case, I will ask at startup or at some other, non-ideal point.

Make rating the app as easy as possible

This is the funnel principle at work. For every action the user has to do in order to leave a review, the fewer users will actually do it. It would be ideal if users could rate the app directly from the alert box. That would involve no extra actions (or leaving the app) and would result in the maximum number of ratings [1]. Unfortunately, that’s not the case, so we have to do the best we can with what we have.

So we want to minimize the amount of actions from the moment the user taps “Rate now” until they can enter the rating. We could send them to the product page on the App Store, but that would still involve them scrolling around, finding the rating button, tapping it, and entering the rating. We can do better than that: We can send them directly to the rating screen. Check out this great post for a dissection of iTunes links and how to create one to the review page. Make sure the technique you choose works on an iPhone (not all of them do!).

Prompt at every update

This one is easy: Just store the number of the last version the user wrote a review for, and if it changes, start from scratch again. That way you don’t have to write any custom code at every release. Also, make sure to respect the “Don’t ask anymore” option if the user selected that in a previous update.

Source code

Here’s the review request source code (released under the MIT license). It consists of only two standalone files, and all the review state is kept in the default, global settings. There are only three functions and it should be very self-explanatory:

	bool ShouldAskForReview();
	bool ShouldAskForReviewAtLaunch();
	void AskForReview();

The ShouldAskForReviewAtLaunch() function simply adds an extra condition of launching it X number of times before asking for review (this is in case the user never triggers the review condition).

Results

Does this prompt for ratings work? This is a screen grab from iTunes 5 days after Flower Garden 2.4 was released:

A few interesting observations:

• Ratings are higher than they were before. Before the free day, Flower Garden had an average of 3.5 to 4 stars.
• Reviews are often shorter and less useful than unprompted reviews. There are a lot of 5-star “I love this app!” reviews without much more substance. (I’m not complaining, really).
• Still a huge number of users chose not to leave a review. There are only 148 ratings, but there have been 30,000 upgrades to the latest version. A lot of people might have upgraded and not launched it, but even so, it’s a very small percentage.

Overall, it was a very small time investment and the results show it was definitely worth it. I’ll be using this approach in my future projects (unless future iOS versions take care of it automatically somehow).

[1] Apple, can we fix that? How about a REST API for leaving ratings and reviews? Pretty please?

And if you haven’t picked up your free copy of Flower Garden, make sure to do it now. It will probably go back to full price after this weekend.

Gracias! Obrigado! Grazie! Σας ευχαριστώ! Tak!

So here we go with the second part. How did Flower Garden fare after the new year? Was it just the Christmas purchasing frenzy that had a momentary effect on sales, or was there something more to it?

Recap and Overview

Here’s where we left off last time. The unusual part was how profits increased as soon as I added in-app purchases to Flower Garden in early December. It made for a very atypical sales plot.

And here is how things look now. This plot includes the previous data so it’s easier to contrast before and after. The area in blue is the new data since the last post.

Just glancing at that chart makes it clear that that the increase around Christmas wasn’t a fluke. It actually wasn’t even done going up. After all the spikes, and the weekly ups and downs, Flower Garden ended up settling to about $1,500/week. And that, even in California, I would consider it to be a comfortable living. What a different from the $50/day it was making last year!

So what exactly happened there? Let’s look at the new data in more detail.

At a glance, there are three, very different sections.

The Feature

The first one starts with a significant increase in sales (A), and is in large part due to being featured on the App Store worldwide as a Staff Favorite. It’s a nice spike, but it’s nothing like the x10 spikes other developers have seen with App Store features. That’s because the Staff Favorite slot, even though it’s a very prestigious one, it does not appear on the iPhone, only on iTunes. I do all my browsing and shopping through iTunes, but apparently I’m in the minority, so the effect on sales is greatly reduced.

That spike is also in part due to my last “numbers post”, which happened right at the same time (not completely accidentally). The page attracted about 15,000 views in a few days, so I’m sure a few of them translated in people checking out Flower Garden out of curiosity.

Valentine’s Day

The second spike isn’t hard to guess: It’s the weekend of February 14th, Valentine’s Day. The more things happen at once and the more something is in the public eye, the more of an impact it has. PR people have known that for a long time, and it was really brought home for me back in December.

Fortunately, I managed to make quite a few things happen in the days leading to Valentine’s Day weekend:

• I released an update with couple new in-app purchases: A new set of seeds (Seeds of Love), and a greenhouse garden, for $0.99 each.
• I sent out a newsletter to the 25,000 subscribers to the mailing list announcing the new items.
• I put Flower Garden on sale for $0.99 (down from the regular price of $2.99).
• Several web sites, including TouchArcade, covered the sale and the new update, giving it lots of visibility.
• Flower Garden Free was the free app for February 14th on the Valentine’s Day Calendar.

All of that combined to cause the big spike in profits (B). You really need to look at the first plot to put it in perspective. Two days in around Valentine’s Day had higher profit than the initial release spike back in April of last year!

Winding Down

Of course, everything that goes up, must eventually come down. So the weeks following Valentine’s Day profits went rapidly down. It was at the very end of April that I started working on Lorax Garden, so for that period of time I wasn’t able to do anything related to Flower Garden. For a while sales were dropping quite rapidly, but they eventually flattened out to about $1,500/week (C).

The good thing about having consumable items as in-app purchases (fertilizer in this case), is that profits are related to active user base, not just initial sales. So even though the amount of downloads decreased significantly during this time, the user base had grown a large amount, and with it, the daily profits.

Flower Garden Free

Back in December, there seemed to be a connection between the amount of downloads of the free version of Flower Garden and profits. They both picked up right around the time I added in-app purchases, although I was never able to tell if it was cause or consequence.

This is the amount of downloads of the free version for this period of time.

It stays pretty regularly at about 800 downloads per day (which is not much compared to a lot of free versions out there), and has a massive spike on Valentine’s Day (caused by word of mouth, lots of sent bouquets, and the Valentine’s Day Calendar).

Notice that the App Store feature for Flower Garden (full version) in January had virtually no effect on downloads of the free version. It seems that people are willing to buy something full price without trying the free version first if Apple features it.

Both versions of Flower Garden (free and full) have in-app purchases in them. Even though there’s a higher percentage of users with the full version that buy in-app purchases, the free version is much more popular and the majority of the revenue comes from the free version (orange). This is a tren that was already noticable around the holidays, but now is much more clear, with over 50% of the revenue coming from in-app purchases in the free version.

The Future

The data for this post stops at May 5th. That’s because on May 6th I released a new set of seeds and gave Flower Garden away for free as part of the Mother’s Day promotion. As soon as the dust settles from that, I’ll write a third post detailing how it turned out.

Surprisingly, there wasn’t that much documentation on how to go about making a universal iPhone/iPad version. There’s a document from Apple showing some initial steps, but that’s about it. So I wanted to share what I learned along the way, including some very useful tips I learned through trial and error or by doing a lot of digging through Twitter and the development forums.

Flower Garden is a strange hybrid of OpenGL and UIKit, which made things more complicated than if it were just one or the other. Learning curve and all, it took two full days of my time. If you’re using just OpenGL or just UIKit, the conversion to a universal app will be significantly faster.

Universal Project

The first thing you need to do is understand what’s going on with all the SDK versions. At this time, the latest iPhone OS version is 3.1.3, but you’re going to be developing the universal app with SDK 3.2. You want the resulting binary to run on both 3.X OS on iPhones and 3.2 OS on iPad. It’s a very similar situation to when we were writing apps that worked on both 3.X and 2.X.

So fire up XCode 3.2.2 (yes, all those version numbers start getting very confusing–that’s the XCode version that comes with SDK 3.2), and load your iPhone project you want to make into a universal one. If you look under the Project menu item, you’ll see an entry called “Upgrade Current Target For iPad”. The Apple documentation even warns you not to create a universal app in any other way.

Go ahead and use it if you want. It will work… as long as you have a single target you want to convert. For some inexplicable reason [1], it will only work once per project (even though it’s phrased as working with whatever the current target is). When would you have more than one target executable? If you have a free and a paid version for example.

Besides, I’m uncomfortable with automated “smart” tools that do things behind my back without me knowing exactly what’s going on, so I upgraded by hand by looking at the diffs of what the upgrade command did. It turns out it’s extremely simple.

1. Under the project properties, set your base SDK to 3.2.
2. Set Architectures to “Optimized (arm6, arm7)”
3. Uncheck “Build Active Architectures Only” 5. In the Deployment section, set it to SDK 3.1.3 (or whatever 3.X you want to target)

That’s it, really. That’s all you need to compile and run your app both on an iPhone and an iPad. When you build for the device, it will compile both arm6 and arm7 versions and create a combined fat executable with both. And yes, this means the size of your executable is going to double (which could be an issue if you’re near the 20MB limit). In the case of Flower Garden, the combined executable is 3.4MB, so that’s not a big deal.

iPad Functionality

What about that new xib file that the upgrade process creates? You don’t really need it. It’s there in case you want to have a different set of xibs for each platform, and it’s hooked up to the info file so the app knows to load it at startup.

For the universal version of Flower Garden, I wasn’t trying to make a whole new brand experience on the iPad. Instead, I was looking for a quick and easy way to make it look much better. Because of it, I decided to reuse the same xib files as for the iPhone version.

That meant I had to go in Interface Builder, and adjust the autosize properties for a lot of the views. Some of them I wanted to stretch and get bigger with the high resolution iPad screen (background views). Others, I wanted to leave at the same size, but remain at the same relative distance from a particular border (buttons). Some other ones, I left the same size and their position just scaled up with the resolution (info boxes). All in all, that was the most time-consuming part of the process. It also required a few tweaks here and there to support the resolution change correctly.

The other big change was updating the resolution of the 3D views. Fortunately, that was just worked without a single line change. The render target code I’m using, takes the view dimensions and creates a frame buffer of the correct size. And it’s not just the dimensions: Remember that the iPad has a different aspect ratio (grumble, grumble), so you need to make sure your perspective and orthographic projections take that into account.

The only other changes I had to make was supporting an upside-down portrait orientation. That was very easy because the frame buffer didn’t have to change. Make sure you support it both at launch time and during gameplay. If you have a root controller whose view is attached directly to the main window from the start, it’s as easy as adding this to your controller:

- (BOOL)shouldAutorotateToInterfaceOrientation:(UIInterfaceOrientation)interfaceOrientation 
{
    return (interfaceOrientation == UIInterfaceOrientationPortrait ||
            interfaceOrientation == UIInterfaceOrientationPortraitUpsideDown);
}

And make sure you add the supported orientations to your plist.

Finally, before you submit it to the App Store, you’ll need an iPad-specific icon. The documentation explains how to explicitly list all the icons in the info file, but at the very least, you can provide a file called Icon-72.png that is a 72x72 image and you’re done.

Running On The Simulator

This is one that should be a lot easier than it is. You’re creating a universal binary with SDK 3.2. Now you want to run it on the simulator. No problem, you run it as usual and you get the iPad simulator. You can debug it and do everything you normally do.

Now you want to run it on the simulator on iPhone mode. It turns out, that’s not so obvious.

You can’t just turn the simulator hardware setting to iPhone, because whenever you launch it from XCode it will override that again with the iPad one. Building and running on the 3.1.3 SDK is a no-no because you’re really running a different build than you’re going to be submitting (3.2) and all 3.2 SDK features you’re using are going to result in compile errors.

So after much searching and tweeting, here’s the solution:

• Build for 3.2 SDK on the simulator platform.
• Change project drop-down to 3.1.3 SDK - Launch with debugger (Cmd + Options + Y). Don’t build and run!

That will launch the simulator on iPhone mode, but still run your 3.2 build. Talk about unintuitive! [2]. The worst thing is, when you’re done, you need to switch the project to 3.2 again or you’ll get tons of compiler errors. That definitely has to go for SDK 4.0.

Here’s an invaluable tip. Maybe it’s obvious to long-term Mac users, but it baffled me for a while. The iPad simulator is very well thought out, and it has a 100% and a 50% mode. That makes it possible to use the simulator on screens without very high resolution. Even my external monitor at 1680x1050 can’t display the iPad simulator at full 1024x768 resolution in portrait mode because of the window borders.

The problem comes in when you want to take a screenshot to add to the App Store or anything else. On the iPhone simulator it was easy, but how do you do that with the iPad since you can’t fit it on the screen? Cmd + Ctrl + C will take a screenshot at full resolution even when running at 50% mode and add it to the clipboard! Switch over to Preview and select New from Clipboard and voila! Full resolution screenshot!

SDK 3.2 Features

Chances are that even if you do a simple iPad port, you’re going to end up using a few 3.2 features. The main one I used in the case of Flower Garden is a UIPopoverController. Here you should follow the steps outlined in the Apple documentation down to a T. But it comes down to this: If a symbol that is defined on 3.2 appears anywhere, even as a member pointer variable in a class, it will compile and run on the iPad fine, but will crash and burn on the iPhone.

So you need to both check that the 3.2 features are available, and you need to “hide” the new symbol so it never appears anywhere: Use an id variable and cast it based on the class info. Even casting it directly to the symbol you want will cause a crash.

Class classPopoverController = NSClassFromString(@"UIPopoverController");
if (classPopoverController)
{
    m_moreGamesPopover = [[classPopoverController alloc] initWithContentViewController:moreGames]; 
    [m_moreGamesPopover setDelegate:self];
    [m_moreGamesPopover setPopoverContentSize:CGSizeMake(320, moreGamesHeight) animated:NO];
    [m_moreGamesPopover presentPopoverFromRect:m_moreGamesButton.frame inView:self.view
                        permittedArrowDirections:UIPopoverArrowDirectionUp animated:YES];
}

Not pretty, uh? It works, but I couldn’t imagine doing this all over the place.

Gotchas

In the process of creating a universal app, I found a couple more things to watch out for.

When you submit the new binary through iTunes Connect and it’s accepted, the application status will change to something like “Missing screenshot”. If you go back and edit the app information, you’ll see there’s a new set of screenshots you can submit. You’ll need at least one for your application to enter “Waiting for Review” state. And if you have localized descriptions of your app, you’ll need to do that for every language.

Last, and perhaps most puzzling because I never figured this one out: I was not able to get in-app purchases to work with a test account from a development version of a universal app running on the iPad. I must have tried everything, but whenever I tried purchasing anything, it never brought up the familiar “sandbox environment” message. Also, items that hadn’t been approved yet did not show up in the list of available items. The exact same code worked fine on an iPhone, so that’s quite puzzling. Is it a major bug on Apple’s part, or did I miss some obscure “enable IAPs in debug mode” checkbox somewhere? It was a bit of a gamble submitting it like that, but fortunately, the approved version on the App Store allows in-app purchases without any problem.

I thought afterwards that maybe that was because the app version on the App Store was no universal, so the Store Kit server was not allowing the iPad version to access the store through the test account. But I tried it again even after it was approved and I had the same problem.

Has anybody managed to use an App Store test account from an iPad on a universal app?

Conclusion

All in all, it was a relatively painless process considering it’s different hardware, with a different resolution, and it’s the first iteration of the SDK. Was it worth it? I think so. Apart from looking a lot better on an iPad, universal apps get ranked on both iPhone and iPad charts. Flower Garden managed to get all the way up to #14 on the free games chart on the iPad (and around #60 free app overall on the iPad). I’m sure it got some exposure from being so high up, which in turn helped the iPhone rankings as well.

I can’t imagine that my next game is going to be iPad-only, but I’ll certainly have the iPad in mind from the beginning and release it on both platforms. Whether I choose to go universal or separate apps will depend on the game and business decisions, but at least it’s good to know that it’s fairly easy to create a universal app.

[1] Actually, there’s a pattern that is clear by looking at the hoops we have to jump to do a universal build: Apple was clearly scrambling to get this out the door. As a result, things are buggy, unintuitive, and clunky. Hopefully all that will be fixed for SDK 4.0.

[2] See? That confirms point [1].

Am I going crazy? Giving away the fruit of my labor for so many months? The App Store will do that to people, but this is a bit more sane. The idea is to encourage many more people to try it out. I’m actually hoping some of them will like it so much, they’ll buy a few of the in-app purchases from the Flower Shop. And even if they don’t, hopefully they’ll spread the (good) word about Flower Garden and they might also sign up for the mailing list. And in this business, expanding your user base is a great strategy that can be leveraged for future projects.

The idea of giving away a free app isn’t anything new. There are whole sites like FreeAppCalendar dedicated to doing exactly that, and other developers, like NimbleBit have been doing that on and off very successfully to reach a wider audience with their games.

How will it work out? I don’t know yet, but Flower Garden is quickly going up the charts even early on Saturday. It’s currently in the top 50 free apps in Japan, and about to get in the top 10 kids games in the US, so it’s looking good.

If you haven’t picked it up yet, this is the time. And if you have, make sure you tell your friends and family to download it and check it out before it goes back to the regular price. I’ll report back on the results of this promotion in a few days.

In late February I agreed to collaborate with Oceanhouse Media on a game project. They had the Dr. Seuss license and I had the flowers. Perfect match for a Lorax-themed game! And so the Lorax Garden was born.

I was originally hoping to write some entries here on the progress of the game as I worked on it, but I quickly realized that I wouldn’t have any spare time for anything other than the project. We wanted to have the Lorax Garden (and the Lorax book app) on the App Store by Earth Day (April 20th), so that left us with a very aggressive 5-week schedule (and GDC somewhere in the middle!). Fortunately, we managed to hit our deadline, and you can see the results today by yourself by downloading it from the App Store.

It was very interesting starting a totally new project from the existing Flower Garden codebase. I had to add some new flower-tech to make flowers more Seussian (curvy, uneven) and make Truffula Trees. Lorax Garden has a definite game structure and progression, so all the game code was completely new. It even has a flower-growing minigame to earn extra care hearts. All that had to be created by ripping out Flower Garden code and writing new one. It was a pleasure to see how easy it was to reuse all the flower-tech code that was well isolated and tested from the beginning though.

The Lorax Garden is currently featured by Apple on the App Store, and it’s on sale for $0.99 (down from the regular price of $2.99). All of that combined with a strong launch and mentions on USA Today and other media outlets have brought it up to the #39 (edit: #37!) paid app overall in the US App Store charts. A fantastic result, and with any luck it will continue climbing for another day or two.

The main difference is that I’m going to go in detail about each of the different cases of mixing OpenGL and UIKit, and what better way to do that than with a live demo. I’ll be switching back and forth between Keynote and XCode and going over the details, which is where a lot of the tricky parts are.

Here is the source code for the demo. That way you can look at it now and come to the session prepared with more questions, or ready to discuss what works and what doesn’t work for you. I love to have interactive sessions, so definitely come ready to ask questions.

If you’re not coming to 360iDev… shame on you! It’s the best iPhone conference around. You’ll be missing not just my session, but tons of other great talks, and the iPhone Game Jam (sponsored by TouchArcade this year). If you can’t make it this time, make a note on your calendar for next time!

• Demo source code. Released under the MIT license.

It was a 30-minute slot, so the material is pretty condensed without the chance to expand on the topics. I’m giving an extended version of this talk at 360iDev in a few weeks, so I’ll be able to get into more details then and have a cool sample app that shows off all those concepts.

It was great seeing a bunch of you at my session and at around the iPhone and Indie Summits. It was a great couple of days!

• Presentation slides (pdf format)
• Sample code

This year will GDC will host the first ever iPhone Summit on Tuesday and Wednesday. It’s great to see the iPhone getting that much recognition as a gaming platform. Definitely way more than last year’s presence at the conference. Both days are going to be packed with awesome sessions, and I will be giving a presentation on mixing OpenGL and UIKit on Tuesday at 11:15am. I’ll definitely be around throughout all the summit (except when I sneak out for a few minutes over to the Indie Summit which unfortunately happens at the same time).

The rest of the conference looks amazing as usual. There are always way more talks that I want to see than I can actually attend. In particular, I’m really looking forward to a lot of the design talks because they’re very inspiring and directly applicable to any kind of game development. Also, rumor has it I might be making a quick guest appearance in a certain session on Saturday.

Finally, if you’re an iPhone developer, make sure to come to the IDU party on Wednesday evening, and meet us for drinks on Thursday evening after the IGF ceremony.

I’ll be running around with a busy schedule, but you can always follow my Twitter updates to know where I’m at every moment. I look forward to seeing everybody again this year!

On a personal level, this nomination means quite a bit for me. Flower Garden has gotten some great reviews, and some incredible user feedback, and it was even featured as an Apple Staff Favorite on the App Store, but it had never been up for an award of this importance.

It’s also the category that it was nominated for. It makes me really proud that it was nominated for Most Innovative Game, because one of my original goals was to create something unique and original.

Finally, these awards are particularly meaningful because they’re not based on popularity, sales, or voting, which are bound to bias awards towards best-selling titles. Instead, these finalists were hand-picked by the editors of Pocket Gamer. With the crazy amount of good apps out there, it really is an honor that they selected Flower Garden as one of the top 5 for 2009.

Thank you, Pocket Gamer!

Shortly after the unveiling, people divided themselves in to two camps: the iPad haters, and the iPad defenders. Each of them was intent in convincing the other camp that their view is the right one. Just like most of the pointless human conflict over the last two thousand years minus the bloody wars part.

Of course, I felt the same way. As a consumer, I was very disappointed by the lack of video camera and the non-widescreen form factor. But I stepped back and realized that the iPad is here, and it is what it is. All the wishing and convincing in the world isn’t going to make a difference. The platform is here to stay, and with Apple’s genius behind it, I’m sure it’s going to be anywhere from “moderately successful” to “a huge success”.

The only question to answer as a developer is, how can I make the most out of the iPad? And for that, we need to understand what the iPad is.

There’s been a lot of talk on the iPad specs: 1024x768 resolution, a custom A4 chipset including a CPU and a modern GPU, etc. But those are just dry facts about the hardware. They will be crucial things to keep in mind to develop for the iPad, but they don’t help us that much understanding it.

Some people are claiming the iPad is a large iPod Touch, but I think that nothing could be further from the truth.

Precisely because the iPad is larger, it’s not a device you’re going to carry in your pocket. It’s something that’s going to be laying on your desk, or piled on top of a few books on the coffee table. And that changes everything.

Whereas on the iPhone/iPod Touch users could use an app for a few seconds or minutes during the day, the iPad is going to encourage longer sessions, just once or twice a day. The kind of apps, and in particular the kind of games that are going to be well suited to the iPad are going to be much more than a higher-resolution iPhone game. For the most part, they’re going to be totally different genres.

For example, Civilization never made much sense for me on the iPhone, but it now becomes a perfect iPad game. On the other hand, Flower Garden, was designed from the beginning to have multiple, short sessions throughout the day, so it will make very little sense as an iPad game. I suppose that makes me one of the first developers going on record saying I’m not going to port my existing iPhone app to the iPad :-)

The differences with the iPod Touch go beyond how frequently people use it. For example, games controlled by tilt are going to be restricted to a more hard core audience. Someone who really wants to sit down and spin the iPad around like a driving wheel controlling a car. Casual users will be less likely to grab the iPad and play a quick game of Scoops or Doodle Jump because it’s going to feel like a bigger deal than slightly tilting the iPhone in your hand.

The target demographic for the iPad is very different than the iPhone and iPod Touch. It will appeal mostly to people in their 30s and older, whereas the iPod Touch is really spreading on the younger demographic of middle and even elementary school. That will also radically change the kind of apps and games people will play on the iPad.

So it’s clear that the iPad is not a large iPod Touch, and it doesn’t pretend to replace it. That’s a key point. Instead of replacing it, us as developers should think about it as complementing the iPhone and iPod Touch. The kind of experience I expect from the ideal iPad application is one that lets me access the same data set or game state I was using with my iPhone during the day. Maybe I’ll put up with the iPhone interface to play some Civilization while I’m commuting to work, but I expect to come back home, sit down on the couch, pick up my iPad and continue the same game with a much better interface. So we’re going to see a trend towards more apps and games that store all their state in the server, and let you access it from any device.

The older demographic, deeper games, and server integration, combined with a higher-resolution screen is going to make people expect a more polished experience, and development costs are going to increase significantly. It’s definitely a new challenge in a market that’s ever changing, and whoever figures it out will have a good chance to have an early iPad hit in their hands.

If you’ve already purchased Flower Garden and you’d still like to help out, you can check out the great lineup of Mac and iPhone apps at Indie+Relief, or donate directly to Doctors Without Borders here.

I want to share the sales data for Flower Garden. Not just the raw data, but a bit of the story behind it, my thoughts, struggles, and why things happened the way they did.

The first thing that you’ll notice is that Flower Garden is a strange in-between app. It’s far from being very successful or being at the top of any chart, but at the same time it probably made more money than 99% of the apps on the App Store. It was also relelased on April 10th, so this represents 10 months of data, an age after which most apps are usually on drip support. So this should be an interesting new data point.

The Full Monty

I know that if I put it off, you’re just going to skip to the end to see the sales plot, so let’s get that out of the way. Here it is.

The vertical axis is daily profit in US $ (after Apple’s 30% cut). Flower Garden generated a bit over $21,500 over a period of 10 months. I would hardly consider that an entry-level salary, much less in California, but it’s enough for someone without a family or mortgage to (barely) make a living. I supplemented that with some teaching, writing, contracting and consulting, so overall I managed do fine and even save a bit of money last year, all while being totally indie and having full creative control over what I did every day. I’m not getting rich, but it’s not a bad life, really.

How many hours of work does Flower Garden represent? It’s really hard to say. It was 6 solid months of work from conception to release, but then there was the fuzzy time spent on marketing, updates, support, new features, etc. I would estimate it was a total of 8 months of full time work, which, at 50 hours per week, makes it 1600 hours. So that’s a depressing $13.44/hour. At least it’s over minimum wage! :-) On the positive side, Flower Garden should continue to sell in the foreseeable future, so that amount will go up a bit over time.

There’s clearly a story behind that graph. It’s not the usual exponential drop off you expect from most (unsuccessful apps) and shows how an aging app can pick up steam on its own after many months on the store, without ever being featured by Apple.

Launch

Let’s start from the beginning. Release day!

After six hard months of work, I submitted Flower Garden to Apple sometime at the beginning of April, and on April 10th I was surprised to see it had been approved. It caught me a bit off guard without my marketing campaign in place, but I announced it high and wide on this blog, Twitter, Facebook, and I even spammed all my friends with an email. The results are the area marked as (A): About $50-$60 in revenue per day (so about 25-30 sales). Not bad considering that was mostly my friends taking pity on me and people randomly seeing it on the new releases list, but it was far from an auspicious start.

I also contacted all the media sites I knew with a press release and promo codes to entice them to write a review. I was lucky that many reviews appeared over the next couple of weeks, but unfortunately they were all spread out, minimizing the PR effect. The biggest effect was when Flower Garden was simultaneously covered on TouchArcade and MacRumors. That’s what caused the big sales spike (B). From there, it was a standard exponential drop off, until, on the last day of the month, just three weeks after launch, revenue dropped below $100/day again. If that was all there was to it, Flower Garden was a big flop and I should start dusting off my resume.

Mother’s Day

I wasn’t ready to throw in the towel yet. I had a couple cards up my sleeve that I was hoping would change things around. After all, May 5th is Mother’s Day in the US. What better time to do some promotion and get Flower Garden noticed?

I decided to run a contest and give away $100 in real flowers to the winner at the same time I put Flower Garden on sale for $1.99 for the week of Mother’s Day. The result? The revenue dip you see in (A).

Mother’s day is probably the small, second spike in that period, but overall, that week was a loss. Lesson learned: Don’t make a sale unless your app is in a visible position (on a top chart somewhere). Flower Garden was nowhere to be seen, so the sale had no effect other than to cut profits by 33%.

I released a couple updates and did the trick updating the release date to get on the new releases list (which no longer works) and I got a couple minor spikes with (B) and (C). Subsequent updates later that summer had almost no effect anymore on sales. The trend was worryingly clear as profits dipped as low as $10/day in mid June (I have no idea what happened that day, but it was the lowest day ever for Flower Garden).

App Treasures

In early June we launched App Treasures. App Treasures is a label for indie iPhone game developers with top-quality games, and one of the main tools we’re leveraging is some cross-promotion for our games, both through the web site, and from an in-game view liking to each other’s games. Initially there was no measurable effect, but then Harbor Master was released and shot all the way to the #2 position on the US store and held its position on the top 50 for a while. The results is the area under (D). It definitely brought in some sales to Flower Garden and reversed the downward trend.

Unfortunately all good things come to an end, and referrals from Harbor Master dropped off and its slipped down the charts, and Flower Garden sales continued sinking.

Facebook and Lite

At this point I had to face reality. Was it time to give up on Flower Garden and move on to another project? I was ready to do that, but every day I would get several emails from users saying how much they loved Flower Garden, how much happiness it was bringing to their lives, etc. Some of the stories were really touching.

At the same time, every time I would show the game to someone, they usually really liked it. Not liked-it-because-I’m-there, but really, genuinely liked it. Why weren’t more people buying it then? Two problems: First, screenshots were not conveying how cool growing, animated flowers you could touch were, and second, most people didn’t even know Flower Garden existed. It had never been featured by Apple on the App Store, and the audience I was trying to reach didn’t read TouchArcade or other iPhone review sites.

It was at this point that I decided to give it one more try. My goal wasn’t adding more features as I had done with the updates so far, but to make Flower Garden more visible. I wanted more people to know that Flower Garden existed.

The first thing I did was to add Facebook integration. Not only could you send bouquets through email, but you were able to send them directly to your Facebook friends. The advantage of that approach from my point of view is that all your friends also saw the flowers so for every bouquet sent on Facebook, hopefully dozens or hundreds of people were being exposed to Flower Garden. The result on sales: Not much. Maybe it made the early part of July a little higher than it would have been otherwise, but no noticeable difference.

The second approach was to release a lite version of Flower Garden in early September. I was confident that Flower Garden was a good app, and I was hoping that once people had a chance to try the lite version, they would purchase the full version. Fortunately I was right and the effect on sales was very noticeable, pretty much doubling sales (E), but it never really took off in any significant way, and sales slowly declined over time.

Second Wind

You’d think that I would give up at this point, wouldn’t you? And I don’t say that with pride. I mean, it probably would have been smarter to quit a long time ago. But somehow, every time I was ready to move on , something else would come up that would entice me to try something else with Flower Garden.

This time it was in-app purchases (IAP). Apple had announced IAP back in June. They seemed like a very natural fit for Flower Garden, but given how few units Flower Garden had sold, I would have a very limited audience for IAP. A small percentage of a small number is a tiny number! :-( However, in late October Apple announced that IAP were finally allowed from free apps as well. That encouraged me to give Flower Garden one… last… try…

To be totally honest, I wasn’t expecting very much. Even including all the units of the Lite version out there, there just weren’t that many units. Especially not that many people using it on a daily basis (I’m sure a lot of the free ones were downloaded and quickly forgotten). But I thought it would be a good experience and if I only spent a week on it and made $1,000 I could call it even.

The result was totally unexpected. The Flower Shop opened on December 6th and revenue immediately shot up (A). On December 21st I released a new IAP called Seeds of Winter with 8 new winter-themed flowers, and people loved it and immediately purchased it. Christmas day came and went with somewhat of an increase in sales (C) but nothing really spectacular. Finally, to wrap up the year, there was a big spike on New Year’s Eve (D) (do people send flowers on New Year’s Eve? Really?).

Afterwards revenue flattened out, but at a much higher amount than before. Before IAP, daily revenue was about $50/day. Now it’s more around $180/day. That’s totally beyond any of my expectations!

But hang on, where is the revenue coming from? There are three possible sources: You can purchase Flower Garden, you can make IAP in the full version, and you can also make purchases in the free version. Here’s the breakdown:

It looks like sales for Flower Garden (blue) continue to be more or less the same, with a slight increase after Christmas. The IAP from the full version of Flower Garden (green) account for most of the extra revenue, especially at the beginning. But it’s very interesting that the purchases coming from Flower Garden Free (orange) are steadily increasing and, as of last week, they became almost as large as the ones from the full version.

That can be explained because more and more people are getting the free version and upgrading it instead of buying the full version (which is exactly what I was hoping for). What’s also really interesting is the downloads of Flower Garden Free.

Flower Garden Free was never a big player. It had the usual big initial spike, but then it settled down at around 100-200 downloads per day (which is very few considering there were 50-60 purchases per day of Flower Garden Full during that period). But, as soon as the in-app Flower Shop was released, downloads started climbing, and on Christmas day they went through the roof (relatively speaking). So it’s no surprise that IAP from Flower Garden Free picked up in these last few weeks.

The Future

I have no idea what the future will hold for Flower Garden. This week Apple selected Flower Garden and featured it on the App Store as a Staff Favorite worldwide (that period is not show in the sales graphs). I’m also already preparing a new update and some more IAP for Valentine’s Day, so I’m sure there will be more ups and downs in the near future. I’d certainly like to continue supporting Flower Garden for as long as it’s profitable.

I’ll follow up this post in a couple of months with the aftermath of the App Store feature and Valentine’s Day. Also, stay tuned for another post with more details of the IAP, what’s successful, purchase patterns, and more.

This is the first time Flower Garden has received any kind of “Apple love”, so it’s particularly exciting, especially considering that Flower Garden was released all the way back in April. I imagine the recent update, which included the Flower Shop with in-app purchases, must have caught someone’s eye.

The only downside is that the featuring might throw off the sales data I was collecting to show the effect on sales that in-app purchases and the holiday season had. But I’m not really complaining. That’s a very nice reason to have the data thrown off :-) Thank you Apple!

Purchase States

When you first create an in-app purchase in iTunes Connect, it is listed as “Pending Developer Approval”. This means that if you ask for info on that product id with a SKProductsRequest, you’ll get the info for the item only on development builds, not in distributions builds.

Once your in-app purchase item is ready, you need to upload a screenshot of it working in the app, and approve it yourself. At that point it enters the queue to be approved by Apple. Once an item has been approved by Apple (marked as Ready for Sale), it will appear both in development and distribution builds.

This behavior is very convenient because you can add in-app purchases normally for development, and not worry about them showing up for all your existing users until they’re done. Just make sure to keep around a distribution build of your app to test what the end user sees.

In addition to that, you can flag items as available for sale or not. That way you have the added flexibility of temporarily removing an item from sale, or making it available at a later time.

Approval Process

The first time you submit any in-app purchases, you’ll probably do it along with a binary update (since you need to code in support for displaying and allowing the purchases of items). You’ll have an option to mark the items to be approved along with the binary, so everything will become available on the App Store at the same time. In that case, the approval process is just like for a regular binary update (it seems to be averaging a week lately).

You can also submit in-app purchase items by themselves, without an app binary update. I did this for the Seeds of Winter pack in Flower Garden. Somehow, I was expecting a turnaround time of a day or two. After all, I figured they would look at the provided screenshot and approve it. Unfortunately, that wasn’t the case. In this particular situation, approval took almost a week, and I’m not sure it was because of the normal process, or because I emailed Apple asking what the procedure was to submit a product by itself (the product was approved within an hour of me getting a response from them saying they would look into it).

I was monitoring iTunes Connect pretty obsessively at the time it was approved (because it was right before my two-week break for the holidays) and I saw that the Seeds of Winter pack went from “Waiting for Review” to “In Review” to “Approved for Sale” in a matter of two minutes. That was one quick check! (I’m not complaining though).

What’s even more interesting is that I didn’t know that an item wouldn’t appear on a distribution build unless it was approved by Apple, so I had added an extra layer on the shop catalog file on my server that allowed me to prevent an item from showing up on the Flower Shop until I turned it on myself. And since that check wasn’t turned on, it means that Apple approved the in-app purchase by looking at the provided screenshot, without running it in the app.

This shouldn’t be too surprising. After all, I’m not even sure what the point is for in-app purchase approvals. For content that comes from the server, I can change it at any time, so the only thing they can approve is the name and description provided in iTunes Connect. I suppose the screenshot is fine for that, but I would also expect really speedy approval times. After all, not having to wait for lengthy approval processes is one of the big draws for in-app purchases.

I did learn one important lesson from going through this process. Next time I submit a binary prepared for more in-app purchases, I’m going to create stand-ins for all possible in-app purchases I’m considering and submit them for approval at the same time as the app binary, but marked as not ready for sale. That way they will all be approved at once, and I can take my time creating the final version for those items and making them available instantly whenever I want.

Purchasing Items

The same way that non-Apple approved items only show up in development builds, the only accounts that can purchase any items on development builds are test accounts using the sandbox environment. These are accounts you create on iTunes Connect and allow you to purchase items in your programs without spending real money (otherwise I’d be in trouble considering how many times I ended up purchasing items during the development of the Flower Shop!). Conversely, distribution builds only work with non-test accounts and non-sandbox environment.

One thing that threw me off was that I started getting support requests from people who had trouble logging in with their account to make in-app purchases. They claimed they were being asked about this “sandbox environment”. That threw me in a panic for a second, because I thought that everybody was somehow accessing the sandbox environment and nobody could make any purchases (because they had full iTunes accounts). It turns out, that’s a problem that happens with jailbroken phones with a particular app installed. It’s nicely documented here so I’ve been pointing people in that direction.

Finally, I believe there’s no provision in iTunes Connect to deal with different binary versions, but that’s something that’s very important to handle. If I release a new binary with some new feature built into the app, and an in-app purchase to unlock it, I don’t want people with an earlier version purchasing it and getting mad that it’s not working. Ideally this should be an option in iTunes Connect, but in the meanwhile, I build a version check into the shop catalog itself. That way only apps with that version or higher can even display a particular item. Or I can even take it a step further and display the item but notify the user that they need to update the app before the purchase it.

Even so, I decided to take a few extra steps that at least will make things a bit more difficult to crack and keep things more secure. I could have gone way beyond this, but I don’t think that would have been worth my time.

Verify the purchase

Right now, you can download programs that will automatically crack an app by extracting its data and resigning it so it works on any jailbroken phone. There’s no doubt that even if you implement in-app purchases just like I described in earlier posts, crackers will have to analyze the code and do some steps by hand, so that might be enough to slow down the pace of cracked apps. Even so, an experienced cracker won’t take long to crack your in-app purchases unless you take some extra steps. All they have to do is find the function that you call in response to getting the go ahead from the App Store and they’ll gain access to all content.

The first thing you should do is to verify that the purchase happened correctly. To do that, once you get the notification in your app that the payment was complete, you send the receipt to your server, and have your server verify it in turn with the App Store. Since crackers don’t have access to your server (if they do you’re in more trouble than a few cracked apps), that step is secure. If your app was cracked and the user didn’t pay for the purchase, the App Store will fail the verification of the receipt and you don’t have to deliver the content.

How do you check that the receipt is valid? First of all, you need to send the receipt of the purchase to your server. I send it along with some other info, like the user’s UDID, what product was purchased, etc so I can keep general usage statistics. The relevant bits of code are the following:

NSString* receiptString = [self createEncodedString:transaction.transactionReceipt];
//...
[postBody appendData:[[NSString stringWithFormat:@"\r\n--%@\r\n", boundary] dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData:[@"Content-Disposition: form-data; name=\"receipt\"\r\n\r\n" dataUsingEncoding:NSUTF8StringEncoding]];
[postBody appendData:[receiptString dataUsingEncoding:NSUTF8StringEncoding]];
//...
[req setHTTPBody:postBody];

The createEncodedString function takes some arbitrary data and creates a base64 encoding. That’s one of the parts that is not particularly well documented in the SDK documentation.

- (NSString*) createEncodedString:(NSData*)data
{
    static char table[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";

    const int size = ((data.length + 2)/3)*4;
    uint8_t output[size];

    const uint8_t* input = (const uint8_t*)[data bytes];
    for (int i = 0; i < data.length; i += 3)
    {
        int value = 0;
        for (int j = i; j < (i + 3); j++)
        {
            value <<= 8;
            if (j < data.length)
                value |= (0xFF & input[j]);
        }

        const int index = (i / 3) * 4;
        output[index + 0] =  table[(value >> 18) & 0x3F];
        output[index + 1] =  table[(value >> 12) & 0x3F];
        output[index + 2] = (i + 1) < data.length ? table[(value >> 6)  & 0x3F] : '=';
        output[index + 3] = (i + 2) < data.length ? table[(value >> 0)  & 0x3F] : '=';
    }    

    return  [[NSString alloc] initWithBytes:output length:size encoding:NSASCIIStringEncoding];
}

That just sends the receipt to my server. Once the server receives that request, it needs to verify the data with the App Store. The exact format of the App Store request is, again, not well documented. It turns out it needs to be sent as anonymous data in a POST request. After some trial and error, this is the php code I’m using in my server to verify the receipt:

function isReceiptValid($purchase)
{
	$data = array ('receipt-data' => $purchase->receipt );
	$encodedData = json_encode($data);

	$url = "https://buy.itunes.apple.com/verifyReceipt";

	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($ch, CURLOPT_POST, true);
	curl_setopt($ch, CURLOPT_POSTFIELDS, $encodedData);
	$encodedResponse = curl_exec($ch);
	curl_close($ch);

	if ($encodedResponse == false)
	{
		logError($purchase, "Payment could not be verified (no response data).");
		return false;
	}

	$response = json_decode($encodedResponse);
	if ($response->{'status'} != 0)
	{
		logError($purchase, "Payment could not be verified (status != 0).");
		return false;
	}

	$purchase->storeReceipt = $response->{'receipt'};
	return true;
}

If things check out, the server returns a valid code, and you can go ahead and make the content available to the user. Otherwise, it means the user is trying to pass up a fake purchase as valid and you can deal with it any way you want.

Get content from the server

Of course, if the content is already embedded in the app itself, the cracker can bypass both the App Store purchase and the receipt verification through your server, and just access the content directly. If you want a better chance of not being cracked, have your server deliver the content. That way, if the receipt doesn’t check out, you never transfer the content.

In the case of Flower Garden, all seed packets are downloaded from the server, but the other purchases (extra pots, extra fertilizer) are just changing a few internal variables, so they would be a lot easier to crack.

One of the easiest ways that crackers can get around this verification is by buying the content once and saving the receipt that comes with it. Then, cracked versions can pass that receipt to your server, they’ll be validated as correctly, and they’ll be able to download the content. So, as an added precaution, you might want to keep track how many times a certain purchase is re-downloaded, especially from different UDIDs. I can’t imagine a legitimate reason for a user to download the same purchase more than 20-30 times from different devices, so denying the purchase at that point in the server seems justified.

Save those receipts

The last step I to make cracking more difficult was to save the receipt itself to disk along with the content that was downloaded. Then, periodically, the game can pick one of those receipts, and verify with the server that it’s still valid. If it’s ever reported as an invalid receipt, the game deletes the content for that purchase. This is not as draconian as it sounds, because even if it’s ever triggered for a legitimate user (and I have no idea how that could be other than data corruption), they can always re-purchase it again for free (and they would want to do that to fix their data corruption anyway).

To be able to delete things easily and cleanly, I keep each downloaded in-app purchase on a separate directory. So it’s very easy to delete the whole directory without having to access files everywhere. The hardest thing is making sure the game deals with the missing data and that nothing is referencing it. I’ll touch on that on a future part of this article.

Keeping purchases in separate directories also makes it easier to reset all purchases. That’s something I didn’t originally include in Flower Garden, but I should have. Sometimes, data can get corrupted or maybe a bug occurs because of some combination of purchases (hopefully not, but you never know!). Allowing the user to reset the purchases and re-download them allows them to fix those problems without having to reinstall the app and lose their progress.

[1] I know from first hand experience that one dedicated teenager with all the time in his hands can spend inordinate amounts of time bypassing the most elaborate anti-piracy schemes. The harder it is, the more of a challenge it becomes, and the more fun it is. To this day, I still remember the thrill of finally bypassing the copy protection of 1943 on the Amstrad CPC after two days of non-stop hacking, which included several loops XORing all the available memory (including the currently executing code), and messages from the developers taunting me!

If you haven’t seen the presentation, go download it right now. It’s really well put together and he has some great detailed examples on how caches are affected with traditional object-oriented design vs. a more data-oriented approach.

[1] Thanks to Christer Ericson for pointing that out.

Purchasing a product

The store is fully populated, the items are displayed and, if all goes well, the user presses the buy button. What now?

At this point I check that in-app purchases are enabled and if they aren’t, I put up a message box. Apple says you can also do this earlier, but I figured I’d rather show users what’s available and entice them to buy it. By the time they get that message, they’ll be much more likely to enable them and continue the purchase than if they never saw the store in the first place.

Actually, before I even do that check, I see if the item is currently offered for free. If it is, the code takes an alternative path that doesn’t involve the App Store because apparently you can’t set an in-app purchase to free. Really! Why not? It’s yet something else we need to implement ourselves. Please, Apple, fix that. In the case of free purchases (like the Bonus Seeds included in the full version of Flower Garden) I contact my server directly and download everything from there. If you’re thinking that a hacker can just set the free flag to true and have access to all your content, it’s not that easy, because the server checks that the product is really marked as free when it’s contacted, and if it isn’t, it declines the transaction. So they have to work a bit harder than that to crack things (more on that next time).

But if the product is not marked as free and in-app purchases are enabled, then you can add the payment to the StoreKit payment queue which is, of course, another asynchronous call. So now we’re back to having to make a decision of what we want the user to do after they attempt to purchase something. Does someone really want to move away from the screen before they complete the purchase? I don’t, so I set another blocking activity screen explaining what’s going on with an activity indicator.

if ([SKPaymentQueue canMakePayments])
{
    SKPayment* payment = [SKPayment paymentWithProductIdentifier:m_product.m_id];
    SKPaymentQueue* queue = [SKPaymentQueue defaultQueue];
    [queue addPayment:payment];
    [m_progressViewController setText:@"Accessing store" completed:NO];
    [m_progressViewController display:YES animated:YES];
}

To be able to test the actual purchase, you need to create a test account in iTunes Connect. It’s slightly annoying that you need to explicitly log out from the Settings | Store screen and you can’t do that from the app itself, but it’s not nearly as much of a big deal as not being able to test purchases from the simulator at all. That’s a big fail as far as I’m concerned, and I can’t imagine a single reason why that was done that way (other than lack of time/manpower on Apple’s side).

Completing the purchase

You think we might be done by now, but not. Far from it. We’re just getting warmed up.

Now that the payment has entered the queue, you’re going to get a callback notification every time its status changes. That means you’ll get one right away because it was just added to the queue, another time when the payment is approved by the App Store, or when it fails, or if it’s detected that it was already purchased and this a re-purchase.

While that’s happening, the StoreKit code will bring up those ugly message boxes asking the user if they’re really sure they want to purchase the item for that price, and possibly ask them for their login info. It’s funny that the first time I implemented this part, I thought that it was up to me to put those message boxes up, so they were showing up twice. You don’t have to do anything for that, whether you want to or not. What’s worse, is that each time one of those message boxes comes up, your app gets a message putting it on hold, which usually means the sound is temporarily muted. Looks very unprofessional and I’m surprised Apple forces that on us while they leave everything else so open ended and requiring so much work.

Once the purchase is marked as complete, you can obtain the content for that purchase and make it available in your app. After that step is complete, you can remove the payment from the queue and the transaction is considered complete. In the case of Flower Garden, most purchases involve accessing my server and downloading some data from there for new seeds. Some purchases don’t require any downloading though, and all they do is twiddle a few bits internally.

It’s important to understand the payment queue well. First of all, it’s persistent, so once a purchase payment has been added to it, it will be there next time the application starts if it was never removed. It was designed that way to make sure an application can deliver on the product that was purchased even if the app is stopped, crashes, or is interrupted in any way. So initially, a payment is added to the queue meaning that the user has initiated the purchase transaction. At that point, the transaction can fail for multiple reasons: User cancels payment during one of the prompts, wrong user ID, the servers are down, the product was removed from the store, etc. Assuming the purchase is successful, the user account will be charged the cost of the product and the payment will be marked as successful. It’s at this time that you have to deliver the goods. Only once you’ve successfully done that you can go ahead and remove the payment from the queue. Otherwise, next time that the app starts, your code should detect a payment there and continue processing it. You really never want to get in a situation that the user pays for something and it’s never delivered to them.

I’m happy with how the StoreKit API will keep track of past purchases, and if a user tries to buy the same item again, they’ll be given the option to redownload it for free. It’s a great relief not to have to keep track of that ourselves. It will also cut down on customer support emails a huge amount because users can always redownload something if they had to uninstall their app, or somehow, in spite of all our precautions, they bought something but it was never updated in their app. In that case a quick re-download should fix it (just make sure your code is pretty stateless and can deal with re-downloading the same content multiple times).

It gets a bit trickier though. It turns out the payment queue can have more than one payment in it. Why is that? First of all, because the buy process wasn’t blocking the way Apple designed it, a hyperactive user could initiate multiple purchases at once. The other, more realistic situation is that the user initiates a purchase, it’s verified with the App Store, but before you can complete the download, your application is shut down. The Store Kit queue will be restored next time the application starts, but at that point the user could go to the store again and start another purchase.

In any case, you really need to deal with that situation, otherwise you risk losing purchases and angering users (and you really don’t want to rely on users re-downloading something for free if you can avoid it). This is trickier than it sounds when you’re dealing with downloads from your own server because it might involve multiple downloads, multiple files, etc. So in my case, whenever I get an update of status in the SKPaymentQueue, I add it to my own queue. Then I can process that queue sequentially, one purchase at the time.

- (void)paymentQueue:(SKPaymentQueue *)queue updatedTransactions:(NSArray *)transactions
{
    for (SKPaymentTransaction* transaction in transactions)
        [m_transactionQueue addObject:transaction];

    [self processNextTransaction];
}

- (void)processNextTransaction
{
    if (m_processingTransaction)
        return;

    while (m_transactionQueue.count > 0)
    {
        SKPaymentTransaction* transaction = [m_transactionQueue objectAtIndex:0];
        [m_transactionQueue removeObjectAtIndex:0];

        switch (transaction.transactionState)
        {
        case SKPaymentTransactionStatePurchased:
            m_processingTransaction = true;
            [self completeTransaction:transaction];
            return;
        case SKPaymentTransactionStateFailed:
            m_processingTransaction = true;
            [self failedTransaction:transaction];
            return;
        case SKPaymentTransactionStateRestored:
            m_processingTransaction = true;
            [self restoreTransaction:transaction];
            return;
        default: break;
        }
    }
}

Whenever a transaction is complete, you can flag it as finished and it will be completely removed from the payment queue: [[SKPaymentQueue defaultQueue] finishTransaction:info.m_transaction];

A whole  step I skipped for now is verifying the purchase with your server. I’ll cover that in the next part of this post when I talk about anti-piracy measures.

The situation improved as soon as Apple announced the availability of in-app purchases in free apps. Flower Garden Free had been out for a few weeks at that point, so that effectively doubled the potential audience.

The final consideration that convinced me to plunge ahead was the realization that while only a small percentage of users would buy extra content, a fraction of those would make multiple purchases, and some people would probably buy every available item. So maybe it would be worthwhile after all.

Having said that, I gave myself one week to completely implement in-app purchases in Flower Garden. That included everything, from implementing the actual purchasing through StoreKit, to server code, and, of course, creating the additional content itself.

For those of you familiar with Flower Garden, the new content I created specifically for in-app purchases was a flower shop offering:

• An extra garden screen with 12 pots
• Liquid plant fertilizer that speeds up plant growth. This is a consumable item with a fixed number of doses and can be repurchased after it’s used up.

Additionally, the flower shop in the free version of Flower Garden had these items that would bring it functionality up to par with the full version of Flower Garden:

• Extra pots in the main garden
• Extra common seeds
• Set of bonus seeds

My initial estimate of one week turned out to be very optimistic, and it ended up taking closer to two and a half weeks. Most of it was spent creating the new content and integrating it smoothly in the game.

In this post and the next few ones, I’ll share my experiences with in-app purchases. From implementation details, tips and tricks, how it might help with piracy, mistakes I made along the way, and even how many sales in-app purchases generated.

Displaying Store Items

Apple did a good job documenting the overview of how to implement in-app purchases. Some of the details are missing, but they’re small enough that I was able to fill in the blanks pretty easily and get things working in a couple of days.

However, I really think the process should have been implemented very differently in Apple’s part. The sequence of actions that needs to be done feels overly complicated. It may be great if someone is trying to do a very customized and integrated store with all sorts of crazy features, but really, in 99% of the cases, developers just want to sell something and be done with it. Forcing us to go through all those steps seems overkill. A simplified higher-level set of helper functions would be a welcome addition to the SDK. For suggestions on how to go about it, look at the SDK for game consoles. They got that part right (or at least much more right than the iPhone SDK).

Getting the catalog

The first step towards implementing in-app purchases, is to get a list of all the products you want to sell in your store. It’s a good idea to keep this list off your app and on a web server instead, that way you can add, remove, or edit any products without going through a full app update. That in itself makes in-app purchases very attractive, doesn’t it? You can have a look at the master Flower Garden shop catalog.

Getting product info

Step two is to go through each of those products, and get the official product information from the App Store. That involves creating a set with the product ids you want to query, and sending a request.

The code in Flower Garden to query for product ids looks like this:

NSArray* products = m_appData->m_shopCatalog.m_products;
NSMutableSet* productIds = [[NSMutableSet alloc] initWithCapacity:32];
for (int i=0; i<[products count]; ++i)
{
    ShopProduct* product = [products objectAtIndex:i];
    if (!product.m_alreadyIncluded)
        [productIds addObject:product.m_id];
}

SKProductsRequest* request = [[SKProductsRequest alloc] initWithProductIdentifiers:productIds];
request.delegate = self;
#ifdef WORK_WITHOUT_APP_STORE
[self productsRequest:request didReceiveResponse:nil];
#else
[request start];
#endif
[productIds release];

There are a couple interesting things in that code. First of all, notice that before we ask for the product info, we check if the product is already included in the app. That’s because some items, such as the extra pots or the common seeds, already come as part of Flower Garden Full. But at the same time, I wanted to show them in the store and mark them as purchased. So I need to have a parallel path for those items that doesn’t go through the App Store. More on that in a future post.

The other interesting bit is the #ifdef. This one really sucks, so get ready: The simulator can’t make any StoreKit calls. It fails with a nice message box explaining why, but it fails nonetheless. To put it in highly technical terms: That blows! Seriously, that means you’re stuck developing on the device. Turnaround times are about 20-30 seconds, and debugging on the device directly is no fun at all. I’m convinced that alone was part of the reason it took me longer than my initial estimate. So that #ifdef was my attempt to try to get as much done on the simulator as possible. At least I was able to display the products on the flower shop. The actual buying had to be done on the device though. No way around it as far as I know. Why, oh, why, doesn’t Apple let us add a test account on the simulator? Maybe a feature for SDK 4.0?

Displaying the products

Some time after sending the request for info on the products, and assuming we have a valid internet connection and that the App Store is in a good mood (to their credit, Apple’s servers have been very reliable), we’ll get a response. The response will include more information for each of the products we requested, including localized name, description, and price. This is all information taken from the in-app purchases you created in iTunes Connect. And yes, that means that you need to keep multiple sets of data in sync: Your shop catalog and the in-app purchases in iTunes Connect.

Notice that because everything we’ve done so far has been asynchronous (and asynchronous is the name of the game for a lot of the remaining steps as well) it introduces a lot of complexity. What do we do during that time? Do we let the user wander around to other parts of the program? Do we block and make the UI modal? I went with the latter approach for simplicity both coding and in the UI for the user.

At this point you can display them in your shop. In my case I ended up using a very similar interface to the App Store on the iPhone. I figured users are already familiar with that kind of interface, so might as well build on top of that. So that meant the flower shop is a table showing all available products. Clicking on a cell brings up a view with details and a description of each product.

To make my life easier, the only elements that are hardwired on the product page are the icon, the title, and the price/purchase button. The actual description and screenshots is just an embedded web view and part of the product description in the catalog has the correct URL for each project. You can even access some of them directly here.

So far all this has done for us is let us populate a list of products that are available to be sold. Next entry will cover the details of the actual transaction and a few things to watch out for.

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

hinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is Thre Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

Picture this: Toward the end of the development cycle, your game crawls, but you don’t see any obvious hotspots in the profiler. The culprit? Random memory access patterns and constant cache misses. In an attempt to improve performance, you try to parallelize parts of the code, but it takes heroic efforts, and, in the end, you barely get much of a speed-up due to all the synchronization you had to add. To top it off, the code is so complex that fixing bugs creates more problems, and the thought of adding new features is discarded right away. Sound familiar?

That scenario pretty accurately describes almost every game I’ve been involved with for the last 10 years. The reasons aren’t the programming languages we’re using, nor the development tools, nor even a lack of discipline. In my experience, it’s object- oriented programming (OOP) and the culture that surrounds it that is in large part to blame for those problems. OOP could be hindering your project rather than helping it!

It’s All About Data

OOP is so ingrained in the current game development culture that it’s hard to think beyond objects when thinking about a game. After all, we’ve been creating classes representing vehicles, players, and state machines for many years. What are the alternatives? Procedural programming? Functional languages? Exotic programming languages?

Data-oriented design is a different way to approach program design that addresses all these problems. Procedural programming focuses on procedure calls as its main element, and OOP deals primarily with objects. Notice that the main focus of both approaches is code: plain procedures (or functions) in one case, and grouped code associated with some internal state in the other. Data-oriented design shifts the perspective of programming from objects to the data itself: The type of the data, how it is laid out in memory, and how it will be read and processed in the game.

Programming, by definition, is about transforming data: It’s the act of creating a sequence of machine instructions describing how to process the input data and create some specific output data. A game is nothing more than a program that works at interactive rates, so wouldn’t it make sense for us to concentrate primarily on that data instead of on the code that manipulates it?

I’d like to clear up potential confusion and stress that data-oriented design does not imply that something is data- driven. A data-driven game is usually a game that exposes a large amount of functionality outside of code and lets the data determine the behavior of the game. That is an orthogonal concept to data-oriented design, and can be used with any type of programming approach.

Ideal Data

Figure 1a. Call sequence with an object-oriented approach

If we look at a program from the data point of view, what does the ideal data look like? It depends on the data and how it’s used. In general, the ideal data is in a format that we can use with the least amount of effort. In the best case, the format will be the same we expect as an output, so the processing is limited to just copying that data. Very often, our ideal data layout will be large blocks of contiguous, homogeneous data that we can process sequentially. In any case, the goal is to minimize the amount of transformations, and whenever possible, you should bake your data into this ideal format offline, during your asset-building process.

Because data-oriented design puts data first and foremost, we can architect our whole program around the ideal data format. We won’t always be able to make it exactly ideal (the same way that code is hardly ever by-the-book OOP), but it’s the primary goal to keep in mind. Once we achieve that, most of the problems I mentioned at the beginning of the column tend to melt away (more about that in the next section).

When we think about objects, we immediately think of trees— inheritance trees, containment trees, or message-passing trees, and our data is naturally arranged that way. As a result, when we perform an operation on an object, it will usually result in that object in turn accessing other objects further down in the tree. Iterating over a set of objects performing the same operation generates cascading, totally different operations at each object (see Figure 1a).

Figure 1b. Call sequence with a data-oriented approach

To achieve the best possible data layout, it’s helpful to break down each object into the different components, and group components of the same type together in memory, regardless of what object they came from. This organization results in large blocks of homogeneous data, which allow us to process the data sequentially (see Figure 1b). A key reason why data-oriented design is so powerful is because it works very well on large groups of objects. OOP, by definition, works on a single object. Step back for a minute and think of the last game you worked on: How many places in the code did you have only one of something? One enemy? One vehicle? One pathfinding node? One bullet? One particle? Never! Where there’s one, there are many. OOP ignores that and deals with each object in isolation. Instead, we can make things easy for us and for the hardware and organize our data to deal with the common case of having many items of the same type.

Does this sound like a strange approach? Guess what? You’re probably already doing this in some parts of your code: The particle system! Data-oriented design is turning our whole codebase into a gigantic particle system. Perhaps a name for this approach that would be more familiar to game programmers would have been particle-driven programming.

Advantages of Data-Oriented Design

Thinking about data first and architecting the program based on that brings along lots of advantages.

Parallelization.

These days, there’s no way around the fact that we need to deal with multiple cores. Anyone who has tried taking some OOP code and parallelizing it can attest how difficult, error prone, and possibly not very efficient that is. Often you end up adding lots of synchronization primitives to prevent concurrent access to data from multiple threads, and usually a lot of the threads end up idling for quite a while waiting for other threads to complete. As a result, the performance improvement can be quite underwhelming.

When we apply data-oriented design, parallelization becomes a lot simpler: We have the input data, a small function to process it, and some output data. We can easily take something like that and split it among multiple threads with minimal synchronization between them. We can even take it further and run that code on processors with local memory (like the SPUs on the Cell processor) without having to do anything differently.

Cache utilization.

In addition to using multiple cores, one of the keys to achieving great performance in modern hardware, with its deep instruction pipelines and slow memory systems with multiple levels of caches, is having cache-friendly memory access. Data-oriented design results in very efficient use of the instruction cache because the same code is executed over and over. Also, if we lay out the data in large, contiguous blocks, we can process the data sequentially, getting nearly perfect data cache usage and great performance. Possible optimizations. When we think of objects or functions, we tend to get stuck optimizing at the function or even the algorithm level; Reordering some function calls, changing the sort method, or even re-writing some C code with assembly.

That kind of optimization is certainly beneficial, but by thinking about the data first we can step further back and make larger, more important optimizations. Remember that all a game does is transform some data (assets, inputs, state) into some other data (graphics commands, new game states). By keeping in mind that flow of data, we can make higher-level, more intelligent decisions based on how the data is transformed, and how it is used. That kind of optimization can be extremely difficult and time- consuming to implement with more traditional OOP methods.

Modularity.

So far, all the advantages of data-oriented design have been based around performance: cache utilization, optimizations, and parallelization. There is no doubt that as game programmers, performance is an extremely important goal for us. There is often a conflict between techniques that improve performance and techniques that help readability and ease of development. For example, re-writing some code in assembly language can result in a performance boost, but usually makes the code harder to read and maintain.

Fortunately, data-oriented design is beneficial to both performance and ease of development. When you write code specifically to transform data, you end up with small functions, with very few dependencies on other parts of the code. The codebase ends up being very “flat,†with lots of leaf functions without many dependencies. This level of modularity and lack of dependences makes understanding, replacing, and updating the code much easier.

Testing.

The last major advantage of data-oriented design is ease of testing. As we saw in the June and August Inner Product columns, writing unit tests to check object interactions is not trivial. You need to set up mocks and test things indirectly. Frankly, it’s a bit of a pain. On the other hand, when dealing directly with data, it couldn’t be easier to write unit tests: Create some input data, call the transform function, and check that the output data is what we expect. There’s nothing else to it. This is actually a huge advantage and makes code extremely easy to test, whether you’re doing test-driven development or just writing unit tests after the code.

Drawbacks of Data-Oriented Design

Data-oriented design is not the silver bullet to all the problems in game development. It does help tremendously writing high-performance code and making programs more readable and easier to maintain, but it does come with a few drawbacks of its own.

The main problem with data-oriented design is that it’s different from what most programmers are used to or learned in school. It requires turning our mental model of the program ninety degrees and changing how we think about it. It takes some practice before it becomes second-nature.

Also, because it’s a different approach, it can be challenging to interface with existing code, written in a more OOP or procedural way. It’s hard to write a single function in isolation, but as long as you can apply data-oriented design to a whole subsystem you should be able to reap a lot of the benefits.

Applying Data-Oriented Design

Enough of the theory and overview. How do you actually get started with data-oriented design? To start with, just pick a specific area in your code: navigation, animations, collisions, or something else. Later on, when most of your game engine is centered around the data, you can worry about data flow all the way from the start of a frame until the end.

The next step is to clearly identify the data inputs required by the system, and what kind of data it needs to generate. It’s OK to think about it in OOP terms for now, just to help us identify the data. For example, in an animation system, some of the input data is skeletons, base poses, animation data, and current state. The result is not “the code plays animations,†but the data generated by the animations that are currently playing. In this case, our outputs would be a new set of poses and an updated state.

It’s important to take a step further and classify the input data based on how it is used. Is it read- only, read-write, or write-only? That classification will help guide design decisions about where to store it, and when to process it depending on dependencies with other parts of the program.

At this point, stop thinking of the data required for a single operation, and think in terms of applying it to dozens or hundreds of entries. We no longer have one skeleton, one base pose, and a current state, and instead we have a block of each of those types with many instances in each of the blocks.

Think very carefully how the data is used during the transformation process from input to output. You might realize that you need to scan a particular field in a structure to perform a pass on the data, and then you need to use the results to do another pass. In that case, it might make more sense to split that initial field into a separate block of memory that can be processed independently, allowing for better cache utilization and potential parallelization. Or maybe you need to vectorize some part of the code, which requires fetching data from different locations to put it in the same vector register. In that case, that data can be stored contiguously so vector operations can be applied directly, without any extra transformations.

Now you should have a very good understanding of your data. Writing the code to transform it is going to be much simpler. It’s like writing code by filling in the blanks. You’ll even be pleasantly surprised to realize that the code is much simpler and smaller than you thought in the first place, compared to what the equivalent OOP code would have been.

If you think back about most of the topics we’ve covered in this column over the last year, you’ll see that they were all leading toward this type of design. Now it’s the time to be careful about how the data is aligned (Dec 2008 and Jan 2009), to bake data directly into an input format that you can use efficiently (Oct and Nov 2008), or to use non- pointer references between data blocks so they can be easily relocated (Sept 2009).

Is There Room For OOP?

Does this mean that OOP is useless and you should never apply it in your programs? I’m not quite ready to say that. Thinking in terms of objects is not detrimental when there is only one of each object (a graphics device, a log manager, etc) although in that case you might as well write it with simpler C-style functions and file-level static data. Even in that situation, it’s still important that those objects are designed around transforming data.

Another situation where I still find myself using OOP is GUI systems. Maybe it’s because you’re working with a system that is already designed in an object-oriented way, or maybe it’s because performance and complexity are not crucial factors with GUI code. In any case, I much prefer GUI APIs that are light on inheritance and use containment as much as possible (Cocoa and CocoaTouch are good examples of this). It’s very possible that a data-oriented GUI system could be written for games that would be a pleasure to work with, but I haven’t seen one yet.

Finally, there’s nothing stopping you from still having a mental picture of objects if that’s the way you like to think about the game. It’s just that the enemy entity won’t be all in the same physical location in memory. Instead, it will be split up into smaller subcomponents, each one forming part of a larger data table of similar components.

Data-oriented design is a bit of a departure from traditional programming approaches, but by always thinking about the data and how it needs to be transformed, you’ll be able to reap huge benefits both in terms of performance and ease of development.

Thanks to Mike Acton and Jim Tilander for challenging my ideas over the years and for their feedback on this article.

This article was originally printed in the September 2009 issue of Game Developer.

Here’s a Korean translation of this article by Hakkyu Kim.

The class is aimed at iPhone developers without previous OpenGL or 3D graphics experience. As part of the class, we’ll create both 2D and 3D OpenGL applications, and we’ll cover a broad range of topics, start with the basics of setting up OpenGL and rendering triangles on screen, to multitexturing and point sprites in the last day. This is definitely a hands-on class, so you’ll need to bring your laptop and be ready to do some coding.

Check out the Mobile Orchard page for discounts and registration details. Feel free to contact me if you have any questions.

Hope to see you there!

The idea was to get together Tuesday evening, starting at around 7PM, and to code all night and have an iPhone game (or at least a prototype) done by morning. About 25 showed up, working on about a dozen projects. Participants were welcome to group into teams or work solo. There were no restrictions as far as themes or technology. The only rules were that you had to finish something by morning (no leaving something that was 5% of a game) and you had to start the game from scratch (no finishing a game you had started a while ago).

The reason behind doing a game jam is to be able to quickly create game prototypes and decide if they’re worth pursuing further or not. Much better to spend a few hours one night and then throwing it away, than spending several months with multiple people involved. Also, the heavy time constraints often help participants by focusing them in the ultimate goal and forcing them to adopt creative solutions. Being all together in one large room added a lot of energy and made it possible to work really focused for many hours without a huge amount of caffeine (although the large amounts of coffee provided by Jay really helped!). In the end, we even ended up getting a late-night fiddle serenade by Phil Hassey. Try that at home by yourself! Owen Goss liked the idea so much, he decided to jam with use remotely through video conferencing!

One specific thing that I wanted to avoid with the game jam was the idea of giving any sort of prices at the end. I really wanted to keep the jam as an environment where people would be free to experiment and fail. If people were thinking of potential prices at the end, they would be a lot less willing to go out on a limb and try different things.

My goal going into the jam was to experiment with game mechanics that relied heavily on multi touch. I felt that very few games made good use of multi touch (aside from pinching) and that it had the potential to create something very different from the single-touch based game that we’re used to from mice for the last 20 years. The flip side of it is that a multi-touch game was much riskier and unknown. Could something that multiple simultaneous touches be fun, or would it be frustrating? Would your fingers get in the way and clutter the screen, or would it be a new experience? In the back of my mind, I was definitely drawing inspiration from the fantastic game Bug Bugs, by Igloo Games.

I did all the coding myself, but I teamed up with Evan McMahon from Veiled Games and he did all the awesome art in the prototype.

Owen Goss joining the iPhone Game Jam remotely

What Went Right

Started with an idea

I went into the jam with the idea of experimenting with multi-touch game mechanics. I also had in mind a setting that would be a good backdrop to those mechanics: You control a large, round spaceship in interstellar travel. As you travel between the stars, you need to collect different resources to power your ship and provide all necessary materials. Each type of resource goes in a different bay (which are located along the outer rim), so with two fingers you rotate the ship to move the collecting bay forward, and with the other hand you drag the different resources you encounter. Some resources aren’t useful, so you need to flick them away so they don’t hit the ship. Finally, other resources need to be combined before they can be processed, so you first need to bring them together, and then drag them to the ship.

You only have 10-12 hours to build a game, so if I didn’t have an idea going into the jam, I would have wasted precious time trying to figure out what to do while I could have been working on the game.

At the same time, I didn’t have the idea fully fleshed out and set in stone. I definitely let things change based on what I saw as I was implementing it. So it’s good to strike a flexible medium between having an idea and letting it evolve during the jam.

Used familiar tools

Just the day before, David Whatley raved about how great Cocos2d was for prototyping, and that same day, my friend Serban gave a presentation on using Cocos2d. I’m on record saying that I can’t stand to work with Cocos2d because it imposes a particular (totally backwards) architecture on you, and I can’t ever see myself shipping a game with it, but I can see the value in using it for prototyping.

The point of this jam (for me anyway) wasn’t to learn some new tech, but to experiment with gameplay. So I really considered it, but in the end, I decided to work with something I was familiar with, and used my (very small) codebase using straight OpenGL. I’m very glad that I did because I never felt that working at that level got in the way (other than one frustrating moment related to the first point in “what went wrong”),

Didn’t worry about performance

This is a hard one for me! The last two talks I’ve given are all about performance on the iPhone, but even thinking about performance here would have been a waste of time and resources. Sure, the game had to be fast enough to run on the actual device because playing a game on the phone is a very different experience than playing it on the simulator. Besides, a game like this requiring lots of multi-touch was impossible to play on the simulator.

But at the same time, I didn’t have to submit it to the App Store come morning, so I decided that as long as it ran on my 3GS, I didn’t care about anything else. That meant it could be very suboptimal and be a total memory hog. I didn’t care! I wasn’t even releasing resources!

The result, it played perfectly fine on my phone, and I spent zero time trying to optimize it or taking a longer route because of performance or memory reasons. On the other hand, just the tiny, single level I created is about 12MB, so it’s clearly very bloated.

Teaming up

As I mentioned earlier, I teamed up with Evan McMahon to create the art for the game. He wasn’t at the conference, so we organized everything through IM and Dropbox. The experience was great because not only did he create some awesome-looking art, but he really motivated me along the way. We played off each other’s energy and came up with new ideas because of the collaboration.

Going solo would have been OK because I was surrounded by a room full of like-minded developers hacking away at their games, so that would have helped keep up the energy levels, but it wouldn’t have had nearly as much of an effect as working directly with someone else. My only regret is that Evan wasn’t there with me.

What about the art itself? Was it a waste of time to add it to the game? After all, it didn’t help me make a better decision about the multi-touch game mechanics, did it? It wasn’t a waste of time at all for me. First of all, it took just as long adding some nice art than it took to add stand-in art because there wasn’t anything special about it. But most importantly, there is a feedback loop between the look of the game and the gameplay and feel of the game. So seeing the art in the game definitely changed how I felt about it and the decisions I made about it during the night.

What Went Wrong

Didn’t have a project ready

This just goes to show my lack of experience with super-rapid prototyping. I didn’t have a project ready to get started with the prototyping. I originally created a blank OpenGL project from XCode, and quickly realized I would be missing all the set up for sound, the view controller structure, and even some of the default touch-handling. So I grabbed one of my current projects in development, stripped it of everything game specific, and turned it into the project for the game jam. Not a big deal, except that it took me a good 20-30 minutes to go through this, and in the process I deleted a couple crucial lines that gave me lots of problems with OpenGL for the next 30 minutes. Doh!

Next time: Come prepared with a blank project, ready to go.

Running up against technical stuff

I went with OpenGL instead of Cocos2d on purpose to avoid having to deal with learning technical stuff sleep deprived in the middle of the jam. Unfortunately, it turns out my understanding of the multi touch event API wasn’t up to snuff to handle 5 simultaneous touches, starting, moving, and ending, each of them affecting a different object (or different parts of the same object). During the jam I wasn’t writing unit tests either, so that made for a pretty confusing time until I stepped back for a second, hit the documentation and Apple’s touch sample program, and tried to gain a better understanding of what was going on. Hint: Apparently looking at the touches in the UIEvent* parameter passed to touchesBegan is not a good idea, and you need to look at the ones passed in NSSet*.

Not enough time!

I certainly didn’t get as far in the prototype as I had hoped I would. There’s only one type of resource (ice chunks) and the control mechanics need to be improved quite a bit.

Is it possible that if I had used Cocos2d I would have gotten more done of the prototype? Yes, it’s possible, although I won’t know until I try it. Is it possible that half way through the prototype I would have encountered something that Cocos2d couldn’t do and I was left stranded? Yes, that’s also possible. And that’s a big reason why I don’t like framework-y type of APIs, and instead I would rather have a toolkit type of API so I can call into it and do the work myself whenever I want instead.

In any case, this is the flip side of “using familiar tools”. I will definitely experiment with Cocos2d in the future and consider it for another game jam if I’m familiar enough with it and I feel it speeds up development.

Everybody still going strong late at night (but not quite so perky anymore)

Conclusions

I think Space in Stereo has potential, but right now it’s too early to tell if it’s any good. The basic mechanic is there, but it’s not fun by any stretch of the imagination. I already learned a bunch about it though, including that some things weren’t nearly as much fun as I was hoping would be. For example, rotating the space station with two fingers… well, sucks. The better control scheme is to move the space station by dragging around the middle, and rotating it by dragging a finger around the edge in a circular motion. That was also what everybody was instinctively doing when they were trying out the game.

The game is just at the point that spending another 4-5 hours on it would answer a lot of questions about how fun it can be. I’m definitely planning on spending that time and then making a decision. If I decide not to go ahead with it, it’s not a big deal. Great games come out of making a lot of different prototypes and throwing away all the so-so ones along the way.

As far as the Game Jam itself, it can only be described in one word: Awesome. It was everything I was hoping for and more. I can’t stress enough how much it helped to have everybody else around you and being so focused for that period of time. I had heard much about game jams, but I had never actually participated in one. After this one I’m sold. I can’t wait until the next one!

Gameplay footage of Space in Stereo

Session description:

The iPhone has a very powerful engine under that shiny hood when it comes to floating-point computations. This is something that surprises a lot of programmers because by default, things can slow down a lot whenever any floating point numbers are involved. This session will explain the secrets to unlocking maximum performance for floating point calculations, from the mysteries of Thumb mode, to harnessing the full power of the forgotten vector floating point unit. Stay away from this session if he thought of reading or even (gasp!) writing assembly code scares you.

• Presentation slides (pdf format)
• Sample project

Thanks for everyone who attended the session and for the great feedback so far. If you’re going to be in China next month, I’ll be giving a very similar talk at GDC China.

Session description:

This session will describe the iPhone performance optimization lessons learned through many hours of tuning. We’ll start with an overview of the performance analysis tools available on the iPhone SDK to help you narrow down your performance bottlenecks. Then we’ll cover the best way to set up your render loops, rendering best practices, how to deal with the limited memory, or even how to drop down to assembly to use the forgotten vector floating point unit.

Presentation slides (pdf format)

#defining The Problem

The first approach that came to mind was to use #ifdefs. Hopefully, if I did things well, it wouldn’t require too many of them. As soon as I gave it some more thinking, it was clear this was a pretty flawed approach.

I like to be able to create a build with zero human interaction: Press a button, wait, and the build is ready. I don’t want to have to do any manual (read, error-prone) steps. So I couldn’t just have an #ifdef that I turned on and off depending on whether I was building the lite version or the full one. I can get around that by creating a new configuration in XCode, except that it wouldn’t be one, it would be one for every existing configuration I already have: Debug, Release, Distribution Ad Hoc, and Distribution App Store. So this would make a total of 8 configurations. That’s rather ugly!

In case the explosion of configurations wasn’t ugly enough, there was one other, even bigger problem: An #ifdef wouldn’t allow me to change the assets included in each version of Flower Garden, and, most importantly, I couldn’t change the Info.plist file to have a different bundle id (although maybe I could change the Info.plist for each of the new configurations and point it to the new one).

Not an attractive option. So how else can we do it?

New XCode Project

The idea is straightforward: Make a copy of the existing project for the full version of the game, and change it in any way you want: Remove resources, add new ones, change Info.plist, etc. You can still share common source files and define a preprocessor macro LITE in the new project to #ifdef parts of the code.

This approach sounds more promising, until you realize that changes made to one project don’t carry over to the other one. So if you’re done developing the full version, and you create the lite version at the very end, maybe that won’t be a problem (although, how can you be sure you’re done?) And making changes to two separate projects is never fun.

You could improve things a bit by starting out the lite version as a branch in source control. That way, changes made to one of the branches can (hopefully) be merged back into the main line or another branch (have fun merging xcodeproj files). I think it’s because I’ve had the displeasure of working in projects that went totally overboard branching, that just the word branch makes me recoil in disgust. Yes, this method might work, but it seemed cumbersome and error prone.

Any other solutions?

Stay On Target

It turns out I kept thinking in terms of a solution I could have implemented in Visual Studio (that’s what I get for spending so many years working with it). But fortunately we’re not in Windows-land anymore. This is XCode, and XCode totally rocks.

XCode’s build system, once you wrap your head around it, is much more flexible and natural than Visual Studio’s. It’s much more like a make file (and I say that in a good way–I do like make files!). You have files, and then you have targets. And there’s nothing that says a file can’t be in multiple targets (and the file only appears in the project once). Once you have multiple target, whenever you add a new file you have the option to add it to both targets the same time if you want. Perfect!

My Flower Garden project contains about 15 targets: 7 “engine” libraries, 7 test executables for each of the libraries, and one main executable target for the game itself. So the (now obvious) solution, was to simply create a new target for the lite version, in that same project, with the same files. It turns out this solution had all the advantages of the previous approaches with none of the drawbacks.

Lite Target Details

Once I landed on the target idea, it really was quite straightforward to create the Lite version by following a few, simple steps:

• Make a duplicate of the target for the full version. Just right click on the full target, and select duplicate.
• Rename new target to something else (Flower Garden Lite–I’m highly imaginative when it comes to naming things like this! :-)
• Make a duplicate of your Info.plist (InfoLite.plist)
• Change the info.plist file setting for the lite target to point to InfoLite.plist
• Edit InfoLite.plist to have a different bundle identifier and maybe a different bundle display name (the name that’s going to show up under the icon on the iPhone).
• Add a compiler define to the lite target called LITE or something like that. That way you can #ifdef a few parts of the code depending on the version you’re compiling.

And that’s almost it. There’s one more trick that made my life a lot easier. It turns out that you will definitely want to change not just some code, but some assets depending on which version you’re using. So maybe you’ll have a Wall.png texture in the full version, but you’ll want to have a different version of that texture in the Lite version. I started out creating a new file called WallLite.png, but then it quickly cascaded into a bunch of code changes loading either Wall.png or WallLite.png depending on the version and it was ugly and time consuming.

A much better approach is to rely on the resource copy step of XCode. By default, XCode will grab the resources in your project and copy them all in the same folder, flattening the hierarchy completely. This can be a pain if you have resources with the same name organized in different folders because they will clash. But here we can use it to our advantage to supply different versions of the resource for the full and lite versions.

Just create a Lite directory and put any variations of your resources there. Then, in the Resources Copy step of the target, remove the resources you’re replacing and add the ones from the Lite directory. Whenever you build, XCode will put those new resources in the same place where the full resources were and your code won’t have to change one bit. This trick works even with xib files, which is great because I had to modify one of them to reduce the number of pots in the garden screen.

Once you have that set up, the rest is just a matter of deciding what functionality goes in the lite version and how it is presented. It can be as simple or as complex as you want.

In the case of Flower Garden, it was very straightforward. I limited the number of pots to two, and the number of different seeds to three. So I just had to change a few resources and add a couple of #ifdefs with the number of total pots and seeds. I also removed the resources that weren’t used in the lite version (the seed data for all the unavailable seeds for example).

Flower Garden lite was finally approved and it has been available on the App Store for a few days. So far it has been downloaded several thousand times, it has been very well received, and has definitely bumped up the sales of the full version. Definitely a few hours well spent!

[1] The lite version took a couple of hours to create and a month to approve by Apple.

[2] It was something I wanted to have created from the beginning, but my hands were tied because email bouquets were being sent through my server and I had a maximum number of emails I was allowed to send per day because of spam limits. Fortunately, as soon as the 3.0 OS was released, apps were able to use in-app email and that restriction went away.

There’s a darker side to Apple’s randomness. Something developers have been complaining for quite a while and it’s just getting worse: The approval process.

This is nothing new though. For a while developers have been reporting random rejections by Apple. Some of them extremely unfair and illogical.

A few months ago, if you got hit with one of those random rejects, it was annoying, but it was bearable. Usually a resubmit would get approved in under a week, so it was possible to fix whatever random thing Apple flagged down (or show them that it wasn’t an issue), and resubmit without wasting too much time.

Now things are different. Supposedly 95% of apps are approved (or rejected) under 14 days (although my informal and biased survey shows very different percentages). A mistake from Apple rejecting an app doesn’t cost you a week anymore, but perhaps as much as an extra 3-4 weeks! That’s a big deal for a company that’s trying to time the release of a product.

So far, I had been one of the lucky developers and I never got a rejection from Apple with Flower Garden. Actually, updates were usually approved withing 7 days to the hour (very suspicious, isn’t it?). But my luck ran out and I joined the group of developers hit hard by random and unfair updates.

Two days ago, Flower Garden 1.7, which had been in review for 24 days was rejected with the following explanation from Apple:

Dear Noel,

Thank you for submitting Flower Garden to the App Store. We’ve reviewed Flower Garden and determined that we cannot post this version of your iPhone application to the App Store because your application contains pricing information in the application, located in the “More Games” page. Providing specific pricing information in this location may lead to user confusion because of pricing differences in countries. It would be appropriate to remove pricing information from your application.

Once these modifications have been made please upload a new binary using iTunes Connect

Regards,

iPhone Developer Program

I was completely floored when I read that email. So apparently displaying a web page with prices for other apps is not allowed unless you localize that based on the language settings of the iPhone. How many apps are doing that? I doubt many (any?) are doing it. I can certainly bring up piles and piles of other games that are showing non-localized prices. But precedence is clearly not an argument with Apple. Even previous versions of Flower Garden had that exact same view with prices and it wasn’t a problem! Random reviewer strikes again.

But wait, there’s more. The prices are shown on a web page fed through my server. I made that change in 30 seconds and emailed them back, hoping they could simply re-test that functionality and approve it on the spot. No, that would be too easy! I had to resubmit a new binary and start the approval process again! I just gave them the change to cut back their work to a fraction of the time (supposedly they had tested other things that they’ll have to re-test because they have no way of knowing I didn’t change/update anything else). Sigh! I resubmitted it and I’m hoping for a quick approval, but somehow I’m ready for another 24-day wait.

So this morning I was excited to receive another email from Apple, I open it and…

Hello Noel,

Flower Garden Lite cannot be posted because it is a feature-limited version. Free or “Lite” versions are acceptable, however the application must be a fully functional app and cannot reference features that are not implemented or up-sell to the full version.

Users can view unlockable seeds on the “Seeds” tab but cannot access them in the Lite version of the application. Please refer to the attached screenshot for an example.

Please upload a new binary and correct metadata using iTunes Connect .

Regards,

iPhone Developer Program

and they attached the following screenshot:

Users can be unlockable seeds but can’t unlock them? If you follow the included tutorial (2 messages) you unlock the first one. Another message tells them that growing the second seed unlocks the last one. Did they read that, or did they see a locked seed and had a knee-jerk reaction?

I replied to their email right away and explained how to unlock them. I’m afraid I’ll get an email back tomorrow asking me to resubmit and throwing Flower Garden Lite at the back of the queue. I hope not, but I’m prepared for the worst.

It might just be slightly annoying for hobbyist developers, but all of this is making the App Store very difficult to build a reliable, sustainable business around it. The lack of visibility, random approval process, unpredictability, and lack of direct contact make it impossible to have a reliable process or combine release dates with other events such as PR or marketing campaigns. At this point, I would even consider a “professional” iPhone developer account that costs $1K per year and gives you a direct contact in Apple and better visibility and predictability with the approval process. Otherwise, if things continue this way, I’m afraid that professional developers will only put up with this for so long and are soon going to consider the iPhone as a secondary platform and they’ll move on to other pastures like the PSP Mini or new opportunities that come up.

iSimulate consists of two different parts: The iPhone app, which you need to purchase through the App Store, and a static library that you link with your program. Once your app is using the library, launching the iSimulate iPhone app lets you connect your device to your app and forward accelerometer, touch, and GPS events. In other words, most of the things the simulator doesn’t do natively.

iSimulate definitely passed my picky installation requirements with flying colors. I was afraid the installation would hook up with existing framework libraries, add new configurations, and in general mess up my system. Fortunately, hooking up the iSimulate library couldn’t be easier: Copy the provided .a library anywhere you want, link with it in your app (don’t forget the -ObjC linker flag), make sure you’re using the Core framework, and you’re good to go. That simple. That’s how I want external libraries to work.

You don’t even need to initialize anything. Just launch your app in the simulator, launch iSimulate on the iPhone and it detects your program running and lets you connect to it. The iPhone app itself even looks very pretty and polished. The overall package is definitely very professional-looking.

From there, any touch on the device is forwarded to the simulator. So you can finally do off-center multi touches, or two touches that happen at different times, or even 5-touch events (the max the iPhone support). The other big one is being able to tilt or shake your device and see the game react correctly on the simulator, which is would have been very handy when writing the accelerometer code to move flowers around in Flower Garden, but would almost be necessary when you’re making heavy use of accelerometer input for games like Sky Burger. Finally, it also claims to give you location information, but I didn’t have an app I could easily test it with.

Flowers affected by gravity on the simulator

Another situation where iSimulate is a life savior is when trying to capture a video of your game. If you rely on multi touch or accelerometer features, then your only options are to either hack in something yourself, or dig out a physical video camera and record it the old-fashioned way (which is surprisingly time consuming and creates less-than-ideal results). With iSimulate you can still use a desktop video capture software on the simulator and create a top-quality in a fraction of the time.

Not everything is perfect though. I thought the forwarding of touch events would be incredibly useful, until I realize I have no idea what I’m about to tap because the program is running on the simulator, not on the device. So I’m just making a guess about where my finger is going to land. It’s not even like a mouse cursor that you see where it’s going and then you click. You’re literally tapping blind hoping you touch what you wanted. That works with a static UI, but if you have moving elements you need to touch, you’re totally out of luck. I wonder if they could extend iSimulate to capture the image from the simulator and forward it to the iPhone. Even if they only refreshed the image a few times per second, that would be enough to make touch input a lot more friendly.

There’s another big problem with touch input though. From their docs page, iSimulate works great with OpenGL apps, but doesn’t work with the following UIKit controls: Keyboard, UIScrollView (including MKMapView), UIPickerView and UITableView. Wow! Apart from the keyboard one (no big deal, we have a real keyboard attached) the other ones are pretty major. I’m sure they have a good reason why, but I can’t imagine what that might be from my knowledge of how the touch even system works (anybody from Vimov care to comment on that?). Heck, I’d even be willing to add some extra code to my app to hook directly into iSimulate input events if it means it would work with those classes. Apart from that, I found one case in Flower Garden where iSimulate failed: Touching the bouquet tag to bring up the editing mode. It’s weird, it would let me touch-drag it to move it, but it wouldn’t go into editing mode.

The accelerometer input worked flawlessly, but here iSimulate has some competition from Accelerometer Simulator, an open-source app plus library to forward accelerometer data from the device to the app. It’s not nearly as slick as iSimulate, but it’s free, so that migth be a more attractive solution for some people.

Overall, is iSimulate worth the $32 price tag (it’s $15.99 at the moment)? I would say yes if you’re developing OpenGL iPhone games with heavy use of multi touch and accelerometer input and you’re planning on making money from them. In that case, you’re better off spending an hour of your time polishing the game (or taking a break from programming) and buying iSimulate. However, if you’re using UIKit objects or you just need accelerometer input, then you might be better off just using Accelerometer Simulator. If Vimov can fix the UIKit input and display the output of the simulator on the device, iSimulate would become an indispensable tool in every programmer’s toolbox.

Obviously I’m biased when I say the class is going to be awesome (which it is), but I can also say with total honesty that 360iDev is an incredible conference. It’s very different from WWDC, but that’s what makes it so great: It has top-quality content and speakers, but it’s small and intimate, so you get to meet and hang out with all the other speakers and participants. It was at the first 360iDev conference back in March that I met Keith and Owen (among many other cool developers) in person for the first time and that’s how App Treasures was born.

So if you’re on the fence, I hope you give it a try. See you in Denver!

Identity Crisis deals with the horrors of a CRC clash on a resource right before submitting the gold master. And The Programming Antihero shows a fairly common trick in the games industry to get that extra memory after the artists and designers swear up and down that they can’t cut any more content.

Nothing like a good dose of reality to make everybody feel better about their own code :-)

By default, the UIWebView will try to open the App Store link itself, which results in an error. What we want to do instead is intercept the request and launch it directly with an NSURLConnection so the call can be redirected to the iPhone App Store app. The best place to implement this is in the -[UIWebViewDelegate webView:shouldStartLoadWithRequest:navigationType:] method:

- (BOOL)webView:(UIWebView*)webView shouldStartLoadWithRequest:(NSURLRequest*)request navigationType:(UIWebViewNavigationType)navigationType
{
    if ([[[request URL] host] isEqualToString:@"phobos.apple.com"])
    {
        NSURLConnection* conn = [[NSURLConnection alloc] initWithRequest:[NSURLRequest requestWithURL:[request URL]] delegate:self startImmediately:YES];
        [conn release];
        return NO;
    }
    return YES;
}

While we’re in the topic of links, I also recently added some LinkShare links to my apps. The 5% kickback for each purchase is nice, but the tracking capabilities it gives you is the real reason I wanted to use it. The way LinkShare works is by providing a custom URL that contains your referral ID to allow LinkShare to track the transaction. Unfortunately, unlike the Amazon referral program for example, LinkShare is a third party company and is not tied directly into the App Store. That means the links they provide go through click.linksynergy.com and then are redirected to the App Store. The result is that trying to follow one of those links from your app will result in Safari coming up, opening up a new page, shutting down, and then opening the App Store app. Not pretty!

Fortunately, there’s a workaround. Just like we did with the App Store links, we can open the LinkShare link directly with a NSURLConnection instead of letting Safari handle it. Then, once the link resolves to the forwarded one, we can open it directly with the App Store app:

- (IBAction)buyItNow
{
    NSString* link = @"http://click.linksynergy.com/fs-bin/click?........";
    NSURLConnection* conn = [[NSURLConnection alloc] initWithRequest:[NSURLRequest requestWithURL:[NSURL URLWithString:link]] delegate:self startImmediately:YES];
    [conn release];    
}

- (NSURLRequest*)connection:(NSURLConnection*)connection willSendRequest:(NSURLRequest*)request redirectResponse:(NSURLResponse*)response
{
    m_iTunesURL = [response URL];
    return request;
}

- (void)connectionDidFinishLoading:(NSURLConnection*)connection
{
    [[UIApplication sharedApplication] openURL:m_iTunesURL];
}

Voila! The user goes directly to the App Store page, there are no annoying Safari transitions, and you still get the referral bonus.

Incidentally, most of this is already documented in an Apple Technical Q&A note, but it’s kind of hidden away and I think most people don’t know about it.

If you’re using LinkShare, you should also check out this post at Mobile Orchard. One of the things I learned there was the use of the &u1 parameter, which allows you to tag links with arbitrary information so you can track where purchases came from. Knowing where people are buying your app from is extremely helpful to know how to maximize sales!

[Edit: Sam Steele pointed me to the source code for the Last.fm app which handles all the URL redirects. Thanks!]

The demo ended up looking pretty good, so I decided to upload a quick video showing the different lighting modes:

• Diffuse texture only
• Diffuse texture plus ambient and diffuse lighting
• Diffuse texture plus ambient, diffuse, and specular lighting (as usually, per-vertex lighting looks pretty bad, even though this is a relatively high-poly model)
• Fully reflective environment map (using the normal environment map technique)
• Environment map added to the diffuse texture and lighting
• Environment map with a reflection mask plus diffuse texture and lighting (two passes on an iPhone 3G–or one pass if you’re not using the diffuse alpha channel)

The chapter in the book will go in detail into each of those techniques, building up to the last one. Full source code will be included as well.

Some of these techniques will also be covered in my upcoming Two Day iPhone OpenGL Class organized in collaboration with Mobile Orchard.

Trouble With Versions

When you install the 3.0 SDK and create a new project, it will be automatically set up to build only for 3.0. To target earlier versions while still having access to 3.0 features, you need to take a few extra step. These steps are described in detail in the readme of the MailComposer sample (iPhone dev account required). My friend Serban also wrote about how to do it when you add Snow Leopard to the mix.

The required steps are:

• Under project properties, set up the Base SDK setting to be OS 3.0

• Also in project properties, change the iPhone OS Deployment Target to OS 2.2 (or whichever version you want to target).

• Go to target properties, and add a 3.0 library with the features you need. Make sure you set its type to Weak instead of Required.

• In your code, check that a feature is available before using it. For example, you can do this check to see if the in-app mail functionality is available:

Class mailClass = (NSClassFromString(@"MFMailComposeViewController"));
 return (mailClass != nil && [mailClass canSendMail]);

• Make sure you set your app to build with the 3.0 SDK and off you go.

If all goes well, it should run under 2.x and 3.0. If all goes well…

Trouble With Libraries

If that was the end of the story, then we would all be happy and I wouldn’t have to write this entry. And for a while I really thought that was everything I had to do to get Flower Garden to use 3.0 features and still work on 2.x devices. Everything compiled fine, but when I went to run it on a 2.2 device, it crashed.

Looking at the crash logs, it was crashing inside a static library that used Objective C and UIKit. Digging further, it seemed that function calls were being sent to the wrong place. What was going on?

At this point I realized the root of this problem was the linker flags I was using. As soon as I started using the 3.0 SDK, I had to add the -all_load linker flag in order to be able to use the static library. I believe this loads all the symbols used by the libraries and links with them at link time. Without it, the library code would crash at runtime as soon as it was executed

The -all_load flag seems fine, except that the 2.x and 3.x versions of the SDK have different libraries and resolve symbols to different locations. So by doing -all_load, we’re linking against the location of the 3.0 version and trying to run it on 2.x. Bad idea.

I thought long and hard on how to get around this. I came up with all sorts of crazy schemes, and after a couple frustrating days, I gave up. Then, all of a sudden, I realized that it had an embarrassingly simple solution: Don’t use a library! I’m not kidding. Just move the files in XCode directly into your main game target and you’re done. No -all_load and everything works fine.

Yes, I’m still embarrassed for not figuring that out after 30 seconds…

Trouble With Compilers

So all happy with that discovery, I rebuild and run the app and… crash again!

The call stack this time looked like this:

Thread 0 Crashed:
0   dyld                              0x2fe01060 dyld_fatal_error + 0
1   dyld                              0x2fe07ca8 dyld::bindLazySymbol(mach_header const*, unsigned long*) + 484
2   dyld                              0x2fe15eb4 stub_binding_helper_interface + 12

What was going on in there? Some Googling and searching in the iPhone forum later, I learned that SDK 3.0 uses a different version of GCC (4.2 instead of 4.0). That means it will try to use some runtime functions that are not available with earlier versions. In particular, my crash was related to subtracting two uint64_t variables. Re-writing the code by casting the values to uint32_t before doing the operation fixed the problem. There’s an ugly “solution” for you!

So how do you know if something will work on 2.0? I don’t have a good answer for that other than test it as much as you can. Does someone have a better solution?

The good news is that, after I made those fixes, Flower Garden was happily running on 2.2 and 3.0. Now I can finally roll out in-app email without giving up on 2.x devices and cutting my potential customer base (or depriving current users of future updates).

Open Questions

Going through this answered a few questions, but also created a few new ones. Maybe someone here will know the answer or will be able to point me in the right direction.

• Does anyone know how to debug your OS 3.0 app on the simulator set to 2.2? Whenever I launch it from the debugger, the simulator gets set to 3.0. Even if I set it to 2.2, I wonder if it will behave the same as a 2.2 device.
• I’ve heard rumours about somehow, packing two versions of the app in the same executable (kind of like the universal MacOS executables with both a PowerPC and an Intel version). Has anyone done something like that with the iPhone? Any docs on that?

At first, things sounded simple enough: Create 20 different seeds, with varied looks and properties. I had a really cool tool to tweak the flower DNA in real time (more on that another day), so how hard could it be?

The main properties I had to balance were amount of care required (as in, how often you had to water them before they would go dry) and how long the flower took to grow. The first few flowers came along just fine, but after the fifth or sixth seed, they all started blurring together. Did I have a fast-growing flower that needed a lot of care already? To get around this, I started writing all the seeds I had done so far and their characteristics in a text file so I could refer to them easily.

After a few more seeds, I knew I was in trouble again. Reading through all the flowers was a pain, but now I was starting to forget what they looked like. Did I already have an orange flower with a few, large petals? How about a small white one with lots of rounded petals and glossy leaves? The text file didn’t cut it anymore, I had to go graphical. So I created a document with a picture of each flower along with all its relevant information (name, growth time, and amount of care).

This helped a lot, but it eventually became inadequate when I started creating the unlock conditions and balancing and the different seeds in order of difficulty. I just wasn’t able to look at a list of 20 entries, each of them with 3 key elements (care, duration, and unlock condition) and keep it all in my head to make intelligent decisions based on it.

I’m a very visually-oriented person. Whenever I can, I try to solve problem visually instead of memorizing lists or plugging equations. In particular, I love to take a multi-dimensional problem and visualize it along its main axes. So I took the next logical step to help me make sense of all that data and created a seed chart.

This seed chart is a two-dimensional arrangement of all the common seeds in the game, along with their picture and the key information we had before (there are another dozen bonus seeds that aren’t listed in this chart). The horizontal axis indicates how much care a plant needs. Specifically, it shows how many segments in the water meter before it dries out (and each segment is 3 hours). The vertical axis indicates the growth duration for the plant in real time. Instead of making it a straight, linear scale, I decided to go with regions, which is how they are presented in the game (“Instant”, “A few hours”, “Overnight”, “A few days”, “About a week”, and “Quite a while!”). Within the region, they’re roughly ordered by duration (shorter towards the top, longer towards the bottom).

Now it was really easy to see at a glance the key characteristics for each seed, see where I had clusters, and where I had empty spaces that no seeds were currently using. As a bonus, I was able to come up with a rough ordering indicating the level of difficulty for each seed. The further down and to the right a seed was, the more difficult it was to grow.

That allowed me to block out some regions (dotted red lines) indicating difficulty ratings, which are listed in the game in the seed information as “Piece of cake”, “Easy”, “Moderate”, “Hard”, or “Experts only”. Notice how the lines separating the regions are not parallel (even though the vertical axis is kind of logarithmic). That’s because moving along the horizontal axis increases difficulty a lot faster than moving down the vertical axis. So even a relatively complex relationship like that can be observed at a glance from the chart.

Another great benefit of the seed chart is that I was able to draw paths between seeds, showing the unlocking relationships between them. Before it was hard to see these relationships in text, but now it was very visual, and I could make sure that easier flowers unlocked the path to harder ones as long as the unlock relationship flowed roughly from the top left to the bottom right.

As a side effect, the chart allowed me to learn even more things about the data than I had expected. For example, I was able to see which colors I had used so far, and which colors I still needed to use. One really interesting thing I noticed is that colors were clustering around specific areas. Red colors seemed to be mostly in the harder flowers. Yellow colors were predominant with the dry plants, and blue colors with the wetter ones. I thought that was pretty neat, so I tweaked a few seeds here and there to make that color connection even stronger.

The lesson learned here is that the old saying applies even more in this age of information: One picture is worth one mega of words.

The class is aimed at iPhone developers without previous OpenGL experience. It’s going to be very hands-on, and you’ll create both 2D and 3D applications during the weekend. You’ll learn all the basics: cameras, transforms, and how to draw meshes, but we’ll also cover some more advanced topics such as lighting, multitexturing, point sprites, and even render targets. Most importantly, you’ll walk away with a solid understanding of the basis, which will allow you to continue learning OpenGL and advanced computer graphics on your own from the docs, samples, or even browsing the API directly.

The main requirement for the class is that you’re familiar with the iPhone development environment and that you have basic knowledge of the C language. Beyond that, to the the most out of the course, you should be familiar with the basics of linear algebra (vector, matrices, and dot products). Anything else, we’ll cover it all during the class.

Registration is now open, and you can get some great discounts by registering early and attending the 360iDev conference. For more details, check the official announcement page.

Hope to see some of you there!

Notice the other iPhone games that flash at the start of the Flower Garden section: Harbor Master, Up There, and Little Red Sled, all App Treasures games! The video also starts with coverage of Fifa 10, so we’re in pretty good company :-)

Today we finally unveiled a project we had been working on for a couple of months: The App Treasures label.

Only indie games that we consider to be really fun, polished, and of high quality are part of the label. We hope that when players enjoy one of our games, they can turn to the other App Treasures games and find something they like.

With the App Store dominated more every day by big developers and publishers, this is a way for small indie developers to stand our ground. With App Treasures we establish a strong brand, share resources, and get more visibility in the App Store and in the eyes of the users. The same benefits we would get with a publisher, but without giving up our independence, creative freedom, or a percentage of the profits.

The current App Treasures members are:

• The Blimp Pilots (Koi Pond, Distant Shore)
• Imangi Studios (Imangi, Word Squares, Little Red Sled)
• Snappy Touch (Flower Garden)
• Streaming Colour Studios (Dapple)
• Veiled Games (Up There, Payday Roulette)

You can read more details in this interview I did for App Craver. Mobile Orchard also published a really good piece with some interesting views and conclusions on App Treasures.

Oh, and don’t forget to follow App Treasures on Twitter to keep up with the latest news.

As far as I’m concerned, virtual memory paging is the lazy man’s caching scheme. There, I’ve said it. Now let me qualify it a bit and justify why I feel that way.

A lot of applications, and especially games, can’t keep in memory every bit of data they need for their execution. There’s just too much data and not enough RAM to keep all the levels, all the textures, all the characters, and all the movies. Even the thought of keeping everything in memory is ridiculous. Instead, games are architected to strike a balance between memory usage and responsive interfaces. Usually that means loading levels, or parts of levels on demand, and keeping them in memory while they can be needed. Some other times it means loading levels of detail for textures, meshes, and animations depending on the player position. The point is, nobody knows as well as the game itself what needs to be in memory, what can be unloaded, and when the best time to do it is.

A different approach would be to ignore thinking about managing memory as a scarce resource and use as much of it as we need. If we ever go over the amount of available physical memory, the virtual memory system will kick in and page out memory to disk to make room for the memory we’ve requested. But there lies the problem: The virtual memory system is a lower-level system that knows absolutely nothing about our game or application. It can only make guesses about what memory is OK to evict and when it’s a good time to do it. If you’re unlucky it will choose to page out memory when you need performance the most, and it might evict a page that you’re about to use in a few milliseconds. If that happens, you can kiss your performance bye-bye.

Because of that, virtual memory paging will always do a much worse job than we could have done ourselves. But hey, it takes no effort or thinking on the part of the programmers, hence the “lazy man’s caching scheme” I was referring to earlier.

However, virtual memory paging is sometimes a necessary evil. The assumption running through the previous three paragraphs is that we, as programmers of the game, can always manage the physical memory so our game runs efficiently on it. That is true for certain platforms, usually with fixed hardware specs, and ones with minimal or no background processes running. Game consoles are perfect examples of this: We know how much RAM we have to start, we know how much the system needs, how much video memory there is, and we can deal with the rest ourselves. We’ll decide what to load and when to load it.

On something like a modern PC (whether it’s Windows, Mac, Linux, or any other flavor), we have none of those guarantees. We don’t know how much memory we’re going to encounter when we run our game, and, what’s even worse, we have no idea how that available memory is going to change during the execution of the program [1]. In a situation like that, we can plan for some minimum memory requirements, but we’re going to need that virtual memory paging to bail us out of tricky low-memory situations. It might sound like a good idea, but that’s one of the main reasons why PC games are often choppy and with inconsistent frame rates (buggy drivers being the other main reason).

Finally coming back to the iPhone, how does virtual memory paging fit there? There’s no doubt that it would help because it would reduce the number of crashes due to programs unexpectedly running out of memory. But at the same time, it would cause most games and apps to be choppy and unresponsive, especially when it’s just launched. But the iPhone is mostly a fixed spec platform, with minimal background processes [2], so we can always do better than the dumb virtual memory system. Maybe that’s not a big deal if you’re writing an app that interfaces with a web site and writes data to a database, but it’s crucial to be able to write responsible games, which are considered near real-time apps.

On the iPhone there should be no need for virtual memory paging (not to mention that it would probably drain the battery a lot faster). Instead of solving the memory problem by throwing more and more complex systems at it, what we really need is a guaranteed amount of memory for our app to run and we can take care of the rest. But Apple needs to take the first step and set that memory aside.

We’re waiting, Apple.

[1] We could go totally hard core and allocate the memory we need and then mark it as not swappable by the virtual memory system. That wouldn’t do any good because if other processes start requesting memory and they run out, the virtual memory system will swap out other pages (and possibly cause other processes to thrash) and kill our performance as well. So things are pretty much out of our hands as soon as virtual memory paging comes into play.

[2] There are a lot of background processes, but they should be relatively well-behaved. None of them should be pulling in massive amounts of data while the game is running.

My recent rant

As a response to that post, some people suggested some intriguing ways to increase the available memory on the iPhone. I’ve been experimenting with that a bit, but I don’t have any conclusive solutions yet. Right now it’s all totally unreliable hacks. Hopefully in a followup post I can present some solutions.

In the meanwhile, I wanted to talk about a piece of the puzzle, which is keeping track of the memory status on the iPhone. How much memory is available? How much memory does the program use? We need to be able to answer those questions accurately in order to do anything about memory. Unfortunately, there doesn’t appear to be a good answer even for this!

Finding Total Memory

Let’s start with the easy part: Finding the overall memory on the device. Right now all iPhones and iPod Touch models have 128MB RAM, so it’s kind of pointless, but it’s interesting anyway. To accomplish that we need to dig into some of the low-level functions that query hardware capabilities. In particular, the function sysctl() reports both physical memory and user memory.

    int mem;
    int mib[2];
    mib[0] = CTL_HW;
    mib[1] = HW_PHYSMEM;
    size_t length = sizeof(mem);
    sysctl(mib, 2, &mem, &length, NULL, 0);
    NSLog(@"Physical memory: %.2fMB", mem/1024.0f/1024.0f);

    mib[1] = HW_USERMEM;
    length = sizeof(mem);
    sysctl(mib, 2, &mem, &length, NULL, 0);
    NSLog(@"User memory: %.2fMB", mem/1024.0f/1024.0f);

The output of that code is:

Physical memory: 116.00MB
User memory: 91.30MB

Interesting. Physical memory is not quite reported as 128MB. That’s probably because the video memory takes up 12MB of the total. The kernel apparently uses 24.7MB, which is not unreasonable if that was all the memory used by the OS. Unfortunately, that’s only for the kernel. The different processes and apps are going to use up more than that. A LOT more than that.

Incidentally, sysctl() is a pretty rocking function. It will give you all sorts of cool information, such as CPU and bus frequency (412MHz and 103MHz respectively for my iPhone 3G), cache sizes, and whether a vector unit is present (it claims it isn’t, so it must not be referring to the vfp).

Finding Available Memory

As cool a function as it is, sysctl() only returns static information about the hardware. If we want to find information about the current status of the memory, we need to look elsewhere.

As far as I can tell, the best function to get this information host_statistics() with HOST_VM_INFO_COUNT to get virtual memory statistics. Querying that function fills out the following structure:

struct vm_statistics {
    natural_t	free_count;		/* # of pages free */
    natural_t	active_count;		/* # of pages active */
    natural_t	inactive_count;		/* # of pages inactive */
    natural_t	wire_count;		/* # of pages wired down */
    natural_t	zero_fill_count;	/* # of zero fill pages */
    natural_t	reactivations;		/* # of pages reactivated */
    natural_t	pageins;		/* # of pageins */
    natural_t	pageouts;		/* # of pageouts */
    natural_t	faults;			/* # of faults */
    natural_t	cow_faults;		/* # of copy-on-writes */
    natural_t	lookups;		/* object cache lookups */
    natural_t	hits;			/* object cache hits */

    /* added for rev1 */
    natural_t	purgeable_count;	/* # of pages purgeable */
    natural_t	purges;			/* # of pages purged */

    /* added for rev2 */
    /*
     * NB: speculative pages are already accounted for in "free_count",
     * so "speculative_count" is the number of "free" pages that are
     * used to hold data that was read speculatively from disk but
     * haven't actually been used by anyone so far.
     */
    natural_t	speculative_count;	/* # of pages speculative */
};

Lots of good info there! (at least for a hardware geek like me). The most interesting one is free_count. That number combined with the page size (which is defined in the handy global variable vm_page_size–which is unsurprisingly 4K on the iPhone) should give us the amount of available memory, right? Right?

Kind of. Unfortunately virtual memory is managed at many different levels. The OS will keep some pages on reserve to use on a rainy day. So it’s possible that there is more memory available than reported in this function. It will however give you a good minimal bound on the amount of free memory. You can count on that amount for sure.

This function puts it together to return the amount of available memory in KB:

int getAvailableMemoryInKB()
{
    vm_statistics_data_t vmStats;
    mach_msg_type_number_t infoCount = HOST_VM_INFO_COUNT;
    kern_return_t kernReturn = host_statistics(mach_host_self(),
                             HOST_VM_INFO, (host_info_t)&vmStats, &infoCount);
    if(kernReturn != KERN_SUCCESS)
        return -1;
    return (vm_page_size * vmStats.free_count) / 1024;
}

To check that available memory roughly matches what I do in the app, I allocate a 1MB block, run it again and… surprise! I get the same amount of memory available. Apparently the OS is waiting to actually set those memory pages aside until you really need them (I hate it when machines try to out-think me). So to really get them to count as allocated, we need to write to every page of the memory we just allocated. The easiest way is just go bzero(mem, 1024*1024);. Running the function again correctly shows that the available memory has gone down by 1MB.

(Not) Making Sense Of Used and Total Memory

An old habit I picked up during my engineering days is to always confirm my calculations through a different path. If the total amount of memory reported by the virtual memory system matched up with the total amount of memory reported by sysctl() I would leave this happy and be able to sleep soundly tonight.

The total amount of virtual memory is calculated by adding together the free, used, wired, and inactive pages. It should come up to be roughly around 91MB. The answer: 89MB. OK, close enough, right? Not really.

Here’s the biggest mystery so far: If I malloc a chunk of memory (and write to it to have it marked as not available), host_statistics() correctly shows that those pages are not available anymore, but they don’t show up as used, wired, or anything else!!! So simply adding free, used, wired, and inactive reports a totally bogus number that doesn’t take into account allocations from your program. What’s going on there? Obviously I’m misunderstanding something, so maybe someone with more knowledge of kernel and vm features can help me out. Otherwise I won’t be getting much sleep I’m afraid :-)

Just to prove that I’m not crazy, here’s a code snippet and its output:

- (void)updateStatus
{
	vm_statistics_data_t vmStats;
	mach_msg_type_number_t infoCount = HOST_VM_INFO_COUNT;
	host_statistics(mach_host_self(), HOST_VM_INFO, (host_info_t)&vmStats, &infoCount);

	const int totalPages = vmStats.wire_count + vmStats.active_count +
                               vmStats.inactive_count + vmStats.free_count;
	const int availablePages = vmStats.free_count;
	const int activePages = vmStats.active_count;
	const int wiredPages = vmStats.wire_count;
	const int purgeablePages = vmStats.purgeable_count;

	NSMutableString* txt = [[NSMutableString alloc] initWithCapacity:512];
	[txt appendFormat:@"Total: %d (%.2fMB)", totalPages, pagesToMB(totalPages)];
	[txt appendFormat:@"nAvailable: %d (%.2fMB)", availablePages, pagesToMB(availablePages)];
	[txt appendFormat:@"nActive: %d (%.2fMB)", activePages, pagesToMB(activePages)];
	[txt appendFormat:@"nWired: %d (%.2fMB)", wiredPages, pagesToMB(wiredPages)];
	[txt appendFormat:@"nPurgeable: %d (%.2fMB)", purgeablePages, pagesToMB(purgeablePages)];

	NSLog(txt);
	[txt release];
}

- (void)viewDidLoad
{
    [super viewDidLoad];
    [self updateStatus];
    const int sizeInBytes = 1024*1024;
    int* mem = (int*)malloc(sizeInBytes);
    bzero(mem, sizeInBytes);
    [self updateStatus];
}

And the output is:

Total: 22128 (86.44MB)
Available: 8124 (31.73MB)
Active: 6361 (24.85MB)
Wired: 6408 (25.03MB)
Purgeable: 359 (1.40MB)

Total: 21880 (85.47MB)
Available: 7888 (30.81MB)
Active: 6344 (24.78MB)
Wired: 6412 (25.05MB)
Purgeable: 359 (1.40MB)

Notice how the available pages go down by about 1MB but everything else stays about the same?

While we’re at it, does someone know of a better, lower-level way to allocate memory than malloc()? I’d like to just allocate memory pages directly. I looked into vm_allocate() but I wasn’t able to get the correct port rights. Anyone?

Update: Thanks to Colin for pointing out that mmap is the low-level memory allocation function I was looking for. You can allocate pages directly without going through malloc. Perfect!

Next Up…

This is a stepping stone towards the memory experiments I’m running. In the next day or so I should be able to report some good ways to clear up as much memory as possible for our apps.

Except when it comes to memory, that is.

I really like most of the design calls that Apple made designing the software for the iPhone: Using C and Objective C (instead of Javascript or web-only apps), building it on top of a Unix-like core, and even providing OpenGL (although I would love to have lower-level access to the graphics hardware).

Developing for the iPhone is a mix of PC and game console development. One one side, it’s a fairly open platform with standard APIs (like PCs), and on the other, it has (almost) fixed hardware [1] and a regulated distribution channel (like game consoles). I would even argue that the iPhone borrows the best of both worlds. When it comes to memory, however, the iPhone adopted the worst of both worlds. I feel that Apple completely dropped the ball there. Completely.

On a PC (whether it’s Windows, Mac, Linux, or whatever), you can’t count on having a fixed amount of memory. When your program runs, it can be running in a machine with RAM to spare, or in one with barely enough to run the OS. Or maybe it’s running on a machine with enough RAM but there are fifty other programs running and there’s no memory left for your program.

Fortunately (or unfortunately depending on your point of view), virtual memory allows the operating system to write memory pages back to disk and free up space for your program. Yes, it can cause the program to halt and crawl while memory is being paged out, but it will run. Definitely not an ideal solution for semi-real time applications like games. That’s one of the main causes for choppy, underperforming PC games, but it has worked for many years and will do in a pinch.

On consoles on the other hand, you have a fixed amount of RAM from the start. You know what that amount is, and you can plan for it. In this latest generation of consoles, you still need to give up part of that precious memory to some other processes for user interface and network updates, but even that amount is known ahead of time. You can plan for it, and make your game fit in the memory that you have left.

The iPhone memory situation is… a mess. It’s like a console in that you have a fixed amount of RAM: 128 MB (although I’m sure Apple doesn’t want developers thinking about it so they can roll out iPhones with more memory without affecting existing programs). Because it has a fixed amount of RAM, the iPhone doesn’t provide virtual memory swapping [2]. So far so good. But here’s the kicker: Unlike a game console, the iPhone makes no guarantees about how much memory your application has available when it runs.

Let me say that again: You’re dealing with a fixed-memory environment and you have no way of knowing how much memory will be available to you. Did that sink in yet?

So how are you supposed to deal with that? Apple wants us to be able to dynamically load and unload anything in our programs in response to low-memory warning events. That might work well if you’re loading web pages and just need to unload some cached ones, but that’s far from ideal for games. What are you going to do? Unload part of the level? Get rid of some sounds? Not every game can do that, and even if you could, it would add a huge amount of complexity to something that should be pretty simple.

You can’t even plan on using a fixed minimum amount of memory. From my experiments on my iPhone 3G, it seems that it’s relatively safe to use between 15 to 18MB. Any more than that, and you start getting low memory warnings, and unless you program does something about it, the OS will promptly terminate you. On an iPod Touch, since it’s not running as many background services, you usually have more memory available. I learned that the hard way, because I developed most of Flower Garden on an iPod Touch, only to find out that it was too slow and running out of memory on the iPhone 3G.

And this is out of a total of 128MB. Where did the rest of the memory go? Come on, even with phone, GPS, and music processes in the background, where did the rest of the memory go? Probably the worse culprits are the mail program and Safari, which are left in memory even after you exit back to the springboard (Bad design decision, no cookie!).

And what exactly does it mean “to do something about it” when you get a low-memory warning? You’re supposed to free up as much memory as you can, but is it enough? Who knows! Maybe you free up one megabyte but the OS still decides that you’re taking too much memory and it will kill you anyway. It seems like the worst memory handling scheme ever designed.

As an extreme situation, I have seen as little as 3.5MB available when starting my own program! What are you supposed to do in that case? Most games will just crash back to the springboard. Want to test this? Launch Safari, load 10-15 heavy web pages, quit, and then launch your favorite game. A crash is almost guaranteed. For my next project, I’ll be tempted to respond to a low-memory warning by killing the Safari and Mail processes. That will free up some memory!

To make matters worse, I had the bright idea of combining OpenGL and UIKit for Flower Garden. Some things worked really well, like being able to quickly create some screens with Interface Builder and take advantage of all of Apple’s UI widgets and fonts. Creating the seeds screen or the settings panel was a breeze with UIKit. However, UIKit uses an undetermined amount of memory behind the scenes, which makes all attempts at staying below a certain memory threshold pure black magic. In particular, drawing images to a context apparently causes mysterious caching of bitmaps taking huge performance hits and memory chunks over which we have no control. And game programmers don’t like to feel they have no control over what’s happening!

Another consequence of such a horrible memory situation is that game developers aren’t able to make full use of the resources available. If we’re aiming for 15MB, but have to drop back to 5-8MB, most developers are not going to try to write a game that uses 30 or 40MB (which might be available). What a shame!

I’ve said before that the iPhone very closely resembles the great Sega Dreamcast (which is my favorite game console of all time). The Dreamcast only had 16MB of main RAM and 8MB of video RAM, but the best games on the iPhone, still pale in comparison to the best Dreamcast games. Why? Mostly because of the memory situation. If we had a guaranteed 16MB of RAM and the possibility of using all the extra free memory, iPhone games would look very, very differently.

My plea to Apple: Please, give us some minimum memory guarantees for OS3.0! That, more than anything else, will make the biggest difference in the look and reliability of future games.

[1] There are slight differences between iPhone 2G, 3G, iPod Touch 1st and 2nd gen (especially the latter, which has a faster CPU and faster file system).

[2] It has virtual memory in the sense that there’s a paging system with virtual addressing (so two memory pages far apart in real memory could be made to seem to be continuous), but there is no automatic paging in and out of memory pages to/from disk.

When: April Wednesday 22nd at 7:30PM until whenever Where: O’Sullivan’s Pub in Carlsbad (home of the Appy Entertainment Secret World Headquarters)

Appy is being a great host again and providing some munchies. You’re responsible for your own drinks though :-)

Hope to see you there!

The interview topics range from high-level topics like business aspects of iPhone development, down to the nitty-gritty of how the procedural flowers were created, or how I had to use assembly language to take advantage of the vector floating point processor on the iPhone to do all the heavy lifting for matrix transforms.

Flower Garden was finally approved and it’s available right now! This is what I’ve been pouring my heart and soul for the last six months. You can buy it directly from the App Store.

If you like it and would like to help out, here are some of the things you can do to promote it:

• Go back to iTunes and write a fair review
• Grow flowers and send bouquets to your friends
• Join the Flower Garden group on Facebook
• Tell all your friends how great it is :-)

Thanks everybody for all the encouragement and support along the way. I hope you enjoy playing it as much as I enjoyed creating it.

And don’t worry, this is just the beginning. I have lots of updates planned, so keep an eye out for lots of fun new features!

With all the excitement last week with GDC and the last push for Flower Garden, I completely forgot to announce that I submitted Flower Garden to Apple last Friday. It’s currently listed “In Review” and hopefully, if all goes well, will be available on the App Store either this weekend or early next week. Keeping my fingers crossed!

The presentation went very well. The room was *completely* packed, with every seat taken and people standing along the walls. In retrospect I shouldn’t be surprised because it was the only iPhone presentation in the main GDC conference. Clearly there is a huge amount of interest in the platform. I’m already pushing to have a lot more iPhone content for next year, so you can all look forward to that.

Because it was the only iPhone presentation in the main GDC, I had to keep it very high-level and focused on the question of “what can you expect if you switch to develop games on the iPhone”? Not like I can really answer that, but at least I can share what my experience was. Hopefully next year I can really dive into some juicy tech topics.

In the meanwhile, if you’re dying for some more technical content, go check out my latest column in Game Developer Magazine dealing with multi-touch input devices with the lessons I learned from Flower Garden.

Update: Ari Braginsky recorded the audio of the session (part 1 and part 2), so if you want, you can follow along with the slides. Thanks Ari! I believe that the synced audio and slides will be available through the GDC web site for registered attendees too.

The point is that as March 20th approached, I was slaying bugs left and right. Flower Garden went through two rounds of beta testing and it had some great feedback. All the crash bugs and top priority tasks were done and taken care of a few days before the deadline. Things were looking good for a submission to Apple on the expected date! Yay!

Then, with just a day to go, I decided to step back a second. Something had been nagging at me for a bit and I had to be honest with myself. Yes, all the crashing bugs and major features were taken care of. But how come my issue tracking system still had 50 entries in it marked as “Final release”? It wasn’t a big deal. Those were all marked as “low priority” or “trivial”. And that’s the great thing about the iPhone, I can submit the application and then start pushing out updates as I get to the little stuff. Right? Right?

This wasn’t a case of taking Old Yeller behind the shed. The app was looking good. Really good. And it wasn’t just a puppy, but a fully grown, beautiful app. It’s just that it was a bit… slow and rough around the edges.

This is were things got really tough for me. On one hand, I was ready to ship this baby. I have been working on it for almost six months (where did time go??) and a lot of people were loving it. But when I allowed myself to give it one hard look, I realized that the experience I was hoping for wasn’t there. Yes, the functionality was there, but not the experience. With a tool that gets a job done, maybe that’s not a huge deal. But Flower Garden is all about the experience.

I tried to rationalize it by thinking how I could get all those little, low priority things done in an update right away: Improving the performance of the rendering, making the transitions more responsive, adding those subtle animations and sounds, making load times shorter… In short, it was missing the key ingredient to a successful experience: polish. I could always add that polish later, but what about all the people who bought it right away? What kind of impression were they going to get? In a saturated market like the iPhone App Store, and especially with a title aimed at a casual audience, first impressions are everything. If people don’t fall in love with your product right away, you might not get a second chance.

So it was with a very heavy heart that I decided to slip my target date. I really needed one more week to get to the point that Flower Garden could be the experience I had envisioned. And of course, to make it twice as hard, this coming week it’s GDC, so that means that the release date is pushed out two weeks.

Coming from the console world, where updating your game is a rare opportunity and you better get everything right for the big release,I was really counting on releasing frequent updates. Does it mean that updates are useless in the App Store? That we shouldn’t take advantage of them? Not at all. I realized that I was looking at them the wrong way. Updates are supposed to provide bug fixes and new functionality. They are not a crutch for sloppy development and an excuse to release products before they’re ready.

Now I have regrouped. Reassessed where I’m at, and prepared the final plan of attack. If all goes well, I will be submitting Flower Garden to the App Store the last week of Mark/beginning of April. When exactly? When it’s ready. Lesson learned.

Yes, GDC is around the corner.

Yes, I’m supposed to ship Flower Garden this Friday.

But none of that is stopping us. Come mingle with other iPhone developers this Wednesday at the Irish Pub in Carlsbad. As a bonus, I’ll do a practice run of my GDC talk about iPhone development (now that’s an old picture!).

When: Wednesday March 18th from 7:30PM until whenever. Where: O’Sullivan’s Pub in Carlsbad (home of the Appy Entertainment Secret World Headquarters)

If you’re thinking of coming, please RSVP by leaving a comment below or emailing me. That way we get a rough idea of how many people to expect.

See you all in a couple of days!

Last night was particularly frustrating because I had just called it “beta” on Flower Garden and celebrated with the requisite victory dance, when the fun of sending out the beta build started. What I thought would be a ten-minute task, turned into a long nightmare and I didn’t get the build out until 1AM.

This is not the first time I run into issues with provisioning profiles and distribution in general. It’s a very finicky process, and other developers are going through the same pains. Mark Johnson suggested I collected all the gotchas I’ve learned into a single place so it can serve as a form of reference. That’s a great idea, because the process is so byzantine that I will have forgotten most of them next time I need to do another big submission.

This is not a “how to” document on preparing your builds for ad-hoc or App Store distribution. Apple has the basic flow documented in their site, and other people have also talked about the process. This is intended to be more of an appendix explaining details, clarifying assumptions, and correcting things that are just plain wrong.

To make this more comprehensive, I’d like to open it up to other people’s experiences, not just mine. If you have gone through some pains with the iPhone Developer Program (and who hasn’t?) and found a way to deal with them, please post in the comments and I’ll update this page to reflect it.

iPhone Developer Program

1. App identifier is case sensitive. This is the string you need to enter when you create an app ID through the developer web site, and the one you need to set in your Info.plist. My usual development id is com.snappytouch.* so I can use it for anything I do. But in preparation for sending out the beta build, I created one just for Flower Garden: com.snappytouch.flowergarden. I thought that everything on the build side of things would be fine because my default app ID was: com.snappytouch.${PRODUCT_NAME:identifier}. Unfortunately, this failed to create a working build without any helpful errors. After much hair-pulling, I figured that was because my app is called “FlowerGarden” not “flowergarden”. Changing the app ID in XCode to com.snappytouch.flowergarden fixed that.

2. Easy UDIDs. Forget about asking your testers to go to iTunes and get the UDID from there. Instead, tell them to get BetaHelper and email you all the information directly. Easy and no typing errors!

3. Bulk device upload. I got a great response to the call for beta testers for Flower Garden, so I was left with about 30 new device UDIDs to enter in the developer web site. Entering them one at a time with a slow-as-molasses interface isn’t my idea of fun so I decided to use the bulk upload option. Unfortunately the format isn’t described exactly anywhere, and the errors displayed if the format doesn’t match exactly are meaningless.

The exact format is:

Device ID	Device Name
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	Whatever name here
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	Lalalala
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	Whatever

So, tab separated, not comma separated. First column is the device ID (40 digits). Second is the name, no quotes around it. First line is ignored. Here’s the killer: You cannot have any blank lines after the last entry. And the error message if you do is simply a “Bad format”. Come on Apple, throw us a bone over here! One last thing: You can’t upload a device ID that is already in your database. Really annoying. Still, beats cut and pasting all those IDs by hand.

4. Ad-hoc zip file for Windows users. Apparently it can’t be created with the Compress option in finder because it puts extra files. I used zip from the command line and it worked correctly. (Thanks to Mark Johnson for that one).

5. App Store zip file has to be created through Finder with Compress. I heard that on Twitter somewhere, but needs confirmation. Anyone?

6. Adding provisioning profiles. Forget about dragging them on XCode and manage them manually. Check out my earlier entry about it.

7. Set correct provisioning profile everywhere. And I mean everywhere. You need to do it both in the Info for the target and the info for the project itself. Yes, both dialog boxes look identical and have 99% the same information, but the provisioning profile setting is different apparently. Fail to set it in both places, and your app won’t load in the device.

I’m sure there are a lot more. If you’ve found some of your own, leave a comment and I’ll update the list.

Session description: This session will cover the experiences of a professional game developer, used to 200+ person teams, multi-million dollar budgets, and 3+ year schedules, who left all that behind to become a one-person indie company developing exclusively for the iPhone. It will explain how things are different and how some things are very much the same, and will show specific examples of graphics technology, development environment, and asset pipeline. I will be using my current iPhone project, Flower Garden, as an example. The audience will learn what the transition is like and what to expect going indie making games for the iPhone.

BecomingIndie.key.zip

Or maybe I just had gadget envy. Whatever.

But I’m not writing to rub in what a cool gadget I have now. I had a couple interesting experiences adding new devices to XCode and I figured I would share them with other iPhone devs.

SDK and firmware compatibility

This is something I noticed a few weeks ago, but I was able to confirm it with my new iPhone. Each new version of the iPhone SDK warns you to upgrade your device firmware to that version before you attempt to run any programs on it with the new SDK. I always took that literally, but I realized it’s not as restrictive as it sounds.

With each version of the SDK, you have the option to target your build to that version or any of the previous ones. I discovered quite by accident that is perfectly fine to use a certain SDK (like 2.2.1) to build for an earlier version (2.0) and run it on a device with older firmware matching that version. It really makes a lot of sense, because otherwise the only way you could test your app on older firmware would be by reinstalling an old SDK. So maybe this is old news to most people, but it was certainly nice to first upgrade the SDK, and only later the firmware to minimize potential for problems.

Updating provisioning profile

So with my new shiny iPhone in hand (actually, it was a $99 refurbished model, but it shines like new and has that new car smell… oh wait), I set to run Flower Garden on it. The process involved the following steps:

• Add a new device with the UDID of my iPhone to the Apple Developer Program account. I made the mistake of having to type it manually from iTunes–didn’t realize that XCode allows you to copy and paste it, doh!. Check.
• Add the new device to the development provisioning profile I was using. Check.
• Download the updated provisioning profile and add it to XCode and my iPhone. Since the iPhone was plugged, all I had to do was drag it onto XCode. Check.
• Clean all, build, run. And….

Yeah, you guessed it, I got an error. Otherwise I wouldn’t be writing this if it had behaved as expected, would I? The error kindly informed me that the provisioning profile I was using to build the app wasn’t available on my device. But I could see it in the organizer window of XCode. I double checked it was the correct one and that Flower Garden was being built with that one. Check. Check.

What was going on?

Here’s the fun part, which took me a few minutes of digging through the docs. When I added the updated provisioning profile, XCode decided that it was a different one from the one I was trying to replace. But, here’s the kicker, they both had the same name and description. So XCode was still building my app with the old one, but it was the new one that was installed on the iPhone.

I discovered this by going to the source. XCode stores all the provisioning profiles in the folder ~/Library/MobileDevice/Provisioning Profiles. You can’t tell much of anything by looking at the filenames since they’re a bunch of UIDS, but you can crack the files open with a good text editor and you’ll see they’re a mix of text and binary data. From there, you can see which file corresponds to which profile by looking at the text between the tags. All I had to do was delete the two development provisioning profiles, re-add the new one to XCode and the iPhone and I was back in business.

I had heard so many horror stories of upgrades to 2.2.1 that I was afraid I was hitting one of those. Fortunately it was quite simple. Next time I’ll know where to look right away.

Now it’s back to work for me. I still need to finish the slides for 360iDev and then I need to fix a couple of things in Flower Garden and get it ready for the first round of beta testing. Sleep? What’s that?

I’ll be sharing my experiences transitioning from traditional AAA console game development, with teams of 100+ people, multi-million budgets, and several years of development, to indie iPhone development. There are a surprising amount of things that carry over from “big game development”, and quite a few that are totally different. I’ll go into what’s involved making games for the iPhone, and what game developers can expect when making the transition.

Apart from my talk, I’m particularly excited about this year’s GDC. It seems that the amount of content on indie game development and iPhone game development has shot through the roof. On Monday and Tuesday we’re treated to not just one, but two great summits: The Independent Games Summit and GDC Mobile! Last year, the Independent Games Summit was the best part of the show. Meeting all the other indie game developers out there and hearing their experiences was great. This year I’m hoping for more of the same plus all the iPhone-specific content.

Of course, the main conference is packed with great content too, but I haven’t had time to go through all of it and pick the sessions I want to attend yet. Too busy wrapping up my current project.

So if you see me around the show, stop by and say hi. I’m always glad to meet other fellow developers, and it’s always nice to put a face with a name for those of you that I know know through Twitter or online blogs.

In the next few days I’m going to start beta testing, so if you’re interested in participating, let me know.

More info coming soon…

It is true that Objective-C can be a bit more verbose than plain C++, but it more than makes up for it with named parameters (which increase readability a huge amount), and not having to duplicate many things between header and implementation files.

Yet, even though I like Objective-C so much, there was one last, really annoying feature that was preventing perfect interop between C++ and Objective-C. Something so simple, yet so fundamental, that it really got in the way. Something that wouldn’t allow me to write something as simple as this:

@interface MyView : UIView
{
    // ...
    SingleTouchEvent m_singleTouchEvent;
}

SingleTouchEvent is a custom C++ class. There’s nothing wrong with that right? There shouldn’t. But if you try and compile it, you’ll get this warning:

warning: type `SingleTouchEvent' has a user-defined constructor
warning: C++ constructors and destructors will not be invoked for Objective-C fields

Some of you might be saying, no big deal, you can dynamically allocate objects and have their constructors called as usual. That’s true, but I hate allocating objects on the heap just because of something like this. My preference is always to make them member variables if I can. It’s faster, more efficient, less error prone, and easier to read. Needless to say, this was bugging the hell out of me.

I looked all over the build options in case there was some setting I could toggle to enable C++ constructors to be called, but no luck. I did a bit more research, and lo and behold, it turns out there is a custom setting you can pass directly to gcc to do exactly that! It’s nowhere in the XCode settings GUI, so you have to enter it by hand. Go to Project | Info, and in the Custom Settings section, enter the following: GCC_OBJC_CALL_CXX_CDTORS = YES

With that setting enabled, Objective-C and C++ now play together better than ever!

Without any barriers to writing new code in Objective-C, I expect it will slowly (or maybe not so slowly) replace C++ as my main language of choice for future projects.

So we’re back at it again. Anybody involved in iPhone development or interested in learning more about it is welcome to come.

When: Wednesday Feb 11th from 8PM until whenever. Where O’Sullivan’s Pub in Carlsbad (home of the Appy Entertainment Secret World Headquarters)

If you’re interested, you might also want to join the San Diego iPhone Developers group to keep up to date with any future announcements.

See you all there!

From UIKit to OpenGL

This is definitely the most common way of sharing image data. Loading an image from disk in almost any format is extremely easy with Cocoa Touch. In fact, it couldn’t get any easier:

UIImage* image = [UIImage imageNamed:filename];

After being used to image libraries like DevIL, it’s quite a relief to be able to load a file with a single line of code and no chance of screwing anything up. So it makes a lot of sense to use this as a starting point for loading OpenGL textures. So far we have a UIImage. How do we get from there to a texture?

All we have to do is create a CGContext with the appropriate parameters, and draw the image onto it:

byte* textureData = (byte *)malloc(m_width * m_height * 4);
CGContext* textureContext = CGBitmapContextCreate(textureData, m_width, m_height, 8, m_width * 4, 
            CGImageGetColorSpace(image), kCGImageAlphaPremultipliedLast);
CGContextDrawImage(textureContext, CGRectMake(0.0, 0.0, (CGFloat)m_width, (CGFloat)m_height), image);
CGContextRelease(textureContext);

At that point, we’ll have the image information, fully uncompressed in RGBA format in textureData. Creating an OpenGL texture from that is done like we always do:

glGenTextures(1, &m_handle);
glBindTexture(GL_TEXTURE_2D, m_handle);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_width, m_height, 0, GL_RGBA, GL_UNSIGNED_BYTE, textureData);

And we’re done.

A word of advice: This is a quick way to load textures and get started, but it’s not the ideal way I would recommend for a shipping product. This code is doing a lot of work behind the scenes: loading the file, decompressing the image into an RGBA array, allocating memory, copying it over to OpenGL, and, if you set the option, generating mipmaps. All of this at load time. Ouch! If you have more than a handful of textures, that’s going to be a pretty noticeable delay while loading.

Instead, it would be much more efficient to perform all the work offline, and prepare the image into the final format that you want to use with OpenGL. Then you can load that data directly into memory and call glTextImage2D on it directly. Not as good as having direct access to video memory and loading it there directly, but that’s as good as it’s going to get on the iPhone. Fortunately Apple provides a command-line tool called texturetool that does exactly that, including generating mipmaps.

Another use of transferring image data from UIKit to OpenGL beyond loading textures is to use the beautiful and full-featured font rendering in UIKit in OpenGL applications. To do that, we render a string into the CGContext:

CGColorSpaceRef    colorSpace = CGColorSpaceCreateDeviceGray();
int sizeInBytes = height*width;
void* data = malloc(sizeInBytes);
memset(data, 0, sizeInBytes);
CGContextRef context = CGBitmapContextCreate(data, width, height, 8, width, colorSpace, kCGImageAlphaNone);
CGColorSpaceRelease(colorSpace);
CGContextSetGrayFillColor(context, grayColor, 1.0);
CGContextTranslateCTM(context, 0.0, height);
CGContextScaleCTM(context, 1.0, -1.0);
UIGraphicsPushContext(context);
    [txt drawInRect:CGRectMake(destRect.left, destRect.bottom, destRect.Width(), destRect.Height()) withFont:font 
                                lineBreakMode:UILineBreakModeWordWrap alignment:UITextAlignmentLeft];
UIGraphicsPopContext();

There are a couple things that are a bit different about this. First of all, notice that we’re using a gray color space. That’s because we’re rendering the text into a grayscale, alpha-only texture. Otherwise, a full RGBA texture would be a waste. Then there’s all the transform stuff thrown in the middle. That’s because the coordinate system for OpenGL is flipped with respect to UIKit, so we need to inverse the y.

Finally, we can create a new texture from that data, or update an existing texture:

glBindTexture(GL_TEXTURE_2D, m_handle);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, m_width, m_height, GL_ALPHA, GL_UNSIGNED_BYTE, data);

From OpenGL to UIKit

This is definitely the more uncommon way of transferring image data. You would use this when saving something rendered with OpenGL back to disk (like game screenshots), or even when using OpenGL-rendered images on UIKit user interface elements, like I’m doing in my project.

The process is very similar to the one we just went over, but backwards, with a CGContext as the middleman.

We first start by rendering whatever image we want. You can do this from the back buffer, or from a different render target. I don’t know that it makes a difference in performance either way (these are not fast operations, so don’t try to do them every frame!). Then you capture the pixels from the image you just rendered into a plain array:

unsigned char buffer[width*(height+30)*4];
glReadPixels(0,0,width,height,GL_RGBA,GL_UNSIGNED_BYTE, &buffer);

By the way, notice the total awesomeness of the array declaration with non-constant variables. Thank you C99 (and gcc). Of course, that might not be the best thing to do with large images, since you might blow the stack size, but that’s another issue.

Anyway, once you have those pixels, you need to go through the slightly convoluted CGContext again to put it in a format that can be consumed directly by UIImage like this:

CGImageRef iref = CGImageCreate(width,height,8,32,width*4,CGColorSpaceCreateDeviceRGB(),
                            kCGBitmapByteOrderDefault,ref,NULL, true, kCGRenderingIntentDefault);
uint32_t* pixels = (uint32_t *)malloc(imageSize);
CGContextRef context = CGBitmapContextCreate(pixels, width, height, 8, width*4, CGImageGetColorSpace(iref), 
                            kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrder32Big);
CGContextTranslateCTM(context, 0.0, height);
CGContextScaleCTM(context, 1.0, -1.0);
CGContextDrawImage(context, CGRectMake(0.0, 0.0, width, height), iref);    
CGImageRef outputRef = CGBitmapContextCreateImage(context);
UIImage* image = [[UIImage alloc] initWithCGImage:outputRef];
free(pixels);

Again, we do the same flip of the y axis to avoid having inverted images.

Here the trickiest part was getting the color spaces correct. Apparently, even though it looks very flexible, the actual combinations supported in CGBitmapContextCreate are pretty limited. I kept getting errors because I kept passing an alpha channel combination that it didn’t like.

At this point, you’ll have the OpenGL image loaded an a UIImage, and you can do anything you want with it: Slap it on a button, save it to disk, or anything that strikes your fancy.

The now gone CrashLander sample, in spite of some atrocious code, was the best example on how to get a simple, OpenGL app on the iPhone. It covered the basics: creating an OpenGL frame buffer, loading some textures, drawing some polys, and presenting the result to the screen. It was very useful because it was a very small and concise and it made for a great starting point for any OpenGL-based app.

Unfortunately, because it was so basic, it didn’t show how OpenGL can play with the rest of UIKit user interface. Instead, it took the approach from many games of taking control of the full screen and viewing it a a single resource. That’s fine as long as you’re making those kind of games, but I found myself having to mix OpenGL and UIKit quite a bit in my current project. And eventually, I even had to use multiple OpenGL views in different parts of the app.

Single OpenGL View

To render with OpenGL to a view, you have to first set the class-static method +layerClass so it creates the right type of layer. Specifically, we need a CAEAGLLayer:

+ (Class)layerClass
{
    return [CAEAGLLayer class];
}

You can create a view of that type in code, or you can lay it out on the Interface Builder. I actually like the Interface Builder quite a bit, so that’s how I end up creating all of mine.

Then, you need to create an OpenGL context:

    m_eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES1];
    [EAGLContext setCurrentContext:m_eaglContext];

Finally, you need to create a valid frame buffer object from the CAEAGLLayer from the view you just created:

    glGenFramebuffersOES(1, &buffer.m_frameBufferHandle);
    glGenRenderbuffersOES(1, &buffer.m_colorBufferHandle);
    glGenRenderbuffersOES(1, &buffer.m_depthBufferHandle);

    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);
    [m_eaglContext renderbufferStorage:GL_RENDERBUFFER_OES fromDrawable:drawable];
    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_WIDTH_OES, &buffer.m_width);
    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_HEIGHT_OES, &buffer.m_height);

    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_depthBufferHandle);
    glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH_COMPONENT16_OES, buffer.m_width, buffer.m_height);
    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);

    glBindFramebufferOES(GL_FRAMEBUFFER_OES, buffer.m_frameBufferHandle);
    glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle); 
    glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES, GL_RENDERBUFFER_OES, buffer.m_depthBufferHandle)

It’s all pretty standard frame buffer OpenGL stuff. The renderBufferStorage:fromDrawable function is the one that actually allocates buffer storage for that particular view.

After that, you’re home free and you can use OpenGL the way you’ve always done. The only difference is that to present, you call this functions instead:

    [m_eaglContext presentRenderbuffer:GL_RENDERBUFFER_OES];

Done.

Integrating OpenGL and UIKit

After you have the basic OpenGL app set up, you can treat it as a full-screen, exclusive resource and do all your output through it. That works fine for a lot of games, but there are a lot of games and apps that benefit from using some amount of UIKit functionality. After all, UIKit is a great UI API, and it seems like a shame to reinvent it all if it’s not necessary.

The good news is that OpenGL rendering is happen in a view, so you can do everything you can do with views: Animate it, do fancy transitions, add them to nav bars or tab bars, etc. The bad news is that, unless you’re careful, performance will suffer.

Apple has some best practices on what to do with OpenGL views, but it comes down to avoid doing transforms on them (instead, use the transforms in OpenGL), and avoid putting other UIKit elements on top of it, especially transparent ones. Although, I seem to be able to get away with small buttons and such without affecting the frame rate, so that’s always a plus.

So having realized that, now you can make your application transition between the OpenGL view, and other views. Maybe you have a table view with settings, or some sort of UIKit dialog box, or a high-score table. You can safely do all of that with UIKit (and add your own graphics and custom rendering if you want to give it a totally different look).

One thing to watch out for: Whenever your OpenGL view isn’t visible, make sure to stop doing frame updates (for simulation and rendering). Otherwise, the rest of the UI is going to be very unresponsive (and you might not find that out until you run it on the device, because the simulator is very fast). The UIViewController viewWillAppear and viewDidDisappear functions are perfect to find out when you should start and stop frame updates.

Multiple Views

Things get more complicated when you want to have multiple views with OpenGL rendering. I don’t mean multiple viewports in the same screen, but multiple views in the sense of multiple UIViews. My current project, for example, has different views connected with a UITabBarController. Some of the are custom UIViews and some of them are OpenGL views.

My first thought was to try to have different OpenGL contexts for each view, but that would complicate things because I wanted to share the same textures and other resources. I know there’s a shared groups option, but that was definitely not the way to go.

The best solution I found was to use multiple frame buffers, binding each of them to the correct OpenGL view. That way, when I switch views, I switch frame buffers and everything works correctly.

On the -viewDidLoad function for each view controller with an OpenGL view, I call the function listed above to create a new frame buffer bound to that view, and it returns an index. Then, in the -viewWillAppear function, I set the frame buffer corresponding to that index:

    const FrameBuffer& buffer = m_frameBuffers[bufferIndex];
    glBindFramebufferOES(GL_FRAMEBUFFER_OES, buffer.m_frameBufferHandle);   
    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);
    glViewport(....);

That’s it. Now you can switch between different OpenGL views and UIKit views without any problems. Just make sure to only render the visible views! That can be trickier than it sounds because the viewWillAppear and viewDidDisappear functions only get called for views that were layed out in the Interface Builder. If you added them by hand, you need to call those methods yourself (why??!?!). So keep tabs on that, otherwise everything will slow down to a halt.

And To Wrap It Up…

You thought I had forgotten about the promise to slowly unveil art from my current project, uh? Fear not, here comes the next teaser. This is an actually screenshot taken a minute ago. Incidentally, this is not one of the views using OpenGL. We’ll have to save those for another day :-)

- (UIView *)pickerView:(UIPickerView *)pickerView
        viewForRow:(NSInteger)row forComponent:(NSInteger)component
        reusingView:(UIView *)view;

Once I did that, the bug mysteriously went away. Ah, the joys of dealing with code you have no source to.

So being the tinkerer that I am, I couldn’t just stop there, and I spent an hour fixing something else that was annoying me to no end: When you start Tea Time! there’s always a “click” sound that the picker plays because I’m selecting the last used tea configuration during initialization.

I looked high and low how to fix that, but there seemed to be no way to do it. Nothing in the published SDK information that I could see. But digging through the header file for UIPickerView, I saw this:

@protocol UIPickerViewDataSource, UIPickerViewDelegate;

UIKIT_EXTERN_CLASS @interface UIPickerView : UIView <NSCoding>
{
 //... snip....
  @package
    struct {
        unsigned int needsLayout:1;
        unsigned int delegateRespondsToNumberOfComponentsInPickerView:1;
        unsigned int delegateRespondsToNumberOfRowsInComponent:1;
        unsigned int delegateRespondsToDidSelectRow:1;
        unsigned int delegateRespondsToViewForRow:1;
        unsigned int delegateRespondsToTitleForRow:1;
        unsigned int delegateRespondsToWidthForComponent:1;
        unsigned int delegateRespondsToRowHeightForComponent:1;
        unsigned int showsSelectionBar:1;
        unsigned int allowsMultipleSelection:1;
        unsigned int allowSelectingCells:1;
        unsigned int soundsDisabled:1;
    } _pickerViewFlags;
}

See that last flag? soundsDisabled. Sounds promising, doesn’t it? Unfortunately _pickerViewFlags has @package protection, which means I can’t get to it. Or so the theory goes.

Objective C is surprisingly “good” about letting you get under the hood and bypass protection levels. So I tried accessing those flags with NSObject function setValueForKey, but the stubborn picker would refuse to let me have access and throw an exception instead. Now he was starting to really annoy me. After all, I’m supposed to be working on my other project, not hacking this little app.

I couldn’t stop though. It’s not in my nature.

Oh yeah, Objective C is refusing to give me access to those flags? Fine. I’ll resort to the hackiest or hacks. But I will turn that damn bit off! An object is just a block of memory really, and each member variable is located at a fixed offset from the start of the object. See where this is going? Are you horrified enough? Yup. In the debugger I saw that _pickerViewFlags was exactly 16 bytes from the start UIViewPicker. So I grabbed the pointer to the picker, moved forward to the _pickerViewFlags variable, and stomped the soundsDisabledBit. Mwahahahaha! That will teach you, annoying picker!

Compile, run, and…. “click”. WTF!?!?!? I went in the debugger and checked that I was changing the right bits. Everything was fine. Except that, somehow, the picker was ignoring that. Defeated, I gave up in disgust.

It was that evening when I saw a tweet from @jeff_lamarche about accessing an undocumented function to turn off the sounds in the clicker. Nice timing! He pointed me to his blog post about how to get a listing of all the undocumented SDK functions and how to access them from your programs. I couldn’t believe that there was a function to do exactly what I wanted, but Apple wasn’t exposing it. And for something so simple too!!

It was as simple as

@interface CustomPicker(hidden)
- (void)setSoundsEnabled:(BOOL)isEnabled;
@end

followed by

    [m_picker setSoundsEnabled:NO];
    [m_picker selectRow:settings.m_teaType inComponent:0 animated:NO];
    [m_picker selectRow:settings.m_teaStrength inComponent:1 animated:NO];
    [m_picker selectRow:settings.m_teaFormat inComponent:2 animated:NO];
    [m_picker setSoundsEnabled:YES];

Finally! I had defeated UIPickerView!

Now the catch is, you’re not supposed to use undocumented SDK functions. Apparently that’s grounds for getting your app rejected by Apple and having to resubmit. But I figured I had nothing to lose, and it might be an interesting learning lesson. After all, how exactly does Apple check for undocumented functionality? Do they scan the submitted executable against a set of forbidden entry points? I couldn’t imagine that’s a manual process. But on the other hand, there are reports of other apps getting onto the App Store using undocumented SDK features.

I figured, what the heck, it was a tiny little thing that wasn’t going to hurt anyone. Might as well try it.

I went ahead submitted the update last Friday, and today (Tuesday) I got the official email with the approval of my new version. Not a bad turnaround time since I imagine they don’t work weekends. It even just made it to the App Store as I was writing this. Shweet!

I’m still left wondering how Apple tries to monitor undocumented SDK usage. Did they not bother with my app since it was so small? Is it really a manual process? (I feel bad for the person in charge of that just thinking about it). Did they catch it and realized it was totally harmless and let it through? Only Apple knows I’m afraid. They really should make their process both more transparent and more consistent and fair for everybody.

But hey, I’m not complaining. No more annoying click at startup :-)

A couple of weeks ago, prompted by seeing a few other indie iPhone devs on Twitter, I decided to give it another try. I’m glad I did! It has been really fun and useful. Unlike Facebook, which is all about connecting with long-lost friends, and seeing pictures people’s new babies, I’m using Twitter exclusively for iPhone development. I keep my follow list to a relatively small amount of people, all of them independent iPhone game developers (so please, don’t feel offended if you’re a friend and I’m not following you).

My routine now consists on sitting to work with a warm cup of tea, firing up the Twitter client du jour, and let it sit in the background (without alerts, thank you). Every so often I check it out, update my status, read what other devs are up to, and get into short conversations. The effect is surprisingly similar to being back working with a team of people and having quick IM conversations with them. Don’t get me wrong, I love working by and for myself, but I do miss the camaraderie, the interactions, and the energy of working with a team of really talented people. Twitter gives me a lot of what I’ve been missing.

Last night, @timhaines put together a Google Spreadsheet of iPhone devs on Twitter, along with a form to add yourself. He sent out a Twitter message asking people to add themselves and pass the word. The effect was stunning. Within minutes there were dozens of devs on the list, and as people continued re-tweeting the message, hundreds more were added! That’s a great resource to hook up with your favorite iPhone developers.

On a similar note, Sam Houston put together a list of video game companies with people on Twitter. That’s not quite as interesting for me right now, but seems like another great resource if you’re into video game development.

So give Twitter a try. Especially if you’re an indie developer working by yourself. Don’t think of it as another Facebook, but as a way to chat and develop connections with other people.

See you there!

So last week, I was excited to learn there was a new iPhone development conference  called 360|iDev (not the greatest conference name–sounds too much like an Xbox development conference). It looks like a very hands-on, for developers by developers kind of conference, instead of half-marketing, half-development like Apple’s offerings (Yes, we’re already on board, you don’t have to sell us on how cool the iPhone is. We know. We’ve banked our life and savings on it actually). The lineup of speakers looks really promising (including Urban Tycoon author, Mike Huntington).

What really excites me the most is meeting the other attendees though. It looks like it’s not going to be a huge conference (hundreds instead of thousands or tens of thousands like GDC), and we’re all staying at the same hotel, so there should be plenty of opportunities to meet people, chat, trade tips, and compare horror stories over beers.

I actually liked the idea so much, that I decided to jump in and submit a session proposal, and the organizers very kindly accepted it (which is great because otherwise I might not have been able to have afforded going :-)). I’ll be talking about my experience going from a AAA console development environment, to a single-person iPhone development team. Both the obvious differences, the not so obvious similarities, and how a single person can really deliver a top-notch iPhone game.

I encourage everybody to have a look at the 360|iDev site and register if you haven’t already. It’s not very expensive (especially if you buy the ticker sooner, rather than later), and it’s going to be a blast. It’s in San Jose in early March, a couple of weeks before GDC, so no conflicts for game developers there. Also, if you’re interested in speaking, check out their call for speakers while they’re still accepting new submissions. I hope to see some of you there!

On a totally different note, a couple of weeks ago, Evan at Veiled Games (Up There), which is one of my favorite indie iPhone developers, promised to put a daily image with a teaser of their upcoming project. This was soon picked up by Gavin at Antair Games, who is letting us peek under the covers of what’s coming up after Sneezies.

I figured it would be fun to follow their steps, but don’t hold your breath of a daily update. Maybe weekly if you’re lucky. Maybe.

So here’s this week’s teaser. Draw whatever conclusions you want from it :-)

I guess I really got out too easy and Apple had to remedy that. Today I found that Tea Time! shows up in the App Store listings, but it’s unavailable when you try to buy it. When I looked in the app management site, much to my surprise, Tea Time! was listed as “Pending Contract”.

What I thought was a relatively fast switch from individual to a company status, just propagated through their database and cancelled my previous contract. So I had to enter all the banking and tax information (exactly the same as before) and wait for the bureaucracy to approve it. So it looks like Tea Time! is going to be out of commission for a few days (or weeks or months?).

It really blows because it will kill whatever little momentum it had. Boy am I glad I went through this now and not with my other project!

The only difference is that now it’s listed with Snappy Touch being the seller instead of my name. Will this be the end of the App Store saga? Let’s hope so.

In the meanwhile, a sales update for those of you curious about it: In the first three days, Tea Time! sold 83 units. On one hand, that’s more than I expect (although I suspect that half of those are people following this blog and my announcement on Facebook). But on the other hand, it’s nothing if you’re trying to make a profit from it, even if it just took a couple of days to put together.

Granted, this is an extreme example. It has gotten exactly zero marketing of any kind, and it’s aimed at a very specific audience. I’m sure if a place like Republic of Tea or Adagio Tea advertised it in their catalogs, sales would go way up. But it gives you a base line of what you can expect in the worst case.

A couple of weeks ago, I decided to take a few days off from my project to enjoy the holidays with friends and family. Of course, I couldn’t stop thinking about the project. Soon my thoughts turned to how much more I had to go, and how some parts of the development process were still unknown to me, like the Apple submission process, or making a build for distribution on the App Store.

At that point, inspiration struck, and I decided to use one day during the holidays to write a quick, one-day app. That way it wouldn’t take any time away from my project and I would learn a lot from it. And maybe, just maybe, I would make a buck or two in the process.

The app itself was a very simple tea timer, inspired by tea maker machines: Enter the type of tea you want, the strength, and whether it’s loose leaves or in a bag, and it starts a timer with the correct steeping time for that particular tea. This was such a simple app, that it ended being exactly the way I had visualized the first time I thought about it.

Coding The App

Just coding the app itself was an interesting learning experience. Up until that point, I had spent most of my time on the OpenGL side of the iPhone. Tea Time! on the other hand, was 100% Cocoa Touch, so that was a nice change of pace.

Why did it take me a day? It’s such a trivial app that I should be able to throw it together in a few hours, right? Not quite. I had to deal with all the little quirks of Cocoa Touch, and get around the crippled functionality that Apple exposes in their API (while their own apps use a different API).

For example, it was a bit of a letdown to see I couldn’t add images to the picker control. But my disappointment was immediately replaced with excitement when I saw how easy it was to subclass it and display whatever you wanted in it. Unfortunately, the excitement gave way to annoyance trying to work around some of the bugs Apple still has on the picker control.

It’s really frustrating not having access to the same API Apple uses internally. I wanted to be able to turn off the iPhone while the timer was counting down, and have the device turn back on when the timer ends, just like the built-in alarm. Oh no, can’t do. Apparently that’s reserved for Apple. Fortunately, the longest you’re going to steep a cup of tea is 6-7 minutes, so I just disabled the power-saving feature of the iPhone so it doesn’t go to sleep while it’s counting down.

Even for such a simple application, getting a polished user interface in place takes some effort and time. The Interface Builder allows me to throw a user interface together in a snap just to get the basic functionality up and running. Later, I can come back and tweak it until I’m happy with it. Surprisingly, there are lots of things that the Interface Builder doesn’t allow you to do, so a lot of controls have extra code to tweak some of their parameters (increasing the target area of the info button on the top right corner, or setting the background images for he start/stop buttons, which require a special stretching pattern). Looking on the bright side, it’s very easy to combine a layout from the Interface Builder with custom code, which is a very important lesson for content-creation tools of any kind (and one that a lot of game tools get wrong!).

The rest of the coding time went into all the little things we don’t usually think about: Saving preferences, remembering the last used combination of parameters, getting the right splash screen, etc. All in all, the coding was about a day’s worth of work.

Creating The Content

There’s more to an app than code and a nice GUI layout, especially if you don’t want it to look like a boring spreadsheet: Graphics and sounds. In my first pass, while I was coding the basic functionality, I just grabbed whatever images I found through Google Images and threw them in. Then, once the app was pretty much finished, I set to replace them.

Original photo of my tea cup that became the icon for Tea Time!

That’s harder than it sounds because I’m not a graphic designer. I’m semi-proficient at the Gimp, but that’s about it. I do quite a bit of photography and I love tea though, so I decided to take pictures of the tea I was preparing at home. The main icon of for Tea Time! is just my tea mug with the tea I was having that morning.

The images had to be processed in the Gimp quite a bit. Tweaked colors and brightness, removed background objects, and even altered to fit the constraints of the GUI. For example, the little cups in the picker are the same picture that was then carefully colorized in the Gimp to match different tea types.

For the alarm sound, I was hoping to be able to play the system alarm sounds that come with the iPhone. But again, Apple lets us down by not giving us access to those (Why not???). So I hit a couple web sites with free sounds, selected a few that I liked, and tweaked them with Audacity to get the right length and sampling rate.

It was a really fun process getting all the content ready. I really enjoyed it and it was a nice break from coding, but it sucked almost another full day of work. So my “one day” app, was already almost up to two days. But at least I was done now. Right?

Submitting The App

I had done ad-hoc builds before, but I hadn’t prepared one for submission to the App Store. So I had to figure that out: Create a new distribution certificate, create a new configuration in XCode, test it… Not too bad.

Then I had to sign up for iTunes Connect and agree to a bunch of legalese non-sense. It was all done online and really quick, so that wasn’t a problem either. Finally I was ready to submit the app and be done with it. I had to write up a description, figure out the categories it would be listed under, take screenshots, and prepare a high-res cover art. I get all that ready, I fill out the submission form, and press submit. Done!

Oh wait, I get a message about not being able to get paid until I get a contract in place. What? Apparently when I signed up for iPhone development that only covered free applications, but I needed to set up a new contract in place with Apple if I was going to sell anything. So I had to enter bank ids, account numbers, tax information, and everything else. Once all that information was in place, my contract was in “Reviewing” state, and a couple days later it was approved without any hitches. Be warned that if you’re not in the US, apparently you have to mail a physical form to them before they approve the contract, so the process is likely to be much longer.

Along with the price (a very obvious $0.99), I had to set the “Availability date” for the app. I had read horror stories of developers who finally had their app approved, just to find it buried ten pages into the “New Releases” list in iTunes. The guys over at Veiled Games, explained how the release date is a combination of when the app is approved and the availability date you set. So I set the availability date to two weeks in the future and was ready to tweak it if necessary.

Are we done yet? Not quite. The submission page wanted a URL for a web page describing the app, and I didn’t have anything ready. I didn’t want to just point to a page in this blog, and I had been thinking that I should set up a company as a real legal entity for doing iPhone development and consulting. Fortunately, I had gone through this process before with Power of Two Games, so it was relatively easy and painless, especially since this was a single-member LLC. So the rest of the day was spent coming up with a company name (Snappy Touch), registering the domain (http://snappytouch.com), registering the LLC (in the state of Oregon no less because it’s a lot cheaper than California), installing Wordpress in the new domain, and setting up a basic page describing Tea Time!.

Finally done! Total time: Three days.

Waiting And Waiting

But Tea Time! wasn’t on the App Store yet. It had just entered the review process. Now I had to wait for Apple to approve it (or reject it so I can fix it). I heard from other developers that the usual wait time was about two weeks, but between this coinciding with the new year, and Apple getting swamped with submissions for the holidays, I figured I would be more like three or four weeks. Glad nothing was riding on this app getting out quickly.

A few days later, when I was checking out the info for Tea Timer! in the Apple submission site, while it was still in review, I was shocked to see it was listed as requiring SDK 2.2. That made no sense! I made a point to develop it for SDK 2.0 so every iPhone and iPod Touch user could buy it. I made really sure I selected SDK 2.0 when I created the iTunes distribution build. What gives? Apparently just building against SDK 2.0 wasn’t enough. I actually had to change the project settings to set SDK 2.0 as the default. That’s a sneaky one!

So what to do? I rejected my own binary and uploaded a new, updated one. This one was correctly listed as requiring SDK 2.0. I wasn’t too happy at having to do that because I’m sure it put me back to the end of the line, but this is exactly the kind of thing I was hoping to learn with Tea Time! in a no-pressure situation.

Waiting is no fun, so I made things a bit more interesting. Since I had gone to the trouble of creating a legal company, I decided to change my Apple Developer account to be a company instead of an individual. That way the company information would show up correctly in iTunes whenever Tea Time! was available. I figured this would be a matter of checking off a checkbox in my account settings, but apparently not. I had to email the Apple Developer Connection support with the request, wait a few days for an answer, and then provide some legal proof that the company existed (like the articles of incorporation). Once I emailed those, they initiated the conversion process, during which time I was locked out of my account for three days.  What does it take to change one bit in my account record??? Fortunately I had nothing else better to do than wait by this time, and it wasn’t impeding any work on my other project.

Finally!

The very next day after getting changed over as a company status, I got the much-awaited email from Apple announcing that Tea Time! was approved. That was just a week and a half after my initial submission. That was fast, especially considering I re-submitted the binary half way through. I reset my availability date to that day, and within a couple of hours, there it was, in the App Store. Mission accomplished!

Now, I get to learn about the sales reports, and, if I ever sell enough units, getting payments from Apple. They won’t make a payment until you accumulate at least $250 in profit from a single territory, so I’m not holding my breath for that one :-)

The “one day” app really took three days of work plus a week and a half of waiting. All in all, two weeks from idea to the moment it was available on the App Store. The process could be faster, but it wasn’t too bad. The experience was a lot of fun, and I certainly learned a lot from my mistakes and what things to watch out for next time, when it really counts.

What am I doing writing a little app like that? Don’t worry, that’s not what I’ve been doing for the last few months :-) It was an experiment to see how quick I could put a simple app together and getting the experience of going through the whole cycle with the App Store. Tomorrow I’ll write an entry with all that I’ve learned so far.

Enjoy your tea!

When: Wednesday January 14th from 7PM until whenever Where: Karl Strauss Brewery in Mira Mesa

Hope to see you all there!

I even ran it through some of the iPhone performance gathering tools and saw that it was spending half the time in the simulation and half the time rendering. A bit surprising, but it’s not like I had much to compare it against since this was my first performance-intensive iPhone app.

That was until this morning. Prompted by some of the comments in my previous post, I did a quick search about the iPhone floating-point performance. I had heard conflicting information on it: does it have hardware floating-point support? do you have to use fixed-point numbers for best performance? Apple is very tight-lipped about the specific hardware specs of the iPhone, which seems very strange to those of us coming from a console background. But people have been able to determine that the CPU is a 32-bit RISC ARM1176JZF. The good news is that it have a full floating-point unit, so we can continue writing math and physics code the way we do in most platforms.

The ARM CPU also has an architecture extension called the Thumb. It seems to be a special set of 16-bit bitcodes that claim to improve performance. I image the performance increase comes from a smaller code footprint and faster code fetching and processing. As a bonus, you also get a smaller memory footprint, so it seems like a win for a lot of mobile platforms. XCode comes with the option to generate Thumb code turned on by default.

But, and this is a huge but, it seems that floating point operations cause the program to switch back and form between Thumb mode and regular 32-bit mode. I would be interesting to look at the assembly generated, but I haven’t had time to do that yet. So the more floating-point calculations you do, the less of performance gain you’ll get from Thumb optimizations. Or, in the extreme, you might even get a performance degradation.

Most 3D games are very floating-point intenstive, and my app is no different, so I decided to turn off Thumb code generation on a whim. The results:

• Thumb code generation ON (default): 15 fps
• Thumb code generation OFF: 39 fps

Whoa!!! That’s a saving of 41 ms per frame! That has to be the optimization with most bang for the time spent on it that I’ve ever done. This also probably means that my app is now totally render-bound, which is good news. I’m sure I can optimize tons of stuff there

So if you’re doing any kind of a 3D game, turn off Thumb code generation. Now!

Edit: I realized I never explained how to turn the Thumb code generation off! Oops. Go to your project settings, add a new User-Defined setting called: GCC_THUMB_SUPPORT and set it to NO. That simple (but surprisingly there wasn’t an already existing setting to check it on and off).

Don’t worry, I’m not going to get into the details of the iPhone API. You can get all that and more from Apple’s iPhone development web site. I’m just going to talk about my experience starting with a new set of APIs and how they hold up in the process of a full project development.

Starting Out

Apple has done a great job holding the hand of developers new to the iPhone. As soon as you sign up for iPhone development, you’ll have access to several “Getting Started” documents and videos. Some of the videos are a bit on the fluffy, advertising side (“look at our new platform, it’s sooo cool!”), but they’re not a bad intro anyway. The docs on the other hand are pure gold. They’re a perfect introduction but with a fair amount of detail as well. They cover everything, from UI, to graphics and sound, to platform-specific features such as the accelerometer. I still find myself referring back to those every so often.

When the “Getting Started” docs aren’t enough, you have a full reference of all the classes and APIs at your fingertips, both online and through the XCode help system. I found the quality of the class reference to be pretty good. Usually, when something isn’t clear from the function name, the help is good at clarifying things.

I have to admit that I’m not a read-the-manual-first kind of guy. So while all those documents are great, and I use them a lot now, that’s not how I started out. Apple provides a bunch of sample code, covering all the major APIs, so I just downloaded the ones I was interested in, compiled them and fired them up. Then I used the docs to answer my questions about what the code was doing. The Crash Landing project was a perfect example of how to get started with games. I must have spent a whole day just pouring over that code the day I got my hands on the SDK.

Keep in mind that while Apple does a great job of getting the experienced developer up and running on the iPhone, they’re certainly not catering to new programmers. It’s a shame because the iPhone is such a fun, little platform, that I could see lots of kids wanting to jump in and getting their first real programming experience there. I hope that books and other resources pop up with that audience in mind.

Low Level Stuff

Things will look very familiar at the bottom of the API chain. Most of the low-level functionality can be accessed through pretty standard POSIX libraries: thread, file IO, sockets, etc,  all in straight C. My code ported over to using this libraries without having to make a single change.

As game developers, the first thing that we worry about is how are we going to do graphics. Fortunately, the iPhone includes an OpenGL implementation, so that’s very familiar territory. Specifically, it’s OpenGL ES 1.1 with a few extensions thrown in. The hardware itself is a fixed-function PowerVR chip with hardware vertex processing and texture combiners. Very 1999 :-) All of that makes writing 3D graphics app on the iPhone a breeze, even if performance isn’t top notch. At least they didn’t throw yet another proprietary API at us.

Surprisingly, the online help falls short with OpenGL. There are actually hardly any docs on that, and they mostly cover a couple functions to interface with their view system. I guess they figured you can just go to the internet and read all about it there, but I would have expected them to integrate them with the rest of the references at least.

OpenGL meshes relatively well with the user interface libraries, in the sense that you can render to almost any surface and combine it with other user interface elements. Unfortunately, it seems that putting any user interface views on top of OpenGL causes pretty major performance hits, so the integration isn’t perfect.

I haven’t used it yet, but Apple also provides the Open AL API for 3D sound. Again, props to Apple for sticking to an existing API.

Cocoa Touch

This is where the real fun starts. At least for me, this is all new territory. Mac programmers should be a lot more familiar with it. Cocoa Touch are the libraries for all the high-level user interface and accessing device-specific features. Since the iPhone is such an interface-centric device, this is where a lot of your time is going to be spent. The exception is if you decide to make a game that is 100% OpenGL, user interface and all. In that case, you’ll only be using this layer to access a few things, like touch inputs and accelerometer readings.

My impression coming to Cocoa Touch from an MFC/WTL/.Net background and no Cocoa experience was a breath of fresh air. Sure, the interface on the iPhone is simpler than the horrible monstrosities you can create with MFC, so maybe that’s why the API seems simpler. But whatever the reason, it just makes sense. You have one window, views, and view controllers. Once you get how they relate, the rest just comes together. I find myself guessing how things should be and getting it right most of the time. That tells me they’re doing something right (for the record, I never manage to guess anything with the .Net framework and it’s always a struggle).

When something isn’t obvious, you can look it up in the extensive online help, or just crack open the header file and look at what properties each class exposes. The class hierarchy isn’t too crazy, and each level adds a manageable number of functions.

It’s also really easy to extend controls and create your own versions to have special behaviors or even draw them in different ways. To me it was always a pain to create custom controls in MFC with their own rendering, but here it’s a breeze.

I also really like how a lot of controls don’t own your data, or even keep a copy that you need to refresh, but simply ask you for it when they need it. For example the picker (which you can quickly guess it’s called UIPickerView) will ask a delegate how many columns and rows to display and what data or views to render in each of them. Nice and simple.

We’re done with the fan boy raving. Not everything is perfect and some things are downright annoying.

Initialization of views and view controllers can be a bit weird. Sometimes the initFromNib gets called, sometimes it’s the viewDidLoad, sometimes it’s none of those and I have to hook up the loadView function, and sometimes I just have to make one myself. All depending on how the particular view gets created. Annoying.

Then there are some bugs and weird behaviors that you have to put up with. Because Apple doesn’t provide the full source code, you can’t just fix them or copy them and make your own version. Some controls are pretty elaborate so writing them from scratch is quite time consuming. For example, the segment control behaves as a toggle if you only have two segments, but as a button if you have more than two. There is a boolean variable that allows you to turn the toggle behavior off, but, guess what, it’s protected. So to make such a simple change, you need to create your own class and extend the UISegmentedControl. Doh!

Sometimes you’re not even so lucky. Looking at the header file for the UIPickerView class I noticed there was a flag to turn off the sound it makes when rotating the wheels. But it was marked as private (as @package actually), so I couldn’t do a thing about it. Double doh!

And it’s more than a flag here or there. Maybe you’d like to change the background of the picker, or the way the selection works? Out of luck. That’s all neatly wrapped up inside the picker class itself and you have no access to it. If you want to make a change, you need to reimplement it all from scratch. Hopefully Apple will continue refining the API and exposing more functionality where needed.

All of this leads me to my number one complaint about the Cocoa Touch API. It’s something that annoys me profoundly and I can’t believe Apple is doing. The API you have exposed to you is different than the one Apple uses internally. Why oh, why?? It violates everything I hold true and sacred about library and API design. I understand their wish to keep thing simple and clean, and to provide good documentation to everything is there. They managed to do those things admirably well. But why not provide whatever API calls they consider to be “expert” without documentation and a big warning saying you’re on your own? Why???

But wait, it gets better. Apple has been actively rejecting apps from the App Store from people who went to the trouble of reverse engineering the libraries and calling undocumented functionality from them. What is up with that? Is it Apple’s way of ensuring future compatibility? Then how are their own apps going to work when changes happen since they’re obviously using a bunch of undocumented features. Not only that, but they get it wrong sometimes!

I thought this was as bad as it was going to get, but wait, there’s more. Apple is playing favorites. So if you’re a big company and have two Os in your name (and your name doesn’t start with M) then you get to use their undocumented APIs without getting rejected. How totally annoying is that?

All In All…

Apart from the questionable decisions of what to expose in the API and a few stray bugs, the iPhone has a great set of APIs to develop for. I found it really easy to get into them and became productive in no time. If only they would expose the full functionality, it would be an almost perfect setup.

I loved the sound of that! I’ve been an indie developer for a year and a half, and I’m seeing more and more people taking tha route. I think indie development is necessary for a healthy game development industry. Where else are people going to try weird ideas, experimental development techniques, an commercially unproven designs? I think it’s a sign of maturity that we’re seeing a need filled by independent development next to the Maddens and Worlds of Warcraft and other chart-toppers.

I think 2008 will go down as the year that indie games made it to the big time: Braid, Schizoid, and World of Goo are just the top of the iceberg. It showed that it was possible to become an indie developer and make a living from it. 2009 has to be the year that indie development really explodes, right? The trend is certainly there. I’m constantly hearing about people or even whole teams leaving large development houses and starting their own, small companies. These developers are not trying to become the next EA, but they want to keep things small, creative, and, above all, fun.

If the only path for indie game development were downloadable console games on XBLA and PSN, the future wouldn’t look so bright. I used to be really excited about those platforms, but the restrictive zeal of Microsoft and Sony, the dingy royalty deal on XBLA, plus the high-development costs required for most games in those platforms, makes them very difficult to develop for and make any money from them. More and more, I’m hearing of donwloadable console titles with budgets around a million dollars. A million dollars!! That’s “just” a tad bit out of my indie pocket range.

Fortunately there are other paths. We have the usual PC market, which seems to be on its way to being abandoned by the major players but leaves a void for indies. Some guys seem to be doing well there, especially if you consider the Mac and Linux markets in addition to Windows.

Then there’s Nintendo’s WiiWare, which seems surprisingly much more open than the Xbox360 or PS3. I don’t have any friends who are personally developing for WiiWare, but I hear the barriers of access are not insurmountable. I’d love to see some sales numbers though. Nintendo certainly has the name to attract players and developers. Maybe they need to open up their DS development a bit too. Bob would like that for sure!

And then we have our last, best hope for indie game development: The iPhone [1].

The iPhone seems to hit all the right marks: Hardware advanced enough that you can develop really interesting and good-looking games, yet simple enough that one or two people can create a top title. Very open (the SDK and tools are available for free to anyone!!) but with a very clear distribution channel.

Given the low prices of games (between $0.99 and $10) on the iPhone, not only is it possible to make great games with a team of two people, but actually having larger teams can make development much less profitable. I can’t imagine how EA or Maxis are making any money from their iPhone games if they put even just 10 or 20 people to work on them for a few months.

In order for indies to succeed in the app store, it’s important that we get enough visibility. Right now, the big money is on the top lists that Apple publishes which are unfortunately out of reach for most of us. A trend I’ve noticed lately is that big publishers are starting to elbow out indies out of the Top Ten charts. I hope that’s not a trend that continues much further.

I’m glad to see that the app store has related links when you browse a product, and also a variety of independent web sites that are appearing trying to sort the iWheat from the iChaff. It’s a start, but we need more initiatives along those lines. In the meanwhile, support your indie iPhone developers. Or at least go out of your way to learn a bit more about them. If you haven’t seen it already, check out the New Year’s App Blowout Sale and pick up Sneezies, Up There, and Mouse House. All three totally adorable, very polished games.

Happy New Year 2009 to everybody! I’m sure this year will bring a lot of surprises and interesting developments.

[1]Yes, I shamelessly stole that line from my favorite sci-fi show.

I’ve only been doing iPhone development for a few months, so I don’t claim to know everything. But they have been some intensive couple of months, that’s for sure. This is my attempt to share my experiences, and hopefully encourage more game developers to take the leap and consider this really fun little machine.

As I explained in a previous post, I was a Mac-virgin. Sure, I had used one here and there, but never as my primary platform. Not only was I able to get up to speed and functional in a matter of days, but I totally love it and switched to it as my main computer. So, with that out of the way, how about the iPhone development itself? Over the next few days (or weeks, depending how busy I get), I’ll try and cover the major areas of iPhone development that I’ve experienced so far.

Let’s get the most obvious one out of the way: Objective C.

Somehow, its very mention seems to fill people with fear. When other programmers hear I’m doing iPhone development, they lower their voice, they get closer, and ask “Do you _have_ to write it in Objective C?” I think the main reason people are scare of it is because it’s not very well known. It’s not a sexy newcomer that the web is buzzing about like Rails, or a language with the backup of corporate giants like C# or Java. Objective C has been around since the early 80s, which is pretty amazing because it feels very modern in a lot of its features. Frankly, I’m surprised that Microsoft chose to create C# instead of adopting/extending Objective C (or maybe I’m not that surprised coming from Microsoft–OK, that was the last dig at MS and C# for today. Promise).

I had never used Objective C before I started doing iPhone development, but it was a breeze to get up to speed coming from a strong C and C++ background. The most important thing is that the interop between Objective C and C/C++ is nearly flawless. It’s a pleasure to call Objective C functions from C++ and the other way around. So much so that the lines get all blurred between what is what. Take that C#! (oops, so much for my promise).

Objective C is organized like C++, with header files and implementation files. That’s both good and bad. Good because it’s familiar, it gives you a good separation between the public interface of a class and its implementation, and allows for fast build times. But it’s bad because there is some annoying duplication between both files. Fortunately, Objective C gets this one right (again), and allows you to keep the header file to purely the public interface of the class. That means I can declare all the private member functions in the implementation file, which keeps duplication to a minimum.

A word of warning: You’ll probably have to change all your .cpp files to be .mm so they compile as Objective C and you can freely include any Objective C headers. Also note that Objective C files are usually .m which only parses C header files. If you need to call C++ files you need to make them .mm, or you’ll get some really bizarre errors.

If you’re really a died-in-the-wool C++ fanatic, you can isolate yourself from all the Objective C stuff by writing a few wrappers that interface with the operating system. If not, you can go with the flow and embrace it and use it when it’s needed.

Assuming you’re willing to look at Objective C, how is it coming from a C++ game programming background? At first glance it’s a bit ugly. I admit it. It looks like it’s full of - and + and [[square:brackets] everywhere]. But it’s very superficial. Objective C is a superset of C++, which means they introduced a different way to create Objective C classes (as opposed to C++ classes). The worst thing about the brackets, which are used for sending messages to objects, is that typing them is cumbersome. I often find myself having to move back to the beginning of the line to add new brackets as I’m making new function calls.

Internally things are much more dynamic than C++, and instead of member function calls they’re really messages that get passed around. That makes it easier to deal with null objects or objects that might or might not implement specific interfaces. I don’t know if you can add new message handlers on the fly, but I suspect you could (although why you’d want to do it other than for geek achievement points is beyond me–I’ve certainly never felt the need to do it).

One of the features that I’m using quite a bit are properties. You can declare member variables as properties, tag them with what kind of access is permitted (read-only, read-write, whether you want to take ownership or not, whether it’s an atomic assignment, etc), and all the code is automatically provided for you. Nice time saver.

Protocols are also a nice, clean way to provide interfaces. Certainly much better than the madness of C++! It makes working with objects that implement certain interfaces very easy. It’s also really easy to pass “pointers” to specific member variables through the use of selectors.

Enough with the raving. What’s not to like? Only a couple of things, really. Or at least so far. Ask me again in a year and I’ll probably have a laundry list of complaints. I doubt it will ever reach my level annoyance with C++ though. That’s pretty much impossible.

My biggest complaint about Objective C is that, get this, C++ constructors for member variables in Objective C classes are not called. Yup. You read that right! I’m sure there’s a good reason, but it does suck. Fortunately most of the Objective C code is high level classes, so it’s OK to new the C++ classes I need to work with. Still, it’s annoying. I also wish that Objective C played better with namespaces, but that’s a relatively small complaint.

This might sound like a strong statement, but if I were to design my ideal development language today, it would be pretty darn close to Objective C.

Something I still don’t know much about: Performance. I haven’t been doing much profiling on the iPhone yet, and all my performance-sensitive parts are written in C++. Still, I’d be curious to see how much overhead Objective C adds on top of C++. I suspect that a fair bit given the message-passing architecture and the more dynamic nature of the language.

Someone asked me what am I using for unit testing with Objective C. Hmm… errr… I’m not doing any unit testing in Objective C. There I said it :-) Not yet anyway. I’m sure I will as soon as I start using more of it. Most of the Objective C code is high-level code interfacing with the iPhone API, so I haven’t felt much the need to use TDD with it. If anybody has some recommended Objective C unit-testing frameworks that are lightweight and minimalistic, I’d love to hear it.

If you’re interested in learning Objective C, this document was the best resource I found to learn Objective C. It’s simple enough that you can use it to get up to speed, but detailed enough that I still refer to it from time to time. I haven’t felt the need for any books or any other tutorials.

Coming up next: The iPhone API, Xcode, and more goodies.

I do want to catch up and share my iPhone development experience with everybody, but it’s hard to make time, even during the holidays. To make things worse, I have a pretty hard deadline I set myself to have my iPhone app ready by early February.

It’s funny, I’ve worked more hours in the last year than any other year during my career, but it has never felt like work. Quite the opposite actually, and I find myself doing “work” instead of other things that used to be more fun and having a blast with it. For example, my game-playing time has gone waaay down. If I have an hour that I can do anything, I think “oh, maybe I can play a bit of World of Warcraft now” followed by “but it would be so much fun to finally add this other feature…” I let you figure out which one ends up winning in the end. If it wasn’t because of my weekly WoW session with Jim, Joey, and Michael, I wouldn’t make any progress in the game!

Not only am I enjoying what I do immensely, but it’s very easy to spend lots of time doing it because I get to do it from the comfort of my own home. I never have to worry about getting to work by a particular time, so I can get in my morning runs or go grocery shopping when the stores are empty, I have all my stuff around me, my work area has an incredible view overlooking the whole of Mission Valley, and my kitchen is stocked up with all my favorite teas. It’s really easy to spend hours and hours working. A bit too easy actually.

Burnout is the evil flip side of fun work. Even with incredibly rewarding and fun work, doing something for many hours a day can burn me out pretty easily. The symptoms are obvious: I start to lose interest, I’m not as productive, my mind wanders, and I find all sorts of other things I’d rather be doing. Fortunately, I can recognize those symptoms early on, so just backing off a bit, or taking an afternoon off, allows me to walk the fine line between work and burnout and keeps me productive.

Initially I was a bit concerned about the thought of working full time from home. Would I miss the interaction with coworkers and the sparks and new ideas from seeing all sorts of stuff around me in a company. It turns out that it wasn’t anything to be worried about. Thanks to the Internet, it’s very easy to reach out and connect with other people. I regularly share my progress with some friends and family members, and I always enjoy hearing their comments and reactions (even the not so positive ones–especially those actually). I also meet in a semi-regularly basis with my friend Dave, who is doing all the graphic design for my app (and he’s the mastermind behind the Power of Two logo).  Those meetings are always very inspiring, and usually result in all sorts of new ideas flying around and a flurry of activity in the days following.

Having non-work related hobbies really helps to keep me from spending too much time on work. Training for half marathons (I keep saying that one of these days I’ll do a full marathon–we’ll see about that), or just getting back in shape to do a century on my bike require quite a time commitment and keep me in shape and in top mental shape.

I suppose this is exactly the feeling that some companies try to encourage in their employees. They hire people who are really passionate about their work, give them the means and the freedom to do it, and hope they pour their heart into it. They just have to hope that their employees are able to keep themselves from burning out. That might be easier if you’re forced to work from an office environment, totally separate from home. On the other hand, being away from your family for longs periods of time eventually takes a toll on happiness and morale.

That’s a totally different beast from the kind of crunch that is rampant in the games industry. I define crunch as having to mandate long work hours (either explicitly or through peer pressure–which is more sublte and infinitely more evil). In one case, you’re so passionate about what you’re doing that you can’t wait to go back to work on Monday morning. In the other case, you’re still stuck to your desk come Monday morning because you were forced to work the weekend and get a build ready for some marketing demo. One is a very rewarding life, the other is soul-draining. One results in happy, ultra-productive people and the other degenerates into a death march, people leaving, and projects being even more delayed.

So, even with the February deadline looming, I want to take a few more hours these next few days and get back into the wrting swing of things. Besides, with the app coming up so soon, I should start thinking about announcing it soon! First I need to wrap up the Game Developer Magazine column due on Monday, but after that, I’ll try to start posting regular updates.

Until then, happy holidays to everybody. I hope you’re taking some time off, or at least doing what you really love.

Clearly, those are all things that need to be done, but if they are so repetitive and mindless, couldn’t we put some of the computers around us to good use and have them do the job for us?

A build server will do all that and more, much faster and more reliably than we could, and it will free us to work on the thing that made us fall in love with this industry in the first place: the game.

Getting Off The Ground

Before we can start thinking about setting up a build server, we need to be able to build the game with a single command from the command line. No clicking around, no GUI apps, no multiple steps, no magical incantations that only work during a full moon. Just type a command and the build for the game and all its libraries starts.

This is not just a necessary step to set up a build server; it’s a very good engineering practice. So if you’re not there, spend some time on it right away and you’ll be glad you did when candidate submission time comes around.

Building the game with a single command should be fairly easy, but the specifics will depend on your environment and build system. If you’re using Visual Studio, you can put the game and all the libraries in a single solution with the correct dependencies. Then you can invoke the devenv.com command line program specifying the solution and configuration you want to build: devenv.com mygame.sln /build Debug. You can wrap that up in a single batch file buildgame.bat for extra convenience and you’re done.

If you’re using another build system, such as make or jam, you can probably already build it with a single command. If you’re using a bunch of mode-made scripts, at least wrap them all up in a single script file so they can be run with a single command.

Just building the game with a single command isn’t enough. We must have a way to automatically detect whether the build succeeded or failed. Fortunately, most build systems (including devenv. com and make) return an error code when the build fails. If you’re rolling your own build script file, make sure to capture build failure and return an error code as well.

Setting Up The Build Server

A build server should be a dedicated machine with access to version control. Whenever a new build is needed, the server syncs to the latest version in version control, starts a build, and notifies the team in case of any errors.

That’s a fine start, but we could make things much better. For example, we could include the error message in the notification email so programmers can see right there what the problem was instead of being forced to sync and build the game themselves. We could also trigger a build in different circumstances (for example, code checked-in, forced by a person, or some other event) instead of only at fixed intervals. We might want to distribute the build across multiple machines, or keep logs and make them available on a web page, or format emails better, or use more direct notification methods …

Put away those Python reference manuals because fortunately, someone has already done all the work for us: CruiseControl (and CruiseControl.Net). It’s a free, open source build server program with all the bells and whistles that you could possibly want. And did I mention it’s free?

There are three main parts to it:

1. The build server. It runs as an application or a Windows service. It’s configured through a very simple XML file that tells it when to sync, where to sync, what to build, and how to report it.
2. The web front end. CruiseControl features a pretty, web-based dashboard showing all the builds, their status, past logs, and other pertinent data.
3. The system tray notificator. This is a little app that runs in the system tray and shows the status of all the builds and notifies you of any changes right away with a message and by playing some sounds. This is my favorite way to keep up to date with the build status. You’ll be up and running in about 10 minutes. The most complicated part is probably installing a web server (if you don’t already have one) and getting the web dashboard running. You’ll spend a few more hours tinkering with it to get it “just right,” and then you’re done. The only time I have to mess with it is to upgrade to a new version every so often. Other than that, it’s virtually maintenance free.

At this point you’ll have a fully featured build server in place. It verifies that the game can be built from the latest checked-in version of the code. It notifies developers of failed and successful builds right away. It increases version numbers, keeps a build history and statistics, archives executables, and emails logs.

CruiseControl and CruiseControl.Net are the two build servers I have most experience with. There are other build servers out there, with slightly different features, integrations with different environments, and so forth. Some of them are commercial and come with full support in case you’re more comfortable with that model.

It’s important to stress that a build server is not intended to be the only machine that builds the game. Every programmer (and maybe every member of the team) should be able to build the game in his or her own machine from scratch. The build server is there to verify that all the checked-in changes build correctly on a clean machine, and to make sure that all platforms and configurations are building successfully. Any official builds should be created exclusively from the build server, though. Especially any builds distributed externally to publishers or manufacturers. This ensures that the build is clean, was created in a repeatable manner, and is free of any idiosyncrasies from a particular machine.

How Often?

Once the build server is in place and is producing successful builds reliably, the question arises of how often to make builds of the game.

It used to be considered good practice to do a weekly build. The team would start ramping things up on Thursday to try and get a build out the door by the end of the day on Friday. Anybody who has done that knows how stressful it can be and how it can easily become a bottleneck.

Why wait a week if you can do one every night? More teams started switching to the daily build, which is much less stressful because there are fewer changes in each new build. It also gives the team a chance to fix anything that was found broken in the previous day’s build. Soon, teams took it beyond the daily build and started making two builds a day, or even one every hour.

The build server has been very appropriately described as the heartbeat of the project. A “green build” is one heartbeat and one small step forward. A “red build” is done when something is wrong and needs to get fixed as soon as possible. If you have a red build several days in a row, the project is in serious trouble. The more often you make a successful build, the better. You’ll find fewer surprises and stay more on course that way.

My favorite approach is continuous integration. With continuous integration, the build server starts a new build as soon as there’s a new check-in. If multiple check-ins come in while the build is in progress, another build starts right after it’s done, with all the new changes queued during that time. When following this practice, programmers sync to the latest version often, make small changes, and check-in code frequently, rather than batching many changes. Very conveniently, Cruise Control has a setting to start builds whenever anything changes.

The main benefit of continuous integration is that you are notified as soon as a check-in breaks the build—not a day later, or even an hour later, but minutes later. It tells you, “The last build was good. This one is not.” You can look through the last couple of check-ins that happened during that short time period and quickly narrow down the problem and fix it. Imagine trying to narrow down an elusive crash bug from all the check- ins for a full day or two!

Another benefit is that all programmers are working on a version very close to the latest one. This means that there are fewer source code conflicts when checking-in code, and fewer surprises lurking in the code. The flip side of that is that working on the latest version is living in the proverbial bleeding edge. It’s not unusual for someone to check-in code that has some accidental bad side effects. As long as those bad check-ins are limited, and that whenever they happen they are fixed right away, I have found the benefits to outweigh some instability in the main branch. Some of the ways to minimize disruptions when working with continuous integration are:

• Make sure that any code compiles before checking it in (that should go without saying!)
• Execute a fast set of unit tests to verify that basic functionality is working correctly, and
• Have the build server notify everybody as soon as there’s a broken build so it can be fixed and so that nobody else syncs or checks-in any code while the build is broken.

Need For Speed

Ideally, I’d like to check in some code and see whether the build server found any problems right away. In the real world, things can be much slower. After all, the build server needs to sync to the latest code, kick off builds for multiple platforms and multiple configurations, and perform some other time-consuming steps. Even so, there is work we can do to get feedback as soon as possible. Perform incremental builds during the day, so only the affected sections of the code need to be built. It’s still a good idea to do a full build at least every night to make sure that everything can be built from scratch.

Set up each platform and configuration as separate builds. That way you get feedback as soon as one of them completes. The only downside is if an error makes it through that causes all the builds to fail, get ready for lots and lots of broken build sounds playing all over the company. Speed up build times through good physical dependencies, modularity, precompiled headers, and good use of forward declarations.

Split up different builds and configurations in different machines. The easiest way is to set up one machine per platform and configuration (or maybe do a couple of configurations per machine). Cruise Control lets you easily integrate several build servers into the same web dashboard and system tray application, so this is a very easy solution.

Don’t Skimp on Hardware

Get the beefiest computers you can afford. Throw fast CPUs, disk access, and gigabit ethernet. Get multiprocessor cores and make sure your build system takes advantage of them. Does it sound like a lot of money? Not when you take into account how few servers you’ll have and how much time you’ll save all the members of the team.

I have tried several distributed build systems, and even though they can sometimes be beneficial for some codebases, I’m still not a huge fan. I find that you can often achieve the same (or better) results by using multiple processors and good build architectures, and you avoid the complexity and overhead of a distributed build system.

One “gotcha” we ran into when we scaled our build farm beyond about 15 build servers was that each of them was hitting our version control repository every few seconds to see if anything had changed. That wasn’t a trivial operation, and so many servers doing it so frequently definitely slowed things down to a crawl.

To remedy that, instead of having the build servers poll the overtaxed source control server, we had the source control server push out a notification. Whenever there was a check-in, the source control server changed a timestamp in a file located on an internal web server. We changed the build servers to constantly monitor the internal web server for changes in that file, and whenever it changed it triggered a build, which completely eliminated the overhead on the version control server.

Beyond The Build

So far, we’ve only been talking about building the game. But the build server is a great tool that we can put to good use for many other purposes. Why restrict ourselves to just the game? All the in-house tools would also benefit from getting the same treatment. We can even take it a step further and deploy the freshly-built copies of all the tools on a network drive or web page so they’re available to the whole team.

The build server can also double up as a symbol server. That makes it much more convenient for programmers to debug an earlier version of the game and libraries and have all the debugging information available without having to rebuild everything locally.

There’s no reason to limit the build server to just building source code. One of the most useful things you can do with it is use it to build game assets as well. Building assets is usually a slow process. Having a fast asset build system that can correctly perform incremental builds is crucial to keep asset build times down.

Build servers are general enough to perform just about any task. Running both unit tests (small tests on each class or function) and functional tests (tests that exercise a larger module or even the whole game) are perfect uses for the build server. Functional tests can be pretty slow, so make sure that they’re treated as a separate build and not as the last step in building the game. Nobody wants to wait for hours for all the functional tests to complete before they can see the successful build status after a check-in. The sky is the limit with what the build server can do. We use it to run static analysis of our source code, checking for spots in the code that can lead to subtle and dangerous bugs (uninitialized variables, implicit type conversions, and the like).

Another great use is to run through the different levels of the game, recording frame rate at different points of each level, logging the results, and failing the build if it ever drops below a certain threshold. Having the performance history for specific levels can be really useful to narrow down why a particular section is chugging at 20fps but was running at a solid 60 a couple of weeks ago. For bonus points, integrate all the collected data into easy-to-visualize graphs available through the web front end.

The build server is definitely the heartbeat of a project. Keep those check-ins coming and those builds green, and you know you’re heading in the right direction.

This article was originally printed in the August 2008 issue of Game Developer.

Starting iPhone development was a bit of a change since it required me to work exclusively under OSX and use a new IDE (XCode). I had talked to people who found the change very cumbersome and found the new environment got in their way, so I approached it with a bit of trepidation.

It’s An OS X World

With my new Macbook Pro under my arm (actually, it wasn’t new, I bought it used off Craigslist because I couldn’t afford a new one), I embarked on an adventure to this strange, new world of OS X development.

Some background is necessary here: I"m a minimalistic, no frills kind of guy when it comes to my preferences when working with computers (although now that I think about it, it probably applies to most other things in my life as well). The first thing I do with a new install of Windows is turn off all the GUI animations (don’t get me started on the search puppy dog!!), most of the sounds, all the auto-complete and auto-spellcheck features, and automatic upgrades. When I want the computer to do something, I’ll tell it to do so. Otherwise I want it to be quiet and responsive.

My ideal Visual Studio setup is also pretty similar: No fancy web startup page, no animations, no autocompletions by default (unless I press CTRL-Space), out with all the toolbars, just give me two text windows side by side and control everything from keyboard shortcuts (which I’m still using the ones from Visual C++ 6.0).

Needless to say, I was afraid of clashing with Mac OSX’s environment. I had a suspicious that it was full of eye candy, GUI frills, and required a mouse for everything. Not a good match.

There wasn’t as much of a learning curve as some getting-used-to. There are some Mac quirkiness that I just don’t get: The disembodied menu on the top of the screen, the lack of change of mouse cursor when you can resize a window, or not being able to automatically restore a file to its original location from the recycle bin.

But those are all relatively small things that don’t overshadow the fact that I just love this new environment. I used to like the Windows fonts, but after a week of using a Mac, I can’t go back to those tiny, jaggedy Windows fonts. Handling of multiple monitors is perfectly integrated and works like a charm. In general, things Just Work (TM).

Then there are the things that I love from my time in Linux that I can’t live without: Multiple desktops with Spaces, a decent command line shell, or being able to tweak settings directly in low-level config files.

And the thing that has changed how I work the most: Spotlight. No need for icons everywhere, or “Start | Programs” or anything. Just start typing what you want and there it is. It’s like the return of the command line on steroids. Between Spotlight, Firefox smart locaton bar, GMarks, and Ubiquity, I feel right at home. Now combine them all somehow and I’ll be in heaven!

But Wait, What About Windows?

I ended up falling in love with OS X so much, that I quickly moved to it as my primary environment at home for everything. Frankly, these days, it’s pretty easy for me to change environments since most of what I do is online. Just give me Firefox (or a good browser) and I’m there: GMail, Google Calendar, Google Reader, Google Docs (hmm… I sense a theme here), and a Wiki cover most of what I do on a daily basis.

For the rest of the apps, almost everything I need runs under OS X: good media visualizer (Preview), photograph organizer (Lightroom), image editor (The Gimp/Photoshop), audio editor (Audacity), music player (iTunes), and a few more odds and ends. OS X comes with some great utilities out of the box too, like Grab, Preview, or Activity Monitor.

Unfortunately, there are still a couple of things that I can only run under Windows. The main thing are games unfortunately, so I need to keep my Windows box around just for that. Although I just started playing World of Warcraft and it runs great on my Mac. Thanks Blizzard! Apart from that, every so often I need to do something in Visual Studio, or run some application I wrote in .Net. For that, I go about it in two different ways.

I can run it on the Windows box itself. Instead of keeping two sets of keyboards and mice (or a switch box), I used Synergy for a while. It was pretty cool being able to move the mouse cursor from one screen on the Mac to the screen on Windows and continue working there, and that might be a great solution if you’re working 50-50 on both platforms. In my case, it was more like 90-10 at the time (more like 99.9 to 0.1 now), so it felt a bit of a waste to have a full monitor dedicated to Windows. Instead, I decided to use Remote Desktop to control my Windows computer from the Mac. Amazingly enough, Microsoft wrote a Remote Desktop app for OS X that works like a charm, much better than some of the VNC programs I tried.

Whenever I don’t need to run something that is performance critical, I reach for Fusion. I’ve tried many products in the past that claim to run programs for another platform in your own computer: Wine, CrossOver Office, earlier versions of VMWare, but they’re always plagued by problems and incompatibilities. VMWare Fusion really surprised me by running anything I threw at it flawlessly, including Quicken (yes, I need to switch to Mint or Buxfer) and some hardware-accelerated 3D programs (those were a bit chuggy though, but they worked). And the coolest feature ever: Unity–I can run Windows apps in a window of their own directly on the Mac desktop. Totally awesome! On top of all that, I can have multiple snapshots, restore earlier states, and I can even run other operating systems like different versions of Linux. A geek’s true dream!

Some people will claim that I have drunk from the Apple kool-aid, and to a certain extent, they’re right. But I’d like to think I was swayed by many good reasons having to do with using the computer and being more productive, rather than Apple’s brainwashy, lifestyle marketing message. That and the smooth feel of the Macbook against the palm of my hands, its sleek profile, and sexy design. :-)

What about XCode and development? That’s another story for another day.

This being Southern California, cars are an inevitable reality. So what do I do while I’m stuck in my tiny subcompact hatchback? Devour every broadcast of TedTalks. I’m totally, irreversibly hooked on that podcast. And now, we finally have our own game industry podcast.

Ryan Wiancko, over at Industry Broadcast, has started recording articles from several of the top game development blogs. I’m happy to announce that he has added Games from Within to the list of articles he’s recording. He started with the oldie-but-goodie Optimizing the Content Pipeline that I wrote for Game Developer Magazine a few years ago (Note to self: I need to write an updated version of that article with my current thinking on the topic). He’ll be adding some of the most popular articles from the site as well as covering new entries.

So fill up your iPod with juicy game development recordings and have them handy wherever you go.

Now I’m 100% feed driven. Content is pushed to me, and not the other way around. Interestingly, the number of sites I keep up with has gone up significantly (over 100 by last count) so it’s still as time consuming as it was earlier. I just get a lot more information during the same amount of time.

So, for those of your who use the RSS feeds (and I hope it’s 99% of you out there, if not, head over to Google Reader right now), I just enabled two new types of feeds:

• Feed with full content. No more annoying excerpts and having to click through.
• Feed with all the comments. So you can follow any discussions more easily.

As a side note, WordPress totally rocks. I’ve used other blog systems before, but the latest version of WordPress is so slick, configurable, and easy to work with that it’s a real pleasure. Adding new themes or plugins is as simple as downloading a file and administration is done 100% through a super-slick web interface. Definitely better than MovableType, and light-years ahead of the hulking, slow beast of Drupal.

To give you an idea of how much I’m liking WordPress, I’m even using it to write posts. Seriously, that’s no small feat for me. I usually loathe online editors, and always composed the text offline and then pasted it into the blog at the last minute. With WordPress I feel totally comfortable composing posts on the fly, which makes the whole process very painless (which hopefully translates into more posts :-).

The big news is that we decided to call it quits on Power of Two Games. It was an amazing experience and I learned a huge amount. I’m sure we’ll write a postmortem with some of the gems we learned along the way sometime soon. Unlike most postmortems, we really don’t have any office politics, bosses to suck up to, or publishers to keep happy, we’ll be able to be brutally honest.

Against my better judgement, especially with the current horrible economy, I’m still burning the indie development candle. This time by myself and on the iPhone. I’ll get into many more details in the future, but so far iPhone development has been surprisingly fun and smooth.

One of the most attractive things about iPhone development is the short projects. Some of the top selling iPhone applications were written in just a week or two. The game/toy I’m working on is a bit more involved than a fart generator or a rotary dial, so I expect it will take me a total of three months. Still, much better than trying to make a game for XBLA or PSN!

Another big plus of iPhone development is the approval process. Yes, Apple needs to approve the application, but the process is infinitely simpler than trying to certify your PS3 or 360 game. I haven’t gone through that myself yet, but that’s what I keep hearing from other developers.

And then you have the App Store. Ah, the App Store. A captive audience of impulse-buyers with their credit cards already in the system because of iTunes. Genius. The cut that Apple takes is very reasonable, just 30%. Let’s hope they keep it that way and don’t pull the Microsoft move on us!

So, what’s not to like? That everybody and their grandmother is writing iPhone apps and throwing them on the App Store on the hopes of making a quick buck. And the worst thing is that it’s actually working. So people keep gambling with crappy apps hoping to become the next iBeer or Koi Pond. As a result, the App Store is flooded with stuff, most of it horrible, some OK, a few good, and a very few great apps. Right now the App Store interface isn’t ideal for discovering new, high-quality games and applications, and most of the exposure is limited to the Top Seller lists. I’m hoping in the next few months they move towards something more along the lines of Amazon.

And speaking of interfaces, whose retarded idea was it to only be able to access the App Store from within iTunes? I like Apple, and I’m totally digging Mac OS X (more on that later), but I hate having to use a slow, bloated application to do something that amounts to browsing a few web pages. Besides, you’d think they would hook more people by making it more accessible. Oh well. That’s not changing any time soon.

Stay tuned for more updates on my iPhone development adventures and maybe, just maybe, an early glimpse of the game :-)

On paper it looks great:

• Built from the ground up for parallel builds
• Uses a real programming language (Python) instead of overly-complicated XML files
• It allows for discovery of assets to build as other assets are parsed

The big question mark hanging over it was whether it was fast enough. I had done some experiments with Scons and other build systems a few years ago, and Scons totally failed by having ridiculously long incremental build times just checking if anything had changed. That was simply not acceptable. If no data has changed, I want the build system to detect it and come back in a second or less.

Since then, I’ve been told that Scons had finally fixed its dependency checking and it was much faster now. Music to my ears!

So I resurrected the old script I wrote back then to generate a nightmare stress test, downloaded the latest stable version of Scons (1.0.1) and fired it up.

I’m sorry to report that Scons is still as unusable as it was back then. I generated a codebase like the one I used to measure all the build systems (50 libraries, 100 classes each, 15 internal includes, 5 external includes). I built everything once, then ran it again and it took 49 seconds. I have some pretty nice hardware, including a fast SATA hard drive and a 2.8GHz Core2Duo with plenty of RAM, and 49 seconds to say that nothing needs to be changed just doesn’t cut it.

Now to be fair, right now we’re looking at Scons to build our assets, not our code. Maybe the nested project configuration is not particularly realistic, so I tried something simpler: 5000 files in a single library without any dependencies among each other at all. This is a very conservative example for an asset build of a large commercial game.

• It’s only 5000 files. A real game can easily have 10 times as many asset files.
• All the files are tiny (cpp files in this case). Assets can get very large when dealing with textures, sounds, and animations.
• There are zero dependencies among assets in this example. You’ll often have models depending on shaders and textures, levels on game entity definitions, etc.

Building assets doesn’t have to be the instant, no delay kind of build that I require building code. I expect most developers will be building assets locally, tweaking things, rebuilding, and trying them in the game. The faster the build is, the better, but a delay of a few seconds can be acceptable. If Scons can’t handle this build in less than 10 seconds, it’s certainly doomed in a full-scale game.

The result: 37 seconds!

For purely masochistic reasons, I decided to see how it scales, so I threw 20,000 files at it (again, without any dependencies). Incremental build time without any changes: 3 minutes and 46 seconds. That’s over 6 times longer for processing 4 times the number of files, so it doesn’t even scale linearly.

Scons is definitely too slow, and MSBuild has its share of problems. What’s left out there? Any recommendations for good parallel asset building systems? Hopefully something lightweight and easily configurable through a real language. Anything?

Let’s look at some of the common problems that can trip up the playback system and what we can do about them. We’ll warm up by starting with the easiest ones and work our way to the hardest ones.

Game Versions

A playback of a game recorded with a different version is almost guaranteed to end up with different results than the original version. Any minor tweaks to AI parameters, pathfinding algorithms, or just about anything is most likely going to generate a different game state. It doesn’t even have to be a change in code either: adding a new character or moving a trigger volume will also cause different playbacks.

In general, you’re best off avoiding using recorded inputs from previous versions of the game. Inserting the game build version at the beginning of the recorded game state file allows you to detect version mismatches during playback. If a mismatch is detected, you can either print a warning or give up on the state verification altogether. This check will be particularly useful if you’re sharing recorded sessions across the office (perhaps through a bug database) and people are on slightly different versions.

The idea of recording a session in release mode (all optimizations enabled, few debugging checks in place), and playing it back in debug mode is very tempting. Unfortunately, mixing and matching configurations is almost guaranteed not to work. Middleware libraries will often have slightly different behaviors in debug and release, which will throw off playback right away. But most likely so does your own code, and often in ways you might not expect. For example, you might have different floating point rounding modes in debug and release. Or even a more subtle difference: depending on the optimizations you have turned on, in debug mode a float variable might be copied back to memory after each operation (which rounds it back to a float), whereas in release the program might execute several operations in a row on that value before writing it back to main memory and causing the rounding only once. The results are going to be very close, but they’ll often be cumulative and the simulation will quickly diverge.

As a general rule, treat debug and release builds as if they were different versions. Make sure to include which configuration it is along with the build number in the game state file so you can check for that as well.

Memory State

Reading data from uninitialized memory is probably the number one cause of non-determinism in games. Uninitialized memory will contain whatever data was stored there before. That can mean anything from sane-looking values to total garbage. What’s worse, they’re going to change from run to run, so if your game ever uses the values contained in uninitialized memory, playback is not going to be deterministic.

The most common case of using uninitialized memory happens by reading a variable that was declared and not initialized. This mistake is easy to see when we’re dealing with a standalone variable, but when it becomes a member variable hidden inside some class, it becomes a lot less obvious. It’s also important to notice that this will only happen when we declare variables on the stack or we create them dynamically on the heap. For global and static variables, the C runtime takes care of initializing them all to zero for us.

Cranking up the compiler warning level to the maximum will allow it to catch some of the most obvious cases of reading uninitialized variables. For extra checks, I recommend a static code analysis program such as PC Lint. That’s almost guaranteed to catch every use of uninitialized variables (along with ten thousand other things, some of them extremely useful, though most of them you probably won’t care about at all).

Another common mistake is to read past the end of a valid memory area, such as an array. The values read from memory in that case are going to depend on where that array is, and what happened before. In any case, it’s going to become a source of non-determinsim, and it’s a clear bug that should be fixed right away.

If you ever find that your game is deterministic in debug mode but not in release mode, start by suspecting access to uninitialized memory. In debug mode, a lot of platforms will fill the memory heap with specific bit patterns describing the memory status: allocated, freed, etc. Those patterns can be really useful to track down problems with memory allocations, but they also set memory values to a consistent state, which will make debug builds seem deterministic when they really aren’t.

Pointer Values

Unless you’re working on a platform over which you have total control, and you have a very strict memory allocation scheme, you should probably never rely on the actual numerical value of your pointer variables. The pointer values will change from run to run, and will depend on what happened before, including what other programs were running at the time.

If you’re puzzled at the idea of using the pointer values as part of the logic in your program (and you should be), you still need to watch out for this possibility. Some well-known, open source compression libraries rely on this technique by hashing pointer values. This means that two consecutive runs of the same program might end up with two slightly different results.

In general, it’s considered a good practice to avoid using pointer values for anything other than dereferencing them and accessing the data they’re pointing to. Making decisions based on their actual numerical values is asking for trouble, and you can almost always achieve the same result by using offsets between pointer values (as long as you know they’re coming from the same memory block), which has none of those problems.

Asynchronous File I/O

This is where things start to get fun. A lot of games perform background loads while the game is running. Whenever the game detects it will need an asset, it initiates an asynchronous load. When the load is complete, the game is notified and, optionally, does some processing on that asset. There is no guarantee about when exactly the load will complete, which means that each playback has the potential to be slightly different.

If the background load just brings in more mipmaps for a texture, it’s not going to affect the simulation whether it happened a frame earlier or a frame later. On the other hand, if it’s loading a more detailed AI navigation path, it can definitely affect the position and state of AI units. Even if you’re not loading data that affects the simulation, it’s very useful to have asynchronous file IO work deterministically to catch bugs more reliably. For example, the game might crash if a background texture load completes the same frame as a line of dialog sound is requested. As any programmer who’s ever had to debug a problem like this will tell you, being able to reliably reproduce that crash is worth a small fortune.

We can turn asynchronous file loading into a deterministic operation by considering the read completions as inputs to the game. Whenever an asynchronous file load completes, we record it in the game input file at the corresponding frame. During playback, the game will request background loads as usual, but completion events are made available only when they’re read from the input stream.

This can be easily implemented at the game level by buffering all background read completions. During recording and normal game operation, background read completion events are made available as soon as they happen. During playback, however, background read completion events need to be available the exact same frame they happened in the original session. If the read finishes earlier, it is simply buffered for a few frames until the correct time. If the read hasn’t completed by the time it finished in the original session, the game blocks until it’s done and it’s made available right away. That means that playback might be a bit choppy at times, while the game blocks for data to be read, but it will ensure that playback is fully deterministic. Notice that blocking the game while waiting for a read to complete is not going to affect subsequent frames with a larger delta time because we’re also reading the system clock from the input stream.

Network Data

What about online games? Our playback method has been concerned exclusively with local players. Playtesting and Q/A on online games is much more expensive and time consuming than single-player games because of the manpower required and the coordination necessary to set the games up. Wouldn’t it be great to be able to replay a 30-person game that resulted in a crash without having to get 30 players again?

Data received from the network most definitely influences the game itself, so it’s another input to the game. We could treat all network traffic like any of the other inputs. Record it along with the other game input and play it back by inserting the network packets as if they had come from the network card. That would work, but it’s a bit more complicated than it has to be. Fully emulating the network traffic, connection status, and everything else can be a bit tricky to get just right.

A simpler approach consists of translating the data received through the network into higher-level game actions. For example, whenever a player fires a weapon, it results in a network packet, which is translated into a game-level action whenever it’s received. Then, the local simulation applies the action that causes that player to fire locally. You probably already have a system like that in place in your multiplayer game anyway. Recording those actions is a much simpler task than the raw network traffic, and playing back a recorded multiplayer game is just a matter of inserting the game-level actions at the same time they occurred, just like any other input.

The game input file is going to grow significantly as soon as you start recording all the actions created by the players through the network. Fortunately, most games are extremely careful to minimize network bandwidth, so the input file will remain at a reasonable size even recording all that data.

Threads And Multiprocessors

In today’s games, multithreading is an inevitable reality. Even if your code isn’t explicitly multithreaded, parts of your engine and middleware are probably using multiple threads. The problem with threads is that you never know exactly when one thread stops working and another one resumes. So it is possible that two events in two different threads will happen in different order in two runs of the game. To make things even more fun, most of today’s platforms have multiple cores, making determinism even more complicated. Do we need to give up determinism in a thread-dominated world?

The totally honest answer is that we’ve already picked all the low-hanging fruit. If you really want your game to be 100 percent deterministic while running in multiple threads over multiple cores, get ready to roll up your sleeves and do some serious work. For the rest of us, we can still get most of the benefits of a playback system without total determinism. That means that there is potential for bugs caused by thread interactions that might not be repeatable from run to run. But at least the game simulation will be the same for every playback, so the recording technique should still be very useful.

In the easiest case, the simulation runs on a single thread, with the rest of the threads dedicated to graphics, sound, and other systems. If we apply all the techniques we’ve covered so far, the simulation itself should be deterministic and we shouldn’t have to do anything different than we did in the singlethreaded case.

However, as soon as the simulation is spread across multiple threads, we need to be a lot more careful. Even if we have no control over the thread context switches, we can at least ensure that major events happen in the same order no matter how they were executed. For example, if a worker thread creates a set of actions to be executed, we can sort those actions before processing them. If that’s not an option (because those actions are processed as soon as they are created, for example), we could record the creation order as part of the input recording, although enforcing that order during playback might have far reaching consequences for many systems deep inside the game.

If achieving 100 percent determinism in a threaded environment is important to your project, have a look at Replay Director. It’s a commercial tool and set of libraries that gives you very accurate recordings and playbacks of your game, including thread context switches, with little extra programming on your part.

More Than Just For Debugging

At this point, if you’ve applied all the techniques we’ve discussed so far, you should have a pretty bullet-proof recording and playback system: lightweight, reliable, and accurate. It will be a great help as a debugging tool, but there’s no reason to stop there. You could use the same system to record demo sequences to show off in presentations without the need to make a movie capture and degrade the image quality. You can also truthfully claim that it’s a live demo, running on the game itself, not a canned movie, which always carries extra weight with most audiences.

You can even integrate the playback system as a key feature in your game. Halo 3 already does this by allowing you to replay any play session, examine what happen exactly, and let you take screenshots of the juiciest moments. It can be a great feature for a lot of games for almost no extra effort over what you’ve already implemented. The main problem will be to make sure future updates to the game don’t affect the playback of older gameplay sessions. This is really hard to achieve, since the most insignificant change can end up causing the simulation to diverge in unexpected ways. So if you’re going to rely on it as a game feature, you need a suite of comprehensive tests verifying that nothing changes whenever the game is updated. Last month we saw how to verify that a playback results in the same game state as the original session, so again, there’s very little extra work there.

With all these techniques in your toolbox, you should be able to make your game pretty close to 100 percent deterministic—enough to have a solid playback system and use it to its fullest during production, and maybe even as a game feature.

This article was originally printed in the June 2008 issue of Game Developer.

How would you like to be able to reproduce every crash report that QA adds to the bug database quickly and reliably? How useful would it be to be able to put a breakpoint the frame before a crash bug happens?

You can do all that and more if your game is deterministic and you feed it the same inputs as an earlier run. Sounds easy? It is, if you implement it early on and you keep it that way during development. If you choose not to make your game deterministic, your team will go insane by Einstein’s definition, and maybe by a few other definitions as well by the time the project ends.

Determinism

A system is said to be deterministic if, given the same set of inputs, it produces the same set of outputs. Think of your game as a big, black box, with some inputs and outputs.

The two main input types for most games are:

• Player input devices. The state of the gamepad, keyboard, mouse, or any other input device is used to update the simulation.
• System clock. Most games query the system clock once per frame to retrieve a current time and delta time. The game then advances the simulation forward by that amount of time. Even console games that expect to run at a rocksolid 60 Hz are often implemented this way to be able to deal with different TV refresh rates or with slower builds during development.

The main outputs of a game system are the pixels on the screen and the generated sounds. Optionally there are other outputs such as network packets or force feedback.

To make the game deterministic we need to make sure that, for a given set of inputs, it always produces the same outputs. In other words, playing the game twice, entering the same button presses at the exact times, the game will end up in the same state both times (same location, same health, number of lives, NPC positions, etc).

What about random numbers, which we rely on so much in games? Aren’t random numbers inputs as well? Yes and no. In games we use pseudo random number generators. That means that the sequence of numbers from a particular seed is always the same. For a given seed, all runs of the game are fully deterministic, so the random numbers are not really an input into the system. The seed to the random number generator is an input, since it will greatly affect the state of the simulation.

Record and Playback

Recording all player inputs and the system clock is as straightforward as it sounds: At the beginning of the simulation loop, sample the clock and the player input devices. Then save those values to a file and continue the game simulation as usual. You want to make sure the file is flushed after writing the values for each frame. That way, if the game crashes, you’ll have all the input leading up to that frame and you’ll be able to play it back right up to the point where it crashed.

Recording the input has a negligible performance impact and the amount of data saved is very small. As an example, in our current game, every frame we save the frame number, the clock time, frame delta time, and a 20-byte structure for each gamepad (see Listing 1). Without any compression, that’s 52 bytes per frame for a two-player game. At 60 Hz, a 20-minute game would be a tiny 3.6 MB, which is small enough to attach to a bug tracking system, email to the team, or archive with the build itself.

Listing 1. Game input structures.

struct FameInput{    uint32 frameNumber;    float time;    float dt;};

struct RawControllerState{    uint32 buttons;    float leftStickX;    float leftStickY;    float rightStickX;    float rightStickY;};

Using the recorded input for playback is almost as easy. Every frame, before the simulation starts, we read the input data from the file and feed it to the game as if it came from the system clock and the input devices. A clean way to do that is to separate the reading of the data from where it comes from. In C++ we can use abstract base classes that define an interface describing how to retrieve the data. For example, a class IGameInput can define an interface, and the class GamepadGameInput reads the data from the hardware, while the class FileGameInput reads it from the recorded file (see Listing 2).

Listing 2. IGameInput, GamepadGameInput and FileGameInput

class IGameInput{public:    virtual void GetInputData(RawControllerState& state) = 0;};

class GamepadGameInput: public IGameInput{public:    virtual void GetInputData(RawControllerState& state);};

class FileGameInput: public IGameInput{public:    virtual void GetInputData(RawControllerState& state);};

In addition to recording and playing back those inputs every frame, we also need to record the seed to the random number generator before the game starts, and read it and apply it during playback. That will ensure that all the random numbers are the same in both runs of the game.

Verification

If you’re going to rely on this recording system, you need to make sure the playback produces the exact same results as the original session. If they aren’t the same, the whole system is worthless. Also, because we’re just recording input, if the state of the game during playback ever starts to diverge, it will continue getting more and more out of sync from there. We need a way to make sure the playback produces the same state as the initial recording.

Earlier we identified some of the outputs of the game as the pixels on the screen or the sound produced by the speakers. We could compare pixels with a previous run of the game, but the storage requirements would be enormous, and the performance less than ideal. Besides, subtle changes in shaders, lighting, or a simple texture change would throw the comparison off. An easier solution is to check the game state itself. If an enemy is in a different location in two runs of the game, of course it’s going to render to a different set of pixels. It will be a lot faster and easier to compare game states than raw output.

During recording, in addition to the game input, we can save some of the state of the game. There’s no need to record it all, just some of the most important and representative state. Good candidates include player and enemy positions, prop transforms, score, etc. Unless it’s crucial to your game, there’s no need to record things like player animations or enemy AI state. Chances are that if any of those diverge, the positions for those entities will also diverge right away.

Once we have all of this state recorded, we can verify the state does not change during playback. Every frame, we compare the current game state with the recorded game state. If they’re different, even by the smallest amount, we know something has gone wrong and we flag it right away. The game isn’t quite deterministic and something needs to be fixed. We can choose to record and verify the game state anywhere in the simulation loop (as long as they’re both done in the same spot), but if we do it after the simulation step instead of before, we’ll be able to see what the inputs were that caused the divergence this frame, which will help when debugging. The main loop is shown in Listing 3.

Listing 3. Main loop structure

while (!done){    SampleClockAndInputs();   // from different sources through the same interface    RecordInput();    UpdateSimulation();    if (recordGameState) RecordGameState();    if (verifyGameState) VerifyGameState();}

Unlike recording game input, recording the game state can be a much more expensive operation. Traversing the game structures can be a significant performance hit due to cache misses, and the data saved to disk often results in large files. Because of this, in our current game player input is recorded all the time, but recording game state is optional, and we can be controlled through a command-line parameter. Game state verification is also optional, since sometimes we want to play back a recorded set of inputs even knowing that the state of the game is going to diverge due to changes we’ve made.

For a few of you, checking the game state won’t be enough. If you’re writing a middleware graphics layer, you probably want to verify that the values you’re generating in the back buffer are the same ones from a previous pass. Or maybe that improving the performance of a rendering algorithms still generates the same image. In that case, you might want to consider something like Perceptual Image Difference, which will be a much less error-prone than comparing exact values for pixels.

Monkeying Around

Once the game is fully deterministic and the playbacks are rock solid, you want to make sure it stays that way. It’s all too easy to introduce bugs that will cause playbacks to diverge. For example, a single, innocent-looking, uninitialized variable can change the simulation depending on the value it happens to have for this run.

The best method I’ve found to stress test the playback system is to use a recorded session of monkey input. The monkey input method consists of feeding the game pseudo-random inputs (as if a monkey were playing the game). Recording both the input and game state of a monkey input play session, and then playing it back verifying the game state kills two birds with one stone: You get some nice automated testing of your game, and you verify that the game is fully deterministic. It sounds too simple to be useful, but you’ll be amazed at how many bugs your first session of monkey input will uncover.

A clean way to implement the monkey input, is writing a new class that implements the IGameInput interface. This new class will generate game input for all the buttons and axes of a game controller, but it won’t come from the hardware game controller or from a recorded session, but from randomly generated values. Apart from being a very clean way to insert input into the game, this approach has the advantage that the monkey input can be recorded just as if it were regular input coming from the controller. That means we can later replay a session and verify its game state, which makes for a very useful functional test.

It turns out that totally random input values are not ideal, as it becomes apparent as soon as you implement the naive random monkey input class. If the state of the jump button is truly random, it will be pressed and released almost every frame, which means the player will hardly ever jump high enough off the floor. A better implementation will use a range with a minimum and maximum press time durations, which will give much better results. You might also want to avoid pressing specific buttons (like the pause button, or the restart button combination). Resist the temptation to make the monkey input too smart and make it behave more like a real player, for example, by limiting the number of buttons it can have pressed simultaneously. Part of the benefit of the monkey input is that it will do very unexpected things, that no sane player will ever do on purpose, and by doing that, it will unearth many more problems and issues with the game.

One day, shortly after I implemented the monkey input system, the playback verification started failing. Some game entities were ending up in a different state than expected. After doing some digging, I realized that I was using the same random number generator for the game simulation and for the monkey input. Since the playback was just reading input values from the file and not generating them on the fly, the random number sequence was getting off sync right away, causing entities with some randomness in them to behave differently. Lesson learned: Make sure that nothing in the monkey input affects the game systems. In this case, I solved it by using two instances of the random number generator, one for the monkey and one for the game.

Playback in The Real World

At this point we can easily reproduce bugs. We can re-run the game and put a breakpoint right before the game crashes—except that the crash happens 20 minutes into the playback and nobody wants to wait that long.

Fortunately, we can make time go by faster. During playback we don’t care about tearing artifacts, so we can turn off the vertical sync, which will speed things up a bit. Most importantly, we often don’t even care whether we render anything or not, so if we turn off rendering completely, we can make the playback run significantly faster, particularly if you were graphics bound. One word of caution: If the rendering part of the game does significant work, and especially if you suspect it might be interacting with the rest of the game to cause a bug or some other source of non-determinism, you might want to do the same work in the rendering system, constructing the push buffer the way you would normally do, but never send it off to the graphics hardware.

Turning off the vertical sync and disabling the graphics rendering, is just reducing the amount of time we spend in each frame. But it still takes 10 minutes to get 10 minutes into the game, we just go through many more frames to get there. The last piece of the puzzle to make time go faster is to be able to set a fixed timestep. Now we can go through the simulation at a rate much faster than real time (and the beefier you computer, the faster it will go). In my current game, we can go through 10 minutes of game time in about 1 minute of real time.

One use for recording and playback that we haven’t mentioned is performance comparisons. When you’re optimizing the game, you can record a gameplay sequence and gather some performance statistics: average fps, longest frames, etc. Then you can apply your optimizations, playback the same input sequence, and compare the performance statistics to see how effective the optimizations really were. Just make sure you use real clock time and not the recorded clock time, otherwise you won’t see any differences, even with all the hard effort you put into the optimizations.

There are a few details we glossed over that might prevent some games from begin fully deterministic: network traffic, asynchronous file I/O, and threading issues. We’ll cover those in detail next month.

You’ll soon find that input recording and playback becomes an essential tool for in development process and you’ll wonder how you ever lived without it. Spend a few hours implementing it early in the development cycle and reap the benefits many times over during production. You’ll be the hero of the day when that dreaded crash bug from QA arrives minutes before a deadline.

Thanks to Jim Tilander for being a great idea bouncing-board and proofreading the article.

This article was originally printed in the May 2008 issue of Game Developer.

Size

Charles and I were priding ourselves in keeping things small and minimal. But truth be told, it’s not like we were keeping track of how many lines of code we had written. Were things as small as we hoped they were?

The most convenient way of counting lines of code that I know is CLOC. It’s an extremely easy to use open source program which counts the lines of code in a codebase, gives very detailed information, strips out whitespace, breaks things down by language, and does just about everything you’d want from a program like that.

Running it on the latest version of our code (not including any 3rd party libraries) produces this:

    1621 text files.    1579 unique files.    3721 files ignored.
-------------------------------------------------------------------------------
Language                    files          blank        comment          code
-------------------------------------------------------------------------------
C++                            485          13577            303          46181
C#                             324           4935            712          22966
C/C++ Header                   407           4153             95          11975
MSBuild scripts                 18              0            126          1490
-------------------------------------------------------------------------------
SUM:                          1234          22665           1236          82612

Almost 60K lines of C++ code seemed very high. At first I thought it was because CLOC was counting files twice: once in their regular location and once in the .svn directory, but apparently it’s already removing all duplicates, so that wasn’t it.

Almost more scary than the amount of C++ code (which is all our runtime and some of our tools) is the amount of C# code. For a language that claims to be of significantly higher level than C++, that’s quite a mouthful of code!

Another surprising count in there is the number of lines with comments. Since we make heavy use of TDD, I really didn’t expect more than a couple dozen lines of code in the whole codebase. Still, I’m kind of proud that we have less than one line of code per file on average :-)

Here’s a more detailed breakdown, with the line count just for our runtime (engine and game):

1089 text files.    1053 unique files.    2338 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            441          11997            245          40943
C/C++ Header                   385           3964             90          11405
-------------------------------------------------------------------------------
SUM:                           826          15961            335          52348

and for our tools:

532 text files.     531 unique files.    1383 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C#                             324           4935            712          22966
C++                             44           1580             58           5238
MSBuild scripts                 18              0            126           1490
C/C++ Header                    23            199              5            591
-------------------------------------------------------------------------------
SUM:                           409           6714            901          30285

Tests

Then I realized that a good chunk of those were tests. So excluding all directories matching *Tests* gets the following result:

1206 text files.    1187 unique files.    4199 files ignored.
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                            283           6464            150          22464
C#                             213           2636            534          12402
C/C++ Header                   380           3824             94          10782
MSBuild scripts                 12              0             84            978
-------------------------------------------------------------------------------
SUM:                           888          12924            862          46626

A bit more than half the C++ code consisted of tests. That’s pretty consistent with my experience with TDD. C# seems to follow a similar percentage as well.

As for the exact number of tests, running a grep for TEST shows all the C++ tests:

C:\pow2>grep -r TEST SweetPea Engine Tools | grep -v svn | wc   2163    3620  221953

And doing the same thing with [Test] brings up all C# tests:

C:\pow2>grep -r \[Test\] SweetPea Engine Tools | grep -v svn | wc   735    1470  52717

That means that our average C++ test is about 11.5 lines long, and C# tests 14.4. Frankly, that sounds rather high. We make heavy use of fixtures whenever possible and each test usually only checks for a single condition (even if it involves a couple check statements). I suppose that number is higher than expected because it probably includes all the lines from #include statements and all the fixtures as part of the average.

* If we only count cpp files, that goes down to 49%

I was curious about that last part of checking a single thing per test, so I ran a grep for the number of CHECK statements in our code:

C:\pow2>grep -r CHECK SweetPea Engine Tools | grep -v svn | wc   3886   15079  399598

That’s 1.8 CHECK statements per TEST, which is about right. Even though we’re checking for a single condition, we’ll often check a couple things about it (i.e. the camera stopped and it reached its final destination).

Build Times

So, given that amount of code, how long does it take to build it? Clearly it depends on your hardware. Since we’re not exactly rolling in money, we don’t have particularly powerful machines. Here at home, I’m using a modest Core 2 Duo E4300 (overclocked to 2.6 GHz) with fast memory and a relatively fast SATA hard drive, so that’s what I used for all my timings.

A full build of our game, plus all the libraries, all the tests, and running all the tests takes exactly 1 minute and 10 seconds. That’s pretty good for two reasons:

• When we work with the game we don’t build and run the unit tests for the engine. We have a separate solution for that. A full build of just the engine, the game, and the game unit tests only takes 43 seconds.
• The game itself is a fairly large project and devenv doesn’t know how to paralellize that build, so it’s only using half the available CPU power for about half the build time.

An incremental build after changing a single cpp file takes slightly over a second (including half a second of unit test execution).

As you can imagine, working with that codebase is a dream come true. Snappy, responsive. Nothing is hard enough that can’t be changed.

Unfortunately that’s where the fairy tale ends. The tools are another story altogether. Our C# tools, with all their unit tests, build in a mere 18 seconds, and the C++ tools in 1 minute and 10 seconds. That’s not too bad, except that it’s a surprisingly large amount of time for the C++ tools since there aren’t that many of them.

Here’s the kicker, doing another build without changing a single thing take 38 seconds. Whoa! We’re doing some C++/CLI trickery and apparently dependency checking is totally broken in VS2005 (either that, or we just don’t know how to set it up right).

Keeping things fast

What’s the secret of a lighting-fast build? Clearly, keeping the code size down is crucial. If your codebase is 2 million lines of code, builds are going to be painful no matter what. But they can be a little less painful with some gentle care.

One of the main build-time killers that we’re avoiding is the use of STL or Boost. Those libraries pull in everything and the kitchen sink, and their heavy use of templates make build and link time go through the roof. No thanks.

Our template use is pretty minimal. We have a couple containers (which I love and I’ll write about it one of these days) and that’s about it.

We’re pretty anal when it comes to keeping physical dependencies to a minimum. We forward declare aggressively, and we only include the headers that are necessary for each cpp file (PC Lint is “kind” of enough to remind us every time we have unnecessary #includes). We’re not using external include guards or #pragma once.

Precompiled headers are either not used, or kept to a minimum. I think the only project that uses them is the game and only for Havok headers. We don’t even have windows.h in a precompiled header (which would be a really bad idea because you’d be putting all the junk in windows.h available to your whole program).

Finally, we are using incremental links whenever possible. I remember a few versions of Visual Studio ago they were pretty broken, but they’re not giving us any problems. The only caveat is that if you modify a static library your program is linking with, it will force a full link. So they’re really only good for modifying the executable itself.

We’re not using any distributed builds. First of all, we don’t have enough computers to make it worthwhile. And second, I had horrible experiences with distributed builds in the past. They would help with a badly structured codebase, at the cost of longer incremental builds and mysterious spurious bad builds. Besides, once they’re in place, they tend to encourage even further disregard for keeping dependencies to a minimum.

How about you?

So, that’s it for the Power of Two codebase. How about you? Want to share your size, build times, or any other data?

Here at Power of Two Games, we write all our code with test-driven development. C++ tests use the fantastic UnitTest++ framework (no big surprise there :-) ) and we run all unit tests automatically as the last step of our build process. That means that every time we build anything, the tests for that library or program get executed. Every time. No exceptions.

None of that would be an issue except that TDD creates lots of unit tests and encourages almost constant incremental builds. In the past I’ve stressed how important it is to keep them fast. But this week that I was able to really feel what a dramatic difference slow unit tests make.

For many months, our unit tests ran under one second. Each library has its own set of unit tests, and so does the game and each tool. So even though we were writing lots of unit tests, we weren’t always running them all at the same time. There was a definite snappiness to coding: write test, compile, fail, write code, compile, pass, move on. Go, go, go.

As the game project got larger, we started accumulating more unit tests and testing times started creeping up into the second range. At the time, I couldn’t quite put my finger on what had changed, but there was a definite change in feel. Things weren’t quite as snappy.

Then, about a couple of weeks ago, we hit the two second mark and things definitely changed.

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 2.13 seconds.

As the tests were running, my mind would wander, thinking about what test I would write next, or whether the code I had just written would be cache-friendly, or whether we’d do Honey’s or Manhattan for lunch. I would be jerked out of my state of flow. It wasn’t just me, Charles felt it too. The difference in productivity was huge.

The Zones

For those of you laughing at those times and dismissing them as insignificant, you need to keep in mind that our build times are very short:

• An incremental build with just a change in a cpp file takes under a second (including link times).
• A full build of *all* our runtime code, including libraries, is around 50 seconds in a dual-core machine [1].

So adding on 2 seconds to the incremental build time is going from “practically instant” to “OK, I’m waiting…”.

Of course, none of this has anything to do with unit tests in particular. It’s total incremental build times (including compiling, linking, and running unit tests) that matter. An incremental build that takes 3 seconds with no unit tests is just as bad as one that is under one second with 3 second unit tests. They both interrupt the flow the same way.

I’ve often thought about how a company size affects communication and “feel”. It’s close to a step function with logarithmic scale: there’s a distinct jump from a company of 3 to a company of 10, to one of 35, to one of 100+.

I think something similar applies to build times [2]

• 0 to 2 seconds. Programmer stays in the flow. Productivity is way high.
• 2 to 8 seconds. Flow is broken. Definite feel of things being a bit painful and sluggish.
• 8 to 30 seconds. Attention wanders to other parts of the code, email, etc. Multitasking kicks in and overall productivity plummets.
• 30+ seconds. Builds seen as batch processing. After each build is started, some other action is started. Maybe even physically get up, get a drink, start a conversation. Full brain reboot between coding tasks.
• 5+ minutes. Programmers bang their heads against the desk, actively consider the merits of different forms of suicide, or at the very least start polishing their resumes.

The Fix

Having been spoiled by sub-second build times, neither one of us wanted to put up with the broken flow of 2+ second builds, so we decided it was time to fix things [3].

The good news is that our slow build times were solely due to running the unit tests. 2.13 seconds to run only about 1100+ tests is really bad, even on a so-so desktop machine like the ones we can afford. That’s almost 2ms per test! Remember that these are unit tests, not functional tests. So I expect to be able to run several thousands of these under one second. Something was way off.

Things had gotten much worse in the last couple of weeks, so it wasn’t a slow, steady creep up. Chances are we had introduced something very slow recently, which meant it was probably easy to fix.

As a first pass, we turned on the option to fail any test that takes over a certain amount of time in UnitTest++:

RunAllTests(reporter, UnitTest::Test::GetTestList(), NULL, 5); // fail any test over 5ms

That brought about tons of failing test, most of them added recently and creating one particular object in their fixtures. Digging a big deeper, that particular class had an array of 25,000 (yes, that many zeros) objects. Those objects were plain-old-data structures, but had a default constructor. Even though the constructor was simply initializing its members to zero, that was causing a significant slow down when so many objects were created. To make things worse, some of the code I had written was iterating over the array resetting the entries or shifting them up or down, repeatedly calling the constructor. Ouch!

A few minutes and lots of code hacking later, and the object didn’t call a single constructor anymore. Our test times went down by almost half. Victory!

We were already at 1 second, which is in the acceptable range, but since we were here, maybe we could improve things a bit more. After all, 1ms per test is still too slow.

UnitTest++ wasn’t reporting failing tests over 1ms very reliably. Different runs would cause very different results. Things were too spread out or other things in the computer were interfering too much.

About a fifth or so of our tests use Havok. Not with mocks, but directly. They each initialize Havok, create a world, and deal with a simple rigid body, constraint, or something of the sort. I suspected all along that the Havok system initialization and shutdown for each test was probably causing quite a slow down, so I changed it to be initialized only once before we ran any tests.

And that did the trick. With that change we were down to 0.16 seconds. Back to snappy, happy land!

> bin\SweetPea_d.exe -unittestSuccess: 1145 tests passed.Test time: 0.16 seconds.

In release mode things are even better, but we do most development in debug mode, so that’s the really important one:

> bin\SweetPea.exe -unittestSuccess: 1145 tests passed.Test time: 0.06 seconds.

Perhaps sub-second build times are not realistic for all projects, but I think there’s a lot to be gained by keeping compile, link, and test times as short as possible. For tips on how to keep compile times down, check out these articles I wrote a while ago. If your unit tests are slow and you really can’t speed them up anymore, consider splitting them into two sets: one that is important or recent ones that runs with every build, and another, more comprehensive one that runs in the build server.

Even if you cut down from 20 seconds down to 10, you’ll be making a huge shift in how you’re able to work and iterate on your programming.

[1] We’d love to keep detailed metrics with each build: executable size, number of lines of code, number of tests, etc, etc. Unfortunately, when the rubber hits the road, that ends up in the “nice to have category” and we simply have no resources to spare on a two-man startup.

[2] For a whole whooping statistical population of one since I’m only talking about my experience. Still, I suspect that the same applies to many other programmers by scaling the function by some constant.

[3] Perfect use of agile mentality. Fix things when you need them fixed, not before. But make sure you do fix them when you need them. Otherwise it’s not agile, it’s just lazy.

C++ enums are great in practice, but they’re a bit lacking once you really start flexing them. Don’t get me wrong, they’re definitely a step up from #defines, but you can’t help but wish for more. Type safety is only so-so (enums get promoted to ints silently), you can’t forward declare them so you’re forced to have extra includes, you can’t split up a declaration of enums across multiple files (the compiler would have to do more work, but it would be awesome to be able to create uids for different classes in their own header files instead of a global one). But the most grating of design decisions is their scoping rules.
Take for example a set of enums describing the game flow:

enum GameFlowType
{
    Run,
    Exit,
    Restart,
    Restore,
};

Now functions can return a GameFlowType to indicate that the main loop should do. So far so good.

Except that, look at how it’s used:

GameFlowType DoMainLoop(bool render)
{
    //...
    if (someCondition)
        return Exit;
    return Run;
}

Ouch! We’ve managed to pollute the global namespace with totally generic keywords such as Run and Exit. What if I want to have an enum that controls what action an AI is going to take:

enum AIAction
{
    Enter,
    Exit,
    Stop,
    Walk,
    Run,
};

That won’t even compile because the symbols conflict with the GameFlowType ones. Even worse, it will compile just fine if they don’t happen to be included in the same compilation unit. So you might be fine all along until you write some AI code that has control over the game flow. Oops!
OK, so you can prefix all your enums like this:

enum GameFlowType
{
    GameFlowTypeRun,
    GameFlowTypeExit,
    GameFlowTypeRestart,
    GameFlowTypeRestore,
};

and

enum AIAction{
    AIActionEnter,
    AIActionExit,
    AIActionStop,
    AIActionWalk,
    AIActionRun,
};

That will technically fix the problem, but it’s a bit ugly, and we’re still polluting the global namespace (what if I wanted to have a class called AIActionEnter?).

Another potential solution is to score the enum inside the class that is using them. So we could have:

class AIAction{
public:
    enum AIActionType
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };
};

Much cleaner, do doubt, but also with its share of problems: First of all, it forces you to associate a set of enums with a class, which is not something you always want to do. But the biggest problem is that as soon as you have more than one set of enums per class, they all get promoted to the same scope:

class AIAction
{
public:
    enum AIActionBehavior
    {
        Enter,
        Exit,
        Stop,
        Walk,
        Run,
    };

    enum AIActionGroup
    {
        None,
        Self,
        Team,
        All,
    };
};

When we see one of those enums in code referred to as AIAction::Self, we really have no idea that it’s referring to that group the action is applied to, as opposed to AIAction::Enter, which is the type of action.
Our solution? Move all enums into a descriptive namespace.

namespace GameFlowType
{
    enum Enum
    {
        Invalid,
        Run,
        Exit,
        Restart,
        Restore,
    };
}

When you have a variable of that enum type, it’s listed as GameFlowType::Enum type, which clearly indicates what it does. And its values are referred to as GameFLowType::Exit. You get the ideal scoping, you’re not tied to any particular class, things are explicit but not verbose, you’re no polluting the global namespace, and it’s doesn’t affect the runtime or compilation times.
Sometimes the best solutions are the simplest ones.

A couple of months ago something unusual happened: The functional test failed but I wasn’t able to reproduce the problem right away. The failure was not a crash, but an object in the world ending up in a different state than expected. That’s always tougher to track down. To make things even more fun, was object was affected changed depending on whether the game was run from the command line or the debugger. Oh, and did I mention it only happened in release mode? I’ve got a baaad feeling about this!

My first instinct was to blame it on Havok. After some investigating, I narrowed it down to one of the enemies that was in mid-air. Sometimes it would fall down, and other times it would stay hovering, as if not affected by gravity. I knew that Havok was supposed to be deterministic, but it’s every programmer’s instinct to always mistrust other people’s code. I should have known better.

I spent a whole day tracking this down. Adding more and more information to the recorded world state to detect the problem as soon as possible. Slowly circling in; silently creeping up on the unsuspecting bug. Eventually, I had my “aha!” moment. I had finally found it.

A member variable, m_alive, in the RobotLogical class was never initialized! That was causing the Havok character controller not to be updated sometimes, causing the discrepancy in object state between the recording pass and the playback pass. What made it even more difficult is that m_alive is a boolean, so any bit pattern other than zero, it’s going to make it true, which is the initial state I expected for the robots. It never failed in debug mode because unused heap memory is filled with patterns like 0xdeadbeef or (the definitely less fun, but more practical) 0xcdcdcdcd, which all make the variable come up as true. Doh!

Yes, C++ sucks because of time-wasters like that, but it didn’t prevent me from feeling really stupid for having sunk a whole day on it.

All the code Charles and I write is done through TDD so we have almost 100% unit-test coverage [2]. How come Test Driven Development didn’t catch that? TDD is not about catching bugs, it’s about designing code. This is the type of error that is only triggered once every thousand times, so running one or two unit tests that cover that path probably won’t trigger it. That’s one of the many reasons end-to-end functional tests are invaluable. Go LeChimp!

Help, LeChimp! Help!

Detecting there’s a problem is good, but if it takes a whole day to fix it, things are going to be pretty painful. The more complicated the game gets, the more difficult life will be.

A good start is to crank up the warning level of your compiler as far as it goes. We’re compiling everything with warning level 4 in VC2005, so we’re already covered in that end [3]. Unfortunately, warning level 4 doesn’t warn us about uninitialized member variables. We need something else.

Fortunately we have a secret weapon in our arsenal we had been ignoring: Lint.

Lint performs static analysis of C++ programs and spews out lots and lots of warnings. Think of it as an extremely pedantic warning level on a C++ compiler. And I mean extremely! It will warn you not only of potential errors and dodgy constructs, but of best practices, and sometimes even of recommendations in books like Effective C++.

As you can imagine, with all those things to check against, Lint is going to find lots to complain about. So taming the output into something readable is a definite challenge.

When it comes to Linting your code, there are a few options available.

Team Edition Compiler

My friend Jim Tilander brought up that there’s a Lint-like tool hidden in the bowels of the Microsoft Platform SDK. Boy, he wasn’t kidding about the hidden part! It’s a whopping 1.4 GB download for a DVD image that then needs to be extracted and installed in order to get to the small free compiler. When will Microsoft adopt a more modular approach?

Once I got past all the installation hurdles, I was pretty excited to give it a test. A free tool is a free tool. And if it has good integration with Visual Studio, so much the better. Strictly speaking, it’s not really a Lint tool, but an optimizing compiler with a special /analyze switch, which performs a lot of the same checks as Lint. But as we know, a rose by any other name…

Right off the bat, it fell flat the moment I started using it. Apparently it doesn’t know how to deal with vcproj files. To work on all the files in a project, I had to feed each cpp file separately. That in itself is only a slight annoyance, but it meant that you also need to pass it all the compiler switches set in the vcproj files: include directories, defines, etc. For a Microsoft product, I really would expect it to understand vcproj files. Oh well. It’s not the end of the world. I can always write a quick script to do that for me.

Unfortunately, the next problem was more serious. There was no global way to control what warnings I wanted reported. I can disable specific warnings in the command line, but if I only care about a few, it gets really cumbersome.

Finally the killer: It simply doesn’t report some of the most important warnings I expect out of Lint, such as uninitialized member variables or variables appearing in incorrect order in the initializer list.

In its defense, it had some decent features I would expect of good Lint tools, like the ability to suppress certain warnings from code or to give extra hits in the code itself. On the other hand, the way to control the behavior of the analyze switch is done with special keywords, which will make other compilers choke, making it harder to write cross-platform code. Somehow, I’m not terribly surprised.

LeChimp’s recommendation: Even for a free tool, it simply doesn’t stand up to any serious use. Skip it.

PC Lint

Discouraged from the few hours I spent in the Team Edition compiler fiasco, I turned to Gimpel Software PC Lint.

Don’t be put off by PC Lint’s archaic web site that seems to be snatched straight out of 1995 [4]. PC Lint has been around forever and it shows. Not just in their web site, but in the program itself. Don’t expect any fancy GUIs, or dialog boxes, or any fancy cyber-amenities developed in the last 20 years.

Still, what it lacks in sophistication and pleasing visuals it more than makes up in functionality: Three minutes, $239, and a 4MB download later, I was ready to go.

In about an hour, I had PC Lint fully integrated with our build server and checking for problems in all of our source code. In just half an hour more, all our code was passing Lint’s rigorous inspection.

PC Lint can be quite intimidating out of the (virtual) box. If you run it on a typical codebase (or even a really high-quality one), it will treat you to megabytes of warning spew. Most of it useless, I have to say. So you have to take some time and tame it. Teach it to be a bit more polite, or more relevant. Or at least not to yell so much.

My preference is to start out with no warnings by starting out the options.lnt file with -e* and then add warning messages that I find useful: uninitalized member variables, unreferenced symbols, gotchas with virtual functions, etc..

Even running Lint with a small number warnings enabled, there were a few places in our code that caused PC Lint to complain but we didn’t want to change (Vec4 really shouldn’t initialize any member variables in its default constructor, or the NonCopyable class shouldn’t have a virtual destructor). In those cases, we can add a few discrete comments directly in the code that tell Lint to be quiet. We know what we’re doing, Lint. Really.

After we got all the code passing the initial set of checks, Charles dug into the Lint options file with a gleeful glint in his eyes and became the Lint-nazi, adding a good couple dozen extra warnings. It only took a few more minutes to get all the code to pass all the new checks. Frankly, the slowest part of the process is reading the thousands of different checks that PC Lint can do and decipher them (there are a couple of them I’m still unsure what exactly they mean).

This is what our options.lnt file looks like today:

-d__debugbreak()=
-e*

+e539  // Did not expect positive indentation from Location
+e549  // Suspicious cast
+e616  // control flows into case/default
+e744  // switch statement has no default
+e750  // local macro 'Symbol' (Location) not referenced
+e751  // local typedef 'Symbol' (Location) not referenced
+e752  // local declarator 'Symbol' (Location) not referenced
+e753  // local struct, union or enum tag 'Symbol' (Location) not referenced
+e764  // switch statement does not have a case
+e766  // Include of header file FileName not used in module String
+e767  // macro 'Symbol' was defined differently in another module
+e773  // Expression-like macro 'Symbol' not parenthesized
+e777  // Testing floats for equality
+e783  // Line does not end with new-line
+e784  // Nul character truncated from string
+e789  // Assigning address of auto variable 'Symbol' to static
+e796  // Conceivable access of out-of-bounds pointer
+e797  // Conceivable creation of out-of-bounds pointer
+e801  // Use of goto is deprecated
+e802  // Conceivably passing a null pointer to function 'Symbol'
+e803  // Conceivable data overrun for function 'Symbol'
+e804  // Conceivable access beyond array for function 'Symbol'
+e806  // Small bit field is signed rather than unsigned
+e814  // useless declaration
+e815  // Arithmetic modification of unsaved pointer
+e817  // Conceivably negative subscript (Integer) in operator 'String'
+e818  // Pointer parameter 'Symbol' (Location) could be declared ptr to const
+e820  // Boolean test of a parenthesized assignment
+e825  // control flows into case/default without -fallthrough comment
+e940  // omitted braces within an initializer
+e954  // Pointer variable 'Symbol' (Location) could be declared as pointing to a const

// TURN THESE ON IF WE CAN FIGURE OUT HOW TO APPLY IT ONLY TO VALUES AND NOT POINTERS!
//+e952  // Parameter 'Symbol' (Location) could be declared const
//+e953  // Variable 'Symbol' (Location) could be declared as const

+e1401 // member symbol 'Symbol' (Location) not initialized by constructor
+e1404 // deleting an object of type 'Symbol' before type is defined
+e1411 // Member with different signature hides virtual member 'Symbol'
+e1413 // function 'Symbol' is returning a temporary via a reference

+e1506 // Call to virtual function 'Symbol' within a constructor or destructor
+e1507 // attempting to 'delete' an array (not a pointer)
+e1509 // base class destructor for class 'Name' is not virtual
+e1511 // Member hides non-virtual member 'Symbol' (Location)
+e1512 // destructor for base class 'Symbol' (Location) is not virtual
+e1516 // Data member hides inherited member 'Symbol' (Location)
+e1520 // Multiple assignment operators for class 'Symbol'
+e1521 // Multiple copy constructors for class 'Symbol'
+e1534 // static variable 'Symbol' found within inline function in header
+e1535 // Exposing low access data through member 'Symbol'
+e1537 // const function returns pointer data member 'Symbol'
+e1538 // base class 'Name' absent from initializer list for copy constructor
+e1539 // member 'Symbol' (Location) not assigned by assignment operator
+e1541 // member 'Symbol' (Location) possibly not initialized by constructor
+e1542 // member 'Symbol' (Location) possibly not initialized
+e1543 // member 'Symbol' (Location) possibly not initialized
+e1544 // value of variable 'Symbol' (Location) indeterminate (order of initialization)
+e1545 // value of variable 'Symbol' used previously to initialize variable 'Symbol' (Location)
+e1547 // Assignment of array to pointer to base class (Context)
+e1552 // Converting pointer to array-of-derived to pointer to base
+e1554 // Direct pointer copy of member 'Symbol' within copy constructor
+e1555 // Direct pointer copy of member 'Symbol' within copy assignment operator
+e1556 // 'new Type(integer)' is suspicious
+e1557 // const member 'Symbol' is not initialized
+e1561 // Reference initialization causes loss of const/volatile integrity

+e1705 // static class member may be accessed by the scoping operator
+e1706 // Declaration with scope operator is unusual within a class
+e1707 // static assumed for String
+e1710 // An implicit 'typename' was assumed
+e1711 // class 'Symbol' (Location) has a virtual function but is not inherited
+e1718 // expression within brackets ignored
+e1719 // assignment operator for class 'Symbol' has non-reference parameter
+e1720 // assignment operator for class 'Symbol' has non-const parameter
+e1724 // Argument to copy constructor for class 'Symbol' should be a const reference
+e1726 // taking address of overloaded function name 'Symbol'
+e1729 // Initializer inversion detected for member 'Symbol'
+e1734 // Had difficulty compiling template function: 'Symbol'
+e1736 // Redundant access specifier (String)

+e1931 // Constructor 'Symbol' can be used for implicit conversions

I originally only had about a dozen warnings enabled, but, as you can see, Charles really went to town with it :-)

To integrate it into our functional test, we just set up a new target in msbuild. For each project, we call it twice: once to generate a lnt file for the project, and once to run it on the files for the project.

<PropertyGroup>
    <LintDir>..\Bin\Lint\</LintDir>
    <Lint>$(LintDir)lint-nt.exe</Lint>
</PropertyGroup>

<ItemGroup>
    <ProjectFiles Exclude="..\Engine\**\*Tests.vcproj"
                  Include="Src\SweetPea.vcproj;..\Engine\**\*.vcproj;"/>
</ItemGroup>

<Target Name="Lint" DependsOnTargets="CleanLint;RunLint"/>

<Target Name="CleanLint">
    <Delete Files="%(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt"/>
</Target>

<Target Name="RunLint" Inputs="@(ProjectFiles)"
        Outputs="@(ProjectFiles -> %(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt)" >
    <Exec Command="$(Lint) -v -os(%(ProjectFiles.RelativeDir)%(ProjectFiles.Filename).lnt) %(ProjectFiles.Identity)"/>
    <Exec WorkingDirectory="%(ProjectFiles.RelativeDir)"
          Command="$(MSBuildProjectDirectory)\$(Lint) -i$(MSBuildProjectDirectory)\$(LintDir) std.lnt %(ProjectFiles.Filename).lnt"/>
</Target>

Whenever PC Lint detects some code that violates one of its rules, it returns an error message, which fails the msbuild target and the whole build is reported as failed by Cruise Control .Net. I love it when things work exactly like you want them to without doing any extra work. To make it even better, because PC Lint can output errors in VC++ format, they’re understood and parsed correctly by msbuild and CCNet, so they show up in the CCNet log like any other error.

PC Lint runs surprisingly fast. It takes about 20 seconds to run on all of our source code. I was almost tempted to add it to the incremental build in the build server, but I really want to keep build times to a minimum there, so it runs once every hour, which seems to be good enough.

If you want to be more hands-on with PC Lint, it’s a breeze to integrate with Visual Studio. Just set up a few custom commands in External Tools and you’re good to go. The Visual Studio Lint options file even details exactly how to set up those commands.

My only minor complaint with PC Lint is the strange behavior of error 766 (Include of header file not used in module). I don’t know what algorithm PC Lint uses to determine that, but it seems not to report some unused headers. That wouldn’t be a big deal, but sometimes just rearranging headers or moving some code around will cause it to recognize a header as unused, triggering a failed build. It’s a very useful warning though, so I’m willing to live with that quirk.

Working with PC Lint is a pleasure. It’s really low level and old school, but that’s precisely what made it so easy to integrate with our code and our build system. It doesn’t need installation programs, databases, registry entries, dll deployments, or anything like that. Just a good-ol’ ini file with options and you’re good to go. We even have it checked it in version control to make it easier to deploy to our development stations and the build server.

PC Lint has the characteristics I want in a tool: Easy to buy, deploy, and integrate into the way I want to work with it.

LeChimp’s recommendation: An absolute must! Worth every penny. Two opposable thumbs up!

Other Lint Alternatives

Lint was originally a Unix utility developed in the late 70s. You would expect that lots of different versions would be available for modern compilers and platforms, but there aren’t that many. Maybe it’s not a really fun project to engage enough Open Source hackers?

I have to admit I haven’t looked at any other alternatives to PC Lint and the Team Edition Compiler that work under Windows. A quick Google search reveals there are a few tools out there. Has anybody tried them? I’d love to hear your experiences with them.

• Splint. Only works on C. Emphasis on security. Open source.
• CodeCheck. Works on C++. Commercial.
• A bunch for Linux/Unix. As much as I’d like to, the reality is that most game development happens under Windows.

Conclusion: LeChimp Rules The Day (Again)

PC Lint should be a requirement for any C++ project. The day you set up a build server (which should be the first day), go ahead and spend the extra hour setting up PC Lint as well. Be safe up front and avoid wasting any time down the line. We should have followed our own advice, but sometimes priorities slip a bit in the rush of getting a whole company off the ground.

Now I can finally sleep better knowing that LeChimp is keeping busy linting my code several times a day.

[1] This is our current broken build sound bite.

[2] There are some things we don’t bother TDDing: main functions, super-high level leaf code, etc.

[3] Of course, you need to have a clean compile in warning level 4 in order to be worth it. That means builds without any warnings or other spew. Incidentally, I do hate the setting to treat warnings as errors because it means you can’t temporarily have warnings while you’re refactoring between checkins, which slows things down.

[5] That was just a wild guess, but it seems it really wasn’t that different in 1996.

Bug Hunting

Unit tests, by their very nature, are limited to a single class or function at the time. There are all sorts of complex interactions between objects and systems that they simply can’t test. Even if you use mock objects and are extremely careful to test all your object interactions, there will be lots of unexpected cases and bugs that crawl out while running the game under real world conditions. Dr. Chaos is alive and well.

Besides, unit tests just test that the code does what you think it should do. So if the algorithm you have in your head is completely wrong, they’ll just make sure that the implementation is as broken as you imagined. Bugs like that might only surface once the code is put in the context of the whole game and interacts with other systems.

Traditionally, game companies rely on QA to uncover and squash most of the bugs created by complex interactions. You know, the type of bug that gets only triggered when the player enter the cave while there’s a blue moon and he spun around in place a seven times. Not only is this is an expensive process, but it’s not even very good at uncovering all the bugs. Most games have millions and millions of combinations of possibilities and interactions, and it’s completely impractical to try to run through all of them by hand.

Our Hero: LeChimp

At Power of Two Games it’s just two of us. So no QA or even interns to play the game endlessly. But, even in preproduction, we can’t afford to ignore those types of bugs. Instead, we enlisted the help of our hero: LeChimp.

LeChimp is our functional test server. It tirelessly runs the game every couple of hours and makes sure it loads and runs without any problem. Sure, ideally it should run more frequently, but LeChimp doubles up as our build server, and we don’t have another good computer to spare (maybe if y’all bought more t-shirts we could afford to buy another cheapo Dell).

Running the game is a good start. It checks that it’s possible to load every level and that nothing crashes. Frankly, that’s a good percentage of what QA does a lot of the time, and a lot of teams would really benefit from having such a simple test and know as soon as a level stops loading. Still, it we can do much better than that.

Monkey business

LeChimp runs the game for a fixed number of frames, and makes sure the game doesn’t crash or hits any asserts. But running the game without any action going on is not very useful, so it runs it with the -monkey switch, which feeds pseudo-random input to the game, as if a monkey were playing the game.

Actually, the input from -monkey is not random at all. I first made it truly random by generating new inputs through std::rand() every frame, but it looked more like a monkey on crack was playing the game and the characters shook violently back and forth and never managed to do anything interesting. Instead, now the monkey input holds the controller sticks and the buttons for some varying time interval, and it looks more like a monkey without its ADD medicine, which is clearly a step up.

When I first heard of this technique at a GDC tutorial a few years ago, I thought it would be pretty useless. How many bugs could feeding random input to the game really uncover? Shouldn’t we try to be doing something more intelligent to mimic how players interact with the game? But since it was really easy to implement I decided to give it a try. Boy, was I in for a treat! Within a few runs, it uncovered several major bugs that nobody hadn’t seen during their runs of the game. Since then, there isn’t a game I work on that doesn’t get treated with some monkey input love..

It’s so simple to implement that if you haven’t done it already, I really encourage you to go and do it right away. Do it over your next lunch break even. I guarantee you’ll be amazed at what it uncovers (or I’ll refund your money for this article :-).

Recording for posterity

In addition to just trying to crash the game, LeChimp records all the inputs and the state of the game at every frame. Then, after running the game for a while, it runs it again, feeding it the recorded input, and verifies that the game is in the same state as it was during the recording session (every enemy is in the same place, every prop has the same orientation, every player has the same health, etc). This verifies that the game is fully deterministic, that is, given the same inputs, it always produces the same output.

We’re planning on some cool features that rely on the game being fully deterministic, so this something very important to us. But even if we weren’t planning on doing anything with it, determinism is an extremely useful feature to have when it comes time to track down bugs. Instead of getting lengthy descriptions of a crash from testers (which somehow always seems impossible to reproduce), they can include the playback file that lead to the crash along with the bug report, allowing us to catch it in the debugger in no time. Of course, it’s not like we have actual testers, but we affectionately think of LeChimp as one, and he’s always very careful to save all his playback files with every functional test run.

Right now we’re just recording the inputs to the game: delta time every frame and controller inputs. Just with that, the game runs exactly the same time in and time out (thanks to Havok for being deterministic!). Things get more complicated as soon as multiple threads are involved, since the exact timing of context switches between threads can affect the output. Some tools out there, like ReplayDirector claim to address this, but I haven’t looked into it very much.

Both the input file and the same state file are opened, written to, and closed every frame. That way there is no data loss if the game crashes unexpectedly and you get all the input leading up to that frame.

As far as checking that the world state is the same, it’s totally an ad-hoc process. We simply pick some of the obvious state and save them to a file: player positions, enemy position, props transforms, etc. If we ever see something get out of sync, we add it to the game state that gets saved and compared so it doesn’t happen in the future.

No waiting around

So far we have LeChimp running the following with every functional test:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name

followed by

sweetpea -playback=functional_test -level=level_name

At 60 Hz, running 10,000 frames is almost three minutes. 10,000 is just a number we pulled out of a hat. The longer you let the monkey loose with the game, the better the chance of uncovering something. Multiply that by the number of levels and sandboxes, and the functional test now takes quite a while to complete.

A good chunk of the time of the functional test is spent rendering each frame and waiting for the vertical sync signal. But nobody is actually looking at the output [2], so why bother?

We added a couple more command line switches to make the game run without rendering or waiting on vsync. To make that really useful, we also added the ability to force the frame time to be a fixed timestep. So we can run the game like this:

sweetpea -frames=10000 -record=functional_test -monkey -level=level_name -render=no -vsync=no -timestep=0.01566

The game will cruise through the simulation as fast as possible, often cramming all three minutes of gameplay into 10 seconds or so. Perfect for poor monitor-less LeChimp.

Testing, testing

There’s even more to LeChimp that just monkeying around. It also runs several other functional tests checking some high-level functionality:

• Player attacks. The player character attacks enemies using each different type of attack and verifies that each attack is successful. This is particularly useful when there are a few types of attacks (which themselves are unit tested), but there are many combinations that can be created between attacks and targets.

• Level restart. LeChimp loads a level and then restarts it hundreds of times, checking for crashes and memory leaks.

• Torture chamber. A tiny level in which the hero (in god mode) frantically mows down hundreds of enemies that get immediately respawned. This is a perfect stress test for performance and hardcoded limits.

We’re writing these tests as we go along. Whenever there’s a feature that seems complex enough, or that it relies on other systems, or that it seems to break repeatedly, we take a few minutes, write a new functional test, and throw it to LeChimp to run with all the others.

Functional tests like these are about as high level as it gets. They deal with actions such as “move the player to the right”, “spawn an enemy here”, or “perform special attack XXX”, so it would make sense to implement them in the same way you implement game logic (which in our case is still C++, although we’re considering a switch to Lua in some not very distant future).

Long Live LeChimp

LeChimp has been invaluable battling against Dr. Chaos. Several times I made a refactoring or introduced a new feature, all the unit tests passed, I checked it in, and a little while later LeChimp screams at us that something is wrong. Once we see the functional test fail, it’s usually pretty obvious how to fix it: a memory pool is too small, or a combination of events that causes the player to enter some unexpected state. Fixing it is a matter of writing a unit test, fixing the logic, and checking it in, all in a few minutes.

A few times, however, the problem hasn’t been that obvious. The world gets out of sync, but only in release mode. Running it from the debugger often results in yet a different state. Sounds like some annoying memory overwrite, or perhaps some uninitialized memory. Any programmer who’s had to deal with this before probably has shivers running down his back. Fortunately, LeChimp has a secret weapon in its arsenal to deal with that. But that’s another story and shall be told another time.

Until then, happy holidays, everybody!

[1] Don’t underestimate the power of keeping programmers happy! At a previous company I used to work for, a manager rewarded one of the programmers by buying him a DVD set of a TV show he was really into. That was only about $40, but they had a huge effect on the programmer’s morale and productivity. Talk about well-spent money.

[2] LeChimp doesn’t even have a monitor, although that sucks sometimes. RemoteDesktop is pretty cool, but it locks up the DirectX surfaces in some weird way, and then the graphics renderer refuses to initialize correctly. So we’re forced to use… get this… NetMeeting! With fake phone rings and all! Ring, ring, calling LeChimp…

Of course, we’re far from loaded with money, so keeping expenses to a minimum is definitely a top priority. It’s not coincidence that one of our most popular lunches is sharing a gigantic $5 sub at Manhattan Giant Pizza or the $3.50 Kealani’s chicken teriyaki sandwich (fortunately, they’re actually delicious too!).

Whenever it makes sense, we’ve opted for the most inexpensive options[1]: plastic workbenches from Costco for our desks, lights from Ikea with unpronounceable names, an outdated P3 that became our file server given to us by a friend, or a free scanner/printer donated by my girlfriend.

So when it came time to set up our office tools, we also looked for the most inexpensive solution that met all our needs.

Email

Our requirements for our email solution were:

• Minimize costs. Not just the software, but the hardware needed to run any severs, support, etc.

• Minimize maintenance. We’re our own IT staff, so the less time we have to spend screwing around with severs and updates, the more time we can spend coding and designing the game.

• Reliable. This, after all, is going to be used to send company emails, contracts out to outsourcing companies, responses to publishers, etc. We don’t want to be missing any emails.

• Good search capabilities.

• Available from any location. I guess that implies a web front end.

• Fully integrated with our domain powerof2games.com.

In the past, the companies we had worked at all used Microsoft Exchange and Outlook or the god-awful Lotus Notes. Apart from having to endure the torture of using these day in and day out, they’re quite expensive, so we looked for alternatives.

We could use the email server provided by our web hosting company. It would give us the bare bones functionality that we needed, including some form of web front end. It could work, but after using Gmail for personal email for several years, it felt like a major step backwards.

Since Charles and I were already using Gmail for our personal accounts, we thought of creating a new account for Power of Two Games, but having pow2games@gmail.com in all our business emails doesn’t exactly look very professional.

That’s when we found out about Google Apps for small business. It’s a free service offered by Google that allows integration of Gmail into your domain and use it as your corporate email. That seemed too good to be true. Gmail is hands down the best email web interface I have ever used, so the thought of using it as the company email was extremely tempting.

Setting up Gmail took a bit of messing around with some DNS advanced settings. All email sent to powerof2games.com had to be routed to Google somehow, so we had to set up a custom MX record in our web host and point it to Google’s servers.

We also wanted to access our email through a nice URL like mail.powerof2games.com instead of some long (and hard to remember) default URL going through Google. For that, we had to add a CNAME record pointing the new mail subdomain to ghs.google.com.

Once all that was set up, Gmail was up and running and fully integrated with our Power of Two domain. Not only that, but it was totally independent of our personal Gmail accounts, so we can even have them both open in different tabs in the same browser. Good job, Google!

Calendar

Gmail was just the tip of the iceberg. By going the Google route, we also get Google Calendar for our domain. And Google Calendar totally rules. It’s not like we were planning on scheduling company meetings at all hours of the day (although, technically, since we’re in the same office I guess we’re in a permanent meeting). But Calendar comes in really handy to put down big milestones, remind us to pay the rent, or keep track who we’re having lunch with this week.

Calendar works out of the “box” without any extra set up. The only thing we did was to create a custom subdomain to access it through calendar.powerof2games.com, so we did the same trick as for our mail subdomain with the CNAME entry.

I can even share the Power of Two Calendar with my personal Google Calendar account and see all my events combined in a single place. My whole life in a single web page. Very handy!

Extras

But wait, there’s more. We’re not always working at the office at the same time. Sometimes we’ll work from home for a few hours in the morning before walking down to the office, or maybe I’ll do some work from E Street late in the evening with an oatmeal chocolate chip cookie and some live music. At those times, it’s very helpful to be within easy reach of communication, and since we’re using Gmail, we get Gmail chat for free.

Documents

I’m usually very hesitant of “integrated solutions”. I always prefer to be able to mix and match programs, components, or whatever instead of going the route of vendor lock-in. And the truth is that the different Google components are totally optional, but it’s curious that we’re biting, hook, line, and sinker all the way into Google’s office suite.

I was thinking we would end up using OpenOffice.org, which is a great Microsoft Office clone. But we gave a try to Google Documents and Spreadsheets and we were totally impressed.

Not only has Google Documents reached point as far as features and user interface that they’re usable for real-world situations, but they offer a lot more functionality.

• It’s a web-based app, so we can edit the documents from any computer with web access.

• We can collaborate on the same document at the same time, which came in very handy when we were both working on the Sony Developer Program application document.

• Because it’s a Google product, you can search through the contents of all your documents and spreadsheets very easily.

• Tagging and sorting. You can tag any document with any set of labels and then sort or filter by them. I’m glad the days of hierarchical trees for organizing information are coming to an end.

• Exporting to a variety of formats (MS Office, pdf, etc) comes in very handy when we need to send a document to somebody else who is not using Google.

There are only a few things we wish it did differently, like being able to upload a pdf document (even if it’s just in read-only mode for archiving and searching purposes). The other odd quirk is that we can’t make all documents and spreadsheets shared by default between the two of us. Instead, we need to explicitly share them. Oh well. I guess they have bigger companies than us in mind. I don’t blame them.

It’s very encouraging to see these applications grow and get better week after week. Just a few months ago, Google Spreadsheets couldn’t do plots, but they added that functionality since then. They’ve also improved the whole front end with the tagging and listing of documents.

Voice communication

How the times have changed. When we started Power of Two Games, we didn’t even bother with a phone. Even though we ended up contracting a DSL Internet connection and getting a voice phone line would have been extremely easy, we simply didn’t need it. We were simply using our cell phones for the few times when email communication or chat wasn’t good enough.

But as we started outsourcing art and talking to console manufacturers, we started having the need to do voice conferences and calling internationally. After a couple, very uncomfortable, conference calls using the speakerphone on our cellphone, we decided that we really had to look for a better solution.

For once, Google didn’t have the solution for us (although wait a couple of years and see). I have been using Skype to call internationally for quite a while with great results, so it was natural that we would look at it as our first option. Again, we were pleasantly surprised: not only could we make very inexpensive international calls (with great voice quality), but for a small yearly fee we can get SkypeIn and have anybody call us from a regular phone.

Conclusion

We’re extremely pleased with how Google Apps and Skype have been working for us so far. They fit our needs perfectly and the cost is right. We were able to set things up in about a morning and never have to fuss with it again. How good is that?

I really don’t see why Google Apps wouldn’t work for a larger company. But companies seem to be stuck in the Exchange/Microsoft Office (or some other heavyweight and expensive server solution) choice. I wonder if there’s a good reason for that other than inertia.

We’re certainly happy to have these services available for free and being about to spend our time and money in what really matters: Creating a kick-ass game and getting it out the door.

Notes

[1] There are some things where going rock-bottom cheap doesn’t make any sense though. The $7 desktop speakers we bought online were quite the surprise. I wasn’t expecting top quality sound, but they sounded like those two way radios that security guards use. I didn’t even know they made speakers that bad! Also, we have “splurged” on 24" LCD monitors and comfortable chairs (and even so, always trying to find the cheapest price point for what we want), because those things make a world of difference when you’re using them many hours every day.

Since both Charles and I are tech guys, we knew we could implement our game idea without any problem and make it sing and dance at a silky-smooth 60 fps. But would it be fun? Now that we have these brand-new designer hats, finding that always elusive fun factor is a big concern. The idea of waiting for a couple of months before we could tell if our game idea was fun seemed too risky, so we tackled that problem first and head-on by prototyping.

We could start at the bottom, write some low-level input handling, some graphics rendering, a basic asset pipeline, and all the other usual suspects. The problem is that, even if we try to keep things as simple as possible, it would still be several weeks or even months before we can actually start implementing the game itself. And even when we do, we’ll always be running up against incomplete technology and having to spend time fleshing it out as we try to make a game come out of the other end.

Instead, we decided to start with a prototype. This isn’t supposed to be a “prototype” that eventually morphs into a shipping game, or a prototype that uses the same technology as the production code, or even a prototype that’s used to impress the big-wigs to squeeze some money out of them (ha!). No, all that stuff detracts from the ultimate goal of our prototype. Our approach was very similar to what Chris Hecker and Chaim Gingold described in their GDC presentation. We had the need to answer one very specific question: “Is our game idea fun?” And we wanted the answer as quickly and cheaply as possible. Everything else was secondary.

Prototyping period

And so we dove into a prototyping frenzy. For our first day, we set up shop at the E Street Cafe. We opened up the place, took over a table, hooked up gamepads to our laptops, and hacked away furiously until late in the evening, fueled all the while by cappuccinos, gourmet tea, and pastries [1]. Yum! By the end of the day, we had the basics of the game implemented and even some of the more advanced features. Heck, we even thought it was kind of fun, but that’s because we were tired and we had been staring at Charles’ horrible programmer art all day long. In all honesty, it really was a pretty sucky game, even if the enemy characters were wearing party hats. But hey, that was just end of day one.

Over the next few days, we had to take a break to deal with all the paperwork and logistics involved with setting up the new company and the office, but we resumed prototyping as soon as possible. The plan was to keep hammering away at the prototype until we saw it was clearly fun, or we decided it was hopeless and we had better think of something new (or go back and beg for our jobs back). Charles took it even further and swore not to shave his beard until the prototype was fun, much to his fiancee’s delight (seriously, the only thing the ladies think is hotter than a game programmer is a bearded game programmer).

A week later, things weren’t looking too hot. The game hadn’t gotten much beyond what we had done the first day. Sure, we had a bunch of advanced features in, but it didn’t really make it much more fun. Were we on the wrong path?

The major breakthroughs happened in the last few days. First of all, we removed a key feature that had been there since the very beginning. We really thought it was going to be key to the whole game, but it really made the game too easy and boring. As soon as we removed it, the feel of the game changed completely and things started to look up.

Then, as we were discussing some other advanced moves and features, the idea for a minor feature came up. Since the caffeine hadn’t kicked in yet and I didn’t have the energy to work on the next big feature, I implemented our little idea in about ten minutes. Charles saw it over my shoulder and he started laughing and begging me to check in the code. It was an instant hit! We ended up liking it so much that we decided to expand on with a few features, and before the end of the day, it’s clear that it’s going to become a main gameplay mechanic in the final game. That day we went home knowing we were nearing the end of the prototype face. Good thing because Charles’ beard was starting to look scary!

We tried about seven different captions, but none of them did the sprite justice.

One more day of tweaking the prototype and we officially called it done. We knew we had gotten there when we tweaked a small feature, fired up the game to see how it felt, and then spent 10-15 minutes going crazy with the controller because we were having too much fun. Another good sign was when we would start playing and then yell at each other to look at the crazy things we had just accomplished.

Finally, Charles was able to shave. Thank God.

The prototype

The prototype process was supposed to be as quick and cheap as possible. None of the code written was intended to be kept or extended to go into production code, and performance wasn’t an issue either. So, with that in mind, what did we use for developing our prototype?

C++ and Allegro. Yes, you read it right the first time. Let me say that again: C++ and Allegro.

With all the other choices available of higher-level languages and extensive game development libraries and frameworks, why on earth did we make that choice? Wouldn’t Python with PyGame or C# and the Game Studio Express have been better choices? The honest answer is that I don’t think either one of those options would have allowed us to get the prototype done any faster for several reasons:

• Small scope. The scope and size of the prototype was quite small, so there wasn’t much to be gained by using slightly higher-level language constructs or more complete libraries.

• Allegro. Even though it might seem like an odd choice and be behind the times, it’s precisely the old-school, fully procedural approach that makes Allegro so valuable. There are no frameworks, no enforced class hierarchies… heck, no classes or private members to get in the way. In Allegro, if you want to do something, you call the function and you’re done. Exactly what we needed for the prototype. In retrospect, SDL might have been a better choice since it would have allowed us to use a DirectX surface and do video capture more easily. But Allegro worked like a charm and we have nothing but good things to say about that experience [2].

• Proficiency. Both Charles and I are quite proficient at C++, so even if another language gave slightly faster development time, we more than made up for it by being very familiar with C++ and its libraries. For the prototype we allowed ourselves to use STL recklessly and with wanton disregard for memory access patterns and spurious allocations.  We also didn’t care about memory leaks at the end, so that helped get things up and running a lot faster.  Destructors?  Delete?  Free?  Cache coherency?  How do those equate to fun gameplay?!

It’s no secret that I’m an avid advocate of good software engineering practices for game development. As with anything, good software engineering practices have some very real and important benefits, but those benefits come with a cost. Because of the small scope of the prototype, and the fact that the code wasn’t going to survive more than a week or two, we decided to throw any software engineering practice that would slow us down in the least straight out the window!  Forget about accessors and private scoping; skip any unit tests; bye bye factories, loose coupling, and proper physical dependency management. Welcome to hackfest, check your engineering discipline at the door!

One of the things that made the prototype very successful for us was to concentrate on a single gameplay mechanic. We didn’t try to prototype the whole game, just the one bit we were most concerned about (and that was going to take 80% of the player’s time and attention). That allowed us to deal with a small subset of the game and iterate on it very quickly without getting bogged down by creeping features or dealing with a larger scope.

The prototype shamelessly used the worst programmer art imaginable (Charles + MS Paint == … you can imagine [edit: Charles] TOTALLY AWESOME SWEET). Yes, it looked horrible, but if the game was so much fun with those graphics, just imagine how it’s going to blow people away whenever we add some real art and animations.

Lessons learned

The most valuable lesson we learned was the value of constantly asking ourselves the questions “Is this fun?” and “What can I do now to make it more fun?“  That was actually really crucial. Having such a specific (even if subjective and nebulous) goal allowed us to remain in the right path. Sometimes it was tempting to add feature X or Y, but when we realized that it wouldn’t make the game any more fun, we quickly dropped it and moved to something else.

It’s important not to over-intellectualize the idea of “fun”, though. Sure, most of the fun comes from the choices the player makes and what happens as a result of those choices (even if the choice is to mash the buttons). But don’t underestimate the value of “fun” in having cool, flashy things on the screen, or to have the character jump with a nice ramp-up curve and land with some squishiness to give it that “just right” feel.

It was also very helpful to run the prototype by somebody else towards the end to make sure we weren’t finding it fun because we were simply delirious. Clearly, it has to be someone you can trust and who can understand what the prototype represents and not be put off by the stick figures. It was great to get confirmation from an outside source that it really was getting to be fun. We also got a bunch of feedback that made it a much tighter experience. A lot of it was pretty obvious, but we were too close to the game to be able to see it by ourselves.

As with most prototypes, it’s always very tempting to not throw it away and continue hammering at it until it looks like something that could be shipped. We didn’t really have a choice because of the technologies we went with (allegro is a fully cpu-based renderer), but we could feel the call of the dark side. It really was a good thing that we didn’t even try. Towards the end, the terrible hacked code we’d written was starting to slow our productivity. Code that bad builds up an “engineering debt” that you have to pay off sooner or later or you hit a wall.  Our particular wall came in the form of extending a pre-existing player action in what seemed like a trivial way.  Something like this would have literally taken about 5 minutes at the beginning of the prototype, but by the point we were at it would have taken closer to an hour. Heck, even having to go on for another week would have started to be painful. So in a way, the prototype came with a built-in self-destruct mechanism, which is a good thing because it discouraged us to continue working on it for too long.

Using existing technology is definitely the way to go. That way there’s very little “wasted” time getting up to speed. The fact that we were able to get most of the core mechanics of the game in the first day goes to show how useful is to hit the ground running instead of getting bogged down in how to load bitmaps or create a window.

Additionally, the procedural, non “framework-y”, non-object oriented approach of the API we chose made it much easier to bend to our will and make it work the way *we* wanted, and not the way somebody decided for us ahead of time. The fewer restrictions your prototyping API has, the easier it will be to do all the crazy things that come to mind during the prototyping phase.

The game

Unfortunately, we can’t really talk about the game right now. Just rest assured that we’re working on it at full steam and we’ll announce some details as soon as we can. Clearly, we’re not going to release the prototype since it has a good chance of scaring away any potential publishers in its current state. But if one day the game sells tons of units, maybe we’ll release the prototype as a “behind the scenes” feature so other people can have a glimpse at how things got started and they can laugh at our artistic skills.

Conclusion

There you have it. Now at least we can be confident that the game is going to be pretty fun. Sure, there are lots of aspects of the game that we haven’t prototyped, but we consider those to be lower risk and we’ll figure them as we go along. At least the core mechanic has proven to be fun. Now it’s just a matter of writing the production code and implementing it all. That simple. What could possibly go wrong? ;-)

[1] Money saving tip #1 for starving startups: If you have to pay for a 24-hour Internet connection in a coffee shop and you need to have several computers with Internet access, connect them with an ethernet cable and share the Internet connection from one of them. Ta-dah! :-)

[2] Actually, I’ve been wanting to create a movie of the prototype in action, but I haven’t been able to. It seems that Fraps doesn’t work because it’s not using a DirectX surface, and saving each frame is too slow. Anybody has any suggestions? Don’t get your hopes up though, we’re not uploading a movie of the prototype any time soon, we’re just saving it to show our grandchildren one day.

The computer was laughably primitive by today’s standards: Z80 4MHz CPU with 64KB of RAM. What made it stand out from other computers at the time was a whopping 16 simultaneous colors (as long as you gave up half your horizontal resolution), three-channel square wave sound generation, and 3” floppy disks. Cell phones these days are hundreds of times more powerful than that computer. Heck, the chip inside your microwave oven is probably more powerful!

What was it that caused love at first sight with that computer? Impressive as they were at the time, it wasn’t the technical specs that attracted me. It wasn’t the silly games either, even though those were fun for a few days. It was being able to give commands to the machine and have it execute them immediately.

The computer booted directly into a Basic interpreter. I started experimenting with a few PRINT statements, then moved to FOR loops, getting input and solving algebra problems as if by magic. I spent endless hours typing game listings that came in magazines (usually with a few printing errors, which made them so much more fun to get working correctly). I experimented with graphics, sounds, and animations.

Before I knew it, the Basic interpreter was too slow and bloated and I had to graduate to assembly. Programming became a bit slower (especially since for some reason I could only save the assembly onto tapes, not disks), but the programs became thousands of times faster. I was finally able to draw sprites without annoying flickering, use all available memory, and even overwrite part of the ROM jump table to squeeze in a few extra KB.

I was hooked.

That experience totally changed the rest of my life. It caused me to study computer engineering and computer science, and eventually to write games professionally (much to my parents’ dismay).

If that little, primitive computer had that effect on me, today’s dazzling multimedia computers with broadband Internet connection must have a hundred times that effect on today’s children, right? Quite the opposite.

Today’s computers might be a lot more powerful, but they’re also a lot more complicated. Windows is very large and intimidating to a newcomer. It doesn’t exactly scream “play with me, experiment!”. Instead, it is a very closed system. It discourages experimentation, and you’re in constant fear of disturbing any of the overly complex configurations (like the registry or system dlls), not to even mention dealing with viruses and other malware. Linux is much more transparent, but it’s still far from an inviting system to play and experiment with for newcomers to computers.

Another problem with modern computers is the lack of a programming language out of the box. Things are actually a bit better now than they were a few years ago. Microsoft offers a free express edition of Visual Studio. Python and other scripting languages are also free downloads, and they also have libraries for game development. Unfortunately none of those tools come pre-installed, which makes it hard to get started and even harder to share your results with other people. Also, a lot of free development tools and environments today are geared towards GUI programming, which is very different from the free-form, take-control-of-the-machine approach that you need not just for games, but to really learn about the machine. There really aren’t many better ways to permanently scare people away from programming than showing them something like this.

Web development is a bit better because it tends to be simpler, and once you have a web server set up, it’s easy to show the results to anybody with an Internet connection (which I hope is everybody these days). Unfortunately it’s not particularly well suited for something like games with the exception of things like Flash.

Although I’m sure that part of it is nostalgia, I’m clearly not the only one who feels this way. Computer science enrollment has been dropping dramatically in recent years and there seems to be a lack of interest in learning more about these machines that surround us, beyond how to use a browser and how to share videos with friends.

Really, when I look at things this way, I feel bad for kids growing up today. They’re missing out on a huge source of enjoyment that I had growing up as a kid. Of course, today there are other new frontiers to explore, and they have the Internet with all the new social aspects, but it’s not the same.

So, is the dream dead?

I have often wondered, isn’t there a platform out there that is more like the 8-bit computers were in their day? The closest parallel today is game consoles, except that they’re very closed and proprietary and they’re only intended to play games. Sony started making some progress along these lines with the Yaroze program for PSX and Linux for the Playstation 2. Unfortunately neither program was particularly successful for a variety of reasons, but it was a start.

A couple of days ago, in the keynote for GameFest, Microsoft announced that they’re taking things further on the Xbox 360 with Game Studio Express. They’re planning to open it up and allow everybody to develop games for it. Even better, the code will be mostly common between Windows and the Xbox 360, so it will be possible to do a lot of development on the PC and then move it over to the 360 relatively painlessly. How cool will it be for kids to write their own games running on the Xbox 360 and be able to show them off in their friends’ living room?

It seems that the only language available will be C#. At first I was disappointed as I was hoping it would work with C++. Then I realized it wasn’t such a bad idea. If anything, it would be better if it were something simpler, more like Dark Basic, that people with very little programming experience can quickly start messing around with. Game Studio Express is supposed to include several tools and APIs that allow you to program things at a really high level, so that might be good enough.

I’m very curious about the details, though. For instance, are we going to be able to get our hands dirty and write as much C# as we want, or are we going to be limited to a very strict framework? Is the link between the PC and the 360 something that they’re going to expose? In other words, will we be able to create other tools that use that functionality to create new types of content?

A really important question is whether anybody will be able to play content created by users, or will they have to have some special payment-based subscription. Having to pay to develop for the 360 is not ideal, but I don’t see it as an unsurmountable barrier. However, if users have to pay and be part of the developer program just to be able to play some user-generated content, that would come close to killing its usefulness. I hope that Microsoft learns some lessons from YouTube and Google Video and realize that the lower the barriers for people to browse the content, the more successful it is going to make the product. People have been able to share videos for a long time already, but it involved downloading them, getting a player that could decode them, and playing them back. Now it’s as simple as clicking on a link, and that’s why it has finally reached critical mass.

I’m definitely keeping a very close eye on this project, and I’ll try to give it a test drive as soon as I get a chance. It’s a bit ironic that with dozens of multi-million AAA titles out there, the reason I might end up buying an Xbox 360 is the chance to do play around with some sprites on the screen and re-create a bit of the old 8-bit days. The dream might indeed be back.

Today I had lunch with an old friend. Even though he’s not a programmer, he’s enough of a tech-head that our conversation eventually turned towards agile development. In the past, I’ve given presentations about agile game development to two distinct groups of people: game developers without much exposure to agile development, and agile developers who were unfamiliar with game development. This morning I realized how interesting it was to explain the goals and reasons behind agile development to someone completely outside those circles. It forced me to go back to basics and think about the fundamentals of a topic I’ve been so involved in that I’d stopped thinking about in those terms..

What exactly is the key concept behind agile development? One possible definition is “a set of techniques and procedures with minimal overhead that allow us to make sound, just-in-time decisions.” [1]

There are several important parts to that definition:

• Decisions: It’s not about the code, or the team, or the tests. It’s about decisions. The ultimate goal is to deliver a useful product, but agile development helps us to make the decisions along the way. What do we create next? How do we want this thing to look? What features do we have to ship with?
• Just-in-time: Maybe the agile community has a better term for this, but the concept strongly reminded me of the just-in-time compiling strategy of Java interpreters. Basically, agile methods will delay decisions until the last moment they have to be made. Not only does this help avoiding design paralysis and having to make too many decisions early on, but it also means that a lot of decisions will never have to be made because things changed from the original expectations.
• Sound: I originally left this one out, but I realized that an 8-ball can make just-in-time decisions with the best. It’s making sound ones that is difficult. Whenever a decision needs to be made, agile teams will draw on all the information they’ve acquired up until then.. Not only will they know more about market trends and competitors’ products, but they’ll also know about their own team strengths, technology limitations, or the shortcomings of past implementations. That’s a lot more information that they would have had available at the beginning of the project, during the traditional waterfall planning phase.
• Minimal overhead: Agile processes are, by definition, very light. The goal is to create a valuable product, not the process itself. Agile techniques introduce the minimal amount of overhead to achieve their purpose.

My friend was very curious about this approach, since it’s quite different from the normal way of working he’s used to in his company. He had a very insightful question: “Don’t you end up doing a lot of re-work?”. And the answer is, yes, absolutely. However, it’s not as wasteful as it sounds. It’s actually quite an efficient approach for a lot of projects.

Let’s visualize the progress made in a project as a path on a 2D surface. The amount of work is the length of the path. The progress is how far from the starting point you’ve gone, and the direction represents different approaches, designs, or implementations. An ideal project would look like this:

The project starts, knows exactly where it’s going, and progresses in a straight line. Euclid would agree: You can’t be more efficient than by taking the straight line between two points.

This diagram can be used to represent any scale: from the whole project spanning several years, to how a particular feature is implemented in a couple of weeks, or how a class is implemented in one morning. The same principles apply in all cases.

As a comparison, I suppose a death march project would look more like this:

You are constantly doing work, but you aren’t getting any closer to your destination. The project eventually gets canned, or development stops somewhere and it gets patched up the best it can and shipped to unsuspecting customers.

You might hope that an agile project looks more like this:

A project like that implies that some changes happened during development, but while always making steady progress. That would be nice, but in practice it often doesn’t work that way. When you change your plans or even some implementation, you often end up with some amount of rework:

You can see all the rework by the zigzags going in and out indicating varying levels of progress. I believe this rework is inevitable (and maybe even desirable) in agile projects. It happens at a low level, like changing interfaces to a class and having up update all the hundred tests you wrote that relied on that interface. But it also happens at a higher level, when you realize that a game feature isn’t as fun as you thought and you drop it in favor of another one that could be more promising.

From the above diagrams, agile development really looks inefficient. Clearly, we have traveled a much longer path to reach our destination (work done). But agile development can actually do better. How can we be more efficient than traveling in a straight line without resorting to strange relativistic warped space?

The diagram doesn’t tell the whole picture. Remember that agile development relies on just-in-time decisions. That includes the final goal: As we’re working towards that goal, we might see other, better or easier-to-reach ones. As a matter of fact, this is something that happens most of the time. In that case, the picture looks quite different:

Now the amount of work done is less than it was in the ideal first case! And we might have a better solution to boot. Who says you can’t have your cake and eat it too?

With this in mind, are there some cases in which it is better not to use agile development? Clearly. If you know things won’t change and you have a clear idea of how to reach your final goal, then go for it. Agile development won’t help much there. I have yet to see a project that worked that way. Maybe a sports game that ships year after year, or another simple port to a mobile platform. But most projects I’ve seen have such a high level of chaos and uncertainty due to all sorts of external factors that they could all greatly benefit from an agile approach.

This can be visualized very well by mapping the characteristics of a project in terms of requirements vs. technology. Projects that fit agile development well fall anywhere in the complicated to anarchy zone (a fair amount of technology uncertainty and requirements disagreement). On the other hand, projects in the lower left corner, which are very well understood and are very certain, would benefit the least from an agile approach. [2]

I don’t usually bash agile development because I truly believe it’s a great way to do development, especially for games. But how would an agile development project gone bad look?

Here there was a huge amount of rework without having any significant impact in the project. The project ended up very near the original goal (a bit further actually), but it spent most of its time thrashing around. You could argue that maybe some of that was exploring different solutions, but it’s probably not a sign of a good project anyway.

This brings up a good point: With agile development, it’s important to let go of the control-freak mindset. It’s good to have an idea of what your ultimate goal is, but it should never take precedence over what you discover along the way.

I see this frequently in people who have just learned the mechanics of test-driven development: they go through the cycle correctly, writing a test, coding, and then refactoring. But they’re still implementing the design they had in their head when they started instead of letting the tests dictate how the design should look. Often they get quite frustrated because they can’t figure out how to test things. This also applies at a larger scale. If a feature isn’t giving you good results, maybe it should be dropped or significantly changed.

This is one of the reasons I try to do my best not to talk about things as concrete as classes or functions during early discussions about some feature we will implement. I find that doing so tends to solidify those concepts too much and I tend to be much more likely to follow them instead of letting design evolve from what we learned during its implementation. So now instead, I prefer to talk about more abstract concepts like operations, modules, or data flow.

Whether you’re an agile developer or not, it’s always important to step back and question what you’re doing and how you’re doing it. Think about it in the context of your projects and needs, and re-evaluate it accordingly. Sometimes you’ll be surprised what new insights you achieve by stepping back and looking at the fundamentals of something you’re intimately familiar with. Just make sure you don’t bore your friends with too much technobabble.

1. Scott Ambler has a very good, much more detailed definition of agile development that describes more the practices rather than the goals.
2. Diagram adapted from Clinton Keith’s Agile Game Development GDC 2006 presentation

With all that going on, I realized I never got around to writing my impressions about GDC. It really was a great conference with some very good highlights. My travel is far from over, and I’m heading out to the airport to catch my plane to London in a couple of hours, but until then, here is my quick take on this year’s GDC.

This year I decided to come for the tutorials happening two days before the main conference. The first day I bounced between “Advanced Visual Effects with OpenGL” and “Software Engineering Issues in Multiplayer Games.”

The OpenGL tutorial was exactly what you would expect: lots of material from previous years and a few good bits of information. The first thing I was happy to learn is that Microsoft changed their decision on how OpenGL will be implemented in Vista. It will no longer be a layer on top of DirectX; instead it will be a first-class citizen in its own right. Now, I’m not an OpenGL fan. Up until recently I never had to work much with it, and now that I have, I admit that I like the Direct3D API much better (I’m just not a fan of implicit global states). But I’m very glad to see that OpenGL is going to be treated correctly in Vista and given the opportunity to compete with Direct3D.

I was also pleasantly surprised by the tools NVidia is releasing for OpenGL development. NVPerfKit is a bit like Pix for DirectX; combined with gDebugger, it starts making OpenGL a lot more attractive. They’re supposed to release version 2.0 soon, but I didn’t find it on their web site. It’s really interesting how much NVidia is pushing OpenGL lately. At one point it felt that it was over for OpenGL, but it really has been making a comeback in the last couple of years. Now if they can only improve Cg…

The tutorial “Software Engineering Issues in Multiplayer Games” managed to surprise me, and I wish I had spent longer there than I did. It really was focused on the software engineering aspects, rather than on the multiplayer part of it. Lots of good information and war stories of large-scale game development.

Tuesday I spent it exclusively in the Direct3D tutorial. The biggest new thing was DirectX 10, with several talks focused on that. Apart from the new cool tech features (such as the stream out feature or the geometry shaders), DirectX 10 is quite a departure from earlier versions of the API. A couple of things make me scratch my head in puzzlement. DirectX 10 supposedly won’t have any caps to query anymore. Hardware is either DirectX 10 compliant or it isn’t. That’s great news for developers. However, I also learned that DirectX 10 is going to be tied to the Vista operating system, and new versions of DirectX will only be released as part of new versions of Windows. Unless I’m missing something, that means that hardware won’t have an opportunity to change and grow while running on Vista. That would be horrible for hardware manufacturers, but at least they have OpenGL to show their new features. Or maybe it means that Microsoft is planning on doing a yearly operating system release. It will be interesting to see how it plays out.

The best session on Tuesday was clearly Natalya Tatarchuk’s talk on the Toy Shop demo. She spent a full hour dissecting the demo, covering every rendering trick and effect they used. Considering how impressive and packed the demo is, it was clear that she could have spent a full day talking about it. Some effects are awesome, like their smeared rain reflections on the road, but it was funny to see how they clearly spent quite a bit of time doing things that are completely lost, like diffraction on the light that goes through the droplets on the store window. In any case, if you haven’t seen the demo, download it right now.

Wednesday was the start of the main GDC sessions. To kick things off, Sean and I gave our talk on test-driven development. I was really surprised at how packed the room was and how well received it was, judging by all the questions and positive comments. It’s great to see that other people are also very interested in TDD and are thinking about applying to it game development.

The next-generation animation panel was pretty interesting as an overview of what could be coming down the pipe for animation. I was already familiar with most of the content except for Ken Perlin’s new foot-placement work, but it was a great overview anyway. I’ve been quite excited about data-driven animation for a couple of years, and Lucas Kovar’s work is very promising. I think the only thing holding that technique back from being applied to games is the memory requirements, but I would love to spend some time trying to make it practical. Victor Zordan’s work is also very promising, combining dynamics with data from motion capture, which would be great for games (using ragdolls to predict and blend into falling animations for example).

Thursday was definitely the big day (for sessions, although that’s always the biggest day for parties too). The day started with the excellent session on “Advanced Prototyping” by Chris Hecker and Chaim Gingold. I really can’t say enough good things about it: it was packed with interesting, non-obvious information, it was very well presented and full of energy (it’s Chris we’re talking about, so of course it is!), and extremely motivational. They did a top-notch job preparing for the presentation and it showed. The pair had a very interesting presentation style that felt almost like a conversation between them two, and it worked very well. It was hands-down the best session of GDC for me. The talk covered things like why you want to prototype, what constitutes a good prototype, how you go about prototyping things, what to prototype and what to leave out, how to put the results together, etc, etc.

I don’t really understand why it was classified under the game design track. It really was more of a programming, production, or just all-around session. Even so, apparently the session was full and they had to turn people away. I really hope that they recorded it on video and they make it available online.

At one point they described what they considered to be a good codebase for prototyping. The funny thing is, I completely agree with every point they brought up, but for me those are characteristics I want in any codebase, not just one for prototyping. They are all things I value very much: ability to change, flexible, code vs. content, use of scripting, etc. In particular, the comment of frameworks vs. toolsets really hit the mark. That’s something I’ve been pitching for a while, but I really consider frameworks to be fundamentally broken for the type of development I want to do. Instead, I want a set of tools (functions, classes, whatever) that I can put together in any way I want instead of being constrained to one predefined structure (even if it has hundreds of callbacks, like MFC).

I think I was the only person in the packed Civic Auditorium left cold by the Nintendo keynote. Yes, it’s great to have Satoru Iwata give a GDC keynote, but I really didn’t connect with his speaking style or his message. I found it content-free and not very engaging (it didn’t help any that I was stuck in a corner of the top balcony—apparently there was a screen in the middle of the stage that I was not able to see that explained a lot of his strange comments).

On the other hand, Will Wright’s keynote was just the bomb. Somehow every year he manages to deliver amazing talk after amazing talk. I knew I really wanted to hear his session, so I chose to stay in the auditorium instead of going out again to get a free DS game and have to wait through that whole line. Good choice! I was able to get the best seat in the house this time around.

Will’s keynote cannot really be described in one paragraph. People have asked me what it was about, and I’m left without words. The point is not what it was about, it was the whole experience! It’s like being 12 years old again and going to a summer blockbuster that hits you from the beginning and doesn’t let you go for the next two hours. I left the auditorium dazzled and super-motivated. Isn’t that enough?

On the surface, the talk was about how he goes about researching new projects, interleaved with what he learned about astrobiology for the development of Spore. You can try getting more of an idea what it was about here and here (and a short video here). That’s another talk I hope GDC puts online very soon.

I have to admit, that after Will’s keynote, the rest of GDC was fairly anticlimactic. Maybe next time they should leave it for the last day to build up the grand finale. The only session that I attended that really stands out was Tim Moss’ God of War talk. I was expecting something a bit different, but it turned out to be a mini-postmortem of God of War. In particular, he talked about how they were organized, what their priorities were, and how they went about solving problems. I found it very interesting on many different levels, but the most interesting part was that they do things very differently to how I would go about doing them (or how we’re doing them at work right now). They also have a very different set of constraints: they have a very small number of very senior programmers, so they do anything they can to minimize using their time. They end up developing very general solutions, while right now I would advocate for the very specific solutions. It’s clear that their approach worked very well for them, so it’s great to see how different teams can tackle different problems in a variety of ways.

One pattern I noticed in this year’s GDC is that, for technical talks, the more general the talk, the more I enjoyed it. As soon as they got bogged down in details, they became much less effective. A lot of it has to do with the dry nature of the topics, and the fact that I can get all those details from a well-written paper. On the other hand, the more general talks (advanced prototyping, Will’s keynote, or the God of War one) were all very motivational and inspirational. In the end of the day, that’s what GDC is about for me: inspiration and socializing. And this year’s GDC was a huge success by that measure.

I had been using a modified version of CppUnitLite for quite a while, slowly fixing the parts in need of mending, and adding new functionality as it became needed. Eventually I released most of those changes as CppUnitLite2, and I thought that would be the end of that. It turns out that was just the beginning.

Shortly after that, several people started emailing me with new functionality, fixes, patches, suggestions, etc. At the same time, Charles Nicholson was starting to get quite a bit of response as well for his own testing framework, sometimes with similar fixes and suggestions.

If this was going to grow to be a real project, developing over time, supporting users’ needs, and accepting code contributions, clearly it would make sense to join forces. And that’s exactly what we did. We grabbed the best features of each framework and created what we think it’s the best C++ unit-testing framework out there (for our needs anyway).

We took the results and put them up in Sourceforge under a very unrestrictive license, and that’s how UnitTest++ was born.

Our ultimate goal is to apply test-driven development to game development. All of the existing frameworks fell short in one area or another. Specifically, the driving forces behind the design of UnitTest++ are:

• Portability. As game developers, we need to write tests for a variety of platforms, most of which are not supported by normal software packages (all the game consoles). So the ability to easily port the framework to a new platform was very important.
• Simplicity. The simpler the framework, the easier it is to add new features or adapt it to meet new needs, especially in very limited platforms.
• Development speed. Writing and running tests should be as fast and straightforward as possible. We’re going to be running many tests hundreds of times per day, so running the tests should be fast and the results well integrated with the workflow.

UnitTest++ is a fully featured testing framework, with many of the features you may expect from Xunit-style frameworks such as fixtures and reporting. Additionally, here are some of the specific features that make UnitTest++ unique:

• Minimal work required to create a new test. For TDD development, we write lots and lots of tests, so creating a new test should be as quick and painless as possible. UnitTest++ does not require explicit test registrations, and creating new tests requires minimal work.
• Good assert and crash handling. When automated tests are executed, it’s very important not to hang the process or abort the tests any time the program crashes or an assert fails. UnitTest++ will catch and report exceptions as failed tests and print a lot of information about them. It will translate signals to exceptions as well as be able to catch invalid memory accesses and other problems.
• Minimal footprint and minimal reliance on heavy libraries. When you intend to run tests on a Cell SPU with a measly 256 KB of RAM, you really want to be as lean and mean as possible. UnitTest++ is very lightweight, and it will get even lighter weight once strstream is replaced.
• No dynamic memory allocations done by the framework, which makes it much easier to track memory leaks and generally more attractive for embedded systems.
• Multiplatform support. Right now it supports Win32, Linux, and Mac OS X out of the “box,” and other platforms will probably work without any changes. The source code makes use of platform-specific functions whenever necessary, so it is easy to hook up special timers, allocators, or reporters for specific platforms. The code comes with a makefile as well as with project and solution files for Visual Studio 2003 and 2005.
• Full set of unit tests for itself. UnitTest++ was TDD’d from the ground up, including some of the ugly macros (imagine how much fun it was bending the framework to test itself that way!). It goes without saying, all the tests are included with the source code, so you can go totally wild with them :-)

This was just the first release. UnitTest++ is ready for full production work, but we already have plans beyond this release. This is a peek at some of the features we have in mind for upcoming versions:

• Out-of-the-box support for game console platforms (Xbox 360, PS3 PPU and SPU, and Nintendo Revolution). Of course, this is dependent on us getting permission from the console manufacturers to release some code that works with their SDK.
• Ability to trade some features (such as rich check reporting) for some extra memory. In some platforms, like Cell SPUs, you really need to go barebones.
• Suites to group tests together.
• Different reporters (HTML, GUIs, etc). Don’t worry, all those modules will be optional and well separated, so they won’t affect the simplicity of the framework.
• Per-test timings. Maybe the ability to flag some tests as not exceeding a certain amount of time, or just failing any test that goes over a threshold.
• Built-in memory leak detection.

So that’s it. Download the source code and give it a try. We welcome any comments and suggestions (best channels would be through the project mailing list or comments here). We hope you find it useful and that it makes your TDDing even more productive.

See how doing something so apparently backwards as writing unit tests before any code can help with all those problems.

This paper will be presented at the 2006 Game Developers Conference

Noel Llopis and Sean Houghton High Moon Studios

Printer-friendly format. Presentation slides. Sample code from presentation.

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. Frederick P. Brooks, Jr.

1. Introduction

Part of the appeal of programming is that, as Brooks describes, it allows us to build ornate castles in the air, where our imagination is the only limit. As our target platforms increase in power and memory, our castles become larger and more intricate.

However, it’s the same facility to weave code out of thin air that can become our greatest danger. Code has a tendency to grow beyond initial expectations, quickly surpassing our capacity to fully understand it. Our castles in the air can quickly become snowballs that are too large to be moved and shaped to fit our needs.

We have all experienced how development slows down to a crawl towards the end of a project. We have seen first-hand the difficulty of squashing insidious many-headed bugs. We have wrestled with somebody else’s code, just to give up or fully re-write it in despair. We have sat in frustration, unable to do any work for several hours while the game build is broken. We have seen countless hours of work go up in smoke as code is thrown away and started from scratch for the next project.

Code can get too complex for its own good. Milestone pressures, a fluctuating game industry, growing teams and budgets, and the breakneck pace of hardware change don’t help an already difficult situation.

This is where test-driven development comes in.

2. What is test-driven development?

Let’s start by looking at the traditional programming process. Once you decide to implement a specific piece of functionality, you break the problem down mentally into smaller problems, and you start implementing each of them. You write code, and you compile it to know whether things are going well. After a while, you might have written enough code that you can run the game and try to see if the feature works. Maybe you run through a level, checking that feature and making sure nothing else looks obviously broken, or maybe you even step in the debugger to make sure your code is getting called and doing what it’s supposed do. Once you’re happy with it, you check it into source control and move on to another task.

Test-driven development (TDD) turns the programming process around: As before, you break down a problem mentally into smaller problems (or at least a single smaller problem), but now you write a unit test for that small feature, see it fail (since you still haven’t implemented anything), and then write the code to make that test pass. Then you repeat the process with another, very small piece of functionality that will get you closer to the full feature you’re trying to implement. The cycles are very short, perhaps only a minute or two.

Let’s have a more detailed look at the TDD cycle:

• Write a single unit test for a very small piece of functionality.
• Run it and see it fail (in C++, it’s likely that it won’t even compile).
• Write the simplest amount of code that will make the test compile and pass.
• Refactor the code and/or tests. Run tests and see them pass.

A unit test is a test that verifies a single, small behavior of the system (also called a developer test). Specifically, in this context, each unit test deals with something almost trivially small, which can probably be done in less than a minute. For example, it can test that a health pack doesn’t add more health past the player’s maximum, or it can test that a particular effect disables depth writes. We’ll see an example of a simple test in the next section.

How does TDD help us deal with the complexity of software, and, more specifically, how does it solve some of the problems listed in the introduction? These are the benefits, listed roughly in our order of importance:

Better code design

What is the first thing you do when writing some code with TDD? You are forced to use that code in a test. That simple fact makes it so all code is created with the user of the code in mind, not the implementation details. The resulting code is much easier to work with as a result, and it directly solves the problems it was intended to fix. Also, because the code was first created by testing it in isolation from the rest of the code, you will end up with a much more modular, simpler design.

Safety net

With TDD, just about every bit of code has some associated tests with it. That means you can be merciless about refactoring your code, and you can still be confident that it will work if all the tests continue to run. Even though we find refactoring to be an extremely important part of the development process, the safety net goes way beyond refactoring. You can confidently apply obscure performance optimizations to squeeze that last bit of performance out of the hardware and know that nothing is broken. Similarly, changing functionality or adding new feature towards the end of the development cycle suddenly becomes a lot less risky and scary.

Instant feedback

The unit tests provide you with instant feedback up to several times per minute. If at any point you thought tests should be passing and they aren’t, you know something has gone wrong: not an hour ago, not ten minutes ago, but sometime in the last minute. In the worst case, you can just revert your changes and start over. This is quite subjective, but the constant feedback has a surprising morale-boosting effect. No problem seems too large as long as you’re taking small steps in its direction and getting feedback that you’re in the correct track. For some of us, TDD has rekindled the same joy of programming that we discovered in the 8-bit computers we cut our teeth on.

Documentation

What’s worse than uncommented code? Code with outdated comments. We all know how easy it is for comments to get out of date, yet there is very little we can do about it. The unit tests created through TDD serve as a very effective form of documentation. You can browse through them to see what kind of use a class is intended to have, you can look up a particular test to see what kind of assumptions a function makes about its parameters, or you can even comment out some code that makes no sense and see what tests break to give an idea of what it does. The best thing about unit tests as documentation: they can never get out of date. We have found that in codebases that are developed with TDD, comments have almost disappeared, being used only to explain why something was done in a particular way, or to document what paper an algorithm we implemented came from.

One thing that should be clear and is important to stress is that TDD is not just writing unit tests. TDD is a development methodology, not a testing one. That’s why TDD’s benefits deal with better code design and structure, ease of refactoring, etc, and not with correctness. TDD ensures that your code does whatever you wanted it to do, not that it does it correctly.

3. Implementing TDD

Let’s put TDD in practice by following the normal practice and writing our first test. Let’s assume we already have a game up and running, and our task is to implement health powerups. The very first thing we want to implement is that if the pl ayer walks over a health powerup, he receives some amount of health. That’s actually even more complicated than what we want for our small testing steps, so let’s start with an even simpler test that requires the least amount of effort: if we have a player and a powerup, and the player isn’t anywhere near the powerup, his amount of health doesn’t change. Simple, right?

Ideally, how would we like to test that? Probably by writing some code like this:

World world; 
const initialHealth = 60; 
Player player(initialHealth); 
world.Add(&player, Transform(AxisY, 0, Vector3(10,0,10))); 
HealthPowerup powerup; 
world.Add(&powerup, Transform(AxisY, 0, Vector3(-10,0,20))); 
world.Update(0.1f); 
CHECK_EQUAL(initialHealth, player.GetHealth());

And that’s exactly what the test will look like, only we’ll have to surround it with a macro to take care of all the bookkeeping and to give it a descriptive name. Our full test looks like this:

 
TEST (PlayersHealthDoesNotIncreaseWhileFarFromHealthPowerup) 
{ 
    World world; 
    const initialHealth = 60; 
    Player player(initialHealth); 
    world.Add(&player, Transform(AxisY, 0, Vector3(10,0,10))); 
    HealthPowerup powerup; 
    world.Add(&powerup, Transform(AxisY, 0, Vector3(-10,0,20))); 
    world.Update(0.1f); CHECK_EQUAL(initialHealth, player.GetHealth()); 
}

The macros TEST and CHECK_EQUAL are part of a unit testing framework. The framework is intended to make the task for writing and running unit tests as simple as possible. As you can see, it can’t get much easier than that. There are a variety of freely-available unit-test frameworks for C++ and other languages. We recommend UnitTest++, which is a lightweight C++ unit-test framework that supports multiple platforms, is easy to adapt and port, and was created after using TDD in games for several years.

Now we compile this code and we get the following output:

 
Running unit tests... 
1 tests run There were no test failures. 
Test time: 0 seconds.

The CHECK_EQUAL macro is the part that performs the actual check, and it takes as parameters the expected value and the actual value. If they are different, it will mark the test as failed, and output a message with as much information as possible. As a bonus, UnitTest++ formats the failed test message so it can be parsed by Visual Studio and it is easy to navigate to the location of the test.

The reason we have macros like CHECK_EQUAL is that unit tests need to be fully automated. There should be no need for visual inspection or reading some text. The test program should know whether any tests failed or not without any ambiguity, that way it can be easily integrated into the build process.

4. How to test

When writing unit tests, there are three main ways of testing your code:

Return value

Make a function call, and check the return value. This is the most direct way of testing, and it works great for functions that do computations and return the computed value. Because this is the easiest and most straightforward way of testing, we should use this approach whenever possible.

For example, a function called GetNearestEnemy() would be a perfect candidate to be tested this way. We can easily set up a world with a couple of enemies, call that function, and verify that it returns the enemy entity we expected.

Beware of testing functions that return boolean results indicating if the function failed or succeeded. You really want to test that the function does things correctly, not just that it reports it did them.

Object state

Make a function call, then check that the state of the object or some part of the system has changed correctly. This tests that state changes directly, so it’s also a very straightforward form of testing.

For example, we could send an event of nearby noise to an AI in “idle” state and check that its state changes to “alert” in response.

Sometimes it might force you to expose some state that would otherwise be private, but if it’s something that you need to test from the user point of view, then making it public is probably not a bad idea anyway. It might lead to larger interfaces than absolutely necessary, but it also sometimes shows that a class really should be split into two, and one of the classes should contain the other one.

Object interaction

This is the trickiest aspect to test. Here we make a function call, and we want to test that the object under test did a sequence of actions with other objects. We don’t really care about state, just that a certain number of function calls happened. The most common testing pattern for this situation is a mock object. A mock object is an object that implements the interface of another object, but its only purpose is to help with the test. For example, a mock object could record what functions were called and what values were passed, or could be set up to return a set of fixed values when one of its functions is called.

Because this is the most complex form of testing, we recommend using it only when the other two forms of testing are not possible. Using mock objects frequently could be an indication that the code relies too much on heavy objects with complex interactions instead of many, loosely-coupled, simpler objects.

A good situation for using a mock object could be testing HUD rendering. We want to verify that the HUD elements are rendered in a specific order, so we create a mock for the rendering canvas which will keep an ordered list of the elements that get passed to it. We pass the mocked rendering canvas to the HUD renderer and make the render call. Then we can examine that the list in our mocked canvas matches our expectations.

5. Best practices

This is a set of best practices we have found to be particularly important for all of our development with TDD.

Run tests frequently

Being able to write tests easily is a requirement for successfully rolling out TDD. Nobody wants to spend extra time writing unit tests when they can be implementing the features that are due for this milestone. But even more importantly, tests should run very frequently, and a failed test should be treated the same way as a failed build.

At High Moon, every library has a separate test project that creates an executable that links with the library and runs all the tests. We have hooked up running the executable itself as the post-build step in Visual Studio (or as the last command in a make file). That means that making any changes to the library or tests and triggering a compile will also run all the tests. Additionally, since the test executable returns a value with the number of failed tests, a failing test will return a non-zero value, which is interpreted by the build chain as a failed build. This makes it so everybody is running unit tests for all code all the time, which greatly improves the build stability.

In addition to running tests locally, the build server also runs all the unit tests at the same time it does code builds. In our case, since the tests are a postbuild step, there was nothing special we had to do in the build server, and a failed test would be reported just as code that didn’t compile correctly.

Test only code under test

This is a key practice for writing good unit tests. The most important reason to minimize the amount of code involved in a test is to keep things simple. Ideally, when something breaks, you want only a small number of tests failing so you can quickly pinpoint the problem. If some tests involve a large amount of the codebase, they will constantly break for unrelated reasons.

Also, if you are able to use the code under test with a minimal amount of other code or libraries, it means you are creating very modular, self-contained code, which will help with the overall design. Finally, another very good reason to avoid involving extra code is that tests that only work on a small set of code are typically much faster than tests that involve large systems and complex initialization/shutdown sequences.

Continuing with the health powerup example, notice that the test involved a World object, a Player object, and a HealthPowerup object. There was no graphics system initialization, databases, etc. The World object can be a very lightweight container for game entities without any other dependencies. Try setting something like that in your current engine and you might be surprised by how many implicit dependencies you find in different parts of the engine.

A test that involves a lot of code across many different systems is usually referred to as a functional test(also called a customer test). Functional tests are extremely useful, especially if they are fully automated, but they fill a very different role than unit tests.

Keep tests simple

This is related to the previous best practice. You want a test to check one thing and only one thing; that way, when something breaks, it’s immediately clear what went wrong.

A good start is to label each test clearly with a name that describes exactly what the test is supposed to do. For example, a test named PlayerHealth is not very helpful. A much better name would be PlayerHealthGoesUpWhenRunningOverHealthPowerup.

Keeping each unit test to just a handful of lines makes it much easier to understand at a glance. In our case, it is unusual to have unit tests that are more than 15 lines long. Needing to write overly long unit tests is usually a sign that the test is involving too much code and could probably be re-written in a better way.

We also found that limiting each unit test to a single check statement or two resulted in tests that were the easiest to understand. Sometimes that means you’ll have to write some duplicate setup code between two tests, but it’s a small price to pay to keep the tests as simple as possible.

Whenever you have some common code in two or more tests, you can use a fixture. A fixture is a set of common code that is executed before and after each test that uses that fixture. Using fixtures can cut down tremendously on the amount of test code you have to write. Any decent unit test framework should support fixtures, so use them whenever they’re needed.

Keep tests independent

Unit tests should be completely independent of each other. Creating an object in a test and reusing it in a different test is asking for trouble for the same reasons as we discussed earlier: when a test fails, you want to be able to zero in on the failing test and the problem that is causing it to fail. Having tests depend on each other will cause chains of failing tests, making it difficult to track the problem down to the source.

Keep tests very fast

The unit tests we write are compiled and executed as a postbuild step. We have hundreds or thousands of tests per library, so that means they have to run blazingly fast or they’ll get in the way. If unit tests are only dealing with the minimum amount of code, they should be able to run really fast. That means they shouldn’t be talking to the hardware, they shouldn’t be initializing and shutting down expensive systems, and they most definitely should not be doing any file I/O.

All our unit tests are timed (another feature of UnitTest++), and the overall time for the test run is printed after it runs. As a general rule, whenever a set of unit tests goes over two seconds, it means something is wrong and we try to fix it (even if a unit test takes a full ms, which is a huge amount for a unit test, you can still run 1000 of them in a single second).

6. TDD and game development

Applying TDD to game development has its own unique challenges that are not usually discussed in the TDD literature.

Different platforms

Most game developers today need to develop for a variety of platforms: PCs (Windows, Macs, or Linux), game consoles, handhelds, etc. Even though at High Moon we develop console games, we use Windows as our primary development environment because of the good development tools and the fast iteration time. That means we always have a version of our engine and tools that runs under Windows, which makes running unit tests very easy and convenient.

We also wanted to run the unit tests in each platform we develop for, but we had to make a few changes to compensate for the shortcomings of those platforms. The minimum support that we needed was the ability to run an executable through the command line and capture the output and, ideally, the return code. Surprisingly, none of the game console environments we develop for supports that simple operation, so we were forced to write a small set of programs using the system API to do exactly that.

After we wrote those small utility programs, we were able to run unit tests on the consoles just like we did under Windows. The main difference was that running any program on the consoles had a noticeable startup time delay. It’s a matter of just a few seconds, but it was too long to run the unit tests automatically as a postbuild step after every compilation. Besides, we don’t yet have one dev kit for every developer station, so we couldn’t count on always having a target platform available. Because of that, our unit tests on platforms other than Windows are only executed manually, and by the build server. It is not ideal, but the large majority of the problems show up in the Windows tests, so as long as those continue being run all the time, we catch most problems on time.

Graphics, middleware, and other APIs

Probably the biggest barrier that people see to doing unit testing and TDD with games is how to deal with graphics. It’s certainly not the textbook-perfect example you’ll read about in TDD books, but it’s certainly possible.

The first thing to realize is that graphics are just part of a game. It is common sense and a good software engineering practice to keep all the graphics-related code in one library or module, and make the rest of the game independent of the platform graphics API and hardware. Once we had that organization, we were able to test any part of the game or engine without having to worry about graphics.

Ideally, we wanted to develop the graphics renderer library with TDD as well, and there’s no way to avoid dealing with platform-specific graphics calls. We tried three different approaches, from most involved to least involved:

• Catch all graphics function calls. We inserted a layer between the graphics renderer and the graphics API that exactly mimicked the platform API. That way we could make any graphics API calls that we needed without having to worry about the underlying hardware. As a bonus, that layer could report which functions were called and with what parameters. This approach made for extremely thorough tests. The testing layer could be removed in the final build and the functions would call directly into the graphics API so there would be no performance penalties. However, it was quite labor-intensive, especially for the Direct3D API because it uses classes and not just plain functions like OpenGL.
• Check for state. Another approach is to actually work on the graphics hardware and check that things are working correctly by querying the graphics AP I state. For example, rendering a certain mesh should set a specific vertex declaration. This approach is much simpler, but there were some things we couldn’t test. For example, we could render a mesh, but we wouldn’t be able to find out how many triangles were sent to the hardware. Also, since OpenGL is so state-based, it was important for most functions to clean up after themselves, so it was often very hard to check for any state. On platforms in which the internals of the graphics API are more open, you might be able to examine more states by looking at the graphics command buffer.
• Isolate graphics calls. When all else fails, this is a useful technique for dealing with calls into external APIs. Isolate an API function call or a group of API function calls into a single function, and test that your function is called at the right time. The function that calls into the graphics API itself won’t be tested, but all it does it make some straight calls. Remember, what we really want to test is that our code behaves correctly, not that the graphics API works as documented.

We started with the first approach, wrapping the API as we were using it, but the amount of extra work we had to do to wrap complex APIs like Direct3D and OpenGL quickly became overwhelming. Maybe if we were working on a commercial graphics rendering middleware it would be worth the effort. For us, after doing that for a few weeks, we realized that the benefits we were deriving from it weren’t worth the time we were spending. It was also discouraging us from using new API functions just because they hadn’t been wrapped before.

In the end, we ended up settling for a combination for the second and third approaches: testing the state whenever possible, and isolating the calls the rest of the time. One of the drawbacks of this approach is that we’re working directly with the hardware, so tests actually do initialize and shutdown the graphics system every time (except for those platforms out there that can only initialize the graphics system once), so they can be more time consuming. On the positive side, because the tests are exercising the graphics API and hardware directly, we catch things that the first approach wouldn’t have caught (for example, sending incorrect parameters to a function).

The same three approaches can be used for any middleware or external API. The important thing to remember is to make sure you’re testing your code, not the API itself. Keeping that in mind helps to write true unit tests and not functional tests for the API.

A good example of this approach is how we wrote our input system with TDD. The input system deals with getting input values from gamepads and other controllers. Even though at first glance it might seem like the whole system is about making system calls to poll the data, there is a lot of common code that is totally platform independent: button mappings, edge detection, filtering, plugging/unplugging controllers, etc.

In our system, we have an interface named GameController with a Sample() function. Each platform implements a platform-specific version of the GameController (for example D3DController) and does the raw sampling of all buttons and axis through platform-specific API calls. That part is fully untested. The rest of the input system works through the GameController interface, but for all the tests we provide a MockGameController() which allows us to control what input values we feed to the tests.

We are hoping that as TDD becomes more common in the games industry, middleware providers will make their APIs more TDD friendly and even ship with their unit tests.

Third-party game engines

If dealing with APIs was not straightforward, working with a full third-party game engine that was not developed with TDD is even more challenging. After all, an API is just a list of classes or functions, and you have a lot of control over how and when they get called. Working with a full game engine, you might end up writing a small module that is fully surrounded by the engine code. If the engine was not developed with TDD, it can be very difficult to use your code in isolation or figure out how to break things up so they can be tested.

At High Moon, we are working on several projects with the Unreal Engine 3, and we’re using TDD on them. Even though the engine is not TDD-friendly at all, we still find more benefit from doing TDD with it. Imagine then how useful TDD can be on a full engine developed with TDD from the start.

The most important thing to do in this situation is to separate the code we write as much as possible from the engine code. Not only does this allow us to unit-test our code in isolation, but we also keep future merges with Epic’s codebase as simple as possible. However, that severely limits the amount of large-scale refactorings we can do to keep things as testable as possible, so it’s a tough trade-off..

One unique aspect of the Unreal Engine is that it makes heavy use of its scripting language, UnrealScript. Because a lot of the high-level game engine is written in UnrealScript, we had little choice but to use it to write most of the game code. The first thing we had to do was to create a unit-testing framework for UnrealScript. The resulting framework is called UnUnit and is freely available for download through UDN (Unreal Developer Network). Several companies working with the Unreal Engine are currently using UnUnit for their game projects.

UnrealScript is a surprisingly good language for unit testing. By default, all functions are virtual, so overriding behaviors and creating mocks is simpler than C++. Also, compilation is very fast, which keeps iteration times low (large, badly organized C++ codebases can take a long time to link). All of that makes TDD with UnrealScript not just possible, but very effective.

Randomness and games

Most games involve a fair amount of randomness: the next footstep sound you play can be any one of a set of sounds, the next particle emitted has a random speed between a minimum and a maximum, etc. As a general rule, you want to remove the randomness from your tests. A few times we caught ourselves starting to write a unit test that called a function many times in a loop and then averaged the result, but that’s totally the wrong way to go. Unit tests are supposed to be simple and fast, so any loops in a test are usually very suspect.

A better approach is to separate the random decision from the code that uses it. For example, instead of just having a PlayFootstep() function that takes care of computing a random footstep and then playing it, we can break it into int ComputeNextFootstep() which is just a random function call, and a PlayFootstep(int index), which we can now test very easily.

Another approach is to take control over the random number generator at the beginning of the test and rig the output so we know what sequence of numbers is going to come up. We can do this either by mocking the random number generator object or by setting some global state on the random number generator, depending on how it is implemented. Then tests can count on a specific sequence of numbers being generated and check the results accordingly.

High-level game scripts

How far is it worth taking TDD? Should TDD be used for every single line of code? How about game-specific script code? The answer depends on your priorities and game.

As a general rule, we find that if any other part of the game is going to depend on the code we are writing, then it’s probably worth doing it with TDD and having a full set of unit tests for it. Otherwise, if it’s a one-shot deal with the highest-level code, then it’s probably fine without TDD. Also, code at that level is often writ ten by designers in a game-specific scripting language, so TDD might not be a viable option.

An example of some code that we would not use TDD for is trigger code: when the player goes around the corner, wake up two AIs and trigger a different background music.

Functional tests are a great complement to unit tests, so it’s important not to forget about them. Automated functional tests can be extremely useful catching high-level problems, gathering performance and memory utilization data, and removing some of the mechanical testing from QA and letting them concentrate on issues such as gameplay balance and flow.

7. Lessons learned

Design and TDD

One of the most important benefits we get from TDD is the better code design that it creates. For this to happen, it’s important to let the tests guide the code and not the other way around. It can be disconcerting to always be looking only a few minutes into the future and implementing the “simplest thing that could possibly work,” but it really works. The key to working with TDD is to realize that refactoring is an integral part of the development process and should happen regularly every few tests. Good design will come up through those refactorings as needed by the tests we have written and the code we have implemented.

Does this mean you shouldn’t do any design ahead of time? That depends on your situation and experience, and the type of code you’re writing. In general, the less known or more likely to change something is, the less design we do. If you’re working on the 10th iteration of a well-known sports game franchise, for a known platform, and you know exactly what you’re going to do, up front design can be more beneficial.

In our case, we prefer to discuss very rough concepts of where we expect something to go, what it should do in the future, and maybe some very, very rough organizational structure. In general, we prefer not to even think of classes or draw UML diagrams on whiteboards before starting, because such discussions can then lead implementation too much. We prefer to let the tests guide us, and if at some point we’re going away from what we had in mind at the beginning, we can stop to reconsider if we’re heading in the right direction. Most of the time we are, and we simply hadn’t thought things through enough at the beginning (or thought through them too much and we simply didn’t need that level of flexibility).

TDD and high-level code

One of the questions we had when we jumped into TDD is whether it was going to hold for high-level code. We had seen in practice from previous projects that we can certainly do TDD to create low-level and intermediate-level libraries (math, collision, messaging, etc). But would it really work for high-level code that would build on low-level code?

The answer is an unconditional yes. We have developed a full codebase doing TDD from the start, and we had no difficulty writing high-level code with TDD. Things like character state machines, game flow, or specific game entities were done through TDD without any problems, and greatly benefited from the TDD approach.

Two tips that helped us apply TDD to high-level code:

• Keep tests as true unit tests. When working on high-level code, it can be tempting to let tests degenerate into functional tests that involve the whole game engine. Don’t do that. Even if it takes a few more minutes to make an enemy character testable in isolation, it is well worth it in the long run.
• Keep the engine architecture as flat as possible. This is a general good practice, but it is more so with TDD. Clearly, avoid having your engine as one big, intertwined module. But even if it’s broken down into libraries or modules, keep them as independent of each other as possible instead of as one long list of dependencies. For example, there’s no reason the AI module needs to know anything about graphics or sound, but it will need to know about a world representation and have access to the messaging system.

Tests as a measure of progress

Software developers have been struggling for a long time to find a good measure of progress or work done. Some projects use code line counts to determine progress, others use features completed, while others just look at the number of hours spent at the office. Clearly, none of those approaches are ideal.

We have found that counting the number of unit tests is a really good measure of progress. Especially when tests are developed through TDD, they deal less with corner cases and more with program features. If a library has 500 tests associated with it, we can say that it’s roughly half as complex as a library with 1000 tests.

As an added benefit, people tend to behave based on how they think they are being measured. So if this is made clear, people will be much more likely to be strict about applying TDD and not falling back on writing code without tests. In our groups, we have a test chart, which is updated every day and shows the total count of tests. If tests aren’t going up, or they’re going up more slowly than other times, we know something is wrong and we’re not making much progress.

Build stability

As we expected, having almost full code coverage with unit tests greatly improves the code stability. Broken builds happen much less frequently, and they’re usually caused by a missing file or a different platform not compiling correctly. What was a pleasant surprise is that broken builds are much easier to fix. By following our best practices for unit tests, it becomes really obvious when something breaks, and we can usually check in a fix right away. Builds are rarely broken for more than a few minutes at a time, which can completely change how you organize your code in source control.

Amount of code

It will probably come as no surprise to anybody that writing code with TDD results in more code being written. Sometimes the test code is as large as the code under test itself. Even though this can initially sound like a scary proposition, it really isn’t a problem. The extra code does not take significantly longer to write initially, and it allows us to move a lot faster once we have accumulated more code and we need to refactor or optimize the code in any way.

Just because we have twice as much code it doesn’t mean we have twice the complexity. Quite the contrary. The test code has been written so it’s extremely simple, so its complexity is minimal. Its only job is to check the non-test code, so it’s keeping an eye out for us, helping us, and letting us know when something breaks. Additionally, the TDD approach results in much simpler, decoupled code. Overall, the complexity of the codebase is greatly reduced with TDD.

A good analogy to TDD is scaffolding in a building construction. It’s not something you’re going to deliver to your customer, but it’s an absolute necessity, it needs some time commitment to set it up, and you wouldn’t dream of doing any complex building without it.

Development speed

This is the million-dollar question: Does TDD slow development? We’re not doing TDD because it’s a “good” thing to do, but because we want to ship a better-quality product faster and cheaper than we did before. If TDD doesn’t help with this, then there’s very little point to it (other than keeping programmers happy).

As with other aspects of software development, it is difficult to make an objective study and measure exactly the effects of TDD versus a control group. Things change too much from project to project and team to team. Still, some initial studies have some interesting initial findings (http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf), even if the study was not very rigorous and had a very small sample.

Our experience is that TDD, like any other new development technique, slows development down at the beginning while the team is learning it and becoming familiar with it. Th is can take as long as a couple of months. Once the team is over the hump, and given the right tools and development environment, the impact of TDD on development speed is minimal.

It is possible to simply write code faster than it is to write the tests first, but as soon as that code needs to be refactored, debugged, used by somebody else, or simply gets more complex, any time savings quickly disappear. The larger the team and the more complex the problem, the more TDD saves time in the long run. It’s very important to not fall for the temptation of skipping TDD for a very short-term gain (aka milestone of the month).

Ideally, TDD and refactoring can flatten out the classical cost-of-change over time curve into something like this. Notice the trade-off between slightly slower short-term speed vs. massive gains later on.

Adopting TDD

We started with TDD by first trying it with a small group on a separate project. In our particular case, not only were we doing TDD, but we were doing all the extreme programming practices (pair programming, continuous integration, collective code ownership, etc). When doing this, it can be extremely useful to have somebody on board with previous TDD experience who can help guide the process, set up the environment, and avoid common early mistakes.

Applying it on a small scale initially was very useful in many different ways. It let the team become more comfortable with TDD and get to the point where they are as productive as they were before. It also makes any bad practices apparent early on and they can be dealt with before rolling it out (making overly complex tests or functional, rather than unit tests).

It also had the effect of creating new evangelists for the new technique. Members of that team were then moved on to other teams, where they became the resident TDD expert.

TDD fits best with other agile development practices. Having an agile mindset fits very well with the idea of finding the design through tests. Pair programming can be extremely helpful, especially at the beginning while rolling out TDD and getting everybody on board. Pair programing also has the added advantage of making programmers less likely to skip writing tests, which can be a common reaction early on.

If you’re interested in applying TDD but you don’t have a commitment from your manager or lead, it is possible to start doing it on the side on your assigned tasks. The most important thing is to make sure your tests run automatically (remember the postbuild trick). Slowly, other people might see how useful the tests are, or how much easier it is to refactor that code, and eventually the time might be right to roll out TDD.

A few people can have a hard time switching over to the TDD mentality, but to take full advantage of TDD, it is best to let the design emerge from the tests and the refactoring rather than trying to plan everything up front. It is very interesting to note though, that most programmers at High Moon quickly accepted TDD, and soon became very enthusiastic about it and started using it in all their code, including their home projects.

There is no doubt that the best situation to roll out TDD is with a fresh new codebase. Not everybody has that luxury, so it is important to learn how to apply TDD even with an existing legacy codebase. Michael Feathers’ book Working Effectively with Legacy Code, explains exactly that situation and gives some very good guidelines on how to go about unit testing with a codebase without existing unit tests. Sometimes it just takes a little bit of refactoring of the existing code and it becomes a lot easier to add new tests.

8. Conclusion

Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and we’re convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development.

9. Resources

This is required reading before you start doing any TDD:

• Beck, Kent, Test Driven Development: By Example. Addison-Wesley, 2002

These other books will also come in handy:

• Fowler, Martin, Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999
• Astels, David, Test Driven Development: A Practical Guide. Prentice Hall, 2003
• Beck, Kent, and Cynthia Andres, Extreme Programming Explained: Embrace Change (2nd Edition), Addison-Wesley, 2004

Very useful mailinglists and web sites:

• TestDriven.com
• TDD mailing list

Resources dealing with TDD and agile game development:

• Noel’s blog, Games from Within
• Clinton Keith’s blog, Agile Game Development

C++ unit-testing frameworks:

• UnitTest++. A C++ unit-testing framework designed with game development in mind.
• CppUnit
• Comparison of C++ unit-test frameworks