One of the unintended side effects of my previous post on the horrible memory situation on the iPhone, was that some people pointed out it was possible to hook up a disk storage back end to the iPhone’s virtual memory system. That’s quite ironic because I think of it as compounding the already dismal situation rather than a solution or a even a stopgap measure.

As far as I’m concerned, virtual memory paging is the lazy man’s caching scheme. There, I’ve said it. Now let me qualify it a bit and justify why I feel that way.

A lot of applications, and especially games, can’t keep in memory every bit of data they need for their execution. There’s just too much data and not enough RAM to keep all the levels, all the textures, all the characters, and all the movies. Even the thought of keeping everything in memory is ridiculous. Instead, games are architected to strike a balance between memory usage and responsive interfaces. Usually that means loading levels, or parts of levels on demand, and keeping them in memory while they can be needed. Some other times it means loading levels of detail for textures, meshes, and animations depending on the player position. The point is, nobody knows as well as the game itself what needs to be in memory, what can be unloaded, and when the best time to do it is.

A different approach would be to ignore thinking about managing memory as a scarce resource and use as much of it as we need. If we ever go over the amount of available physical memory, the virtual memory system will kick in and page out memory to disk to make room for the memory we’ve requested. But there lies the problem: The virtual memory system is a lower-level system that knows absolutely nothing about our game or application. It can only make guesses about what memory is OK to evict and when it’s a good time to do it. If you’re unlucky it will choose to page out memory when you need performance the most, and it might evict a page that you’re about to use in a few milliseconds. If that happens, you can kiss your performance bye-bye.

Because of that, virtual memory paging will always do a much worse job than we could have done ourselves. But hey, it takes no effort or thinking on the part of the programmers, hence the “lazy man’s caching scheme” I was referring to earlier.

However, virtual memory paging is sometimes a necessary evil. The assumption running through the previous three paragraphs is that we, as programmers of the game, can always manage the physical memory so our game runs efficiently on it. That is true for certain platforms, usually with fixed hardware specs, and ones with minimal or no background processes running. Game consoles are perfect examples of this: We know how much RAM we have to start, we know how much the system needs, how much video memory there is, and we can deal with the rest ourselves. We’ll decide what to load and when to load it.

On something like a modern PC (whether it’s Windows, Mac, Linux, or any other flavor), we have none of those guarantees. We don’t know how much memory we’re going to encounter when we run our game, and, what’s even worse, we have no idea how that available memory is going to change during the execution of the program [1]. In a situation like that, we can plan for some minimum memory requirements, but we’re going to need that virtual memory paging to bail us out of tricky low-memory situations. It might sound like a good idea, but that’s one of the main reasons why PC games are often choppy and with inconsistent frame rates (buggy drivers being the other main reason).

Finally coming back to the iPhone, how does virtual memory paging fit there? There’s no doubt that it would help because it would reduce the number of crashes due to programs unexpectedly running out of memory. But at the same time, it would cause most games and apps to be choppy and unresponsive, especially when it’s just launched. But the iPhone is mostly a fixed spec platform, with minimal background processes [2], so we can always do better than the dumb virtual memory system. Maybe that’s not a big deal if you’re writing an app that interfaces with a web site and writes data to a database, but it’s crucial to be able to write responsible games, which are considered near real-time apps.

On the iPhone there should be no need for virtual memory paging (not to mention that it would probably drain the battery a lot faster). Instead of solving the memory problem by throwing more and more complex systems at it, what we really need is a guaranteed amount of memory for our app to run and we can take care of the rest. But Apple needs to take the first step and set that memory aside.

We’re waiting, Apple.

[1] We could go totally hard core and allocate the memory we need and then mark it as not swappable by the virtual memory system. That wouldn’t do any good because if other processes start requesting memory and they run out, the virtual memory system will swap out other pages (and possibly cause other processes to thrash) and kill our performance as well. So things are pretty much out of our hands as soon as virtual memory paging comes into play.

[2] There are a lot of background processes, but they should be relatively well-behaved. None of them should be pulling in massive amounts of data while the game is running.

My recent rant about the dismal situation of memory in the iPhone quickly became quite popular. Most people were either unaware of the situation, or completely agreed that it was a major stumbling block for any app that tries to make good use of the hardware.

As a response to that post, some people suggested some intriguing ways to increase the available memory on the iPhone. I’ve been experimenting with that a bit, but I don’t have any conclusive solutions yet. Right now it’s all totally unreliable hacks. Hopefully in a followup post I can present some solutions.

In the meanwhile, I wanted to talk about a piece of the puzzle, which is keeping track of the memory status on the iPhone. How much memory is available? How much memory does the program use? We need to be able to answer those questions accurately in order to do anything about memory. Unfortunately, there doesn’t appear to be a good answer even for this!

Finding Total Memory

Let’s start with the easy part: Finding the overall memory on the device. Right now all iPhones and iPod Touch models have 128MB RAM, so it’s kind of pointless, but it’s interesting anyway. To accomplish that we need to dig into some of the low-level functions that query hardware capabilities. In particular, the function sysctl() reports both physical memory and user memory.

    int mem;
    int mib[2];
    mib[0] = CTL_HW;
    mib[1] = HW_PHYSMEM;
    size_t length = sizeof(mem);
    sysctl(mib, 2, &mem, &length, NULL, 0);
    NSLog(@"Physical memory: %.2fMB", mem/1024.0f/1024.0f);

    mib[1] = HW_USERMEM;
    length = sizeof(mem);
    sysctl(mib, 2, &mem, &length, NULL, 0);
    NSLog(@"User memory: %.2fMB", mem/1024.0f/1024.0f);

The output of that code is:

Physical memory: 116.00MB
User memory: 91.30MB

Interesting. Physical memory is not quite reported as 128MB. That’s probably because the video memory takes up 12MB of the total. The kernel apparently uses 24.7MB, which is not unreasonable if that was all the memory used by the OS. Unfortunately, that’s only for the kernel. The different processes and apps are going to use up more than that. A LOT more than that.

Incidentally, sysctl() is a pretty rocking function. It will give you all sorts of cool information, such as CPU and bus frequency (412MHz and 103MHz respectively for my iPhone 3G), cache sizes, and whether a vector unit is present (it claims it isn’t, so it must not be referring to the vfp).

Finding Available Memory

As cool a function as it is, sysctl() only returns static information about the hardware. If we want to find information about the current status of the memory, we need to look elsewhere.

As far as I can tell, the best function to get this information host_statistics() with HOST_VM_INFO_COUNT to get virtual memory statistics. Querying that function fills out the following structure:

struct vm_statistics {
    natural_t	free_count;		/* # of pages free */
    natural_t	active_count;		/* # of pages active */
    natural_t	inactive_count;		/* # of pages inactive */
    natural_t	wire_count;		/* # of pages wired down */
    natural_t	zero_fill_count;	/* # of zero fill pages */
    natural_t	reactivations;		/* # of pages reactivated */
    natural_t	pageins;		/* # of pageins */
    natural_t	pageouts;		/* # of pageouts */
    natural_t	faults;			/* # of faults */
    natural_t	cow_faults;		/* # of copy-on-writes */
    natural_t	lookups;		/* object cache lookups */
    natural_t	hits;			/* object cache hits */

    /* added for rev1 */
    natural_t	purgeable_count;	/* # of pages purgeable */
    natural_t	purges;			/* # of pages purged */

    /* added for rev2 */
    /*
     * NB: speculative pages are already accounted for in "free_count",
     * so "speculative_count" is the number of "free" pages that are
     * used to hold data that was read speculatively from disk but
     * haven't actually been used by anyone so far.
     */
    natural_t	speculative_count;	/* # of pages speculative */
};

Lots of good info there! (at least for a hardware geek like me). The most interesting one is free_count. That number combined with the page size (which is defined in the handy global variable vm_page_size–which is unsurprisingly 4K on the iPhone) should give us the amount of available memory, right? Right?

Kind of. Unfortunately virtual memory is managed at many different levels. The OS will keep some pages on reserve to use on a rainy day. So it’s possible that there is more memory available than reported in this function. It will however give you a good minimal bound on the amount of free memory. You can count on that amount for sure.

This function puts it together to return the amount of available memory in KB:

int getAvailableMemoryInKB()
{
    vm_statistics_data_t vmStats;
    mach_msg_type_number_t infoCount = HOST_VM_INFO_COUNT;
    kern_return_t kernReturn = host_statistics(mach_host_self(),
                             HOST_VM_INFO, (host_info_t)&vmStats, &infoCount);
    if(kernReturn != KERN_SUCCESS)
        return -1;
    return (vm_page_size * vmStats.free_count) / 1024;
}

To check that available memory roughly matches what I do in the app, I allocate a 1MB block, run it again and… surprise! I get the same amount of memory available. Apparently the OS is waiting to actually set those memory pages aside until you really need them (I hate it when machines try to out-think me). So to really get them to count as allocated, we need to write to every page of the memory we just allocated. The easiest way is just go bzero(mem, 1024*1024);. Running the function again correctly shows that the available memory has gone down by 1MB.

(Not) Making Sense Of Used and Total Memory

An old habit I picked up during my engineering days is to always confirm my calculations through a different path. If the total amount of memory reported by the virtual memory system matched up with the total amount of memory reported by sysctl() I would leave this happy and be able to sleep soundly tonight.

The total amount of virtual memory is calculated by adding together the free, used, wired, and inactive pages. It should come up to be roughly around 91MB. The answer: 89MB. OK, close enough, right? Not really.

Here’s the biggest mystery so far: If I malloc a chunk of memory (and write to it to have it marked as not available), host_statistics() correctly shows that those pages are not available anymore, but they don’t show up as used, wired, or anything else!!! So simply adding free, used, wired, and inactive reports a totally bogus number that doesn’t take into account allocations from your program. What’s going on there? Obviously I’m misunderstanding something, so maybe someone with more knowledge of kernel and vm features can help me out. Otherwise I won’t be getting much sleep I’m afraid 🙂

Just to prove that I’m not crazy, here’s a code snippet and its output:

- (void)updateStatus
{
	vm_statistics_data_t vmStats;
	mach_msg_type_number_t infoCount = HOST_VM_INFO_COUNT;
	host_statistics(mach_host_self(), HOST_VM_INFO, (host_info_t)&vmStats, &infoCount);

	const int totalPages = vmStats.wire_count + vmStats.active_count +
                               vmStats.inactive_count + vmStats.free_count;
	const int availablePages = vmStats.free_count;
	const int activePages = vmStats.active_count;
	const int wiredPages = vmStats.wire_count;
	const int purgeablePages = vmStats.purgeable_count;

	NSMutableString* txt = [[NSMutableString alloc] initWithCapacity:512];
	[txt appendFormat:@"Total: %d (%.2fMB)", totalPages, pagesToMB(totalPages)];
	[txt appendFormat:@"nAvailable: %d (%.2fMB)", availablePages, pagesToMB(availablePages)];
	[txt appendFormat:@"nActive: %d (%.2fMB)", activePages, pagesToMB(activePages)];
	[txt appendFormat:@"nWired: %d (%.2fMB)", wiredPages, pagesToMB(wiredPages)];
	[txt appendFormat:@"nPurgeable: %d (%.2fMB)", purgeablePages, pagesToMB(purgeablePages)];

	NSLog(txt);
	[txt release];
}

- (void)viewDidLoad
{
    [super viewDidLoad];
    [self updateStatus];
    const int sizeInBytes = 1024*1024;
    int* mem = (int*)malloc(sizeInBytes);
    bzero(mem, sizeInBytes);
    [self updateStatus];
}

And the output is:

Total: 22128 (86.44MB)
Available: 8124 (31.73MB)
Active: 6361 (24.85MB)
Wired: 6408 (25.03MB)
Purgeable: 359 (1.40MB)

Total: 21880 (85.47MB)
Available: 7888 (30.81MB)
Active: 6344 (24.78MB)
Wired: 6412 (25.05MB)
Purgeable: 359 (1.40MB)

Notice how the available pages go down by about 1MB but everything else stays about the same?

While we’re at it, does someone know of a better, lower-level way to allocate memory than malloc()? I’d like to just allocate memory pages directly. I looked into vm_allocate() but I wasn’t able to get the correct port rights. Anyone?

Update: Thanks to Colin for pointing out that mmap is the low-level memory allocation function I was looking for. You can allocate pages directly without going through malloc. Perfect!

Next Up…

This is a stepping stone towards the memory experiments I’m running. In the next day or so I should be able to report some good ways to clear up as much memory as possible for our apps.

I love developing for the iPhone. It’s a really fun machine. Small enough to allow very small teams to create great apps, but at the same time powerful enough that you can do some really impressive games. The tools are great, the iteration time is great. It’s was a pleasure all around developing Flower Garden.

Except when it comes to memory, that is.

I really like most of the design calls that Apple made designing the software for the iPhone: Using C and Objective C (instead of Javascript or web-only apps), building it on top of a Unix-like core, and even providing OpenGL (although I would love to have lower-level access to the graphics hardware).

Developing for the iPhone is a mix of PC and game console development. One one side, it’s a fairly open platform with standard APIs (like PCs), and on the other, it has (almost) fixed hardware [1] and a regulated distribution channel (like game consoles). I would even argue that the iPhone borrows the best of both worlds. When it comes to memory, however, the iPhone adopted the worst of both worlds. I feel that Apple completely dropped the ball there. Completely.

On a PC (whether it’s Windows, Mac, Linux, or whatever), you can’t count on having a fixed amount of memory. When your program runs, it can be running in a machine with RAM to spare, or in one with barely enough to run the OS. Or maybe it’s running on a machine with enough RAM but there are fifty other programs running and there’s no memory left for your program.

Fortunately (or unfortunately depending on your point of view), virtual memory allows the operating system to write memory pages back to disk and free up space for your program. Yes, it can cause the program to halt and crawl while memory is being paged out, but it will run. Definitely not an ideal solution for semi-real time applications like games. That’s one of the main causes for choppy, underperforming PC games, but it has worked for many years and will do in a pinch.

On consoles on the other hand, you have a fixed amount of RAM from the start. You know what that amount is, and you can plan for it. In this latest generation of consoles, you still need to give up part of that precious memory to some other processes for user interface and network updates, but even that amount is known ahead of time. You can plan for it, and make your game fit in the memory that you have left.

The iPhone memory situation is… a mess. It’s like a console in that you have a fixed amount of RAM: 128 MB (although I’m sure Apple doesn’t want developers thinking about it so they can roll out iPhones with more memory without affecting existing programs). Because it has a fixed amount of RAM, the iPhone doesn’t provide virtual memory swapping [2]. So far so good. But here’s the kicker: Unlike a game console, the iPhone makes no guarantees about how much memory your application has available when it runs.

Let me say that again: You’re dealing with a fixed-memory environment and you have no way of knowing how much memory will be available to you. Did that sink in yet?

So how are you supposed to deal with that? Apple wants us to be able to dynamically load and unload anything in our programs in response to low-memory warning events. That might work well if you’re loading web pages and just need to unload some cached ones, but that’s far from ideal for games. What are you going to do? Unload part of the level? Get rid of some sounds? Not every game can do that, and even if you could, it would add a huge amount of complexity to something that should be pretty simple.

You can’t even plan on using a fixed minimum amount of memory. From my experiments on my iPhone 3G, it seems that it’s relatively safe to use between 15 to 18MB. Any more than that, and you start getting low memory warnings, and unless you program does something about it, the OS will promptly terminate you. On an iPod Touch, since it’s not running as many background services, you usually have more memory available. I learned that the hard way, because I developed most of Flower Garden on an iPod Touch, only to find out that it was too slow and running out of memory on the iPhone 3G.

And this is out of a total of 128MB. Where did the rest of the memory go? Come on, even with phone, GPS, and music processes in the background, where did the rest of the memory go? Probably the worse culprits are the mail program and Safari, which are left in memory even after you exit back to the springboard (Bad design decision, no cookie!).

And what exactly does it mean “to do something about it” when you get a low-memory warning? You’re supposed to free up as much memory as you can, but is it enough? Who knows! Maybe you free up one megabyte but the OS still decides that you’re taking too much memory and it will kill you anyway. It seems like the worst memory handling scheme ever designed.

As an extreme situation, I have seen as little as 3.5MB available when starting my own program! What are you supposed to do in that case? Most games will just crash back to the springboard. Want to test this? Launch Safari, load 10-15 heavy web pages, quit, and then launch your favorite game. A crash is almost guaranteed. For my next project, I’ll be tempted to respond to a low-memory warning by killing the Safari and Mail processes. That will free up some memory!

To make matters worse, I had the bright idea of combining OpenGL and UIKit for Flower Garden. Some things worked really well, like being able to quickly create some screens with Interface Builder and take advantage of all of Apple’s UI widgets and fonts. Creating the seeds screen or the settings panel was a breeze with UIKit. However, UIKit uses an undetermined amount of memory behind the scenes, which makes all attempts at staying below a certain memory threshold pure black magic. In particular, drawing images to a context apparently causes mysterious caching of bitmaps taking huge performance hits and memory chunks over which we have no control. And game programmers don’t like to feel they have no control over what’s happening!

Another consequence of such a horrible memory situation is that game developers aren’t able to make full use of the resources available. If we’re aiming for 15MB, but have to drop back to 5-8MB, most developers are not going to try to write a game that uses 30 or 40MB (which might be available). What a shame!

I’ve said before that the iPhone very closely resembles the great Sega Dreamcast (which is my favorite game console of all time). The Dreamcast only had 16MB of main RAM and 8MB of video RAM, but the best games on the iPhone, still pale in comparison to the best Dreamcast games. Why? Mostly because of the memory situation. If we had a guaranteed 16MB of RAM and the possibility of using all the extra free memory, iPhone games would look very, very differently.

My plea to Apple: Please, give us some minimum memory guarantees for OS3.0! That, more than anything else, will make the biggest difference in the look and reliability of future games.

[1] There are slight differences between iPhone 2G, 3G, iPod Touch 1st and 2nd gen (especially the latter, which has a faster CPU and faster file system).

[2] It has virtual memory in the sense that there’s a paging system with virtual addressing (so two memory pages far apart in real memory could be made to seem to be continuous), but there is no automatic paging in and out of memory pages to/from disk.