Trying Out Multisampling On iOS

I only recently broke free of iOS3.x for Flower Garden, so I’m finally adding all the features I had been itching to add that required higher OS support. I had already added some iOS4+ features, but I was keeping them to a minimum because it’s always a huge cause of bugs to target multiple versions of the OS at once.

One of the first features I looked into adding was multisample antialiasing (MSAA) support for OpenGL, which was originally introduced in iOS 4.0. The geometry generated for the petals in Flower Garden is fairly high contrast, and since it’s not like the textures were carefully created and laid out by an artist, the result is pretty bad aliasing around the edges. Perfect candidate for multisampling!I based everything on the Apple documentation on multisampling. It was very straightforward and it works for both off-screen and view-based render targets. It was also extremely helpful to keep both the ability to use regular and multisampled render targets. That way I can easily run performance and visual comparisons, and, if necessary, I can disable multisampling on a particular device.

The main gotcha was making sure I used the GL_RGBA8_OES color format on the multisampling color buffer (otherwise it won’t work, and all you’ll get are 1282 “invalid operation” OpenGL errors). Also, if you’re in the same boat as Flower Garden, which uses OpenGL ES 1.1 and the fixed function pipeline, you’ll have to add _OES to just about every constant in the documentation.

Visual results

I will let the screenshots speak for themselves.

In this screenshot you can see the rendering of the pots using off-screen render targets.

Pretty impressive improvement, even if I say so myself! The improvement is even more striking in-game because the animation of the flowers moving in the wind doesn’t have jaggies popping in and out.

The improvements are more noticeable on the iPad because the pixels are bigger, and, not surprisingly, less so on retina-display iPhones. But even on the retina displays, jaggies are less noticeable during the flower wind-swaying animations.

Performance

So, no doubt: They look pretty. But how about performance? Under the hood, the rasterizer is generating four samples for every pixel, and then combining them at the end in a separate step. So you’ll get more of a performance hit with complex pixel shaders. Edit: My bad. I spaced out and I forgot that MSAA only evaluates the pixel shader once, so performance doesn’t depend on pixel shader complexity. Instead, the performance hit probably comes from the extra writes (four per pixel) to the render target, and should be fairly similar between games.

It turns out that Flower Garden is still using OpenGL ES 1.1, which is implemented using shaders at the driver level. Fortunately, even though I’m using some texture combiner operations and several input textures, those pixel shaders aren’t all that complex.

These are the frames per second I recorded with one particular flower pot on the different devices I have for testing. Notice that the lowest device I have listed is the iPhone 3GS. That’s because I have also stopped supporting arm6 CPUs to keep things simpler (and the market share is minimal).

Update: I was only running the game loop at 30 fps, so the initial performance numbers I had listed are pretty meaningless. Here are the correct numbers running at a max of 60 fps.

Of the devices I tested, MSAA didn’t slow down things much on the iPhone 3GS, and the iPhone 4S was maxed out at 60 fps both ways. The iPad 1 was a different story, and performance crashed from 60 fps to 18 fps. Like Rory pointed in the comments, looking at the actual time instead of the fps gives a better insight. Using MSAA adds 38.9 ms to each frame! The iPhone 4 also suffered from a big performance hit due to all the extra pixels in the retina display. I don’t care how pretty the flowers are, that’s just not acceptable (and no, I’m not adding user-tweakable graphics settings like PC games).

I should also add that Flower Garden is hugely CPU bound. It generates all that geometry every frame on the CPU, and that’s a lot of vector transforms. So if your game is GPU bound, you’re likely to see higher performance hits.

I was initially very surprised to see that the performance on the simulator tanked big time (as in, going from 30 fps to 5 fps). This was news to me, but apparently the iOS simulator runs completely in software, so the quadrupling of pixels brings it to its knees. I’m really, really, impressed at how smoothly it usually runs for being in software though. It had me fooled thinking it was using OpenGL under the hood!

As far as memory goes, after an animated discussion on Twitter, it seems that the iPhone does create full buffers for the multisample frame buffers, so there is a significant increase in memory usage. It’s not something easy to track because it all happens at the driver level, but it’s something to be aware of. Because of this, I might consider turning multisampling off on retina displays, since the improvement is not mind-blowing (and the extra memory is very significant because of the amount of pixels). I’ll also probably turn it off in the simulator so I can achieve a decent frame rate during development.

Conclusions

Multisampling only required adding a few lines of code and resulted in an impressive visual improvement and minimal performance impact in some devices. It’s a no-brainer on the 3GS and iPad 2. I’m wishing I had implemented it earlier!