There were even a couple of cases during Casey’s Contraptions that we wanted a particular effect, and couldn’t get it quite right, but it would have been trivial to whip up a shader if we had been using OpenGL ES 2.0. In the end, we had to resort to texture combiners (yuck), and it wasn’t exactly what we had in mind.OpenGL ES 2.0 requires that you write shaders for any different kind of “thing” you render, whether it’s a simple 2D quad, or a fancy skinned character. I wanted to provide at least a basic 2D rendering shader in my graphics library, but the workflow was a bit awkward.

First of all, shaders on iOS are compiled at runtime [1], so we need to access the shader source and compile it. Initially I was thinking of having the shader source files as text in the graphics library, but for the game to access those files, it would have to explicitly add them to its resources so they can be copied during the build phase. That’s error prone (I would always forget to do that with every new project) and just plain ugly.

Another possibility would be to include the source text as a string in the library. That would simplify things, but it would mean that:

1. You would lose shader syntax highlighting.
2. You would need to have the annoying \ at the end of every line.

After asking on Twitter and getting some good suggestions, I decided the easiest way would be to use xxd. It’s a shell tool that comes preinstalled in all Macs (and Linux) that takes a file and converts it into a C array that can be included in a program directly.

I liked xxd even better than the old standby of bin2c because it already comes installed, and it generates the whole file, not just an array that then needs to be included from another file. Yes, it would be easy enough to modify, but why bother when you have a perfect tool for the job already?

Now, I can set up a custom rule in Xcode 4 like this:

Update: The array generated by xxd isn’t null-terminated, so we need to add a zero to the array. The easiest way is to use sed. Thanks to SixEcho Studios for the heads up. Also, I updated it to use the varaibles INPUT_FILE_DIR and INPUT_FILE_NAME thanks to Miguel’s comments.

Now, whenever the shader source changes, it generates a header file like this:

unsigned char PassThrough_fsh[] = {
  0x0a, 0x0a, 0x76, 0x61, 0x72, 0x79, 0x69, 0x6e, 0x67, 0x20, 0x6c, 0x6f,
  0x77, 0x70, 0x20, 0x76, 0x65, 0x63, 0x34, 0x20, 0x44, 0x65, 0x73, 0x74,
  0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6c, 0x6f, 0x72,
  0x3b, 0x0a, 0x0a, 0x76, 0x6f, 0x69, 0x64, 0x20, 0x6d, 0x61, 0x69, 0x6e,
  0x28, 0x29, 0x0a, 0x7b, 0x0a, 0x20, 0x20, 0x20, 0x20, 0x67, 0x6c, 0x5f,
  0x46, 0x72, 0x61, 0x67, 0x43, 0x6f, 0x6c, 0x6f, 0x72, 0x20, 0x3d, 0x20,
  0x44, 0x65, 0x73, 0x74, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x43,
  0x6f, 0x6c, 0x6f, 0x72, 0x3b, 0x0a, 0x7d
,0x00};
unsigned int PassThrough_fsh_len = 91;

To compile that shader in the game, all I have to add is two lines:

#include 
//...
const int fs = ShaderUtils::CompileShader(PassThrough_fsh, GL_FRAGMENT_SHADER);

Done! Clean, simple, and minimal cruft.

Update: I haven’t tried it yet because I haven’t had the need for any expensive shaders, but I hear really good things about Aras Pranckevičius GLSL optimizer. In that case, I would add it before the xxd step. You would still use the shader the same way, but it would be optimized and perform faster.

[1] That’s a huge mistake and I hope Apple fixes that ASAP. Loading times for any amount of non-trivial shaders adds up really quickly. That’s the last thing you want on a mobile device! The OpenGL ES 2.0 standard provides an optional way to load binary shaders, and all that Apple has to do is provide a tool that will compile shaders for the iOS hardware. To be completely future-proof, you they can even mandate that if the binary load fails, you still need to compile from source.

One of the first features I looked into adding was multisample antialiasing (MSAA) support for OpenGL, which was originally introduced in iOS 4.0. The geometry generated for the petals in Flower Garden is fairly high contrast, and since it’s not like the textures were carefully created and laid out by an artist, the result is pretty bad aliasing around the edges. Perfect candidate for multisampling!I based everything on the Apple documentation on multisampling. It was very straightforward and it works for both off-screen and view-based render targets. It was also extremely helpful to keep both the ability to use regular and multisampled render targets. That way I can easily run performance and visual comparisons, and, if necessary, I can disable multisampling on a particular device.

The main gotcha was making sure I used the GL_RGBA8_OES color format on the multisampling color buffer (otherwise it won’t work, and all you’ll get are 1282 “invalid operation” OpenGL errors). Also, if you’re in the same boat as Flower Garden, which uses OpenGL ES 1.1 and the fixed function pipeline, you’ll have to add _OES to just about every constant in the documentation.

Visual results

I will let the screenshots speak for themselves.

In this screenshot you can see the rendering of the pots using off-screen render targets.

Pretty impressive improvement, even if I say so myself! The improvement is even more striking in-game because the animation of the flowers moving in the wind doesn’t have jaggies popping in and out.

The improvements are more noticeable on the iPad because the pixels are bigger, and, not surprisingly, less so on retina-display iPhones. But even on the retina displays, jaggies are less noticeable during the flower wind-swaying animations.

Performance

So, no doubt: They look pretty. But how about performance? Under the hood, the rasterizer is generating four samples for every pixel, and then combining them at the end in a separate step. So you’ll get more of a performance hit with complex pixel shaders. Edit: My bad. I spaced out and I forgot that MSAA only evaluates the pixel shader once, so performance doesn’t depend on pixel shader complexity. Instead, the performance hit probably comes from the extra writes (four per pixel) to the render target, and should be fairly similar between games.

It turns out that Flower Garden is still using OpenGL ES 1.1, which is implemented using shaders at the driver level. Fortunately, even though I’m using some texture combiner operations and several input textures, those pixel shaders aren’t all that complex.

These are the frames per second I recorded with one particular flower pot on the different devices I have for testing. Notice that the lowest device I have listed is the iPhone 3GS. That’s because I have also stopped supporting arm6 CPUs to keep things simpler (and the market share is minimal).

Update: I was only running the game loop at 30 fps, so the initial performance numbers I had listed are pretty meaningless. Here are the correct numbers running at a max of 60 fps.

Of the devices I tested, MSAA didn’t slow down things much on the iPhone 3GS, and the iPhone 4S was maxed out at 60 fps both ways. The iPad 1 was a different story, and performance crashed from 60 fps to 18 fps. Like Rory pointed in the comments, looking at the actual time instead of the fps gives a better insight. Using MSAA adds 38.9 ms to each frame! The iPhone 4 also suffered from a big performance hit due to all the extra pixels in the retina display. I don’t care how pretty the flowers are, that’s just not acceptable (and no, I’m not adding user-tweakable graphics settings like PC games).

I should also add that Flower Garden is hugely CPU bound. It generates all that geometry every frame on the CPU, and that’s a lot of vector transforms. So if your game is GPU bound, you’re likely to see higher performance hits.

I was initially very surprised to see that the performance on the simulator tanked big time (as in, going from 30 fps to 5 fps). This was news to me, but apparently the iOS simulator runs completely in software, so the quadrupling of pixels brings it to its knees. I’m really, really, impressed at how smoothly it usually runs for being in software though. It had me fooled thinking it was using OpenGL under the hood!

As far as memory goes, after an animated discussion on Twitter, it seems that the iPhone does create full buffers for the multisample frame buffers, so there is a significant increase in memory usage. It’s not something easy to track because it all happens at the driver level, but it’s something to be aware of. Because of this, I might consider turning multisampling off on retina displays, since the improvement is not mind-blowing (and the extra memory is very significant because of the amount of pixels). I’ll also probably turn it off in the simulator so I can achieve a decent frame rate during development.

Conclusions

Multisampling only required adding a few lines of code and resulted in an impressive visual improvement and minimal performance impact in some devices. It’s a no-brainer on the 3GS and iPad 2. I’m wishing I had implemented it earlier!

Every so often, something happens that bumps up the priority of one of the items in my list. Maybe it’s another related blog post, or a game coming out with something relevant to what I wanted to write about. In this case it was a tweet from Paul Pridham [1]:

Customizing colors in a sprite or texture is very frequent in games, from changing player characters into blue and red teams, to creating color variations of an armor piece, to letting the player pick the exact shade for their pet’s fur color. Or, in the case of Flower Garden, to change the colors of the petals on the fly.

There are two requirements for this:

• We want to change colors dynamically.
• We only want to affect certain areas of the original texture.

That rules out creating texture variations ahead of time, although that might be a valid approach sometimes if you have lots of art resources, don’t mind increasing the download size, and you have a fixed number of variation to deal with. It also rules out modulating/blending the texture by a particular color because it would tint all the texture, and we want to limit the effect to particular areas (leave the player’s arms their normal color, but change their shirt color).

This is one of those funny cases that it was a lot easier to do many years ago, when we used palletized color modes. You could set all the custom color areas to a particular palette entry, and then update that entry on the fly. Ah, all the awesome tricks palettes opened up the door to! I still miss them to this day.

In modern hardware it’s also really easy to do with a shader, but Paul wanted to use it across any iPhone device, and the majority of them are still stuck on OpenGL ES 1.1, so fixed-function pipeline it is.

The simplest approach would be to just render the model twice: First pass renders the texture, and second pass renders the custom color bits (you can render them with a white texture modulated by the global color to get the right color). The main drawbacks are that you’re doubling the number of draw calls, and, with 3D objects, it gets a bit tricker because the second pass needs to use the glDepthFunc(GL_EQUAL) depth comparison function.

The better way to do this is using the texture combiners. Texture combiners allow us to perform a limited number of operations to control the final look of a pixel. We can add two textures, or multiply them, or even do a few more complex operations. The true power of the combiners is that they can be chained together, so the output from one feeds into the input of another, allowing us to create much more complex operations.

The iPhone 3G is limited two two texture combiner units [2], but even two combiners are good to create a good range of effects.

Let’s think about what we want to accomplish. We want to leave some parts of the texture completely alone and display the original pixel value. In some other parts of the texture, we want to replace the pixels there with a custom color. Actually even better, we probably want to multiply those pixels by a custom color. That way we can author the part of the texture that is going to change with grayscale details, and our color adds the tint to it.

Let’s express it mathematically. Let’s make a function M that is 1 for every pixel we want to color, and 0 for the ones where the original texture is supposed to be displayed. Our desired color is c and the texture it t. In that case, the final pixel color (p) is:

p = M*(c*t) + (1 – M)*t

We could express that with two combiners: The first one is a modulate (multiply) operation with c and t, and the second one an interpolation operation between the result of the previous combiner and the original texture, based on the function M.

Obviously M is just a mask texture. We can paint it white where we want to color the texture, and black elsewhere. We could even use the alpha channel of the original texture, but there’s one big thing to watch out for: If you have your texture as a png and process it through the default iPhone resource operations, the image will be premultiplied for you (whether you want it or not), so your color information will be set to zero everywhere that the alpha channel is zero. Oops. You’ll probably want to use the alpha channel to store transparency anyway, so we’ll keep the mask separate. If not, make sure you encode the image yourself (as raw or PVRT formats) so it’s not premultiplied ahead of time.

Are we ready transfer that formula to the texture combiners? Not quite. Apparently (and this was just trial and error, I haven’t seen it documented), the texture assigned to a combiner can only be the one at that stage. If you look at the second combiner, we would need to have the first texture as one of the parameters, in addition to the mask.

So instead, we can reorganize the function above like this:

p = c*(M*t) + (t – M*t)

What did we gain by that? The color is what’s going to change dynamically, but the mask and the texture always stay the same. We could precompute the M*t term by simply multiplying the texture and the mask. We can call that new term A. We can do the same thing with the (t – M*t) term, which just means turning black all the pixels in the texture where mask will go. That one will be B. The easiest way to “precompute” those values is just doing it in Photoshop and exporting it as a new png.

Our new formula is now:

p = c*A + B

Nice and simple! Now we can really add that to the texture combiners like this:

// c
glColor4f(m_customColor.r, m_customColor.g, m_customColor.b, 1);

glActiveTexture(GL_TEXTURE0);
// A = M*t (precomputed)
glBindTexture(GL_TEXTURE_2D, m_maskHandle);
glEnable(GL_TEXTURE_2D);
// c*A
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_MODULATE);

glActiveTexture(GL_TEXTURE1);
// B = t - M*t (precomputed)
glBindTexture(GL_TEXTURE_2D, m_textureHandle);
glEnable(GL_TEXTURE_2D);

// c*A + B
glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_COMBINE);
glTexEnvi(GL_TEXTURE_ENV, GL_COMBINE_RGB, GL_ADD);
glTexEnvi(GL_TEXTURE_ENV, GL_SRC0_RGB, GL_PREVIOUS);
glTexEnvi(GL_TEXTURE_ENV, GL_SRC1_RGB, GL_TEXTURE);

One more thing to watch out for: Because we’re using two textures, you need to have two sets of texture coordinates. In this case, we want them to be the same, so we can just point them to the same set of data:

glClientActiveTexture(GL_TEXTURE0);
glTexCoordPointer(2, GL_FLOAT, sizeof(Vertex), &vertices[0].u);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
glClientActiveTexture(GL_TEXTURE1);
glTexCoordPointer(2, GL_FLOAT, sizeof(Vertex), &vertices[0].u);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);

That’s it! You can see the results in the included project and play with the register combiners to achieve different operations.

At this point I was going to describe the texture combiner setup I use in Flower Garden to render the petals, but this post ended up taking longer than I had hoped for (I’m trying to shoot for an hour per post, but this has taken me already two hours between the code and the the post itself), so I’ll save that for another time.

• Demo source code. Released under the MIT license.

[1] Paul developed Sword of Fargoal, by far my favorite iPhone RPG.
[2] The 3GS allows up to eight I believe.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

The main difference is that I’m going to go in detail about each of the different cases of mixing OpenGL and UIKit, and what better way to do that than with a live demo. I’ll be switching back and forth between Keynote and XCode and going over the details, which is where a lot of the tricky parts are.

Here is the source code for the demo. That way you can look at it now and come to the session prepared with more questions, or ready to discuss what works and what doesn’t work for you. I love to have interactive sessions, so definitely come ready to ask questions.

If you’re not coming to 360iDev… shame on you! It’s the best iPhone conference around. You’ll be missing not just my session, but tons of other great talks, and the iPhone Game Jam (sponsored by TouchArcade this year). If you can’t make it this time, make a note on your calendar for next time!

• Demo source code. Released under the MIT license.

It was a 30-minute slot, so the material is pretty condensed without the chance to expand on the topics. I’m giving an extended version of this talk at 360iDev in a few weeks, so I’ll be able to get into more details then and have a cool sample app that shows off all those concepts.

It was great seeing a bunch of you at my session and at around the iPhone and Indie Summits. It was a great couple of days!

• Presentation slides (pdf format)
• Sample code

The demo ended up looking pretty good, so I decided to upload a quick video showing the different lighting modes:

• Diffuse texture only
• Diffuse texture plus ambient and diffuse lighting
• Diffuse texture plus ambient, diffuse, and specular lighting (as usually, per-vertex lighting looks pretty bad, even though this is a relatively high-poly model)
• Fully reflective environment map (using the normal environment map technique)
• Environment map added to the diffuse texture and lighting
• Environment map with a reflection mask plus diffuse texture and lighting (two passes on an iPhone 3G–or one pass if you’re not using the diffuse alpha channel)

The chapter in the book will go in detail into each of those techniques, building up to the last one. Full source code will be included as well.

Some of these techniques will also be covered in my upcoming Two Day iPhone OpenGL Class organized in collaboration with Mobile Orchard.

The class is aimed at iPhone developers without previous OpenGL experience. It’s going to be very hands-on, and you’ll create both 2D and 3D applications during the weekend. You’ll learn all the basics: cameras, transforms, and how to draw meshes, but we’ll also cover some more advanced topics such as lighting, multitexturing, point sprites, and even render targets. Most importantly, you’ll walk away with a solid understanding of the basis, which will allow you to continue learning OpenGL and advanced computer graphics on your own from the docs, samples, or even browsing the API directly.

The main requirement for the class is that you’re familiar with the iPhone development environment and that you have basic knowledge of the C language. Beyond that, to the the most out of the course, you should be familiar with the basics of linear algebra (vector, matrices, and dot products). Anything else, we’ll cover it all during the class.

Registration is now open, and you can get some great discounts by registering early and attending the 360iDev conference. For more details, check the official announcement page.

Hope to see some of you there!

From UIKit to OpenGL

This is definitely the most common way of sharing image data. Loading an image from disk in almost any format is extremely easy with Cocoa Touch. In fact, it couldn’t get any easier:

UIImage* image = [UIImage imageNamed:filename];

After being used to image libraries like DevIL, it’s quite a relief to be able to load a file with a single line of code and no chance of screwing anything up. So it makes a lot of sense to use this as a starting point for loading OpenGL textures. So far we have a UIImage. How do we get from there to a texture?

All we have to do is create a CGContext with the appropriate parameters, and draw the image onto it:

byte* textureData = (byte *)malloc(m_width * m_height * 4);
CGContext* textureContext = CGBitmapContextCreate(textureData, m_width, m_height, 8, m_width * 4,
            CGImageGetColorSpace(image), kCGImageAlphaPremultipliedLast);
CGContextDrawImage(textureContext, CGRectMake(0.0, 0.0, (CGFloat)m_width, (CGFloat)m_height), image);
CGContextRelease(textureContext);

At that point, we’ll have the image information, fully uncompressed in RGBA format in textureData. Creating an OpenGL texture from that is done like we always do:

glGenTextures(1, &m_handle);
glBindTexture(GL_TEXTURE_2D, m_handle);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_width, m_height, 0, GL_RGBA, GL_UNSIGNED_BYTE, textureData);

And we’re done.

A word of advice: This is a quick way to load textures and get started, but it’s not the ideal way I would recommend for a shipping product. This code is doing a lot of work behind the scenes: loading the file, decompressing the image into an RGBA array, allocating memory, copying it over to OpenGL, and, if you set the option, generating mipmaps. All of this at load time. Ouch! If you have more than a handful of textures, that’s going to be a pretty noticeable delay while loading.

Instead, it would be much more efficient to perform all the work offline, and prepare the image into the final format that you want to use with OpenGL. Then you can load that data directly into memory and call glTextImage2D on it directly. Not as good as having direct access to video memory and loading it there directly, but that’s as good as it’s going to get on the iPhone. Fortunately Apple provides a command-line tool called texturetool that does exactly that, including generating mipmaps.

Another use of transferring image data from UIKit to OpenGL beyond loading textures is to use the beautiful and full-featured font rendering in UIKit in OpenGL applications. To do that, we render a string into the CGContext:

CGColorSpaceRef    colorSpace = CGColorSpaceCreateDeviceGray();
int sizeInBytes = height*width;
void* data = malloc(sizeInBytes);
memset(data, 0, sizeInBytes);
CGContextRef context = CGBitmapContextCreate(data, width, height, 8, width, colorSpace, kCGImageAlphaNone);
CGColorSpaceRelease(colorSpace);
CGContextSetGrayFillColor(context, grayColor, 1.0);
CGContextTranslateCTM(context, 0.0, height);
CGContextScaleCTM(context, 1.0, -1.0);
UIGraphicsPushContext(context);
    [txt drawInRect:CGRectMake(destRect.left, destRect.bottom, destRect.Width(), destRect.Height()) withFont:font
                                lineBreakMode:UILineBreakModeWordWrap alignment:UITextAlignmentLeft];
UIGraphicsPopContext();

There are a couple things that are a bit different about this. First of all, notice that we’re using a gray color space. That’s because we’re rendering the text into a grayscale, alpha-only texture. Otherwise, a full RGBA texture would be a waste. Then there’s all the transform stuff thrown in the middle. That’s because the coordinate system for OpenGL is flipped with respect to UIKit, so we need to inverse the y.

Finally, we can create a new texture from that data, or update an existing texture:

glBindTexture(GL_TEXTURE_2D, m_handle);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, m_width, m_height, GL_ALPHA, GL_UNSIGNED_BYTE, data);

From OpenGL to UIKit

This is definitely the more uncommon way of transferring image data. You would use this when saving something rendered with OpenGL back to disk (like game screenshots), or even when using OpenGL-rendered images on UIKit user interface elements, like I’m doing in my project.

The process is very similar to the one we just went over, but backwards, with a CGContext as the middleman.

We first start by rendering whatever image we want. You can do this from the back buffer, or from a different render target. I don’t know that it makes a difference in performance either way (these are not fast operations, so don’t try to do them every frame!). Then you capture the pixels from the image you just rendered into a plain array:

unsigned char buffer[width*(height+30)*4];
glReadPixels(0,0,width,height,GL_RGBA,GL_UNSIGNED_BYTE, &buffer);

By the way, notice the total awesomeness of the array declaration with non-constant variables. Thank you C99 (and gcc). Of course, that might not be the best thing to do with large images, since you might blow the stack size, but that’s another issue.

Anyway, once you have those pixels, you need to go through the slightly convoluted CGContext again to put it in a format that can be consumed directly by UIImage like this:

CGImageRef iref = CGImageCreate(width,height,8,32,width*4,CGColorSpaceCreateDeviceRGB(),
                            kCGBitmapByteOrderDefault,ref,NULL, true, kCGRenderingIntentDefault);
uint32_t* pixels = (uint32_t *)malloc(imageSize);
CGContextRef context = CGBitmapContextCreate(pixels, width, height, 8, width*4, CGImageGetColorSpace(iref),
                            kCGImageAlphaNoneSkipFirst | kCGBitmapByteOrder32Big);
CGContextTranslateCTM(context, 0.0, height);
CGContextScaleCTM(context, 1.0, -1.0);
CGContextDrawImage(context, CGRectMake(0.0, 0.0, width, height), iref);   
CGImageRef outputRef = CGBitmapContextCreateImage(context);
UIImage* image = [[UIImage alloc] initWithCGImage:outputRef];
free(pixels);

Again, we do the same flip of the y axis to avoid having inverted images.

Here the trickiest part was getting the color spaces correct. Apparently, even though it looks very flexible, the actual combinations supported in CGBitmapContextCreate are pretty limited. I kept getting errors because I kept passing an alpha channel combination that it didn’t like.

At this point, you’ll have the OpenGL image loaded an a UIImage, and you can do anything you want with it: Slap it on a button, save it to disk, or anything that strikes your fancy.

The now gone CrashLander sample, in spite of some atrocious code, was the best example on how to get a simple, OpenGL app on the iPhone. It covered the basics: creating an OpenGL frame buffer, loading some textures, drawing some polys, and presenting the result to the screen. It was very useful because it was a very small and concise and it made for a great starting point for any OpenGL-based app.

Unfortunately, because it was so basic, it didn’t show how OpenGL can play with the rest of UIKit user interface. Instead, it took the approach from many games of taking control of the full screen and viewing it a a single resource. That’s fine as long as you’re making those kind of games, but I found myself having to mix OpenGL and UIKit quite a bit in my current project. And eventually, I even had to use multiple OpenGL views in different parts of the app.

Single OpenGL View

To render with OpenGL to a view, you have to first set the class-static method +layerClass so it creates the right type of layer. Specifically, we need a CAEAGLLayer:

+ (Class)layerClass
{
    return [CAEAGLLayer class];
}

You can create a view of that type in code, or you can lay it out on the Interface Builder. I actually like the Interface Builder quite a bit, so that’s how I end up creating all of mine.

Then, you need to create an OpenGL context:

    m_eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES1];
    [EAGLContext setCurrentContext:m_eaglContext];

Finally, you need to create a valid frame buffer object from the CAEAGLLayer from the view you just created:

    glGenFramebuffersOES(1, &buffer.m_frameBufferHandle);
    glGenRenderbuffersOES(1, &buffer.m_colorBufferHandle);
    glGenRenderbuffersOES(1, &buffer.m_depthBufferHandle);

    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);
    [m_eaglContext renderbufferStorage:GL_RENDERBUFFER_OES fromDrawable:drawable];
    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_WIDTH_OES, &buffer.m_width);
    glGetRenderbufferParameterivOES(GL_RENDERBUFFER_OES, GL_RENDERBUFFER_HEIGHT_OES, &buffer.m_height);

    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_depthBufferHandle);
    glRenderbufferStorageOES(GL_RENDERBUFFER_OES, GL_DEPTH_COMPONENT16_OES, buffer.m_width, buffer.m_height);
    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);

    glBindFramebufferOES(GL_FRAMEBUFFER_OES, buffer.m_frameBufferHandle);
    glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle); 
    glFramebufferRenderbufferOES(GL_FRAMEBUFFER_OES, GL_DEPTH_ATTACHMENT_OES, GL_RENDERBUFFER_OES, buffer.m_depthBufferHandle)

It’s all pretty standard frame buffer OpenGL stuff. The renderBufferStorage:fromDrawable function is the one that actually allocates buffer storage for that particular view.

After that, you’re home free and you can use OpenGL the way you’ve always done. The only difference is that to present, you call this functions instead:

    [m_eaglContext presentRenderbuffer:GL_RENDERBUFFER_OES];

Done.

Integrating OpenGL and UIKit

After you have the basic OpenGL app set up, you can treat it as a full-screen, exclusive resource and do all your output through it. That works fine for a lot of games, but there are a lot of games and apps that benefit from using some amount of UIKit functionality. After all, UIKit is a great UI API, and it seems like a shame to reinvent it all if it’s not necessary.

The good news is that OpenGL rendering is happen in a view, so you can do everything you can do with views: Animate it, do fancy transitions, add them to nav bars or tab bars, etc. The bad news is that, unless you’re careful, performance will suffer.

Apple has some best practices on what to do with OpenGL views, but it comes down to avoid doing transforms on them (instead, use the transforms in OpenGL), and avoid putting other UIKit elements on top of it, especially transparent ones. Although, I seem to be able to get away with small buttons and such without affecting the frame rate, so that’s always a plus.

So having realized that, now you can make your application transition between the OpenGL view, and other views. Maybe you have a table view with settings, or some sort of UIKit dialog box, or a high-score table. You can safely do all of that with UIKit (and add your own graphics and custom rendering if you want to give it a totally different look).

One thing to watch out for: Whenever your OpenGL view isn’t visible, make sure to stop doing frame updates (for simulation and rendering). Otherwise, the rest of the UI is going to be very unresponsive (and you might not find that out until you run it on the device, because the simulator is very fast). The UIViewController viewWillAppear and viewDidDisappear functions are perfect to find out when you should start and stop frame updates.

Multiple Views

Things get more complicated when you want to have multiple views with OpenGL rendering. I don’t mean multiple viewports in the same screen, but multiple views in the sense of multiple UIViews. My current project, for example, has different views connected with a UITabBarController. Some of the are custom UIViews and some of them are OpenGL views.

My first thought was to try to have different OpenGL contexts for each view, but that would complicate things because I wanted to share the same textures and other resources. I know there’s a shared groups option, but that was definitely not the way to go.

The best solution I found was to use multiple frame buffers, binding each of them to the correct OpenGL view. That way, when I switch views, I switch frame buffers and everything works correctly.

On the -viewDidLoad function for each view controller with an OpenGL view, I call the function listed above to create a new frame buffer bound to that view, and it returns an index. Then, in the -viewWillAppear function, I set the frame buffer corresponding to that index:

    const FrameBuffer& buffer = m_frameBuffers[bufferIndex];
    glBindFramebufferOES(GL_FRAMEBUFFER_OES, buffer.m_frameBufferHandle);   
    glBindRenderbufferOES(GL_RENDERBUFFER_OES, buffer.m_colorBufferHandle);
    glViewport(....);

That’s it. Now you can switch between different OpenGL views and UIKit views without any problems. Just make sure to only render the visible views! That can be trickier than it sounds because the viewWillAppear and viewDidDisappear functions only get called for views that were layed out in the Interface Builder. If you added them by hand, you need to call those methods yourself (why??!?!). So keep tabs on that, otherwise everything will slow down to a halt.

And To Wrap It Up…

You thought I had forgotten about the promise to slowly unveil art from my current project, uh? Fear not, here comes the next teaser. This is an actually screenshot taken a minute ago. Incidentally, this is not one of the views using OpenGL. We’ll have to save those for another day