Google App Engine As Back End For iPhone Apps

As soon as a game involves servers, there’s no such a thing anymore as “ship and forget”. Flower Garden put this in evidence about a month ago when I started getting complaints from users that the Flower Shop kept going down. Sometimes they weren’t even getting the items they were purchasing! (fortunately they can always do a free re-download, but they don’t always know that).

Flower Garden was using a shared Dreamhost server to upload bouquets, host in-game news, and, most importantly, to serve the Flower Shop (which involves hosting the static shop files, redeeming promo codes, recording transactions, verifying receipts with Apple’s server, and delivering purchased content). All the Flower Shop functionality was implement in PHP and using a mySQL database.

After some stressful sorting through the logs, it looked like the server was running out of memory and killing PHP processes. Dreamhost’s technical support didn’t help much and just claimed that I was simply something that happened and they would be happy to sell me a virtual private server. I was surprised to hear it was using up a lot of memory, and certainly nothing I could see with the tools available to me showed that. I certainly hadn’t done any changes in a while and traffic was constant. Doing some Googling at the time brought up lots of other angry Dreamhost users in a similar situation, so I suspected server configuration problems.

Whatever the case, I couldn’t let it be that way and I had to fix it somehow. I briefly considered a virtual private server, or even a dedicated host in some highly recommended providers. But in the end, the simplicity, scalability, and affordability of the Google App Engine won me over.

Now that Flower Garden has been using the Google App Engine for several weeks, I still think it’s fantastic. I wish someone had whacked me on the head when I started writing server code and forced me to use it. Hopefully this post will be more pleasant than a hit to the head and will still have a similar (good) effect.

Google App Engine Overview

appengine_lowres.gifEven though I was pretty much completely new to the Google App Engine (I had looked into it briefly before the launch of Flower Garden but dismissed it because they couldn’t support the amount of email traffic I needed), it took me two days to port over everything. Actually flipping the switch from the old system to the new one took a while longer, but that required more courage than work. More on that later.

This is not a Google App Engine tutorial. Instead, it’s going to be an overview of what it can do and why you should consider it as a backend for your iPhone app instead of using a shared or even dedicated server.

The first shock (at least for me) when looking into the Google App Engine is that you can’t run PHP. You’re limited to Java or Python. It turns out Python is my go-to scripting language, so I was thrilled at the idea. PHP is fine, but Python is a whole class above.

Next, it took a bit of adjusting to the fact that the environment to run anything on the Google App Engine has to be carefully controlled. You need to create a configuration file indicating the entry points and their handlers. Fortunately, all of that is well covered in the extensive online documentation.

This is what the Flower Garden config file looks like, indicating both static files and several execution entry points:

application: st-flowershop
version: 1
runtime: python
api_version: 1

handlers:
- url: /catalog
  static_dir: catalog

- url: /emailassets
  static_dir: emailassets

- url: /news
  static_dir: news

- url: /moregames
  static_dir: moregames

- url: /.*
  script: purchase.py

If you hit one of the URLs listed as static_dir, you get those files directly (like the in-game news). Anything else is handled by the Python script purchase.py. Google even provides the webapp framework that easily allows you to connect different parts of the script to handle different requests and easily get to the input parameters.

Once you’re past that and you have the requisite “Hello World” up and running, then it’s all fun and games.

One of the great features of the Google App Engine is that it comes with a fully-featured, local environment. This environment is installed automatically with the Google App Engine SDK, and there’s nothing to configure. It’s certainly nothing like setting up Apache + mySQL + PHP in your dev station! That way, you can do all your work locally, and only update the servers when you’re confident everything is working.

Beyond this, the only other thing that is different from what you may be used to is the datastore. They provide a query language called GQL, which is not exactly SQL, but it’s very similar. Close enough that I had no trouble porting over my very simple queries anyway.

Adding data to a data store couldn’t be simpler. For example, to add a promo code, I need the following class definition:

class PromoCode(db.Model):
	code = db.StringProperty()
	productid = db.StringProperty()
	amount = db.IntegerProperty(default=1)
	singleuse = db.BooleanProperty(default=True)
	enabled = db.BooleanProperty(default=True)

And then I can add it this way:

	promocode = PromoCode()
	# fill it up here
	promocode.put()

How do I check if a promocode is valid? I can use the SQL-like syntax, or something even simpler:

	q = PromoCode.all()
	q.filter("code =", code)
	return q.get()

Once you know that, you can go to town with your Python programs.

Google App Engine Advantages

Scalability
This was the big reason that forced me to move away from Dreamhost. I have no way of testing how scalable it really is, but I’m going to take Google’s word for it. They know a tiny little bit about scalability. And besides, Flower Garden is nothing compared to other apps (it serves about 1.5 requests per second on average).

Since I don’t know how things are implemented under the hood in the Google App Engine, I don’t have a good feel for performance best practices. So I’m hoping that my very simple queries (add to the end of a table, or see if a record is present) are just fine. If not, I can do some tuning down the line (and I won’t have to wait for any approval to make the code go live!).

Local environment
The local environment is simply great. It makes developing a pleasure. You can add any test data you want to the local datastore through a web interface without affecting the live server. Probably my favorite feature is the console: Since the server is running locally, you can print out any debug info in the server code and you can see it live as you perform some action. That saved me hours of debugging compared to doing it on a remote server with php!

Server synching
In the past, I’ve synced my server scripts through ftp and I kept meaning to write an automated script to do that. In this case, Google provides a script (and a GUI tool) to do the actual synching with the servers, which is great. It even waits a few seconds until it can verify that the new version is live on the server.

Python
I’m not a big Java fan (although I used it a bit way back in the day when it first came out), but I do love Python. For a program with some complexity like this, it’s definitely a much better choice than PHP.

Real-time stats
The Google App Engine web console displays real-time stats of access to your site. You see it expressed in requests per second and you can visualize it in the past few hours, days, or even month. You also have access to all the logs, and you can filter by type of log message (debug, info, warning, error).

Price
This is one category where the Google App Engine definitely shines. Every day you’re given a free quota of bandwidth, disk usage, and CPU usage. If you stay under that quota you don’t pay anything. If you go over, you pay per unit as you go (at very reasonable prices). You can even set a maximum cap on daily spending so you don’t encounter any nasty surprises at the end of the month.

So far, Flower Garden has been hovering right at the edge of the outgoing bandwidth free quota (the rest of the metrics it doesn’t even come close). So even if traffic were to double, expenses would be very reasonable.

Development Tricks

Along the way, I figured out a couple of interesting tricks that helped me during development.

Bypass the App store for testing
For me, few things are more annoying than doing development in the actual device. Iteration is slow and stepping with the debugger is mostly impossible (seems to depend on the mood of the debugger that day). So I try to do the bulk of the development on the simulator and just test on the device when things are ready.

Unfortunately, the device can’t access the App Store, which makes testing all the server functionality dealing with the App Store painful. What I did was to add a define that, when present, the program bypasses the App Store but still functions as usual.

Local server
During development, you want to be using your local server. In my case, I set it up so that Debug versions of the code access the local server, but Release and Distribution ones access the live one.

There is one problem: The Google App Engine server, by default, binds itself to localhost, not to the IP address of your network card. That meant I was able to hit it from the simulator just fine accessing http://localhost:8080, but not from the device. In order to access the development server from the device, I had to explicitly tell the server to bind itself to another address like this:

dev_appserver.py --address 192.168.1.150 FlowerShop

Now I can access the server at 192.168.1.150:8080 from both the simulator and any device.

Switching servers
Porting the code took just a couple of days. Making the switch on the live server was a lot scarier though. I couldn’t wait for a new update to be released because that seems to take over a week these days, so I forwarded requests from the old server to the new one.

I did it one system every day, avoiding weekends which is when there’s the most traffic. News was easy, and so was the web view with more games. But I ran into a snag with promo codes and Flower Shop purchases.

Initially, I was just redirecting requests in .htaccess this way:

Redirect permanent /Shop/catalog http://st-flowershop.appspot.com/catalog
Redirect 301 /Shop/catalog http://st-flowershop.appspot.com/catalog

As I learned the hard way, POST variables aren’t preserved in a redirect of that kind (I guess to avoid state being changed in multiple servers). I tried a bunch of things, but in the end, I wrote a quick php script that gathered all POST variables and re-posted them to the new request. A bit ugly, but it did the trick:

<?php
function PostRequest($url, $_data) {

    $data = array();    
    while(list($n,$v) = each($_data)){
        $data[] = "$n=" . urlencode($v);
    }    
    $data = implode('&', $data);
    // format --> test1=a&test2=b etc.

    $url = parse_url($url);
    if ($url['scheme'] != 'http') { 
        die('Only HTTP request are supported !');
    }

    $host = $url['host'];
    $path = $url['path'];

    $fp = fsockopen($host, 80);

    fputs($fp, "POST $path HTTP/1.1\r\n");
    fputs($fp, "Host: $host\r\n");
    fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
    fputs($fp, "Content-length: ". strlen($data) ."\r\n");
    fputs($fp, "Connection: close\r\n\r\n");
    fputs($fp, $data);

    $result = ''; 
    while(!feof($fp)) {
        $result .= fgets($fp, 128);
    }
    fclose($fp);

    $result = explode("\r\n\r\n", $result, 2);
    $header = isset($result[0]) ? $result[0] : '';
    $content = isset($result[1]) ? $result[1] : '';
    return $content;
}


print PostRequest("http://st-flowershop.appspot.com/purchase", $_POST);
?>

Conclusion

I hope this post is enough to at least make you interested in checking out the Google App Engine and help you get over a couple of the initial hurdles. I’ll definitely be using it in any future projects.


This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

  • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

    New blog post: Google App Engine As Back End For iPhone Apps http://gamesfromwithin.com/google-app-en… #idevblogaday

    • flarb

      @SnappyTouch Do you have any fears about unpredictable database usage pricing? That’s my one reservation about AppEngine.

    • flarb

      @SnappyTouch Oh wait, you’re using the Python Datastore and not BigTable. So it’s free. Pretty neat. Did you ever consider Amazon EC2?

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @flarb No, I think you still pay for the Python Datastore. I’m storing a tiny amount of data so prob not an issue for me.

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @flarb Amazon confused me w/ their pricing. You see to pay for uptime, not for b/w or CPU usage?

      • flarb

        @SnappyTouch You pay by the hour for uptime. It’s pretty simple–you also pay for bandwidth. That’s about it.

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @flarb Right. So EC2 better for one-off jobs, but I need 24h/day uptime, so very expensive!

      • flarb

        @SnappyTouch Well, depends on how you define expensive. It’s a good way to scale up a traditional web app (PHP, Apache, etc.) on demand.

      • flarb

        @SnappyTouch The cost is going down–but, the price advantage is with the on-demand scaling. So you can spin up Apache instances as needed.

      • flarb

        @SnappyTouch The one thing about AppEngine is you are glued to the platform. If you need to move off it, it’s kind of tricky.

      • flarb

        @SnappyTouch Although there is some kind of python framework you can use to port AppEngine apps to a traditional LAMP stack.

      • flarb

        @SnappyTouch AppEngine is kind of confusing to me–since you pay for CPU usage. It’s less predictable pricing IMHO. But easier environment

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @flarb It’s not 100% predictable, but when w/ thousands of users, things average out pretty nicely. Also, can set max spending cap.

      • http://www.mindjuice.net Ken Carpenter

        Not sure that a max spending cap is such a good idea. What happens when people make an in-app purchase and your server has reached its max?

        You don’t want to lose that sale.

        Can you programatically check your current usage and then make a decision?

        For example, stop serving news and moregames when you reach your max, but allow anything for a purchase.

      • http://www.gamesfromwithin.com Noel

        Good question about the max cap. It would be nice to throttle things somehow. I think it’s there to reduce the fear of getting a $5K bill at the end of the month, but you’re right that on a live system, you might want to remove it completely. After all, if people are using the store so much, chances are I’m making a lot more money than the cost of GAE (unless someone wrote a bot that is just refreshing it :-)

    • barog

      @SnappyTouch Love Google App Engine also – great post. We’re working on a few bits using it at the moment (did some work with it last year)

    • ChuckSmith

      @SnappyTouch Most informative #idevblogaday post I’ve ever read, thanks! Do you think App Engine would work well as a board game server?

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @ChuckSmith Thanks! Yes, I imagine it would be perfect as a board game server. No reason why not. Table for users and a table for games.

      • ChuckSmith

        @SnappyTouch Awesome! I was thinking that as well. Kept meaning to do some games, but didn’t want to manage a server. Thx for inspiration!

    • Godzilab

      @SnappyTouch excellent post. We have been using GAE for both iBlast Moki and Stardunk, and I totally agree with you on its awesomeness

    • dzamir

      @SnappyTouch great post, thanks for sharing!

    • loadedwino

      @SnappyTouch You mention promocodes. Are these codes you create and redeem? I thought this was against Apple policies. Section 3.3.3

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @loadedwino Oh, oh. Don’t tell me that! I’m going to go read it now…

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @loadedwino I think 3.3.3 is intended to avoid you setting up your own IAP. Promo codes have no new content and they’re free. So far OK.

      • loadedwino

        @SnappyTouch I was going to implement them in a recent app but decided against it. I feared apple could assume the trading of codes.

      • loadedwino

        @SnappyTouch maybe you’ve implemented it different to how I did. If you’ve been approved I’ll have to check it out and see what you’ve done

    • JimmyS

      @SnappyTouch really liked your Google App Engine article thanks for posting!

      • http://twitoaster.com/country-us/snappytouch/ SnappyTouch

        @JimmyS You’re welcome! I really liked Borderlands, so we’re even :-b

  • http://www.mindjuice.net Ken Carpenter

    Thanks for another great blog post Noel!

    Sounds like Google App Engine is pretty nice.

    Wish it would support Lua!

    Ken

  • Michael Seare

    You had me at “local environment.” I wasn’t looking forward to setting up an AMP environment on my own. Thanks for the post!

  • Matt Rix

    Just wondering what you think about Dreamhost PS? Obviously you didnt choose it, but I’m considering switching to that for Trainyard, because I really don’t have time to convert all my php stuff over to python.

    • http://www.gamesfromwithin.com Noel

      I didn’t really look into it given the bad experience of having this problem with their shared host. If it was a server being misconfigured, what’s there to stop the VPS being any better? I’d go with a reputable hosting like Linode or Slicehost instead if you’re going to go that route.

  • http://www.cannonade.net/blog Tom Horn

    Thanks very much for the post Noel. I have been procrastinating about moving my backend stuff from PHP/MySQL to App Engine for a while now and your post might just provide the impetus I needed. The section on switching servers was particularly useful (that is definitely the scary part).

  • Ranieri Argentini

    I have seen people complain that random datastore errors made (their apps on) Google app engine completely unusable for production work. Google acknowledges in their documentation (http://code.google.com/appengine/articles/handling_datastore_errors.html) that about one in 3000 datastore access fail due to underlying bigtable issues. Assuming this number is accurate and that a given interaction (e.g. a purchase) needs multiple datastore accesses, I can see a significant proportion of users experiencing issues.

    I was sort of worried by this as I’m ramping up work on my own datastore-backed application and I would hate it to have it fall over once the load starts getting interesting.

    Are you seeing these kind of issues pop up? Is so, can you give us an idea of how often this happens? Do you take any special precautions to safeguard your data in the event of transient datastore issues (e.g. using a task-queue based writeback)?

    Thanks!

  • http://snappyco.de Dave Newman

    I’d also recommend http://heroku.com for a really easy ruby based cloud host. It starts out free and you can ramp up if your traffic grows. Plus you get to write your app using http://www.sinatrarb.com/ which is the greatest little web framework around : )

  • http://www.gandogames.com Nicolas

    Great post! , our servers are currently hosted in VPS.net ( which I think it’s the best cloud service, linode style ). But for the backend of our future apps, we have been looking at App Engine for a while now!.

    Thanks for sharing this Noel :)

  • http://blog.notdot.net/ Nick Johnson

    Nice post! I’ve added it to the App Engine reddit (http://www.reddit.com/r/appengine). I’m glad to see you’re having success with App Engine

    One observation: You can pass –address 0.0.0.0 to the dev_appserver to have it listen on all interfaces, which is usually simpler than specifying your IP manually, and means you can still refer to it as ‘localhost’ in your browser.

  • http://am22tech.com Soan

    I also found Google App Engine to be far better than other players in the market. I am using it as a image CDN though as i have my wesites coded in asp.net.

  • Eric Snider

    It’s three years later and I’m looking for tips for using Google App Engine with iOS and Mac OS X apps… and I found your great article. My big question today: Did you do anything to protect st-flowershop.appspot.com from the world? I’m trying to figure out how developers try to ensure only their app can talk to their GAE server. I want to prevent the rest of the world of web browsers etc from doing bad things on my server (e.g. Posting high scores etc). Any ideas?