in Game tech

Life In The Top 100 And The Google App Engine

A few weeks ago I wrote about using the Google App Engine as a back end for Flower Garden. One of the comments I heard from several people is that the Google App Engine pricing scheme is “scary” due to its unpredictability. What if your server gets very busy and you find yourself with a huge bill?

Google App Engine Costs

At the time I wrote that post, the bandwidth of Flower Garden (and Flower Garden Free) was just under the free bandwidth allowance, so I didn’t have to pay anything. However, this last week, after riding the Pocket Frogs rocket, Flower Garden Free shot up to the #56 overall app in the US. And boy, did that make a difference in server usage!

serverload.jpg

That screenshot shows the Google App Engine dashboard with the number of requests per second for a 4 day period. Can you guess when the Pocket Frogs cross-promotion started? It went from an average of 1.5 requests per second, to a peak of about 16. That’s a factor of 10 right there.

Let’s see what that means in terms of cost. With the Google App Engine, you get charged for different resources you consume: CPU usage, incoming bandwidth, outgoing bandwidth, data storage, and recipients emailed. Of those, the only one that Flower Garden makes any dent on is outgoing bandwidth (mostly the HTML pages and images in the Flower Shop).

For each of those resources, you get a free amount every day. In the case of outgoing bandwidth, we get 1GB before we have to start paying anything. Afterwards, it’s $0.12 per GB. The peak day for Flower Garden Free, when it was all the way at #56, it consumed 7.8 GB. Now that it’s back down, it’s using up at around 2-3 GB of outgoing data every day.

That means, that the busiest day I was charged $0.82 in bandwidth, but the profits for that day were over $1500! I can only hope for days just as busy!

charges.png

As a reference, these are my Google App Engine charges for the busy week during the promotion. $3.28 for the week? Bring it on! 🙂

Minimizing Bandwidth

When I first saw the first day’s bandwidth, I was pretty surprised it was taking up that much. I hadn’t tried to minimize it first, but 8 GB seemed like a significant amount.

The first thing I did was to forward any requests to the “More Games” and “News” section back to the Dreamhost account. After all, if that server goes down, it doesn’t really matter, and they can take up significant bandwidth with all the images.

To forward something, I couldn’t just do it from the configuration file. Instead, I set a handler for the moregames directory like this:

- url: /moregames/.*
  script: redirect.py

And the redirect script is very simple:

import os
import sys
import logging
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app

class RedirectMoreGames(webapp.RequestHandler):
	def get(self, params):
		url = "http://flowers.snappytouch.com/moregames/Snappy_Touch_More_Games/More_Games.html"
		self.redirect(url, permanent=True)
	
application = webapp.WSGIApplication(
							[(r'/(.*)', RedirectMoreGames),
							 ],
							debug=True)

def main():
	run_wsgi_app(application)

if __name__ == "__main__":
	main()

Another thing you can do is to turn on explicit caching of files in the app.yaml file for your Google App Engine app. I don’t know how much that helps with data requested from within an iPhone app, but it can’t hurt.

Finally, I made some changes to Flower Garden. I never had bandwidth minimization in mind, so for example, I was happily requesting updated news and other data at the beginning of every session. I wised up a bit and now it only requests that data at most once every few hours or days.

In case you’re ever getting close to using up your quotas, I learned that you can query them at runtime. I haven’t done anything about it yet, but I suppose I could start forwarding image fetches to a backup server whenever I get close to the limit. But for that, I’ll have to sell many more units of Flower Garden!

Google App Engine Shortcomings

Overall, I’m very happy with the Google App Engine. But it’s not all a bed of roses over here. I think the Google App Engine excels at scaling under heavy load, but surprisingly, it doesn’t have anything close to 100% uptime. I’m going to say it probably doesn’t even have 95% uptime! With such a massive system and all the redundancy underneath, I’m surprised they can’t have better uptime. Supposedly their status page makes you think everything is perfect, but it’s really far from it.

Apart from downtime, people are also reporting severe latency issues sometimes, and I’ve seen some errors like that in my log every so often. Chalk it up to being a beta I suppose. I imagine it’s only going to get better with time.

I would still use the Google App Engine for any new game I’ll release in the future, whether it has big or small backend needs. If nothing else, the ease of development and testing beats every alternative for me.

This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.

  • Interesting stuff, and very applicable to me because I think my server is about to get hit really hard this week. At least I’m lucky that all my server based stuff is more of a “nice to have” than required for the game to operate.

    I also decided to upgrade to Dreamhost PS rather than switch to a different provider. The upgrade process to PS was almost too easy, and I’m only using around 10% of my resources, even though I’m on the minimum ($15/month) plan. My server was hit way harder this week than ever before (18000 solutions submitted), but so far it’s holding up well. Over the next few weeks I’ll really put Dreamhost’s unlimited bandwidth and disk space claims to the test.

  • emerix

    I also use appengine (java) and I confirm that uptime is quite disapointing : I often have the error “Request was aborted after waiting too long to attempt to service your request.”… it’s a shame for a service hosted by Google !
    But anyaway I’m under the quota so it’s free for me… I guess I have what I paid for 😀

    • Jeffrey Considine

      Doesn’t that error message mean your code took too long running?

      • Maxime Rafalimanana

        Yes it should mean that indeed. The problem is I often have this just for getting a single simple object in the datastore…

  • New blog post: Life In The Top 100 And The Google App Engine http://gamesfromwithin.com/life-in-the-t… #idevblogaday

    • pearapps

      @SnappyTouch Can I ask you a question about the iAP sandbox?

    • OneManLeft

      @SnappyTouch Nice info, We’ve got a future project that needs a backend and knowing that certainly clears up some worries 🙂

    • GeekAndDad

      @SnappyTouch nice post! Interesting. Surprised at uptime and intermittent latency – you’d think they could/would do better…

    • renderpaz

      @SnappyTouch A nice data-driven piece. While i’m not happy with outages, uptime is probably closer to 99% – but wtf – it’s Google!

    • rodstrougo

      @SnappyTouch So where is the post about breathing new life into Le Chimp as a build/test server? Was thinking of using a mini+hudson 😉

  • Nice post! I’m sorry to hear you’re not happy with the uptime, though – how are you estimating that? 95% uptime is over 8 hours a week downtime, and I’m pretty sure we’re doing a whole lot better than that. 🙂

  • Great post – interesting comment re uptime.

    We have a business app that get much lower usage, but is quite critical, so we watch each and every failure… I think it’s actually quite high – we’re seeing a 99.9+ really… How are you counting uptime?

    What we have found is that any apps that rely on aggregation of services will struggle. We rely on a number of other Google services (Apps, Storage, etc.) and all up, it’s not at all uncommon that some part of the network is struggling.

    I also think that we now live in the world where at any point in time, something is always down; just need better ways of coping with it!

    J

  • Is the uptime really that bad? A 95% uptime would mean the site is down for a whopping 36 hours every month. Did you use anything to measure the uptime?

  • Hi Noel

    We have success using the Amazon Elastic Cloud service (http://aws.amazon.com/ec2/). It is great when you need to scale quickly to support a marketing campaign or something similar.

    Best regards
    Jesper