I’ve long been a fan of minimal game runtimes. Anything that can be done offline or in a separate tool, should be out of the runtime. That leaves the game architecture and code very lean and simple.
One of the things you potentially give up by keeping the game runtime to a minimum is an editor built in the game itself. But that’s one of those things that sounds a lot better than it really is. From a technical point of view, having an editor in the game usually complicates the code a huge amount. All of a sudden you need to deal with objects being created and destroyed randomly (instead of through clearly defined events in the game), and you have to deal with all sorts of crazy inputs and configurations.
The worse part though, is having to implement some sort of GUI editing system in every platform. Creating the GUI to run on top of the game is not easy, requiring that you create custom GUI code or try to use some of the OpenGL/DirectX libraries available. And even then, a complex in-game GUI might not be a big deal on a PC, but wait and try to use that interface on a PS3 or iPhone. After all, making games is already complicated and time-consuming enough to waste more time reinventing the widget wheel.
Remote game editing manages to keep a minimal runtime and allow you to quickly create native GUIs that run on a PC. It’s the best of both worlds, and although it’s not quite a perfect solution, it’s the best approach I know.
Debug Server
Miguel รยngel Friginal already covered the basics of the debug server, so I’m not going to get in details there.
The idea is that you run a very simple socket server on the game, listening in a particular port. This server implements the basic telnet protocol, which pretty much means that it’s a line-based, plain-text communication.
The main difference between my debug server and Miguel’s (other than mine is written in cross-platform C/C++ instead of ObjC), is that I’m not using Lua to execute commands. Using Lua for that purpose is a pretty great idea, but goes against the philosophy of keeping the runtime as lean and mean as possible.
Instead, I register variables with the server by hand. For each variable, I specify its memory address, it’s type, any restrictions (such as minimum and maximum values), and a “pretty name” to display in the client. Sounds like a lot of work, but it’s just one line with the help of a template:
registry.Add(Tweak(&plantOptions.renderGloss, "render/gloss", "Render gloss")); registry.Add(Tweak(&BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3))
And yes, if I were to implement this today, I would probably get rid of the templates and make it all explicit instead (ah, the follies of youth ๐
TweakUtils::AddBool(registry, &plantOptions.renderGloss, "render/gloss", "Render gloss"); TweakUtils::AddFloat(registry, &BouquetParams::FovY, "bouquet/fov", "FOV", Pi/32, Pi/3);
The debug server itself responds to three simple commands:
- list. Lists all the variables registered in the server.
- set varname value. Sets a value.
- print varname. Gets the value for that variable.
For example, whenever the server receives a set command, it parses the value, verifies that it’s within the acceptable range, and applies it to the variable at the given memory location.
Telnet Clients
Because we used the standard telnet protocol, we can start playing with it right away. Launch the game, telnet into the right port, and you can start typing away.
However, most telnet clients leave much to be desired for this. They all rely on the history and cursor manipulation being handled by the shell they assume you’re connected to. Here we aren’t connected to much of anything, but I’d like to be able to push up arrow and get my last command, and be able to move to the beginning of the line or the previous word like I would do in any text editor. The easiest solution I found for that was to use a telnet client prepared for that kind of thing: A MUD client! Just about any will do, but one that works well for me is Atlantis.
So far, we’ve implemented the equivalent of a FPS console, but working remotely. And because the code is fully portable, our game can be in just about any platform and we can always access it from our PC. Not just that, but we can even open multiple simultaneous connections to various development devices if you need to run them all at once.
Custom Clients
Game parameter tweaking is something that is OK through a text-based console, but really comes into its own when you add a GUI. That’s exactly what we did at Power of Two Games. We created a generic GUI tool (based on WinForms since we were on Windows at the time), that would connect to the server, ask for a list of variables, and generate a GUI on the fly to represent those variables. Since we knew type and name of each variable, it was really easy to construct the GUI elements on the fly: A slider with a text field for floats and ints, a checkbox for bools, four text fields for vectors, and even a color picker for variables of the type color.
It worked beautifully, and adjusting different values by moving sliders around was fantastic. We quickly ran into two problems through.
The first one is that we added so many different tweaks to the game, that it quickly became unmanageable to find each one we wanted to tweak. So, in the spirit of keeping things as simple as possible (and pushing the complexity onto the client), we decided that the / symbol in a name would separate group name and variable name. That way we could group all related variables together and make everything usable again.
The second problem was realizing that some variables were changing on the runtime without us knowing it on the client. That created weird situations when moving sliders around. We decided that any time a registed variable changes on the server, it should notify any connected clients. That worked fine, but, as you can imagine, it became prohibitively expensive very quickly. To get around that, we added a fourth command: monitor varname. This way clients need to explicitly register themselves to receive notifications whenever a variable changes, and the GUI client only did it for the variables currently displayed on the screen.
During this process, it was extremely useful to be able to display a log console to see what kind of traffic there was going back and forth. It helped me track down a few instances of bugs where changing a variable in the client would update it in the server, sending an update back to the client, which would send it again back to the server, getting stuck in an infinite loop.
You don’t need to stop at a totally generic tool like this either. You could create a more custom tool, like a level editor, that still communicates with the runtime through this channel.
Flower Garden Example
For Flower Garden, I knew I was going to need a lot of knobs to tweak all those plant DNA parameters, so I initially looked into more traditional GUI libraries that worked on OpenGL. The sad truth is that they all fell way short, even for development purposes. So I decided to grab what I had at hand: My trusty tweaking system from Power of Two Games.
I’m glad I did. It saved a lot of time and scaled pretty well to deal with the hundreds of parameters in an individual flower, as well as the miscellaneous tweaks for the game itself (rendering settings, infinite fertilizer, fast-forwarding time, etc).
Unfortunately, there was one very annoying thing: The tweaker GUI was written in .Net. Sure, it would take me a couple of days to re-write it in Cocoa (faster if I actually knew any Cocoa), but as an indie, I never feel I can take two days to do something tangential like that. So instead, I just launched it from VMWare Fusion running Windows XP and… it worked. Amazingly enough, I’m able to connect from the tweaker running in VMWare Fusion to the iPhone running in the simulator. Kind of mind boggling when you stop and think about it. It also connects directly to the iPhone hardware without a problem.
VMWare Fusion uses up a lot of memory, so I briefly looked into running the tweaker client in Mono. Unfortunately Mono for the Mac didn’t seem mature enough to handle it, and not only was the rendering of the GUI not refreshing correctly, but events were triggered in a slightly different order than in Windows, causing even more chaos with the variable updates.
Here’s a time-lapse video of the creation of a Flower Garden seed from the tweaker:
Drawbacks
As I mentioned earlier, I love this system and it’s better than anything else I’ve tried, but it’s not without its share of problems.
Tweaking data is great, but once you find that set of values that balances the level to perfection… then what? You write those numbers down and enter them in code or in the level data file? That gets old fast. Ideally you want a way to automatically write those values back. That’s easy if the tool itself is the editor, but if it’s just a generic tweaker, it’s a bit more difficult.
One thing that helped was adding a Save/Load feature to the tweaker GUI. It would simply write out a large text-based file with all the variables and their current values. Whenever you load one of those, it would attempt to apply those same values to the current registered variables. In the end, I ended up making the Flower Garden offline seed file format match with what the tweaker saved out, so that process went pretty smoothly.
Another problem is if you want lots of real-time (or close to real time) updates from the server. For example, you might want to monitor a bunch of data points and plot them on the client (fps, memory usage, number of collisions per frame, etc). Since those values change every frame, it can quickly overwhelm the simple text channel. For those cases, I created side binary socket channels that can simply send real-time data without any overhead.
Finally, the last drawback is that this tweaking system makes editing variables very easy, but calling functions is not quite as simple. For the most part, I’ve learned to live without function calls, but sometimes you really want to do it. You can extend the server to register function pointers and map those to buttons in the client GUI, but that will only work for functions without any parameters. What if you wanted to call any arbitrary function? At that point you might be better off integrating Lua in your server.
Future Directions
This is a topic I’ve been interested in for a long time, but the current implementation of the system I’m using was written 3-4 years ago. As you all know by now, my coding style and programming philosophy changes quite a bit over time. If I were to implement a system like this today, I would do it quite differently.
For all I said about keeping the server lean and minimal, it could be even more minimal. Right now the server is receiving text commands, parsing them, validating them, and interpreting them. Instead, I would push all that work on the client, and all the server would receive would be a memory address, and some data to blast at that location. All of that information would be sent in binary (not text) format over a socket channel, so it would be much more efficient too. The only drawback is that we would lose the ability to connect with a simple telnet client, but it would probably be worth it in the long run.
This post is part of iDevBlogADay, a group of indie iPhone development blogs featuring two posts per day. You can keep up with iDevBlogADay through the web site, RSS feed, or Twitter.
New blog post: Remote Game Editing http://gamesfromwithin.com/remote-game-e… Quick post on my exp w/ network tweaking/editing #idevblogaday
@SnappyTouch That’s a very cool article, thanks. I think I’ll be suggesting we try something similar, our curr. solution is very heavyweight
@SnappyTouch NICE – amazing – its *exactly* like the version I have 9with better UI). I did do the “straight to mem” blasting tho, but…
@SnappyTouch ..in future like you say, I’m looking at pokign function calls (somehow), esp. as I do live syncing of textures
@SnappyTouch So tempted to ask you for that tweak UI code so I can port it to Cocoaรขโฌยฆ but since you’re going to rewrite the whole thing ๐
@mysterycoconut I would love noting more than for someone to re-write it in Cocoa. Almost thought of hiring an intern for that!
@SnappyTouch I wouldn’t mind to have fun to try bring it to Cocoa, so where is the source so I can get going? ๐
@frederictessier @mysterycoconut Let me remove a few dependencies and check with Charles, and release it so you can make a Cocoa client ๐
@SnappyTouch Is @frederictessier going to be your intern now? What he has to do is work on his game! *whip sound*
@SnappyTouch see. this is what we mean w uber-productivity. this time of the day, we just start thinking about topics & start coffee makers
@MarkusN To be fair, I wrote it last night (although it only took an hour–and you can tell–too busy w/ house stuff right now :-()
@SnappyTouch The one thing that got my interest was your aside that you wouldn’t use a template these days. Can I ask you to elaborate?
@groby Only thing template does is makes the var type implicit. I like explicit better these days (and faster build times).
@SnappyTouch It also reduces code duplication (you write function once, this means easier fixes/changes).
@SnappyTouch Explicit templates should be as fast, but less code to write. I like templates, if I can constrain them to a CPP file.
@groby Agreed about templates constrained to cpp. These aren’t. And they don’t save much typing either ๐
@groby Besides, I only support a fixed set of variable types, so might has well have a function for each of them ๐
I recall having read somthing like that earlier … yep here it is !
http://blogs.msdn.com/b/shawnhar/archive/2009/05/01/motogp-tweakables.aspx
We have something similar, although there are a few difference; we split our data between debug tweakables, and config data; as we consider the tweakables inherently temporary (e.g. wireframe on/off), and we want the config data (e.g. speed of car) to go into version control.
Debug tweakables are declared using self registering classes rather than explicit registration as the goal was to make them as easy to use as possible, basically just declaring a few debug vars at global scope is enough to have them appear in our tool. They look something likeรขโฌยฆ
Debug::Bool gDebugWireframe(“render.world.wireframe”, false);
We then just use them like normal variables, each variable has a revision number that gets incremented when it’s changed, this allows us to recognise when a value is modified and sent it back over to the tool and also allows us to detect when the game and tool are mirroring edits back and forth between one another.
We’ve also hooked our shader constants into this system, so we can fiddle with them and see the results in real time.
Our config data is handled by a separate system that’s based on reflection (we use a macro approach), and we have a separate schema that our tool parses which contains extra information about how to visualise the data (e.g. as a form/nodegraph/chart/etc), this tool also has a socket connection to the game so as you tweak stuff in the tool things update in real time.
Great advice as always Noel. I take it the list command also provides details of the variables type and min/max values?
Hi George, Yes, you’re totally right. The list command gives all the info the client needs to generate the GUI. Here’s an example of a few parameters exposed in Flower Garden:
Notice there’s even a RW parameter I forgot to mention. You can tag variables as Read/Write or Read-Only, so it’s clear which ones the client can modify.
Personally I prefer the power of having a Lua server in my game, but more interesting is the fact that you can easily setup a minimal HTTP protocol (get) and return JSON instead of custom formats over telnet, then you can hook up to any browser and use all sorts of html widgets, works everywhere and is highly customizable!
Very interesting article on how you have interface your game with an external editor via telnet. Will keep this in mind when building our new debugging tools. This solution indeed is a bit more flexible on the way you can edit values, you can go primitively via a telnet session or built an GUI like you did for FlowerGarden.
The previous tools that @fcycles built for working on the GP2X he went the opposite direction and it was all built inside a library that we link against when building an apps. But it was pretty low-level, you could switch the different resolutions supported by the graphics on the fly, display different hardware register and poke to them, it was just missing a Machine Language monitor ;-p. Yes input were limited to using joypad and buttons, and if you plug in a dev board you then had access to the keyboard to type in command.
You could see here how our tools look like in action on the actual hardware ;-).
http://twitpic.com/tajpw
If you add function calls, you’re getting close to RPC ๐
Ahh, brings back memories. I see this has striking resemblance to the old liquid tweaker ๐ I still say we should hit Charles for luring us down the C# route. Like unsuspecting little chickens to the slaughter.
Yes, it’s very similar to the Liquid one (minus the binary channels). C# made a lot of sense on Windows, so there’s no regrets there.
Actually, I just talked to Charles and we’re going to open source the server and the client and hopefully other people can help with native Mac (or Linux or whatever) client implentations.
Looking forward to the code – would be great to have something to jumpstart my own debugging client.
I’m going to go on record here that basing an editor around your game engine, even if the tweaking tool is remote, is “a bad idea”. Of course I’m not knocking realtime tweaking, it’s useful, especially if you’re making a very simple game for a small team.
But there’s another side to it.
Think of it this way. Let’s say you wrote a program to display an image. Assume the images are rgab 1 byte per channel. I’m sure that’s a pretty simple program regardless of how you imagine writing it. Consider this your “image display engine”. Now, following the logic above, you could make an in game tool to edit that image. You might have options to draw a single pixel at a time, select a color, maybe something larger than 1 pixel, draw with the joypad. Or you could do this remotely. You write a front end, and edit, now at least you can use the mouse.
Following that plan, how long would it take get to the power of Photoshop? Without going through every detail I hope it should be pretty clear that if you want Photoshop, you don’t start with an image displaying engine and start tacking on features.
To get the power of Photoshop you have to design an editor, not an engine. An editor is designed for ease of editing. An engine is designed for speed of display or play. You’re not going to get layers, blending effects, layer styles, undo, 9 kinds of plugins, text layers, vector layers, importers, and all the other thousands of Photoshop features that make artists so productive by starting with the image displaying engine.
The same is true of games. All these game editors that start with the game and then add editing on are starting from the wrong place entirely. This why editors like Unreal, Unity, etc are not even close to the level of sophistication as Photoshop, Maya, XSI etc. They were never designed first and foremost as editors, they were designed as games. They don’t have the tens of compounded layers that let the designers build things in editable components and then “bake it” into game data the same way photoshop takes 50 layers and bakes it into a jpg.
It’s that “designed for editing” that makes “editors” like Photoshop, Maya, XSI, Illustrator, so powerful and let’s them make artists so productive. And, it’s that lack of “designed for editing” that makes tweakers based on the game engine a path that eventually leads to a deadends and poorly designed tools.
Gregg, The idea of a remote editor or a built in one is not to replace tools like Photoshop or Maya. This kind of editor is for higher-level data: game parameters, debug info, AI options, data relationships, etc. Assets will of course be generated there by the artists and baked into a game-friendly format. And if you do it right, you’ll be able to hot swap the assets currently running in the game as soon as the artist saves any changes, so they can preview their work in semi-real time on the target platform. Really invaluable if you get a system like that in place (which can again be based around this remote tweaking protocol).
I understand the idea of a remote editor. My point is it’s exactly the mentality that leads to bad tools.
The programmer thinks “it’s just take me a few lines of code to expose the entire game tree to be tweakable”. After that each new editing feature always seems like a just few more lines of code. The program never realizes the fact that basing editing on the game engine was the path to bad tools, inflexible tools in the first place.
I’m not saying you want your game editor to replace Photoshop and Maya. I’m saying that a game editor that is as powerful at editing games as Photoshop is editing images and Maya is editing 3D is what you want. I’m saying the path starting from tweakable game data is NOT the path that will get you there. In fact that path that will lead you to keep thinking you have good tools when in reality you’re seriously hobbling your designers because right from the start.
If you want good tools to edit “game parameters, debugging info, AI options, data relationships” and other game data you have to start from the “how should I design the tools to make the data editable” and not “how should I tweak the final game data”. The tools you end up with should have the layers and other abstractions that let designers create these AI options and data relationships with as many tools at their disposal as Maya gives for 3D and Photoshop gives for images.
Gregg, I understand what you’re saying now. I actually agree with you: Tools are very important and you don’t want to create a tool a particular way just because you can tweak it remotely through this channel. For some data, the dumb GUI tweaker is as ideal as it gets: turning on wireframe, toggling AI updates, etc. For most data though, you really want good, specialized tools.
At the same time, real-time (or almost real-time) live updates to the game are extremely important. What this (or any other) remote channel allows you to do is to change any content through any editor you want, and send it to the game to hotload (whether it’s just a ping saying reload a file from disk, or actually sending all the binary data). Maybe you can get away with just updating some files and have the game monitor them, but that’s a pretty coarse method for data that could be pretty small (especially if you’re using some sort of pak files). In other words, there’s nothing stopping you from getting the best of both worlds and hooking up the remote tweaking to your tool of choice and edit any data on the fly.
Agreed, but I see this as very similar to the TDD argument. For TDD the argument is you should always write the test first because it leads to better design. The same is true for editors. As just one example, the fact that you already spent time writing a serializer for the game data means you’re already hobbling any editor based off the tweaking. The editor needs to serialize the data in it’s editable state, not it’s final game format. It needs to save all the data of layers or splines or curves or other abstractions that are helping to make the data editable. Of course you can just write a separate serializer for the editor but to me that’s like arguing you can write unit tests after you’ve written your code. Starting from a certain approach adds momentum to that choice. Momentum which in the end is leads to poorer design.
This article reminds me of stories of Forth developers and their three-instruction system for talking to devices.