in Agile development

A Day in the Life

High Moon Studios is an unusual company in the games industry. We’re applying agile methodologies for all of our development. My team in particular is using both Scrum (an agile management methodology) and Extreme Programming (an agile engineering methodology). And yes, that means we’re doing pair programming, test-driven development, and all the other often controversial practices. I expect that in a few years, these practices will be a lot more common than they are today.

This article was first published in the 2005 Career Guide issue of Game Developer Magazine.

I lead the R&D team here, and my team’s primary responsibility is to come up with the technology that game teams use for different projects. Nowadays, that means putting a lot of middleware programs through their paces and choosing the ones that suit our needs. But it also means getting down and dirty and writing a lot of code for our engine and tools.

With that in mind, come on and follow me through a typical workday.

8:10 AM

I roll in to work on my bicycle like I do every day. Even though I’m an early bird, Jim, a programmer in my team, arrived a few minutes earlier and is already at his desk.

I quickly catch up with my email. I also notice that the PCLint pass on our codebase last night caught a couple of minor warnings, so I quickly fix those and check them in.

8:20 AM

Today is Tuesday and our two-week iteration ends on Friday. An iteration consists of a fixed period (usually two weeks in our case) during which the team commits to deliver a set of functionality described through customer stories. The customer (in our case the other internal teams in our company) creates and prioritizes a set of stories. My team then breaks down those stories into tasks and estimates how long they will take to complete (tasks vary between 1 and 8 hours).

Jim and I walk over to the “war room” (the room with all the task cards corresponding to user stories pinned to the wall) and we choose the task that reads “Blend ragdoll and animation,” which is part of the user story listed as “Throwing a rigid body at a character and seeing a hit reaction.” The task was originally estimated at 3 hours, but now that we know that the ragdoll system is a bit more complicated than we thought, we re-estimate it at 4 hours.

8:23 AM

In addition to our own personal desk areas, we have pair-programming stations in the R&D lab, with two monitors, two keyboards, two mice, two chairs, and plenty of room for two people. All production code is written by pairs of programmers. We grab a station and start working on the task.

Since we’re using test-driven development, we first write a very simple unit test for what we want to do, and only then write the code to make it pass. In this case, our first test checks that we create a blender object without inputs and that it produces no output. Then we write the blender class and make the test pass. It’s a tiny step, but it’s a step in the right direction. The whole cycle of write test, write code to make test pass, refactor, takes only 5-10 minutes, and we do it over and over.

8:39 AM

We have implemented a small amount of functionality, and all the code builds and all tests pass, so we check it in source control. This is called continuous integration, and it requires that programmers work on the latest version in source control and check in their own changes many times throughout the day.

8:50 AM

Other people from the team roll in, grab other pair programming stations, and start going at it. On their way in we have a quick chat and find out what we’re all working on.

Work hard...
Work hard…

9:37 AM

I overhear Joel and Gary discussing how they’re going to test something that requires updating the physics simulations. I just did that a couple of days ago so I jump in the discussion. It turns out they need to do something that is almost the same as what I already did, so I point them to what I wrote and they will modify it to suit their needs.

10:05 AM

Things are moving along very nicely. We’ve checked in four times already this morning. When a pair gets really going, they might check in as many as 20 or more times in a single day. At this rate we might be done sooner than the four hours we had estimated.

10:14 AM

We have the daily Scrum meeting at 10:15, so we head over to the war room. Scrum meetings are very short, standup meetings with the whole team (eight people plus Brian, our producer). We quickly go around the room discussing what people are doing to get everybody up to speed.

10:23 AM

During the meeting the topic of how we’re loading physics assets came up. So we return to the pair programming area and have a quick discussion with everybody involved. We draw some quick UML charts on a whiteboard, think about how the data is going to be passed around, and after ten minutes we reach an agreement and go back to work on our tasks.

11:15 AM

We got the blending working correctly. All the unit tests pass, although we haven’t implemented it in the demo yet. There’s another task card for that. We check the code in right away. The code definitely can stand to improve in a few places because we were just concentrating on getting things working, so we spend some time refactoring. We have lots of unit tests, so we’re confident our refactoring isn’t introducing any bugs.

11:56 AM

The code is now in a much better state. We check it in and wait a few minutes for the build server to report that everything built correctly and all tests passed. During that time we talk about what the next task should be.

12:05 PM

We have nice shoulder-high surf today, so I’m going out surfing at lunch with several of my teammates. If it’s not surfing, it’s a basketball game, cycling, running, or even yoga. If all else fails, there’s always a game of Guild Wars with the rest of the High Moon clan.

1:10 PM

Back at my desk I eat a quick lunch while I catch up on my email (which I haven’t read since this morning because I’ve been at the pair programming station). I read an email from a middleware developer in response to one of our queries earlier in the day and fire back a quick response.

1:25 PM

Sean stops by my desk. He’s ready to go back to work, but the programmer he was pairing with in the morning got pulled to work on some last-minute issues with Darkwatch, which is about to go gold and has priority over anything we’re doing.

Sean quickly brings me up to speed on what they had been working on that morning, which was to display the exact memory usage for our physics system. I remember them mentioning that in this morning’s Scrum meeting. I also worked on the physics system last week, so I’m pretty familiar with the code. In a couple of minutes we’re already making progress.

...rock hard
… rock hard 🙂

2:07 PM

After writing several unit tests and implementing some functionality, we’re ready to add the memory display to the demo program. A couple of minutes later, we have a display in the demo indicating how much memory the physics system is using, and we see it go up and down as we add and remove rigid bodies from the demo.

But something is wrong. When we remove rigid bodies, the memory doesn’t come down to the same level it was before. We exit the demo and we see a long memory leak dump. First things first, we check in our changes, and then we dive in and look for the code that is leaking memory. It’s probably not caused by the code we just wrote, but we have “collective code ownership,” which means that everybody is expected to fix anything that needs fixing anywhere.

2:12 PM

The build just broke! The build server detected a failed build and notified us through a system tray application. I bring up the latest build log and I see that one of the unit tests failed in release mode. Tyson, who is sitting at the station next to ours, says “Oh yeah, I know what that is. I’ll fix it right now.” In less than 30 seconds he makes the change, and checks it in. A few minutes later, the build system reports a passing build and everything is back to normal.

2:17 PM

We found the memory leak. It was a misuse of reference counting. We first wrote a unit test that showed it failing, and then we fixed in the physics library. We check in our code.

2:18 PM

We go to the war room and grab the next task. This one has to do with being able to expose different variables and functions on the demo to tweak them through a GUI. We sign up for the task and start working on it.

3:43 PM

We’re totally in the zone. We’ve been writing tests and code like crazy and this task is going really well. We’ve done three check-ins for this task alone in the last hour.

4:12 PM

Another pair is discussing how to handle errors for some particular case. This is an important topic and it should be done consistently across all the code, so we have a quick discussion about it involving most of the team. Five minutes later we’ve made a decision and we all resume working on what we were doing before.

5:40 PM

We do our last check in for the day. We’re almost done with the task, but not quite there yet. Even though we could stay another hour and try to finish it, we’re both quite tired by now and we’re starting to not think as clearly and make some mistakes. We can wrap this up tomorrow morning as soon as we get in. The important part is that we got to a state where we could check in.

We have a rule that says nobody can check in code and leave. You have to wait for the build server to build the code successfully and pass all the tests. We keep build times short, so that usually means hanging around an extra 4-5 minutes. If anything breaks, you need to fix it or revert what you did, but there’s no excuse to leave with a broken build.

While I’m waiting for the build server to finish, I check the pile of email that accumulated in my inbox during the day.

5:44 PM

After a few minutes we get the green go ahead from the build server. Today was a pretty productive day, and at this pace we will definitely complete all the user stories by Friday. The demo we’re putting together is also starting to look very cool.

One of the things that agile development, and especially pair programming, does is to make each day very intense. There are no little breaks to read email, check a web site, or just goof around. We get a lot accomplished in a work day, but we can’t keep that pace for a long time, so it’s important to call it quits and go home. That leaves me with time at home to read technical books, prototype different ideas, or work on side projects in addition to unwinding, spending time with my family, and enjoying other hobbies.

I hop on my bike and head home with a big smile on my face.

  • ent

    Really interesting. What software are you using for your server builds? Currently we are migrating from a monolitic perl script to CruiseControl.NET + Nant. I’ve already read your articles about the perfect build system but I can’t deduce the system you are using.

    Thanks.

  • Sounds like a real nice way to work

    Noel Llopis, over at Games From Within, has written a nice piece about a typical workday at his work place, High Moon Studios. It certainly sounds like a nice way of working. Unfortunately I haven’t been involved in quite that…

  • Yes, our build server is CruiseControl.NET + Nant. I wanted to go into more detail about it, but I had to keep the article within certain word limits for Game Developer Magazine. We use Nant mostly as a collection or projects/solutions to build, and then call an exec task with devenv.com or make on them (no, we still don’t have a unified build back end unfortunately–one of these days).

    CruiseControl.NET + the system tray applet is just an amazing set of tools. I don’t know how I worked without it for so long. I’d highly recommend it to everybody, even if you’re not doing continuous integration.

  • Nice! We just about have a continuous build over three platforms, but not using anything as complex cruise control. It’s just cron’d to start at 9am and terminate at 9pm on linux, solaris, and win32. Our PC-lint batch picks out *thousands* of issues we can’t even begin to address yet. Our back-end is also not unified. If only. Bash on *nix, and msdev.com + jscript on win32. Any progress with SCons and the like?

  • Thank you for posting this. That helps quite a bit to visualize an agile workflow. I am curious about details about actual code, builds and other things.

    Not knowing your code or the exact requirements, 3-4 hours to blend ragdoll and animations sounds very short. Of course I’m imagining from zero. You are adding to an existing system so there was already probably lots in place that made it take only 3-4 hours. I would be helpful to see actual code or maybe # of lines written or something. I know # of lines is possibly not a useful metric but you mention writing a new class, some tests, testing it and checking it in, all within 15 minutes. That sounds like a lot. Was there code generation? Did you actually document the class? Were you copying a pre-existing class? Had someone already designed this class beforehand?

    You also mention checking in 4 times in the in the first 1 hour and 20 minutes. How did you make that possible? For example on my current project just getting the latest code and compiling and checking in probably takes 5-10 minutes. That’s possibly 40 minutes lost in that 80 minutes. Of course if I was checking in that fast maybe there would be no other changes and it would go quicker. I know one method you clearly use is your build server(s), something my team doesn’t have but if you could go over that.

    PS: I’m not asking you for a quick answer in the comments here, rather I’m hoping maybe you’ll have time in the future to post more detailed articles about your processes. Thanks again for the great article.

  • This sounds a lot like how our days go. I hadn’t heard of anyone else doing the dual monitor/keyboard/mouse thing, but it does seem pretty obvious once you switch to it. Our office is too overcrowded for war rooms and home stations (we just make at least half of the machines pairing stations) and we’ve definitely felt the pain of not doing those things as well as we could.

    What does “informative workspace” look like at High Moon? One of the benefits of not having a war room is that you are living with the visible artifacts of the team’s progress. It’s very reassuring to see the user stories burn down as the day (and week) passes.

    Thanks for the wonderful article and hopefully I’ll see you and Clint at GDC.

  • Actually, quite a few things have changed since I wrote that article. I think I wrote it close to a year ago, so it’s natural that things have continued to change.

    One of the main changes is that, at least for my team, we’ve moved all the project information (what was before in the war room) into the work area. That alone has made a huge difference. There’s no substitute for looking up and seeing the remaining tasks, or seeing another pair go grab a task and start a discussion based on that.

    Another big change is that we’ve been adding more build servers, so now we have one server per platform (Win32, Xbox360, PS3, tools). That makes it so that the average check-in takes about 30 seconds to build, which allows us to check in furiously all day long.

    A lot of people have been asking, so I’ll definitely write an update to this article, specifically looking at the build servers and the pair programming aspects.

    And yes, Clint and I will be at GDC giving talks on agile development, so stop by if you’re there! 🙂

  • What I’d like to know is how your user stories are generated. My experience with game development is that you get a lot of issues entered like, “The monster in room 47 doesn’t have realistic enough movement.” Hard to translate into a set of tests!

  • I know I always seem to have a bad attitude when I post in your blog and not to disappoint…

    I really am having a hard time believing that what you describe is typical daily development. The description reads like a best case theoretical day from a book and not a typical average dev day. If you really can have a majority of your dev days like this then that is just freaking amazing to me and my hat is off to you and your team. However.. I’m a skeptical puppy.

  • Billy, it might seem hard to believe, but I’d say that what I described is actually a pretty typical day. Not a bad day for sure, but nothing exceptional (other than the shoulder-high surf at lunch). Notice that we had to re-estimate tasks to take longer than we had expected, and we left for the day without finishing the last task.

    One of the consequences of agile development is that you’re always focused on things that matter, and you have small, conquerable tasks in front of you. You’re always getting feedback from your tests, the other programmers, and by finishing up tasks and taking them down.

    I’ve never had to sit for days tracking down a crash bug, trying to stabilitize the build, spending the day in pointless meetings, working silly hours, or waiting forever for builds or data. To be fair, we are using a codebase that was written with TDD from the start, so it’s a dream to work with. If something gets in the way, we fix it (for example slow builds).

    Thinking back, a “bad” day for us was spending almost a full day trying to get the memory allocation just right to pack the mip tails on non-power of two textures on the Xbox360 (we way underestimated that), or even today, fighting with make and sed to get custom build rules into our PS3 builds. Other than that, I stand by that day being pretty representative.

  • Videogame development e Extreme Programming

  • One more question that popped into my head from reading the comments above.

    You mentioned at the top of the article you head up the R&D team. I am correct in assuming that means you write libraries and other teams different teams make games. If so, that would suggest that maybe what you are doing is not game progamming but library programming and that maybe because of that your stories are easier.

    Is that the case? In my game programming experience typically a designer sits a desk or two away from me and we run ideas back and forth as in…

    Designer: “hey Gregg, can you make the blue blob fade out when xyz happens?”

    Me: “I can make the blue blob fade out when it gets a signal to its in_fadeout input. You can then connect it to anything you want including when XYZ happens.”

    Designer: “great!”

    Me: (5-10 minutes later). “Okay, I checked it in. If you update you should be able to make the blue blob fade out.”

    How does that “game programming” fit with agile developement?

  • I give you my hat Noel. If I lived anywhere near highmoon I’d consider applying just so that I could get the experience of working in an environment like that.

  • A nice day in the life diary

    This is a nice story of a team using agile development techniques. They even have pictures of the work environment… http://www.gamesfromwithin.com/articles/0602/000104.html Very nice….

  • Definitely being in the R&D team makes applying agile development simpler. But it doesn’t mean we just write libraries. In a true agile way, we try to deliver vertical slices of functionality to our customer (in our case, usually some of the directors). So inevitably we’re writing and delivering demos that show particular features they asked for. We do have artist involvement, but not nearly as much as a full game production.

    However, the game production teams at High Moon are also using agile methodologies (even if not full XP yet). To learn more about it, check out Clinton Keith’s web site (http://www.agilegamedevelopment.com/blog.html) and his GDC presentations. That’s exactly what he talks about.

    Maybe the key insight is that the designer isn’t a few desks away and comes back to ask for something. The designer is in your team and you’re working together on a particular feature.

  • I do appreciate that you threw in the picture of me absolutely dominating Rory at Guitar Hero.

  • Great article, Noel. I’m very happy to see Agile methods work well in game development. I’m not a game developer but an enterprise app developer and have been using Agile/XP techniques for many years. Sorry, I couldn’t attend your talk at GDC but I think it’ll be impressive.

    If you have some down time, we could share our agile experiences. Good luck!

  • Tell me a story

    This morning I read a post from Noel Llopis about one of his typical days. It got me thinking about agile development again.

  • Recommended blog: Games from Within

    I just stumbled across Games from Within, Noel Llopis’s blog on agile technologies and processes applied to game development.

  • nda-gamedev

    So how do you unit test graphics code? Let’s say you discover a bug that you forgot to reset/initialize a particular OpenGL state which caused funky rendering artifacts. Then you fix this bug.

    How can you write a test which fails with the old code and passes with the new? Have the test code run glReadPixels and compare pixels? Or is there some better way?

  • > Maybe the key insight is that the designer isn’t a few desks away and comes back to ask for something. The designer is in your team and you’re working together on a particular feature.

    Could you elaborate? I’m not sure how the artist siting next to me and the artist sitting behind me are not “in my team”. On features we need to work on together we work on them together but for the most part we, the programmers, are providing “smart items?” that the designers and artists can use to build the “game”. Those items though or often thought up by the designers at the momment, not planned days or weeks in advance.

    For more complicated items like a particular creature there is some discussion about what it should do, then a paper sketch of the assets needs (meshes, motions, hot spots, whatever). Then the artist goes off on her own for 1-5 days making those assets. When they are done or at least usable the programmer, by himself, will use them to get the creature working. Once it’s working they may sit together and tweak it.

    Maybe your definition of “feature” would help?

  • I really need to follow this up with a more detailed article given the amount of questions this is generating. Here are some quick answers:

    – Testing OpenGL is tricky. You want to test your code is doing the right thing (not that OpenGL is doing the right thing–you hope it is!). So you can just test that you’re calling a function you wrote that sets a set of render states and that function is untested. You can also write a wrapper around OpenGL and test that the functions are called, but that’s probably more work than it’s worth unless you’re doing a middleware renderer. Testing the pixels is more of a functional test (which also has a place, but not as part of fast unit tests).

    – Working with a designer. Check out Clint’s web site and presentations. That’s exactly what he talks about in detail and can explain it a lot better than I can here. But briefly, you’re working with a small team of programmers/designers/artists towards a small set of goals for an iteration (2 weeks). The goals (stories) well defined (but not micromanaged). For example, fully implement the boss in level 3 to shippable quality, or implement grapple moves in the fighting system. With that, you break them down into tasks (things you need to do that take anywhere between 1 and 8 hours) and everybody in your team works on them.

    My work with designers/artists has been much more iterative than you descrive. We talk about what we need, and they give us a rough version in an hour or two. We bring it in the game (or see that are some problems and need to be changed). Now we can work with those assets and they can see it in the game. They might continue adding assets incrementally and working with programmers to try different things, discuss performance/memory implications, etc. But certainly, nobody dissapears for 1-5 days and comes back later with assets. Maybe you can do that once you’re in full production and the pipeline/technology is well known though.

    Also, check out the Agile Game Developmen group (http://groups.google.com/group/Agile-Game-Development). Perfect place to bring up questions like that.

  • Ray

    Are people always paired with the same people or do you ever mix it up?

  • Ray, pairs rotate quite frequently. At least once per day, sometimes more frequently if two pairs finish a task roughly at the same time. Rotating pairs is really one of the key aspects of pair programming. Without it, you wouldn’t get many of the benefits (better communication, spread of knowledge/culture/values, etc).

  • A Day In The LIfe of an Agile Shop

    Doesn’t hurt that they are developing games – something every programmer secretly wishes he was…

  • Ray

    Its got me very interested your article has (excuse the newb questions) Do you get problems that people can never specialise in an area (ie would you never have roles like – physics programmer or audio programmer).

    Also initially did you get problems where some people (the know it alls or senior experianced guys) needed to be in paired with most of the task creating bottlenecks?

  • Games and Setter Based Injection

    ConstructorInitialization “Constructor Initialization is my first choice. There are cases when it’s difficult to set things up this way and I do occasionally prefer setter initialization, but most of the time constructor initialization is the best bet…

  • XP and Game Development

    Noel Llopis has a fascinating article about using Extreme Programming at a game company. I found it interesting because it’s not only an example of picture-perfect XP, it’s also about a team that’s doing game development, not something you see every day.

  • Oliver Charles

    Hey, I stumbled across this blog a while ago, when I was researching test driven development. Primarily being a game dev’er I didn’t know how to apply it, but your blog showed me how, and very nicely!

    That entry makes your company/life sound so fun! Is every day like that? How hard is it to find a company like that? I’m only young atm, not old enough to get a job yet – but when I can – thats exactly the kind of place I want to work in (agile development, etc).

    Any tips?

  • Oliver,

    Agile development is not very common in the games industry… yet. Give it a few more years and you’ll be seeing many more companies adopt it (OK, so that’s just my prediction).

    You might want to read what I wrote a while ago about advice for people looking to get in the games industry: http://www.gamesfromwithin.com/articles/0501/000066.html

    In the meanwhile, the best thing you can do is to write games and mods of your own. Good luck!

  • A day in the life of an agile developer

  • A day in the life of an agile developer

    I actually believed that these were just bedtime stories, but it seems that Noel

  • A Day In The LIfe of an Agile Shop

    Doesn’t hurt that they are developing games – something every programmer secretly wishes he was…