The Video Game Pipeline

action-superhero-mmo-games-marvel-heroes-december-beta-screenshot-11

The Problem

I tend to be a big advocate of strong and frequent communication with video game players.

Partially, this is because I think frequent communication provides a more human face for a development team, making players more tolerant of the inevitable missteps and more understanding of the limitations of the development process, as well as tending to improve the overall tenor of a game’s community. People like knowing that they’re being listened to, even when you often have to say, “Sorry, we can’t do that for X and Y reasons”. Partially, as well, frequent communication helps the developer by providing a strong stream of subjective feedback from a game’s most passionate players which is a good way of identifying legitimate problems that should be investigated and addressed.

It is true that the players who comment on forums, for example, represent a distinct minority, but even if they are not a demographically precise sampling of a game’s user base, if a significant majority of these players, self-selecting as they are – are evincing pain, it’s probably something worth looking at. Players (and sometimes executives) frequently lack the necessary context to be able to identify what are realistic solutions and what are not, so where a developer is willing to open up and communicate more aggressively, the average quality of proposed solutions tends to go up.

simpsons-angry-mob1

One of the areas where players and developers often clash is over the speed of changes, new content, and bug fixes. Developers are (rarely) actually lazy, but the process and the pipeline to minimize risk of deployed issues is generally quite intense, but most players have very little insight into this. To a player, a bug that sits around for a month before being fixed can conjure up an engineer sitting with his or her feet up sipping Mountain Dew while playing League of Legends, when the truth is (usually) that there are a lot more steps a fix has to go through than players are aware of.

The exact process varies from development studio to development studio – length of release cycles, platform, company development philosophy, and other considerations all have a major impact on the exact structure, but here’s one example – in this case, for Gazillion‘s Marvel Heroes, where I currently work as the game’s Lead Content Designer – of some of the specifics of a development and deployment pipeline for those who have never been exposed to something like this.

The Pipeline

Gazillion’s development pipeline can be thought of as the synchronizing of two interlocking wheels, sort of like an Aztec calendar. One wheel is the code and data set that developers (designers, engineers, artists, audio folks, etc.) are all working on (the Development wheel), while the other wheel is the production deployment (the Deployment wheel).

Both wheels are turning at the same time, but they interlock with each other only occasionally; periodically, the deployment wheel takes a snapshot of what is on the development wheel, kicks that off to QA who then reports on what critical issues there are. Developers then have to fix – sometimes on each wheel separately through what is called a merge, sometimes directly into the wheel – any found issues.

The deployment wheel turns again and kicks off another wheel, it goes back to QA, rinse and repeat until you have a “good” build. Obviously, this takes a variable amount of time based on what issues arise. Once there is a good build, the deployment wheel turns yet again, and this time pushes it to the Test Center on a white list. Rinse, fix, repeat. Then another turn of the wheel and it goes to the Test Center sans white list. Rinse, fix, repeat. Once that’s finally good, then it goes to the Live service.

All this time the development wheel is turning, but you can see here how the development and deployment wheels can – depending on the timing – either be in synch or out of synch by as much as two – or even more – weeks.

2070376490_ac67348870_o

A two week cycle is in the industry very aggressive and very risky; more common (and more safe) is more like a month to two months. Any single gamestopping issue can halt the entire deployment wheel, and until that issue is unjammed, it can’t go forward.

Looking at the patch notes is, as well, only a part of what is going on. The patch notes include everything that is (or is supposed to be, at least) player-facing, but there are a ton of things that go on under the hood from fulfillment services, integration with outside services (like Steam), website integration, server and client optimization, and so on. Any of these things can halt the entire process which is why it is impossible for us to absolutely guarantee a deployment date for any specific piece of content.

For major releases, as you can imagine, things can get very tight and very stressful since there you have an enormous amount of external visibility from the community, from Marvel, from Gazillion’s board, and from game sites. To be sure, we bake in as much extra time as we can to account for things, but by definition you can’t anticipate a Black Swan event.

8 thoughts on “The Video Game Pipeline

  1. Blogs with explanations like these can work miracles when it comes to creating an environment (or even community in some cases) in which customers don’t go for the torches and pitchforks when something goes horribly wrong and isn’t fixed in 3 minutes.

    Knowing the ‘why’ creates understanding in many people. One example that made a huge difference for me personally was the change in communication that the Dutch Railways implemented years ago. Before, you could sit in a train that stopped in the middle of nowhere for 20 minutes and not know why. Nowadays, after about a minute, someone will inform the passengers of the reason for the delay and an estimated duration. Where you could feel the tension rise with every passing minute before, now you mostly see people mutter a bit for a minute and then lean back in resignation. Passengers even try calming down others ‘because there’s nothing that can be done really’. And on-board staff don’t get their heads bitten off when they pass through a compartment.

    Whenever I’m in a delayed train, I thank the communications expert that convinced management to implement this policy.

    If you tell me that we’re in stage 3 of a 5-stage dev/QA process that cannot be short-circuited to deply a fix, the same thing happens as on the train. It’s not me waiting on you anymore, but we’re in it together: both waiting for the wheels to turn and the train to move again. I’m staying away from the torches (unless your dev cycle makes no sense to me, but that’s where communication comes in again), leaving you room to keep me informed instead of going crisis-management mode.

    Thanks for an insightful read, I’d consider posting it on the forums as well some day.

    • Thanks for the feedback.

      Yes, I agree that simply knowing the why and understanding the process can help a lot to defuse the anger. Maybe not the frustration, but understanding helps a bit, at least.

  2. What a wonderful perspective. As a long time player of MH, I just want to comment that I for one greatly appreciate your approach to communication with the player base. It is, to be frank, one of the things that has kept me playing this whole time. You guys have not always been perfect but I find it refreshing how frequently, and honestly, you communicate with the player base. Keep it up!

  3. I think your desire for open communication comes clear on the forums, and I applaud the desire and effort. I do not code for a living, but I have coded and have been in charge of feature rollout in platforms I support– so I can certainly understand the Aztec calendar metaphor– the interlocking wheels of plan and action, development and production.

    I’m certainly glad to see you offering your perspective here and hope that Gazillion has an open mind to these kinds of generalized postings. This may not be appropriate for the Marvel Heroes forums– not all players will receive this insight with an open mind– but I certainly appreciate hearing your thoughts.

    What helps me most in being patient is seeing the to do list and having a sense of where my critical pain points are in the to do list. Unfortunately there isn’t much transparency at this level– sure at a more general level there is– but not at this level of detail, although one of my (subjective) biggest complaints right now is that Nightcrawler is in dire need of a mini-update. According to TribalSage’s thread, he’s slated for update immediately after the AoU event is over. Hopefully that plan holds.

    Aside from that subjective issue, I think revisiting survivability for all non-health/non-defense characters (those who rely on some kind of mitigation mechanic such as dodge) is paramount– especially as you guys are moving into releasing the red version of the Genosha raid. But at the same time, releasing mechanics that penalize particular builds is extremely painful. (Iron Man’s CC break causing death in the Onslaught fight; pet builds causing deaths; etc.) Since not all builds are equal, this creates an inequality in hero desirability.

    And, an even wider perspective (which probably takes time to finish since you’re onboarding a new staff member responsible for hero balancing) is to figure out a way to gradually adjust all heroes when core game mechanics are changed so that you don’t end up in the place you are now (where it is a merry-go-round as to who is bottom of the proverbial barrel.) This would allow you to roll out new mechanics such as the Genosha raid (or future events) with specific mechanics that penalize certain builds, or tweak core mechanics, and then adjust hero viability so that everyone is closer to the desired norm.

    Considering how much flex there is in hero builds, I can certainly appreciate how difficult that goal is to achieve. But it’s (subjectively) a highly desirable goal to ensure hero’s aren’t categorized as “flavor of the month.”

    Thanks for posting your thoughts– I hope my thoughts are received as they are intended (i.e., helpfully analytical, as opposed to simply critical.)– I know you’re not the right person to talk to about this kind of thing. And I certainly think the content you’ve been adding has been top notch (even if it hasn’t all gone in bug free on day one. I see the work being done to clean up bugs quickly and try to treat the player base fairly when problems do arise. Kudos for that!)

    • Thanks for the feedback.

      As you’ve noted, yeah, much of what you’re saying I can’t really comment on as it applies to Marvel Heroes, since those aren’t my team’s areas of responsibility.

      It’s certainly the case that the game has set a very challenging set of goals to hit, though.

  4. Thank you so much for you time in writing this. I think I understood the basics of the completions. Especially the ” fix one thing break another.” As a huge fan I thing the disconnect comes from a few areas ( although obviously I can’t speak for everyone, I am making comments on the things that have stuck out to me ). Lets touch on content – While you’ll never please everyone, and nothing will ever be ” perfect ” because the variables are such that bugs seek in; I can’t help but refer to time periods and speed. You’ve got two schools of thought – 1. Get the content out there NOW! 2. Try and get in right, take your time within reason! While some of the torch bearers can be the most vocal ones and Marvel Heroes makes money off of pushing new heroes ( which I love adding new heroes to play ) there have been clear cut times were the time and player have clashed. Whether it’s pushing out the hero ” under-tuned ” damage wise, or maybe the unqiue’s haven’t been tuned; there are to many variables to name. While you’ll never people EVERYONE there are times when it’s ok to say, ” Let’s delay it, and briefly explain it to the player base ” Again it’s never going to be perfect and I love the game, but as my grandfather would say ” control what you can control. ” Anyways that’s my two cents, thanks for the read 🙂

    • It’s true, while there’s a temptation to say, “Fix all the bugs!” the reality is that that is an economically unviable option unless you are working for NASA.

      The closer one gets to bug-free, the more expensive it is to find each additional bug.

      As a result, at some point Production and management have to make the call as to whether 97% bug-free is not acceptable, but 98% is. Whatever you choose, some people aren’t going to agree with it, but if at the end of the day one has a viable business, I think there is an argument one made “a” right decision.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s