March 1, 2015 at 4:59 am #361
Over six years experience in QA on a professional level, the majority of which was in games (over 70 games, numerous platforms). I’ve just about ran the gamut of test responsibilities, so if you have any questions about considering QA, tracking coverage, or how to document an issue, feel free to let loose.
April 1, 2015 at 10:10 am #2010
I work as a QA analyst for a desktop application company, so a lot of mine are going to be comparisons, sorry. 😛
What would you say is the biggest difference between QA for traditional software and games?
Is there any difference between lodging defects/bugs for traditional software (ie. What were you doing? What happened? What did you expect to happen?) and games?
Do you think the misconception that people going into game testing have (bro, get paid to totally play games all day) has had a detrimental effect on the quality of games?
Does the recent trend of Early Access put QA roles into jeopardy?
How does a game as buggy as Assassin’s Creed: Unity pass QA?
Last one: What’s the funniest bug you’ve found?
April 12, 2015 at 3:41 am #2137
The biggest difference is the finite limits and thus the ability to cover an application versus a game. Applications are meant to be tools, finite, discoverable, consistent, with no surprises. Games however mask their variables and functions as much as possible, through intentionally complex layers, dice rolls, and aspects like AI and loot drops. Because of this, game testing involves a lot more concerns about race conditions.
With traditional software I would favor going for coverage of features en masse, and that is where I would start with games – but ultimately you’ve got to look at 3d matrices of combinations, using a pairwise iteration. And within that, repeated attempts, because games are designed to not be 100% consistent.
Bug logging is largely the same, though so far I’ve found game bug logging to be more exacting in terms of screenshotting issues, and noting exact error messages. Traditional software developers react strangely when I give them a picture of the bug, or when I call out exact errors. That honestly weirds me out.
The misconceptions definitely have a detrimental impact. Games QA can always find lots of testers. Kids fresh out of high school or college, or adults who’ve yet to grow up. Because of this glut of potential hires with low expectations, it is in the immediate fiscal terms more viable to throw lots of inexperienced testers at a project than to cultivate and maintain a professional team. Professionals tend to marry, raise families, and require more pay for their experience, and their families. So what you get is a boom bust cycle with the release cycle. Every spring or summer, you hire a bunch of gamers at cheap rates, your few experienced people have to train them into something more than just playtesters, which usually takes a few months. At that point, you’re approaching the 3rd quarter, so there is a crunch of testing with a focus on console certification. After the game goes live, many bugs are too risky to fix on the user’s machine without damaging data, so the scope of fixes that would be accepted shrinks. Naturally, the QA staff shrinks as well to what is needed for sustainment – unrisky patches, DLC. During this time there are successive layoffs, and the experienced staff gear up for the next wave of hires – some of which are rehires – to begin it all again. The result is that most games are tested by about 5-10% people who know what they’re doing, 30% who’ve done it before, and the rest are completely fresh. Those numbers aren’t based on any actual study, just personal observations. If people had a more accurate picture of what game testing was, it would reduce the glut of potential hires so management wouldn’t be lured into a meat grinder mindset, and it would increase the receptiveness of new hires to the training.
I see little threat posed by Early Access. Users are notorious for not finding substantial bugs, and when they do, for not reporting them in any useful way. The bigger threat is when established companies eschew an actual QA team or role/rely on their community, and the growing number of indie developers who don’t take QA seriously and thus could mold a generation of developers who see no reason to budget for dedicated QA. In the past 7 years I’ve offered professional part time free games QA to indie devs countless times. Only one project has seriously taken me up on it. Even with the currently horrible standard of QA in games, traditional developers and publishers still trounce indies on the subject of quality, because they acknowledge it as a subject worth investing in, in any form.
QA isn’t always a gatekeeper, sometimes all they can do is shout, and hope someone hears them. If producers don’t read emails from QA, don’t mind the bug database – or if they simply don’t care, it doesn’t matter how good or how bad of a job QA does. For all we know the Unity db was jammed full with excellently written bugs covering the lion’s share of issues the public complained about. Much of this is determined by publisher culture. I can’t speak for Microsoft, and wouldn’t dare, and will note that it is a large company with many different cultures inside of it, but the ones I’ve encountered had a mindset of QA as a gatekeeper, for the most part. If we yelled, they listened. Sometimes if we so much as grumbled they listened. A lot of bugs were intentionally not fixed, but it was 99.999% of the time due to the risk of the fix to the codebase or the user, or the amount of time it would take to fix versus the importance of the bug versus the bugs that could be fixed in that same time. When a studio only has the budget for X number of hours a week on bug fixes for a released game, do you correct the dozens of typoes in a localization, the crash that is very difficult to find, or the periodic animation error which warps limbs temporarily?
Funniest bug? Gads, I’ve found so many bugs. I have a hard time remembering which games I’ve tested. Years ago I had to start a spreadsheet just to track them – and I created that by browsing marketplaces by platform to jog my memory. I more so remember particular bugs of projects I managed the testing on – partially because I would audit the bugs and send them back asking for additional details.
I tend to not find Funny Bugs so much, as I like to get in early and find as many structural vulnerabilities in a design, and log those to head off bugs. I don’t want high bug counts, I want low ones.
Actually, one of the bugs I found which made me laugh wasn’t in a game I was testing. I was just playing. There is a mission in CoD Ghosts where you are sneaking along in the grass through a busy compound with lots of soldiers and vehicles moving about. If you idle too long in an area, well, they only scripted so much walking for the soldiers. They stop in the roadway, but they scripted more for the trucks… What is the sound of a dozen special ops guys being driven over by a truck going 10mph?
You must be logged in to reply to this topic.