7/10: The review problem

[Brother None]

CVG offers a must-read feature on some of the key problems of videogame reviewing in today's world, a topic often discussed here on NMA. For those thinking videogame journalists get "bribed", you're wrong, the problem is much more structural than that, and CVG identifies a number of key problems: the way marketing approaches review scores and aggregate sites, and the way reviewers are forced to take aggregate sites into account if for no other reason than the huge backlash when they "dare" to give an AAA game a "low" score like 7/10.<blockquote>From the perspective of a reviewer, review aggregation sites need to be irrelevant. If you're going to let the opinions of the crowd shape and shift your own, the job you're being paid to do has suddenly become irrelevant. Whilst it's understandable that the number-crunchers in publishing will be obsessed with Metacritic, the fact that anyone else buys into this fuzzy-science is nothing short of bizarre.

It's odd enough that we see people actively dismiss games that average at anything less than an 8, but the extent to which aggregated scores have been championed by gamers is genuinely a little bit frightening.

If you're one of the people who've been devastated by a 'rogue' publication damaging the score of your favourite game, then I've got an important message for you: That game's publishers bloody love you. Seriously - they can't get enough of you. I guarantee that you've caused at least six people in marketing to get up out of their chair and do a funny little dance on the table.
For all these years, they thought the only people who'd care about these numbers was them - after all, they're the guys that actually often get paid a cash-bonus for achieving a Metacritic score. Obviously they'll be slightly bemused about why you do care, but they're utterly thrilled that you do.

Considering the disdain that most respectable gamers will have for sites that they don't personally enjoy reading, It's odd enough sometimes that people will accept a tainted average like Metacritic. It's even weirder when people start to get annoyed by an individual website's decision.</blockquote>Thanks GameBanshee.

 

[Guiltyofbeingtrite]

This is a great article, thanks.

It seems like videogames are attaining a sense of fandom and homerism that sports have had for decades, at least here in the US. Just sub the specific sports for consoles and teams for games/publishers.

Ever since NCAA football rankings have been published on every sports news site you've had hordes of people go online and either brag or whine over "their" team getting/not getting the respect it deserves.

 

[PainlessDocM]

I miss the days when games were rated on a logical scale from 0-10. I wonder when 7 (or even 8) became the new 5/10.

I prefer scoreless reviews but I imagine those would generate less attention/ traffic and are therefore not a viable alternative

Will read the article later tonight.

 

[Brother None]

GameBanshee has scoreless reviews, at my insistence. But people don't really want to read multiple full reviews, and that's where scores and score aggregates come in.

 

[UncannyGarlic]

Score aggregate's aren't inherently as valueless as CVG suggests, in fact it's a great way to get a general idea about the critical reaction to any media with solid critical reviews. Metacritic is pretty good for seeing the distribution of movie reviews but less good at quick reads as to why (their quotes are pretty unreliable). The issue is that game reviews have been overinflated for more than a decade but has been steadily getting worse.

The problem is that anything less than a 7/10 is considered a negative review, largely due to critics going by the A-B-C-D-F scale. Another problem is that said scale isn't exactly universal. There are some critics who truly due use the whole scale. I'd argue that Metacritic doesn't necessarily have a properly calibrated scale for games and is probably not well equiped for dealing with the massive amount of score variation that happens between AAA games and other titles. It's valuable in similar ways to how wikipedia is, as a start and library of links for further reading and understanding.

 

[tekhedd]

I think Metacritic is a nice way to browse through the reviews, but you definitely have to take their aggregate score with a WCC-sized grain of salt. Or just ignore it.

Actually, the Mean by itself is relatively useless if we also don't have the standard deviation, and expected distribution, and if we don't always include all of the same review sites in every score. Metacritic's values are statistically pretty much useless.

 

[TheWesDude]

each publication needs to set standards for reviews.

10 criteria that are based upon the "genre" of the game marketed. some will be shared among all, but for each genre they should have some variability.

the reviewer then spends 1 paragraph going over that criteria and giving a review of that aspect of the game.

then they close with another paragraph to wrap it up.


if i care about story over graphics, then that is the category i would care more about. i dont care if a game got 7.5 or 8.0 if i dont know what aspects they are saying is not up to snuff. and in most articles they dont actually provide specifics of what aspects were lacking.

thats what i want.

 

[Sub-Human]

I agree. I remember seeing some of the older reviews at, say, IGN and being amazed at how games would get a 7.

I mean, come on, stop giving Call of Duty 9s. Someone over at IGN said the lighting effects are impressive. What?! You said the same thing 6 years ago, in '05...

 

[grayx]

Good read. Thank you for posting.

Yup, it is structural, if not even more general in its nature. I was thinking about this for some time and I don’t see one silver bullet to solve these problems. Aside the fact that there is only dozen sites nowadays that most of the people go for reviews, situation now looks something similar to a situation in which a lot of us are standing in the queue, but people start to break the line and “rules”. So now, it's "grab whatever you can" situation. I don't want to say before was better, still, people have been consistent much more in their reviews because there are not that many incentives to twist the narrative. Now, until there is some sort of penalty for people/sites who take money and benefits in exchange for a positive review, there is no chance in the world to solve this, in its core, “inflatory” problem – and I don’t see any way to change that atm. The moment they got money (or any perceived “bribe”) from publisher – they are getting unfair advantage and in world of cyberspace today, that influence many things very fast.

The most general solution imo would be education, education, education, but in a world in which 12-14 year old kid can compete with you in purchasing power because his/hers dad gave them credit card (or something similar) – there is no power in the world to turn the tide about that. Less educated – more chance for manipulation – better chance to sold some shitty product.
And game “journalists” play their part, adding insult to an injury.

To cut this short, there is so much wrong in the whole system that it would take books to even mention everything, but I think the rest of us must be patient and for now, satisfied with a fringe role, waiting for industry to clear away. Until then, enjoy your empty but brand new and shiny 89,99$ deer hunting game and this time with a new engine and directX 12!

Yee-haw!

 

[ericjones]

I have a select few sites which I have learned to trust over time, and I also find the Metacritic user score to be a fairly accurate marker as to how good a game is. I suppose the reason for the latter being fairly trustworthy is the fact that the scores accumulate over time, making pre-release hype less and less important. The same can sometimes be seen in game reviews that come out some time after game release as well. When a reviewer receives an early review copy and needs to quickly finish up a review for release date, it might be difficult to ignore the hype.

Another thing to have in mind is that many game sites let shitkids blinded by hype set the agenda - e.g. the fanboys who call on reviewers getting the boot if giving skyrim less than A++, even before the game is out. Are these kids actually a majority or a very vocal minority? As the average gamer is supposed to be 30+ years old, I'd hope for the latter

 

[Morbus]

Metacritic is an interesting thing.

And to say that people often dismiss a game simply because it's bellow 8, well that's a very real situation. I myself don't, partly because I have time to try games out and check if I like them (and time here probably has many an underlying meaning), but also because I have a brain of my own and can decide for my own whether I like a game or not. Besides if a game gets a brilliant score, likelier than not I won't like it.

It all depends on what type of games we like.

Personal score is one thing. I give Gears of War a personal score of 6/10. But game score, that's a whole different matter. If I try to be fair, I'd give Gears of War a solid 9/10, because it's just a very well executed game, and it's a bloody damn good shooter. Personal opinion disclaimers may apply where relevant.

Now, problem is, gamers and "journalists" alike forget the difference between personal tastes (not opinions) and the quality of the game itself, as a game in itself, seen in the context of both its genre and the industry as a whole. And the end result is what we all well know. Top scores are given to good mainstream games, and lower scores are given to good niche games and bad mainstream games alike. If it's bad and niche, well, then that'll warrant a bad score, won't it?

I draw a comparison. AnimeNewsNetwork.com is a pretty huge anime-related site, news and all, but they have this rating system where users submit their score, and based on a number of factors they weight the averages and whatnot, and come up with a top 10/50/100/and so on list of anime series and movies. I myself do not feel repulsed by Japanese animation so I watch quite a bit of it (going for my 60th watched series, so not as otaku-y as others, but still). And I judge what I watch mostly because of how well it scores on AnimeNewsNetwork.com. Mind you, I will drop a series pretty fast if I personally don't like the theme or the genre (I might even not try it out at all), so my opinion is definitely not shaped by their ratings (and, in fact, there are plenty of examples where I would consider a series to deserve a higher or lower place, personal tastes or not), but the way they weight the results returns a relatively better balance than what we see in the games industry. What I mean is, a good niche series will tend to score higher than a not so good mainstream series. There are plenty of exceptions, of course, but chances are, whether you like sci-fi, fantasy, romance or drama, you will probably find the best series of "your" genre have consistent high scores. If you like old-school RPGs, well, you probably won't find them at all, but when they are released, the amount of good reviews is by far lower than it should be. Because reviewers mix up personal taste with personal opinion. Those are two completely different things.

I ramble.

 

[Brother None]

(snip)

I think what you (and others) in this thread miss is that the crux of the problem is that the system has no reason to change because everyone is happy with it. Major journalists are happy to give good reviews to get early access as early access is the absolute key to getting traffic in this cutthroat internet world, publishers are happy because good reviews can boost good sales, consumers are happy because the vast majority of them are fanboys who don't want to hear anything negative about their favorite games. It's a self-enforcing spiral downwards, and as long as that doesn't change there are no methodological hotfixes.

 

[Eternal]

What a funny coincidence. Just 2 days ago I had done some number crunching on metacritic after reading that Skyrim had received over 50 "Perfect" reviews.

Here is what I did. I went to the "highest rated games of all time" section. I then took the the first two pages of games listed which had a metacritic score range of 98 at the highest, all the way down to 91 at the lowest.

I took these titles and then removed any duplicate entries for games released on multiple platforms, that way there would only be the highest rated version of the game counted. After I did that, I was left with a total of 169 different games with a release date ranging from November 2011 all the way back to 1996 (only 3 games from 96 were on the list. Quake 1, Diablo 1, and Civilization 2)

So the 169 top reviewed games of all time. Of those 169 games, 47 of them came out in 2010 or 2011. Nearly 30% of the highest reviewed games ever on metacritic came out in only the last 2 years.

Even discounting the fact that their collection of data before around the year 2001 is fairly sparse. It is still staggering.

What if we just take the top 100 games released in the last 10 years? (2002-end of 2011)

Here is the break down of the years vs number of games in the top 100
Year:
2002 - 9games
2003 - 7games
2004 - 8games
2005 - 4games
2006 - 9games
2007 - 9games
2008 - 9games
2009 - 13games
2010 - 17games
2011 - 15games

 

[Guiltyofbeingtrite]

The spike in the last 3 years could be attributed to a lot of things but I'd bank on marketing departments having a stronger say in the direction games are made. Making sure the games that are published have a good chance to be embraced by reviewers/public.

 

[Brother None]

Yes, trite? How would that not still be wrong?

That said, Metacritic doesn't log well back to older games, so it's not really well-suited for a longer comparison.

 

[UncannyGarlic]

Another thing to have in mind is that many game sites let shitkids blinded by hype set the agenda - e.g. the fanboys who call on reviewers getting the boot if giving skyrim less than A++, even before the game is out.

I'm with the author of the article, I don't get why people get so upset about bad reviews, least of all when they make valid points.

Personal score is one thing. I give Gears of War a personal score of 6/10. But game score, that's a whole different matter. If I try to be fair, I'd give Gears of War a solid 9/10, because it's just a very well executed game, and it's a bloody damn good shooter. Personal opinion disclaimers may apply where relevant.
Now, problem is, gamers and "journalists" alike forget the difference between personal tastes (not opinions) and the quality of the game itself, as a game in itself, seen in the context of both its genre and the industry as a whole.

Both scores are very important and I'd love to see both of them. That said, I feel like a lot of video game critics don't know enough about game design to do a good job on the latter. No, they don't need to be experts but compared to book and movie critics, they know pretty much nothing about their media.

The spike in the last 3 years could be attributed to a lot of things but I'd bank on marketing departments having a stronger say in the direction games are made. Making sure the games that are published have a good chance to be embraced by reviewers/public.

Actually it would be interesting to see how much was spent on advertising (adjusted) to see if it surged at the same time. It would also be interesting to look at the scoring spread to see if scores in general started trending higher.

 

[Morbus]

Personal score is one thing. I give Gears of War a personal score of 6/10. But game score, that's a whole different matter. If I try to be fair, I'd give Gears of War a solid 9/10, because it's just a very well executed game, and it's a bloody damn good shooter. Personal opinion disclaimers may apply where relevant.
Now, problem is, gamers and "journalists" alike forget the difference between personal tastes (not opinions) and the quality of the game itself, as a game in itself, seen in the context of both its genre and the industry as a whole.

Both scores are very important and I'd love to see both of them. That said, I feel like a lot of video game critics don't know enough about game design to do a good job on the latter. No, they don't need to be experts but compared to book and movie critics, they know pretty much nothing about their media.

You'll be stoned to death in some corners of the web for saying stuff like that. I would be big bucks that more than 75% of the gaming population that cares about game reviews doesn't even know there ARE book reviews. And most of them will also not agree with you in thinking game reviewers need to know about game design.

It's completely idiotic.

You wouldn't believe the amount of slack I get sometimes for talking as if people know what's it all about. Online and offline.

Then again, maybe you would.

 

[TychoXI]

Year:
2002 - 9games
2003 - 7games
2004 - 8games
2005 - 4games
2006 - 9games
2007 - 9games
2008 - 9games
2009 - 13games
2010 - 17games
2011 - 15games

You should relativize those numbers against the total number of games Metacritic has for each year.

 

[memetics]

Good article and observations, BN and others.

Consider this example: if asked to "rate" how hot it is outside, ratings of the temperature from independent reviewers should all be within a couple of degrees of each other; the "ratings" shouldn't be all over the map based on how hot or cold the reviewer likes it, whether they grew up in a warm climate or not, or whether they've grown tired of similarly-warm days and want to encourage the weather to change to a temperature that suits them better.

For a scoring system to be effective and meaningful, it must have both objectivity and interrater reliability.

Objectivity means that the criteria used have to be defined in such a way that scoring is based on the object - the attributes of the thing itself - not the subject (the reviewer's personal tastes, etc.) or irrelevant things (the attributes of other objects, for example).

Interrater reliability means that independent raters should be able to arrive at similar scores most of the time because they're using objective criteria and making objective judgments, and (usually) because they've gone through some kind of calibration process together to ensure that they're using the same scale.

The example from the English history instructor in the article's comments area is a great example of the calibration / scale problem: the student complaining about receiving a 73/100 and demanding at least a 90 is judging the score as abysmal because her judgment is made within the frame of a different grading system, one that sees 75 as average (or more accurately, well below average; as William Zinsser puts it, in the typical American inflated-grade system most students think "A = average; B = borderline"), whereas the instructor him(?)self noted that a 70 is considered "excellent" in the English system.

If two raters are using such different ratings assumptions, then there's no way they will come to agreement. Calibration is necessary if aggregations like metacritic's scores will have any meaning whatsoever. When people still ascribe value to such scoring, that's where the real problem lies, and the only effective solution is to better educate people about how scoring and averaging work, so they can ignore the meaningless ratings and focus on objective elements of the reviews themselves.

-m

 

[Janmanden]

I think you forget that a review is not an examination, but a highly subjective piece of work. The understanding of such a review is derived from the understanding of the reviewer. Knowing what the reviewer stand for makes it easier to decide if his opinions are representative of your own.

I am not sure how you would go about quantifying taste and feel with just one number.. All the review sites really have to do is to stop making numbers and add a different kind of summary, something short and literal, I think.

Metacritic can be a remarkably deceptive service, and it's important that you're able to understand the primary function it serves.
...
Metacritic acts as a valuable metric for publishers aiming to shift more copies.

For a highly competitive gaming business it also makes sense why they would try to undermine any bad review that conflicts with their profit and maybe they get a bit of help from gaming zealots and fanbois.. But I just don't see the point of it other than trying to invalidate the point of view, because Metacritics is full of useless numbers pointing towards reviews that has ceased to exist..

Find the sites you love and trust, and stick to what you know. Use score aggregation sites by all means, but please - don't let yourself start caring about them. You don't need to. People get paid for that.