Testing Gaming sites for Bias and Inconsistency

The Study

The integrity of gaming websites has come into question recently, so I thought it would be fun to turn the tables on sites such as 1UP and IGN and review them using some statistical measurements on their review scores. The study I conducted was to test if they were biased when it came to their Wii game reviews, and to test how consistent they were.

The sites I chose to test were:

• 1UP
• IGN
• Gamespot
• Gametrailers
• Eurogamer
  

To do this I used data from Metacritic to find the scores of the following 12 games, and their respective averages based on all the reviews collected:

1. Super Mario Galaxy
2. The Legend of Zelda: Twilight Princess
3. Zack & Wiki: Quest for Barbaros' Treasure
4. Resident Evil 4 Wii Edition
5. Metroid Prime 3: Corruption
6. Super Paper Mario
7. Trauma Center: Second Opinion
8. WarioWare: Smooth Moves
9. Resident Evil: Umbrella Chronicles
10. Mario & Sonic at the Olympic Games
11. Mercury Meltdown Revolution
12. The Godfather: Blackhand Edition

Bias was measured by calculating how the site's review score differed from the Metacritic average across all 12 titles.

Consistency was measured by calculating the Standard Deviation from the Metacritic average across all 12 titles.

The Results:

Bias

The sites are listed below ranked from first to last starting with the least biased. The positive and negative signs indicated whether the site had a positive or negative bias.

1. Gamespot -1%
2. IGN +2.1%
3. Gametrailers +2.2%
4. Eurogamer -2.6%
5. 1UP -3.4%

Consistency

Next the sites are ranked from first to last, starting with the most consistent. The most consistent was the site with the lowest Standard Deviation; basically the site whose game scores varied from the Metacritic average the least.

1. Gametrailers 3.3
2. IGN 4.2
3. Gamespot 5.2
4. 1UP 6.6
5. Eurogamer 9.8

Conclusion

What's most striking about the bias measurements is how little the scores vary from one site to another. The difference between Gametrailers' 2.2% positive bias to 1UP's 3.4% negative bias is a meager 5.6%. If a bias exists in these sites towards Wii games, be it positive or negative, it is negligible at best.

The consistency measurements were more interesting, with Eurogamer and 1UP coming up lowest on the list. They were the sites that differed most with the media's general opinion of these Wii games. The other sites held within about 5% of the average Metacritic rating, with Gametrailers showing remarkable consistency with the media's general opinion.

Overall IGN, Gametrailers and Gamespot appear to be very reliable places for Wii game reviews, with 1UP and Eurogamer not too far behind.

Talking Points

For the sake of this study I set the Metacritic average as the gold standard rating for each game. This only works under the assumption that the reviewer for each site is simply giving his opinion, rather than trying to guess what the Metacritic/GameRankings average will ultimately be. This may not be the case as a freelancer for Gamespot posted this on NeoGAF after the firing of Gerstmann:

As for the future, well... even if they don't get pressured to cut me loose because I've made a few posts here, I personally cannot write for a site that, for whatever reasons, wants to bring its review scores and tone more in line with the Metacritics/GameRankings medians, and is willing to pressure its writers to nudge in that direction. The point of an average is that there are highs, lows, and in-betweens. If every publication strives to be in the middle, then there's no point in having multiple publications anymore. Wasn't one of GameSpot's BEST aspects that it was tougher on games?

While review scores are interesting, one of the reasons I visit different gaming sites is that they have their own distinct voices and opinions. If they all begin to align their scores to what they think will be the media average, I know I personally will be visiting these sites less. It's definitely a fine balance.