It can be incredibly difficult to get a feel for the “critical consensus” for a new film, if there even is such a thing. But, online review aggregators like Rotten Tomatoes and Metacritic do their best to provide their readers with a general idea of the quality of a film, which I have discussed at length before. Today, I’d like to show a crystal clear example of why another metric, the User Ratings from IMDB.com, borders on absolute uselessness. Put bluntly, the site does not require that a person giving a rating has even seen the movie. The result is blatant vote-brigading, either artificially elevating a substandard film through the sheer size of the fanbase of its underlying intellectual property, or unjustly punishing a film for its perceived transgressions that are unrelated to the quality of the filmmaking. In the former case, we’ll look at Batman v. Superman: Dawn of Justice, and in the later, the more recent A Dog’s Purpose.
In the case of Batman v Superman: Dawn of Justice, there is a simple observation which suggests that there has been blatant vote-brigading towards higher scores, and that is the number of users who gave the film a 10/10 score, thereby artificially raising this film’s user rating. One might expect that there are some viewers that really enjoyed the film, thought it deserved a 10/10 score, and that neglecting those people is elitist, snobby, and out-of-touch. While I hated the film, I nevertheless understand that some people liked it, and others absolutely loved it. But not this many. To wit:
It is fairly reasonable to assume that any collection of ratings like these will follow a normal distribution, and while I would argue that the mean of this distribution is too high (about 7.0 in the case of BvS), that’s not worth arguing over. What is worth arguing over is the number of users that gave the film a 10/10 rating: 75,473, out of a total 442,686 reviewers, for 17.0%. This is absurdly high, and one can see that this is the case simply by looking at the rest of the distribution, which is an almost picture-perfect example of a normal distribution with a mean of 7. Take a look at the number of users that gave the film a “4”, 3 points lower than the mean, and you start to get an idea for how many user should have given the film a 10, if there was no manipulation (because “10” is also 3 away from 7, just in the other direction). Using that admittedly simplistic comparison, it looks like a more reasonable number for 10/10 scores would be something in the neighborhood of 20,000 – around 55 thousand fewer votes.
This is a crystal clear case of a devoted fanbase ballot-stuffing “Perfect, 10/10!” user ratings and inflating the final score. For what it is worth, the exact same principle is at work with the 1/10 grades, but to a much lesser degree. These are likely “backlash” votes – users recognizing that the virtual ballot box is being stuffed with unearned 10/10 votes and trying to “correct” the overall score. We’ll see a much clearer case of these votes with A Dog’s Purpose.
This behavior reflects ascribing a score for the quality of a film based on something unrelated to the film. In this case, it is blind fandom combined with a reactionary force intent on “fixing” the rating. The IMDB user rating isn’t so much a barometer of a film’s value as it is a barometer of its most passionate “viewers” emotional reactions. Batman v Superman: Dawn of Justice is hardly the only example of this exact phenomenon (though it may be the most egregious). Check out Suicide Squad and the Transformers films for other instances of fanbase vote-brigading.
The other clear issue is that an IMDB user rating does not require the voter to have actually seen the film. This has been shown before, as overly-excited fans quickly vote-up highly-anticipated films, but it also can be seen in efforts to discredit a film before its release. The latter has clearly happened with A Dog’s Purpose:
This film releases next week (1/27/2017). Nearly 1900 “reviews”, and probably zero have seen the film. This couldn’t be more blatant. The film has been barraged by an overwhelming number of 1/10 votes, on account of a behind-the-scenes video leaked by TMZ that shows poor treatment of one of the dogs on set. Since the release of the video, the film has been denounced by PETA and another animal rights groups, and the vote-brigading is in. Once again, we also recognize a backlash effort: there are also a number of 10/10 ratings in a meager effort to “rescue” the rating, likely by those with a financial interest in the film.
This piece is not meant to express a stance on the leaked video by TMZ, the treatment of the dogs on set, or even the actual quality of A Dog’s Purpose (which, judging from trailers, will be low). It is meant to argue that the IMDB User Rating metric is fundamentally flawed: it does not require that a user have actually seen the movie, and it has no mechanism for adjusting for the inevitable vote-brigading. These flaws suggest that the IMDB User Rating is essentially useless.
Rotten Tomatoes and Metacritic are not perfect. Neither are established or amateur movie critics; even if you happen to find some you agree with, there’s bound to be times you simply appreciate a film differently. But these systems are a far cry better than IMDB’s meaningless example of democracy gone stupid.