It can be incredibly difficult to get a feel for the “critical consensus” for a new film, if there even is such a thing. But, online review aggregators like Rotten Tomatoes and Metacritic do their best to provide their readers with a general idea of the quality of a film, which I have discussed at length before. Today, I’d like to show a crystal clear example of why another metric, the User Ratings from IMDB.com, borders on absolute uselessness. Put bluntly, the site does not require that a person giving a rating has even seen the movie. The result is blatant vote-brigading, either artificially elevating a substandard film through the sheer size of the fanbase of its underlying intellectual property, or unjustly punishing a film for its perceived transgressions that are unrelated to the quality of the filmmaking. In the former case, we’ll look at Batman v. Superman: Dawn of Justice, and in the later, the more recent A Dog’s Purpose.
In the case of Batman v Superman: Dawn of Justice, there is a simple observation which suggests that there has been blatant vote-brigading towards higher scores, and that is the number of users who gave the film a 10/10 score, thereby artificially raising this film’s user rating. One might expect that there are some viewers that really enjoyed the film, thought it deserved a 10/10 score, and that neglecting those people is elitist, snobby, and out-of-touch. While I hated the film, I nevertheless understand that some people liked it, and others absolutely loved it. But not this many. To wit:
It is fairly reasonable to assume that any collection of ratings like these will follow a normal distribution, and while I would argue that the mean of this distribution is too high (about 7.0 in the case of BvS), that’s not worth arguing over. What is worth arguing over is the number of users that gave the film a 10/10 rating: 75,473, out of a total 442,686 reviewers, for 17.0%. This is absurdly high, and one can see that this is the case simply by looking at the rest of the distribution, which is an almost picture-perfect example of a normal distribution with a mean of 7. Take a look at the number of users that gave the film a “4”, 3 points lower than the mean, and you start to get an idea for how many user should have given the film a 10, if there was no manipulation (because “10” is also 3 away from 7, just in the other direction). Using that admittedly simplistic comparison, it looks like a more reasonable number for 10/10 scores would be something in the neighborhood of 20,000 – around 55 thousand fewer votes.
This is a crystal clear case of a devoted fanbase ballot-stuffing “Perfect, 10/10!” user ratings and inflating the final score. For what it is worth, the exact same principle is at work with the 1/10 grades, but to a much lesser degree. These are likely “backlash” votes – users recognizing that the virtual ballot box is being stuffed with unearned 10/10 votes and trying to “correct” the overall score. We’ll see a much clearer case of these votes with A Dog’s Purpose.
This behavior reflects ascribing a score for the quality of a film based on something unrelated to the film. In this case, it is blind fandom combined with a reactionary force intent on “fixing” the rating. The IMDB user rating isn’t so much a barometer of a film’s value as it is a barometer of its most passionate “viewers” emotional reactions. Batman v Superman: Dawn of Justice is hardly the only example of this exact phenomenon (though it may be the most egregious). Check out Suicide Squad and the Transformers films for other instances of fanbase vote-brigading.
The other clear issue is that an IMDB user rating does not require the voter to have actually seen the film. This has been shown before, as overly-excited fans quickly vote-up highly-anticipated films, but it also can be seen in efforts to discredit a film before its release. The latter has clearly happened with A Dog’s Purpose:
This film releases next week (1/27/2017). Nearly 1900 “reviews”, and probably zero have seen the film. This couldn’t be more blatant. The film has been barraged by an overwhelming number of 1/10 votes, on account of a behind-the-scenes video leaked by TMZ that shows poor treatment of one of the dogs on set. Since the release of the video, the film has been denounced by PETA and another animal rights groups, and the vote-brigading is in. Once again, we also recognize a backlash effort: there are also a number of 10/10 ratings in a meager effort to “rescue” the rating, likely by those with a financial interest in the film.
This piece is not meant to express a stance on the leaked video by TMZ, the treatment of the dogs on set, or even the actual quality of A Dog’s Purpose (which, judging from trailers, will be low). It is meant to argue that the IMDB User Rating metric is fundamentally flawed: it does not require that a user have actually seen the movie, and it has no mechanism for adjusting for the inevitable vote-brigading. These flaws suggest that the IMDB User Rating is essentially useless.
Rotten Tomatoes and Metacritic are not perfect. Neither are established or amateur movie critics; even if you happen to find some you agree with, there’s bound to be times you simply appreciate a film differently. But these systems are a far cry better than IMDB’s meaningless example of democracy gone stupid.
What a great article – I really, really enjoyed reading it and you make a lot of excellent points.
Another interesting case study to look at is the latest Ghostbusters film. It was absolutely attacked with 1/10 on its first worldwide release day with probably less than 0.1% of people actually seeing it and feeling it should be awarded that grade, what with the backlash against the reboot/female cast/whatever else the haters wanted to blame it on. 19% of voters have awarded it a 1/10; the second most popular score is a 10/10 (14.6%); followed by a 6/10 (14.4%) in third – talk about all over the place! I will admit that, even after scoring it probably an 8/10 or 9/10 on my blog (I really liked it, if you couldn’t tell), I actually awarded it a 10/10 on IMDB as a way to combat the overwhelming negativity I don’t think it deserved. For films without an in-built fanbase, it’s a great system – otherwise, it is a flawed system with no easy ‘fix’.
Ahh, I should’ve thought of “Ghostbusters”! You’re exactly right, it is a perfect example of vote-brigading, in this case for an even stupider reason than blind fandom. Hence, you get 1/10s from “offended” fools who feel their childhoods are being “ruined”, 10/10s from SJWs trying to “correct”, and (I would guess) 6/10s (on average) for the people that actually saw the film and are rating it on their merits.
Also, be wary of the system even for films without a “built-in” fanbase, per se. This was off topic for the piece, so I took it out, but imagine an ideological film, something with a message and a point of view. For our purposes, let’s just assume it is a heavily religious (or anti-religious) sentiment. Despite the film having practically no “built-in” fanbase, it doesn’t take much imagination to realize that religious nuts and fervent atheists will be encouraged to vote-brigade the film in an extreme direction. All based on the IDEOLOGY of the film, not its quality.
That kind of thing can be VERY dangerous if people take it too seriously. Hopefully, after realizing how imperfect the system is, that wont happen.
I built a site to combat the exact issues you brought up. Not only by standardizing ratings across users, bringing an “average movie” to a true 5, instead of 6.8…but also filter ratings based on each users’ taste. Try it out and let us know how we’re doing.
http://www.taste.io