In the previous post, we wrote about the unreliability of aggregated ratings for movies. A similar problem is the rift between the audience and critics. For instance, a very recent movie got 3/10 from the critics, but 7/10 from the audience. Without a doubt, a professional reviewer has different demands for a movie than an ordinary visitor, at least on average. The problem is that we get a very condensed summary of a movie by just a single number, like 7.1 and it is not obvious what factors lead to this number. In other words, if the opinion of the critics is 5/10, but we just want to see a silly movie and we do not care for the grotesque humor, the rating might not be very useful for us. In contrast, a written review would be more valuable, because then we could figure out what aspects lead to this rating. However, the assessment of a written review takes more time than to look at the rating.
This is very similar to magazines where the editorial staff creates star ratings for movies which allow a simple categorization, but also requires that a user agrees on the standards how these ratings were created. If we consider the extremes, it is much easier, because if a movie is totally bad, or mind blowing, it is usually more straightforward to form a consensus than to agree on movies that flow in the average region of the Gaussian distribution. Even if hand picking movies might not be up-to-date, the problem is also present in technological solutions, like recommender systems where ratings are not calibrated and driven by very different motives.
The point is, it would be silly to not use as much as possible from the knowledge and experience the crowd gathered so far, but we have to make sure that this data is transformed into the “preference space” of the user to provide additional benefits. Otherwise, the data shadows the actual preferences of a user and suggestions merely reflect the wisdom of the crowd.