Category Archives: Business

An Alternative Way of Understanding Reviews

What is the measure of a creative work? Is it the $1,065,825,966 it makes, the 2-stars a top citic assigns it, or the 82% of people who didn’t enjoy it?

And if viewers’ positive experiences supposedly lead to an increase in sales, then shouldn’t a movie that leaves 84% of its audience members satisfied, be more successful? Shouldn’t it generate more income than its poorly received cousin?

I’ve been thinking a lot about reviews for a while now. What they mean, what they don’t. What they imply in this hyper-connected world where anyone with a laptop and an internet connection can be a reviewer.

In some way, you could say I’ve been thinking about them ever since I watched Donnie Darko and found it boring and self-important. My friends, and fellow cinephiles, have never disagreed so aggressively on anything else (even politics). I thought Donnie Darko was a pile of nothing, they thought it was the most brilliant piece of celluloid ever created.

We diverged on Pacific Rim too.

How, I wondered, could this be possible?

How could good novels go unread and their creators be forced into obscurity while questionable ones became popular beyond imagination?

First, I decided that what made me think something was 5-stars differed from other consumers. We all had our preferences. If we stated them at the outset, then we could all be friends once more. We could agree to disagree.

Some of my fellow consumers were happy with this arrangement. Others, not so much. They yelled at me, via digital chat, that I was wrong. They were right. They were always right, and it was my lack of cultural understanding that lead to our contrasting views. Essentially, they were one up on me. I needed to study more and come to their level of understanding.

Which, when you think about it, is circular logic. Their assessment was correct because they had always been correct in the past (or ignored contrary evidence). As they had always been correct, they had to be correct this time.

Of course, there’s a fifty-fifty chance that they could’ve been wrong, and I right. My analysis and probing intellectualism had seen through the film’s failures, and their love for their director had distorted their critical faculties. How could we possibly know?

Rotten tomatoes, they might argue. Look at the wisdom of the crowd. Look, it is you who’s incorrect. Your opinion is invalidated because you are just not there. Eventually, you will be and will join us up above.

Except, I listened to their arguments, watched the movie again and still got bored. Still disliked it. Still wanted to be viewing something else. The movie was 2 stars. 2.5 if I felt generous.

In my mind. 

So, who’s wrong and why is this important?

Well, it’s important when you create. George Lucas dismissed the outcries of his Star Wars’ viewers as unimportant. Other authors have done the same when confronted with negative reflections. “Ahead of its time,” they say. “Cultural imperialism blocking their understanding of its fundamental nature.”

Nor have they been without precedent, in some cases. Citizen Kane took a while before it became one of the top movies of all time. Roger Ebert upgraded his review of Groundhog Day decades after the fact. Plus, some (bad) movies are making truck loads of cash when they (technically) shouldn’t.

If a creative work is poor, shouldn’t it be ignored? If negative word of mouth turns people off a product, shouldn’t it fail?

The problem I think is that the majority understanding of what a review represents is limited in its scope. And what a collection of reviews can tell us is more complex (and nuanced) than we give it credit for.

Currently, the review system is treated as if we consume media in a world with a fixed quality scale, and not a probabilistic one. There is a ranking system (say 1-5) and the movie’s quality fits within this scale. There are very poor pieces of cinema (Plan 9 from Out of Space) and excellent ones (The Incredibles). The critic’s job to ascertain where a particular piece fits on this spectrum.

Yes, some critics agree, it’s comparative. But that’s the point, if you watched a 2-star and a 4-star movie directly after each other, you’d clearly identify the differences.  If everyone built up a sufficient database of cinematic (or literature-based) experiences then we would all be in complete agreement.

I think this is incorrect, and it’s suppressing our ability to innovate in our fields.

Nate Silver in his book, The Signal and the Noise, discusses how we are poor at forecasting. He advises that we are often overconfident in our ability to predict the world and this leads us to make ill-informed (and biased) choices when faced with a complex environment.

And make no mistake, generating a creative work is a complex endeavour. The ultimate goal for many authors is to earn a living off their work and quit their day job. To do this, they have to consider a variety of variables:

  1. What genre should they write in?
  2. What characters / story / underpinning narrative arc should they use?
  3. What are popular genres at the moment?
  4. What ideas / themes are being under-utilised and still have some novelty value?
  5. Should they self-or-traditionally publish?
  6. What should they price their work at?
  7. How large is the readership base for their work?
  8. How does their writing compare to others in the genre? Are they as good as the others? Or are they better?
  9. How much time should they invest in a series before they realise it is no longer worth pursuing?
  10. Is this a fad or a trend?
  11. Will a ‘one-off’ phenomenon translate into additional sales for other works in the same area?
  12. Who are their readers? Where do they live? What topics are they interested in?

Authors are faced with issues that have plagued every product-designer since the dawn of human-creation. We need to forecast. We need to have the ability to predict the trends and changes in taste despite our limited data set; even as our competitors–who (usually) have better access to resources–seek to modify the market.

I’ve met a few authors here and there, some of them are quite astute, but some work on the following assumptions, “If I could get my work into a reader’s hands, they will like it. It is a good book. The trick is marketing.”

This is based on the idea we can assess our own abilities, and determine how well we stack up against our competitors. Except, we are generally poor at this, even in a static test.  In a comparative world, things become much more problematic. In a (very simple) model, the pool of books might go something like the following:

  • 10% of novels: 1 star
  • 20% of novels: 2 stars
  • 40% of novels: 3 stars
  • 20% of novels: 4 stars
  • 10% of novels: 5 stars

This is the harsh reality of the market. 30% of writers will be 1-2 stars.  70% will be 1-3 stars. Only 1 person in your writing group of ten individuals will be a great author.

Yet, it is rare to meet a writer who believes their works are destined for the mediocrity pile. They often point out how much effort they’ve put into their chosen craft, how inadequate other (popular) authors are and how they are better. Don’t I understand that they are just that much better?

No, I don’t. Not anymore.

The world is complex. Yes, they work hard. Yet, so do others. Who is working harder? Who is producing more documents? More literature? Who is improving their marketing skills faster?

Who has accidentally created a new genre and is reaping the benefits of not being a ‘great’ author, but simply being the best (and only) creator in a field that has a large, untapped demand? Who truly is more skilled in a tightly competitive market and is that 1 in a 1,000 individuals who deserves to be read?

Why is the idea challenging? I believe, it’s because authors are competitive. They don’t want to be just an author, they want to be the best. Ratings are their guidepost. A 5-star novel is a sign they are equal to their heros and heroines. A 3-star document a sign they are not good enough.

All fields where money is involved and there are comparative metrics, attract individuals who enjoy the shove and punch of competition. They want to be recognised, they deserve to get that royalty cheque because they’re simply better. That’s their nature. Top athletes’ are expected to be aggressive, authors less so. Yet, if you spend any time with a group of them, you’ll discover just how driven they are.

So the construct that most of them (70%) will only produce good to poor works is probably not news they want to hear. Yes, other people may, but not them. They are better, stronger, faster and therefore I should be quiet and take my pills.

There’s an incentive for them to treat star ratings as important. It shows their ability, it shows how much more elegant with words they are than others, it tells the world about their skills. This is why, in my humble opinion, there’s been such a furor about books that have 5-star ratings because of the creators’ friends liking it. Or where somewhat dubious ethical practices have been used to prop up a work and advise it’s a quality piece, equal to theirs, and yet it isn’t.

These things are perceived to be lowering the value of their work. And not only lowering the value of them, but also costing them money. If people can’t trust a 4-star book to be 4-stars, then how will they choose what they want? If a critic’s hard-earned rating is the same as everyone else’s, how can we tell what’s good and bad?

But, what do review scores really tell us?

Essentially, I (now) think of a review as a forecast, not an indicator of quality. A book that has 1,237 4-star reviews and 50 1-star ones is advising that if I read the book, it’s highly likely I will have a 4-star experience.

The important point is that there’s no guarantee in this.

For the sake of ease, let’s say there’s a novel, The Windswept Hair of McGuff, and it has 100 reviews. The breakdown is as follows:

  • 1 one-star
  • 5 two-star
  • 70 three-star
  • 9 four-star
  • 15 five-star

Traditionally, we’d say it’s a 3.32-star book. It’s medicore-to-good. If you read it and hated it (1-star!), you’d be another outlier. If you discussed it with your peers, they might be dismissive of your opinions. Or argue that you didn’t understand it, after all, it’s a 3-star work. Nothing to write home about, but you shouldn’t hate it either.

However, if we take a probability-based reading, things come into much clearer focus:

  • 1% of people who read it will think it’s 1-star.
  • 5% of people who read it will believe it’s 2-stars.
  • 70% of people who read it will conclude it’s 3-stars.
  • 9% of people who read it will say it’s 4-stars.
  • 15% of people who read it will advise it’s 5-stars.

More significantly though, they will have an X-star experience with the book.

In this scenario, no one is right or wrong per se, but there is an increased likelihood of them having a certain level of experience with the book. So I can look at these stats and say, “If I spend $9.99 on this work, there’s a 94% chance I will enjoy it.”

And for someone who is interested in finding something they can be satisfied with, this is the data they are looking for. They want to know that they are going to get value for money. In addition, the info for an author is important too, at $9.99, I know that I am satisfying 94% of my customers.

Also, as we’re no longer looking at things as a single unchangeable reality, we can dig further into the stats. Who are those 15 people who liked my work? Who were those 4-star readers?

If we find out that 14/15 of the 5-star raters were our family, then we can adjust our estimates accordingly. For example, we can extrapolate that 80% of people who don’t know me will be satisfied with the work.

Perhaps we discover, through the use of targeted surveys, that 10 people are science-fiction fans while everyone else (except our family) usually consumes paranormal romance. Also, our four-star reviews have only come from the science-fiction fans.

We might form some new hypotheses:

  1. 90% of science-fiction readers will think this work is 4-stars or more.
  2. 92% of paranormal-romance readers will consider this to be a 3-star work.

Knowing this information allows us to target our markets more carefully. If we have enough data (unlikely) we could calculate the potential sales of a 3-star paranormal-romance vs. a 4-star science fiction.

In conclusion, reviews are helpful to authors (and product creators) in the creative fields, but not in the way we currently use them. We should adapt a more probability-based understanding of how reviews work and what they mean for potential sales.

Also, please read The Signal and the Noise, it’s very good.