The Star Rating System
Tue, 16 Apr 2002, 11:20 amThe Review Master10 posts in thread
The Star Rating System
Tue, 16 Apr 2002, 11:20 amHi all,
It was brought to my attention last night that I had apparently given Playlover's JCS two stars in the review I wrote about it.
Not true at all. I have no idea what the stars mean. What do they mean Grant?
The Review Master
Rate Expectations
Mon, 22 Apr 2002, 12:00 amGrant Malcolm wrote:
> I'm curious to know what barriers people might see to their
> rating a post?
Hi Grant
As has probably become obvious, I've been trialling this star-rating system with a fair number of posts. I like the concept, and have voiced some ideas about how it should work once it gains momentum.
But I also see (and have possibly created) a few problems that weren't immediately obvious.
Although I read every post, via the email digest, I'm not always motivated to visit each of those posts on the website. I only go if I'm inclined to reply, or want to review what was written in the same thread. Obviously, by then I've formed a strong opinion about the post. So I am most likely to rate it either highly or lowly, but not so inclined to visit to give an "average" rating.
(The concept of rating something lowly came from your initial post, Grant..."You may even choose to rate somethin poorly if it's rude, contributes little to the discussion, or you simply disagree!".
Also it reminds me of the way we often rate movies - high stars is good, low stars is bad).
Because the rating stars suggested a spectrum of 1 to 5, I decided that an "average" post deserved a 3. The more interesting, informative, or entertaining posts get a 4, and the posts that are all of the above and eloquently expressed get a 5. A post that was under par would get a 2 and one that was poorly thought out or obnoxious (we've had a few, and in fact maybe I authored some) should get the lowest mark.
It was also interesting to discover I sometimes rated a post in opposition to whether I agreed or disagreed with the content. It was entirely possible to disagree with a statement (in which case I could dispute it in a reply) and yet still rate it highly as a better-than average post. It lent slightly more objectivity to the way I read the posts and responded to them.
...All fairly obvious, although the actual process of allocating these marks is still a subjective function of personal taste.
The problem occurs because there is no "zero" rating, or rather that all posts start with no rating and so they APPEAR to be awarded zero. When you look at the home page, a three-star rating looks pretty good, but in my reckoning it's an "average" post and is probably worth the same as one with no rating.
A post on the home page with one star is actually pretty inferior, but it looks better than the posts with no ratings.
This is unlikely to improve, because the "average" post is least likely to provoke a strong rating in either direction, and so is just as likely to never be rated.
Perhaps this could be solved by automatically rating each new post as a 3, until modified by further reader's ratings? Or by making it obvious that the rating is a score out of five? Perhaps there should also be a score of zero, as opposed to "unrated" ?
But perhaps I need a complete paradigm shift, and the stars should ALL be considered as merit badges, in which case my awarding criteria needs to be overhauled.
Maybe getting 1 star is slightly above average, 2 or three are notable, 4 or 5 are excellent, and there should be NO distinction for below-average or unrated. You can only give positive stars, like your teacher did at school.
This method looks the neatest on the surface, but the trouble with it is that if someone rates a post unrealistically high (5), and you believe it is mediocre or below par (zero), you can only influence the average rating by voting it a lower but still positive rating (eg 1 star), and the average will always be positive. Many posts will be rated higher than they perhaps deserve.
For this reason I think the "score out of 10" method would be more accurately representational.
What other thoughts? Do you think the system is working? Is it meaningful? How can it be made more consistent?
Cheers,
Craig
[%sig%]
> I'm curious to know what barriers people might see to their
> rating a post?
Hi Grant
As has probably become obvious, I've been trialling this star-rating system with a fair number of posts. I like the concept, and have voiced some ideas about how it should work once it gains momentum.
But I also see (and have possibly created) a few problems that weren't immediately obvious.
Although I read every post, via the email digest, I'm not always motivated to visit each of those posts on the website. I only go if I'm inclined to reply, or want to review what was written in the same thread. Obviously, by then I've formed a strong opinion about the post. So I am most likely to rate it either highly or lowly, but not so inclined to visit to give an "average" rating.
(The concept of rating something lowly came from your initial post, Grant..."You may even choose to rate somethin poorly if it's rude, contributes little to the discussion, or you simply disagree!".
Also it reminds me of the way we often rate movies - high stars is good, low stars is bad).
Because the rating stars suggested a spectrum of 1 to 5, I decided that an "average" post deserved a 3. The more interesting, informative, or entertaining posts get a 4, and the posts that are all of the above and eloquently expressed get a 5. A post that was under par would get a 2 and one that was poorly thought out or obnoxious (we've had a few, and in fact maybe I authored some) should get the lowest mark.
It was also interesting to discover I sometimes rated a post in opposition to whether I agreed or disagreed with the content. It was entirely possible to disagree with a statement (in which case I could dispute it in a reply) and yet still rate it highly as a better-than average post. It lent slightly more objectivity to the way I read the posts and responded to them.
...All fairly obvious, although the actual process of allocating these marks is still a subjective function of personal taste.
The problem occurs because there is no "zero" rating, or rather that all posts start with no rating and so they APPEAR to be awarded zero. When you look at the home page, a three-star rating looks pretty good, but in my reckoning it's an "average" post and is probably worth the same as one with no rating.
A post on the home page with one star is actually pretty inferior, but it looks better than the posts with no ratings.
This is unlikely to improve, because the "average" post is least likely to provoke a strong rating in either direction, and so is just as likely to never be rated.
Perhaps this could be solved by automatically rating each new post as a 3, until modified by further reader's ratings? Or by making it obvious that the rating is a score out of five? Perhaps there should also be a score of zero, as opposed to "unrated" ?
But perhaps I need a complete paradigm shift, and the stars should ALL be considered as merit badges, in which case my awarding criteria needs to be overhauled.
Maybe getting 1 star is slightly above average, 2 or three are notable, 4 or 5 are excellent, and there should be NO distinction for below-average or unrated. You can only give positive stars, like your teacher did at school.
This method looks the neatest on the surface, but the trouble with it is that if someone rates a post unrealistically high (5), and you believe it is mediocre or below par (zero), you can only influence the average rating by voting it a lower but still positive rating (eg 1 star), and the average will always be positive. Many posts will be rated higher than they perhaps deserve.
For this reason I think the "score out of 10" method would be more accurately representational.
What other thoughts? Do you think the system is working? Is it meaningful? How can it be made more consistent?
Cheers,
Craig
[%sig%]
- ···
- ···
- ···