The Star Rating System
Tue, 16 Apr 2002, 11:20 amThe Review Master10 posts in thread
The Star Rating System
Tue, 16 Apr 2002, 11:20 amHi all,
It was brought to my attention last night that I had apparently given Playlover's JCS two stars in the review I wrote about it.
Not true at all. I have no idea what the stars mean. What do they mean Grant?
The Review Master
The Review MasterTue, 16 Apr 2002, 11:20 am
Hi all,
It was brought to my attention last night that I had apparently given Playlover's JCS two stars in the review I wrote about it.
Not true at all. I have no idea what the stars mean. What do they mean Grant?
The Review Master
Babar's SisterThu, 18 Apr 2002, 01:55 am
>>>Rate Me!<<<
`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'
I assumed we were allowed to rate posts out of 5 stars. One star being shoddy, five being brilliant.
From this, people can flick through and see which posts are popular, and which are not. Or which posts are noteworthy, and which are ignored.
If I'm wrong, then you can all point and laugh at me and make me cry.
Babar's Sister
`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'`'·.,.·'
Babar's SisterThu, 18 Apr 2002, 02:02 am
Post Script...
Click on the star next to the post....
...you should get something like this :
RATING DETAILS
The rating for this post is an average of the ratings provided by the following members. Only registered members are able to rate posts. Registration is FREE!
Rating for The Star Rating System posted by The Review Master on 16-04-02 11:20
Member : Justin Hammond
Average Rating : 1.5
Number of Ratings : 15
Time Rated : 16-04-02 20:59
crgwllmsFri, 19 Apr 2002, 03:42 pm
The Rate Debate
Babar's Sister wrote:
>
> I assumed we were allowed to rate posts out of 5 stars. One
> star being shoddy, five being brilliant.
>
> From this, people can flick through and see which posts are
> popular, and which are not. Or which posts are noteworthy,
> and which are ignored.
>
At first, the ratings are going to be highly inaccurate. If I love something and give it 5 stars, that only represents my one very singular opinion. If someone else hates it and gives it a 1, that only brings the average back to three. If the whole board hates it and keeps giving a 1 star rating, the average will creep down to the low 1.5's.
At the moment there's no such thing as a zero rating - it merely means the post has not been rated at all.
The more we all get into the habit of rating each post we read, the more accurately the star system will reflect popular opinion.
...Whether popular opinion equates with quality is a subject previously disputed...however, it's still a useful and informative guage.
Cheers
Craig
<8>-/=====/--------
>
> I assumed we were allowed to rate posts out of 5 stars. One
> star being shoddy, five being brilliant.
>
> From this, people can flick through and see which posts are
> popular, and which are not. Or which posts are noteworthy,
> and which are ignored.
>
At first, the ratings are going to be highly inaccurate. If I love something and give it 5 stars, that only represents my one very singular opinion. If someone else hates it and gives it a 1, that only brings the average back to three. If the whole board hates it and keeps giving a 1 star rating, the average will creep down to the low 1.5's.
At the moment there's no such thing as a zero rating - it merely means the post has not been rated at all.
The more we all get into the habit of rating each post we read, the more accurately the star system will reflect popular opinion.
...Whether popular opinion equates with quality is a subject previously disputed...however, it's still a useful and informative guage.
Cheers
Craig
<8>-/=====/--------
Grant MalcolmSat, 20 Apr 2002, 09:22 am
To Rate or not to Rate
Hearty agreement with your post, Craig. You've neatly summed up the reasons for including a ratings system.
There are a couple of other issues people might like to consider - and while it's under the broad subject of ratings, the Reviews section is probably as apt as any other.
:-)
I'm curious to know what barriers people might see to their rating a post?
I wonder if some people won't rate posts because the rating they apply is visible for everyone to see?
Would more people rate if their rating wasn't visible for all to see? Should ratings be "anonymous"?
On the second matter, if you check a person's profile, you may notice that it records the number of posts they've made (under that login) and also the number of ratings they've made, together with the average rating they've given.
After the average rating they've given, there is sometimes a number in brackets. I've not been quite sure what to call this, but it is the average of the ratings that this person has received - rather than given. I'll shortly be using this figure and a few other factors such as recent activity on the site - e.g. updating/adding events/companies and the number of recent posts - to allocate registered people a personal star rating. This will appear next to their name at various points around the site.
Any ratings you apply to yourself are ignored when calculating a personal rating. To prevent people buddying or teaming up to rate each other highly, the system weights in favour of the number of different people that have rated you. And if you persistently rate other people down, this might even begin to drag down your personal rating.
:-)
This type of feature is becoming common on many web sites - a rough and ready meritocracy that attempts to "reward" and encourage productive, social behaviour.
Cheers
Grant
[%sig%]
There are a couple of other issues people might like to consider - and while it's under the broad subject of ratings, the Reviews section is probably as apt as any other.
:-)
I'm curious to know what barriers people might see to their rating a post?
I wonder if some people won't rate posts because the rating they apply is visible for everyone to see?
Would more people rate if their rating wasn't visible for all to see? Should ratings be "anonymous"?
On the second matter, if you check a person's profile, you may notice that it records the number of posts they've made (under that login) and also the number of ratings they've made, together with the average rating they've given.
After the average rating they've given, there is sometimes a number in brackets. I've not been quite sure what to call this, but it is the average of the ratings that this person has received - rather than given. I'll shortly be using this figure and a few other factors such as recent activity on the site - e.g. updating/adding events/companies and the number of recent posts - to allocate registered people a personal star rating. This will appear next to their name at various points around the site.
Any ratings you apply to yourself are ignored when calculating a personal rating. To prevent people buddying or teaming up to rate each other highly, the system weights in favour of the number of different people that have rated you. And if you persistently rate other people down, this might even begin to drag down your personal rating.
:-)
This type of feature is becoming common on many web sites - a rough and ready meritocracy that attempts to "reward" and encourage productive, social behaviour.
Cheers
Grant
[%sig%]
crgwllmsSat, 20 Apr 2002, 04:38 pm
Rate's Progress
Grant Malcolm wrote:
> I'm curious to know what barriers people might see to their
> rating a post?
People might discover what I think.
...in fact, I might actually have to think.
> I wonder if some people won't rate posts because the rating
> they apply is visible for everyone to see?
Probably. But there's probably some people who WILL rate posts for the same reason.
> Would more people rate if their rating wasn't visible for all
> to see? Should ratings be "anonymous"?
No. It's easy enough for posts to be anonymous anyway. When you create a log-in name, you don't have to identify yourself. But I think making ratings attributable to their authors forces more responsible ratings.
Cheers
Craig
<8>-/====/--------
> I'm curious to know what barriers people might see to their
> rating a post?
People might discover what I think.
...in fact, I might actually have to think.
> I wonder if some people won't rate posts because the rating
> they apply is visible for everyone to see?
Probably. But there's probably some people who WILL rate posts for the same reason.
> Would more people rate if their rating wasn't visible for all
> to see? Should ratings be "anonymous"?
No. It's easy enough for posts to be anonymous anyway. When you create a log-in name, you don't have to identify yourself. But I think making ratings attributable to their authors forces more responsible ratings.
Cheers
Craig
<8>-/====/--------
crgwllmsMon, 22 Apr 2002, 12:00 am
Rate Expectations
Grant Malcolm wrote:
> I'm curious to know what barriers people might see to their
> rating a post?
Hi Grant
As has probably become obvious, I've been trialling this star-rating system with a fair number of posts. I like the concept, and have voiced some ideas about how it should work once it gains momentum.
But I also see (and have possibly created) a few problems that weren't immediately obvious.
Although I read every post, via the email digest, I'm not always motivated to visit each of those posts on the website. I only go if I'm inclined to reply, or want to review what was written in the same thread. Obviously, by then I've formed a strong opinion about the post. So I am most likely to rate it either highly or lowly, but not so inclined to visit to give an "average" rating.
(The concept of rating something lowly came from your initial post, Grant..."You may even choose to rate somethin poorly if it's rude, contributes little to the discussion, or you simply disagree!".
Also it reminds me of the way we often rate movies - high stars is good, low stars is bad).
Because the rating stars suggested a spectrum of 1 to 5, I decided that an "average" post deserved a 3. The more interesting, informative, or entertaining posts get a 4, and the posts that are all of the above and eloquently expressed get a 5. A post that was under par would get a 2 and one that was poorly thought out or obnoxious (we've had a few, and in fact maybe I authored some) should get the lowest mark.
It was also interesting to discover I sometimes rated a post in opposition to whether I agreed or disagreed with the content. It was entirely possible to disagree with a statement (in which case I could dispute it in a reply) and yet still rate it highly as a better-than average post. It lent slightly more objectivity to the way I read the posts and responded to them.
...All fairly obvious, although the actual process of allocating these marks is still a subjective function of personal taste.
The problem occurs because there is no "zero" rating, or rather that all posts start with no rating and so they APPEAR to be awarded zero. When you look at the home page, a three-star rating looks pretty good, but in my reckoning it's an "average" post and is probably worth the same as one with no rating.
A post on the home page with one star is actually pretty inferior, but it looks better than the posts with no ratings.
This is unlikely to improve, because the "average" post is least likely to provoke a strong rating in either direction, and so is just as likely to never be rated.
Perhaps this could be solved by automatically rating each new post as a 3, until modified by further reader's ratings? Or by making it obvious that the rating is a score out of five? Perhaps there should also be a score of zero, as opposed to "unrated" ?
But perhaps I need a complete paradigm shift, and the stars should ALL be considered as merit badges, in which case my awarding criteria needs to be overhauled.
Maybe getting 1 star is slightly above average, 2 or three are notable, 4 or 5 are excellent, and there should be NO distinction for below-average or unrated. You can only give positive stars, like your teacher did at school.
This method looks the neatest on the surface, but the trouble with it is that if someone rates a post unrealistically high (5), and you believe it is mediocre or below par (zero), you can only influence the average rating by voting it a lower but still positive rating (eg 1 star), and the average will always be positive. Many posts will be rated higher than they perhaps deserve.
For this reason I think the "score out of 10" method would be more accurately representational.
What other thoughts? Do you think the system is working? Is it meaningful? How can it be made more consistent?
Cheers,
Craig
[%sig%]
> I'm curious to know what barriers people might see to their
> rating a post?
Hi Grant
As has probably become obvious, I've been trialling this star-rating system with a fair number of posts. I like the concept, and have voiced some ideas about how it should work once it gains momentum.
But I also see (and have possibly created) a few problems that weren't immediately obvious.
Although I read every post, via the email digest, I'm not always motivated to visit each of those posts on the website. I only go if I'm inclined to reply, or want to review what was written in the same thread. Obviously, by then I've formed a strong opinion about the post. So I am most likely to rate it either highly or lowly, but not so inclined to visit to give an "average" rating.
(The concept of rating something lowly came from your initial post, Grant..."You may even choose to rate somethin poorly if it's rude, contributes little to the discussion, or you simply disagree!".
Also it reminds me of the way we often rate movies - high stars is good, low stars is bad).
Because the rating stars suggested a spectrum of 1 to 5, I decided that an "average" post deserved a 3. The more interesting, informative, or entertaining posts get a 4, and the posts that are all of the above and eloquently expressed get a 5. A post that was under par would get a 2 and one that was poorly thought out or obnoxious (we've had a few, and in fact maybe I authored some) should get the lowest mark.
It was also interesting to discover I sometimes rated a post in opposition to whether I agreed or disagreed with the content. It was entirely possible to disagree with a statement (in which case I could dispute it in a reply) and yet still rate it highly as a better-than average post. It lent slightly more objectivity to the way I read the posts and responded to them.
...All fairly obvious, although the actual process of allocating these marks is still a subjective function of personal taste.
The problem occurs because there is no "zero" rating, or rather that all posts start with no rating and so they APPEAR to be awarded zero. When you look at the home page, a three-star rating looks pretty good, but in my reckoning it's an "average" post and is probably worth the same as one with no rating.
A post on the home page with one star is actually pretty inferior, but it looks better than the posts with no ratings.
This is unlikely to improve, because the "average" post is least likely to provoke a strong rating in either direction, and so is just as likely to never be rated.
Perhaps this could be solved by automatically rating each new post as a 3, until modified by further reader's ratings? Or by making it obvious that the rating is a score out of five? Perhaps there should also be a score of zero, as opposed to "unrated" ?
But perhaps I need a complete paradigm shift, and the stars should ALL be considered as merit badges, in which case my awarding criteria needs to be overhauled.
Maybe getting 1 star is slightly above average, 2 or three are notable, 4 or 5 are excellent, and there should be NO distinction for below-average or unrated. You can only give positive stars, like your teacher did at school.
This method looks the neatest on the surface, but the trouble with it is that if someone rates a post unrealistically high (5), and you believe it is mediocre or below par (zero), you can only influence the average rating by voting it a lower but still positive rating (eg 1 star), and the average will always be positive. Many posts will be rated higher than they perhaps deserve.
For this reason I think the "score out of 10" method would be more accurately representational.
What other thoughts? Do you think the system is working? Is it meaningful? How can it be made more consistent?
Cheers,
Craig
[%sig%]
Amanda ChestertonMon, 22 Apr 2002, 01:17 pm
Re: Rate Expectations
This one got points from me for:
i) sheer volume - is there nothing you can't discuss ad infinitum?! I'm terribly impressed, as I always thought I was a pretty good bullsh-tter...
ii) consistency of brilliant, yet witty and apt titles - you only did Comedy Sportz to sharpen your pun gland for this website, now, didn't you?
[%sig%]
i) sheer volume - is there nothing you can't discuss ad infinitum?! I'm terribly impressed, as I always thought I was a pretty good bullsh-tter...
ii) consistency of brilliant, yet witty and apt titles - you only did Comedy Sportz to sharpen your pun gland for this website, now, didn't you?
[%sig%]
SarahTue, 23 Apr 2002, 09:03 am
Re: Rate Expectations
I have to say I don't like the rating system - yet. I say yet, because maybe I am missing something, or maybe it will grow on me?? I find the stars annoying and irrelevant. I will tend to read a post whether it has one, five or no stars. I think Graig's post highlights the "complicated" nature of the stars. Why do we have to make a judgement on someone's post anyway?
They also tend to be ambiguous - The RM's review on JC received two stars. We later found out this was because it was posted on more than one board. I can appreciate this, but how are we to know that this is why someone chose to rate it thus?
I think this is a great site and I visit it often (the stars will not deter me). I will follow with interest the rate debate.
Sarah
They also tend to be ambiguous - The RM's review on JC received two stars. We later found out this was because it was posted on more than one board. I can appreciate this, but how are we to know that this is why someone chose to rate it thus?
I think this is a great site and I visit it often (the stars will not deter me). I will follow with interest the rate debate.
Sarah
AuctorTue, 23 Apr 2002, 11:04 am
Re: Rate Expectations
Sarah wrote:
> I find the stars annoying and irrelevant. I
> will tend to read a post whether it has one, five or no
> stars. I think Graig's post highlights the "complicated"
> nature of the stars. Why do we have to make a judgement on
> someone's post anyway?
I also feel there are some problems with the rating system and I have to ask if it is really needed at all.
IMHO the only reason to have any form of rating/censorship system is to cut down the noise level. My experience with this site is that, apart from scam artists seeking to profit on the hopes and dreams of actors, the noise level is negligible. This is a great group.
I also feel that any system needs to have a mechanical facility. There needs to be a way that registered users can set their viewing preferences based on the rating (ie. only 5-star posts, everything above 3-stars, etc). Without that ability there's no need for a rating, because most of us will probably read anything that interests us no matter what score it has.
[%sig%]