Judging the judges

I understand that the ITA committee are due to discuss some ideas realted to the training and selection of award adjudicators.Barry Lucas and I had a very fruitful discussion last week at the ITA office.Amongst the ideas to emerge from our conversation was a suggestion from me that the adjudication process had been rendered too anonymous and in the process was "protecting" the adjudicators from criticism that may at times be warranted. After all, what does anyone really know about our adjudicators and the marks they give?I suggested that each adjudicator's marks should be subject to a process of peer review - discussed and if necessary voted on at a meeting of all adjudicators. A similar process already takes place for adjudicator's certificates, why not for other awards?Opinions often differ so widely on a show, it is vital that these differences be aired and discussed. Through this process of debate and discussion the adjudicators can achieve concensus. Along the way, their outlooks will be broadened, their critical thinking tested and their opinions revised.I'd almost line up to adjudicate in a process like that! It would be very challenging for all involved.If adjudicators are going to sit in judgement over other people productions, i say, let them be judged, too!CheersGrant

Grant - have you decided that 1999 would be a good year to live dangerously??!!The whole idea of the current process is that the marks are confidential, that is Adjudicators DON'T know each others marks. They do discuss among themselves only at the Adjudicators meetings, but not the marks.I see that maybe that system should change but I can also see a great cry from "outside" that "everyone knows what marks Club X got for play Z" Remember the angst caused by some leakage at the training session at WKSC?should be a lively meeting on Monday. hope I get home by midnight.> Hi Leah> How exciting to have such a talented debater (not to mention> all your other extraordinary talents, wit, intelligence, good looks...> i could go on) here with us!> trust you to draw out the point where i was being deliberately> vague - well, even vaguer than the rest ;)> i'm concerned that faced with the difficulty of deciding exactly> how to go about a peer review and with the ruckus that implementing> such a system might cause, we might dismiss the notion altogether.> i was attempting to focus optimistically on the strengths of such> a system.> why am i trying to avoid discussing the logistics?> take a look at some of my thoughts (that will doubtless change!)> on how the system might work:> 1) A similar process of attendance and individual marking to> that currently in place.> 2) A sheet detailing the marks given to each eligible production> and who gave the marks is distributed to the adjudicators prior to> their all meeting near the end of the year.> 3) At the meeting the shows are divided into those for which> there was concensus amongst the adjudicators that marked and those> amongst which there was not.> 4) I'd suggest that if the marks from the three adjudicators> were within a range of ten (say, 65, 69 and 60) then there is some> concensus on the production and this production can skip to the next> stage.> 5) If the marks vary more widely, each adjudicator that gave> a mark is given an opportunity to explain why they feel the mark they> gave is appropriate. They may answer questions from the other adjudicators.>> 6) All the adjudicators then vote on whether to accept the highest> score, lowest score or an average of all three scores.> 7) At the next stage, any adjudicator that saw a production they> did not mark may query the votes given to that production.> 8) The adjudicator querying the marks explains why they think> the marks are inappropriate and offers their own mark for the show.>> 9) The adjudicators that marked the production each explain the> marks they gave.> 10) All adjudicators vote on whether to accept the highest score,> lowest score or an average of all four scores.> 11) At the final stage, the productions all have a provisional> ranking.> 12) Starting at the lowest ranking production, this production> is compared with the one above it.> 13) Any adjudicator may dispute the ranking of one of these two> productions as better than the other.> 14) Adjudicators for and against changing the ranking are heard.>> 15) All adjudicators vote on whether to change the ranking of> the lower ranked production to that of the one above.> 16) This production is then paired with the next highest production> and the process is repeated from step 13.> Phew! wouldn't that be fun!> There could be a number of shortcuts in the process that wouldn't> significantly harm the outcomes, but it would still take at least> a full day to complete the process.> Unworkable? Probably... but in an ideal world we might not even> have awards :)> hehe> the potential for leaked posts to Gossip Central "I KNOW> THE WINNER!" would be enormous!> (that's a wicked grin)> Cheers> Grant

Re: How to judge the judges?

Thread (6 posts)