Judging the judges

I understand that the ITA committee are due to discuss some ideas realted to the training and selection of award adjudicators.Barry Lucas and I had a very fruitful discussion last week at the ITA office.Amongst the ideas to emerge from our conversation was a suggestion from me that the adjudication process had been rendered too anonymous and in the process was "protecting" the adjudicators from criticism that may at times be warranted. After all, what does anyone really know about our adjudicators and the marks they give?I suggested that each adjudicator's marks should be subject to a process of peer review - discussed and if necessary voted on at a meeting of all adjudicators. A similar process already takes place for adjudicator's certificates, why not for other awards?Opinions often differ so widely on a show, it is vital that these differences be aired and discussed. Through this process of debate and discussion the adjudicators can achieve concensus. Along the way, their outlooks will be broadened, their critical thinking tested and their opinions revised.I'd almost line up to adjudicate in a process like that! It would be very challenging for all involved.If adjudicators are going to sit in judgement over other people productions, i say, let them be judged, too!CheersGrant

Well this is my first time on this page and I must say so far it has been averitable landslide of cyber fun, what a great idea - a forum for the ideaswe usually express in green rooms and bars out of the earshot of fellowthespians. How very year 2000 of us.Grant, thrilled to see your well considered and aired (and justified) viewson our adjudication system are finally being given a good think about by the ITA. Little more explanation required for me though I think -> I suggested that each adjudicator's marks should be subject to> a process of peer review - discussed and if necessary voted on at> a meeting of all adjudicators.Does this mean that the adjudicators who didn't see the show in question will be voting on accepting the marks of the adjudicators who did see it, or that all the adjudicators who saw a show will reach a consensus about it, or something I'm just not grasping at all?

Hi LeahHow exciting to have such a talented debater (not to mention all your other extraordinary talents, wit, intelligence, good looks... i could go on) here with us!>> I suggested that each adjudicator's marks should be subject to>> a process of peer review - discussed and if necessary voted on at>> a meeting of all adjudicators.>> Does this mean that the adjudicators who didn't see the show> in question will be voting on accepting the marks of the adjudicators> who did see it, or that all the adjudicators who saw a show will reach> a consensus about it, or something I'm just not grasping at all?trust you to draw out the point where i was being deliberately vague - well, even vaguer than the rest ;)i'm concerned that faced with the difficulty of deciding exactly how to go about a peer review and with the ruckus that implementing such a system might cause, we might dismiss the notion altogether. i was attempting to focus optimistically on the strengths of such a system.why am i trying to avoid discussing the logistics?take a look at some of my thoughts (that will doubtless change!) on how the system might work:1) A similar process of attendance and individual marking to that currently in place.2) A sheet detailing the marks given to each eligible production and who gave the marks is distributed to the adjudicators prior to their all meeting near the end of the year.3) At the meeting the shows are divided into those for which there was concensus amongst the adjudicators that marked and those amongst which there was not.4) I'd suggest that if the marks from the three adjudicators were within a range of ten (say, 65, 69 and 60) then there is some concensus on the production and this production can skip to the next stage.5) If the marks vary more widely, each adjudicator that gave a mark is given an opportunity to explain why they feel the mark they gave is appropriate. They may answer questions from the other adjudicators.6) All the adjudicators then vote on whether to accept the highest score, lowest score or an average of all three scores.7) At the next stage, any adjudicator that saw a production they did not mark may query the votes given to that production.8) The adjudicator querying the marks explains why they think the marks are inappropriate and offers their own mark for the show.9) The adjudicators that marked the production each explain the marks they gave.10) All adjudicators vote on whether to accept the highest score, lowest score or an average of all four scores.11) At the final stage, the productions all have a provisional ranking.12) Starting at the lowest ranking production, this production is compared with the one above it.13) Any adjudicator may dispute the ranking of one of these two productions as better than the other.14) Adjudicators for and against changing the ranking are heard.15) All adjudicators vote on whether to change the ranking of the lower ranked production to that of the one above.16) This production is then paired with the next highest production and the process is repeated from step 13.Phew! wouldn't that be fun!There could be a number of shortcuts in the process that wouldn't significantly harm the outcomes, but it would still take at least a full day to complete the process.Unworkable? Probably... but in an ideal world we might not even have awards :)hehethe potential for leaked posts to Gossip Central "I KNOW THE WINNER!" would be enormous!>:)(that's a wicked grin)CheersGrant

Grant - have you decided that 1999 would be a good year to live dangerously??!!The whole idea of the current process is that the marks are confidential, that is Adjudicators DON'T know each others marks. They do discuss among themselves only at the Adjudicators meetings, but not the marks.I see that maybe that system should change but I can also see a great cry from "outside" that "everyone knows what marks Club X got for play Z" Remember the angst caused by some leakage at the training session at WKSC?should be a lively meeting on Monday. hope I get home by midnight.> Hi Leah> How exciting to have such a talented debater (not to mention> all your other extraordinary talents, wit, intelligence, good looks...> i could go on) here with us!> trust you to draw out the point where i was being deliberately> vague - well, even vaguer than the rest ;)> i'm concerned that faced with the difficulty of deciding exactly> how to go about a peer review and with the ruckus that implementing> such a system might cause, we might dismiss the notion altogether.> i was attempting to focus optimistically on the strengths of such> a system.> why am i trying to avoid discussing the logistics?> take a look at some of my thoughts (that will doubtless change!)> on how the system might work:> 1) A similar process of attendance and individual marking to> that currently in place.> 2) A sheet detailing the marks given to each eligible production> and who gave the marks is distributed to the adjudicators prior to> their all meeting near the end of the year.> 3) At the meeting the shows are divided into those for which> there was concensus amongst the adjudicators that marked and those> amongst which there was not.> 4) I'd suggest that if the marks from the three adjudicators> were within a range of ten (say, 65, 69 and 60) then there is some> concensus on the production and this production can skip to the next> stage.> 5) If the marks vary more widely, each adjudicator that gave> a mark is given an opportunity to explain why they feel the mark they> gave is appropriate. They may answer questions from the other adjudicators.>> 6) All the adjudicators then vote on whether to accept the highest> score, lowest score or an average of all three scores.> 7) At the next stage, any adjudicator that saw a production they> did not mark may query the votes given to that production.> 8) The adjudicator querying the marks explains why they think> the marks are inappropriate and offers their own mark for the show.>> 9) The adjudicators that marked the production each explain the> marks they gave.> 10) All adjudicators vote on whether to accept the highest score,> lowest score or an average of all four scores.> 11) At the final stage, the productions all have a provisional> ranking.> 12) Starting at the lowest ranking production, this production> is compared with the one above it.> 13) Any adjudicator may dispute the ranking of one of these two> productions as better than the other.> 14) Adjudicators for and against changing the ranking are heard.>> 15) All adjudicators vote on whether to change the ranking of> the lower ranked production to that of the one above.> 16) This production is then paired with the next highest production> and the process is repeated from step 13.> Phew! wouldn't that be fun!> There could be a number of shortcuts in the process that wouldn't> significantly harm the outcomes, but it would still take at least> a full day to complete the process.> Unworkable? Probably... but in an ideal world we might not even> have awards :)> hehe> the potential for leaked posts to Gossip Central "I KNOW> THE WINNER!" would be enormous!> (that's a wicked grin)> Cheers> Grant

I don't see the marks being discussed among the adjudicators as being a huge problem as long as discussions take place AFTER the marks are submitted.Discussion before submission leaves open the theoretical possibility of adjudicators pushing their own agenda and controlling the adjudication.The problem at the moment is that there is no real way to tell if a particular adjudicator is way off beam. Theoretically, at the moment Muriel Boggins ( imaginary adjudicator ) could either have no idea and be submitting marks that bear no relation to the plays viewed OR be harbouring a grudge against Woop Woop Amateur Dramatic Society ( imaginary theatre company ) and purposely be marking them at less than they deserve.I don't believe that that IS happening but the potential is there.The marking system allows for adjudicators who consistantly mark too high or too low and adjusts accordingly, but there is no safeguard against - "Muriel the Idiot" or "Muriel who doesn't like Woop Woop".Allowing our predominantly intelligent, talented adjudication team to peer assess may eliminate the possibility of this occuring.Kimberley

I am again thrilled at seeing some great ideas thrown around on the gossip page,but after reading Grants lenghtly explaination of how a new adjudicators system might work..phewww...all sounds great on paper, but wow, a big process to implement.I will be watching with intest any further debate on this topic!I hope u get to bed before midnight too Norma!!!!!Sue Thomas