At 7/17/17 07:25 PM, EDM364 wrote:
Guys, I've got over 8,000 words of reviews typed up. Send me a PM if you want 'em on the express train.
At 7/17/17 06:09 PM, EvilRaccoon wrote:
So I think having 7 judges, then have them pick one or the other is a much better, for the judges, for the participant and just in general much fairer. It would be interesting to hear what everyone else though of this matter, as it's something which could be implemented in 2018.
Honestly, through the last round, I felt that not immediately seeing someone's competitor in the line-up to draw a direct comparison was much fairer. It allowed me to focus on the quality of the work and not think about anything but the song I was judging at that exact moment. Do you realize that if we focused on the 1v1 matchups at the same time, so many more biases could enter the equation?
As a judge though, How you draw your own conclusions to who should win is up to yourself. The same way you are currently flexible to build your own criteria for scoring. If you chose to score them both individually, then give your decision based on your evidence, that would probably be a needed anyway since you are all still reviewing. This would be most likely the way some of the judges would work. But how it's brought together is where the impact lies.
Cutting out the scores also cuts out important statistics such as the overall statistical trends from our most lenient judge, @TaintedLogic, and our most critical judge, @SkyeWint. The consistency of these scores reads like a litmus test of sorts, the results informing that yes, we're all paying attention and being honest. A simple Y/N is not as simple as it sounds, and it's not going to disclose much -- or even help me remember what stars to score your submission when I go back through and paste my reviews in.
Statistical trends do not support evidence of more appropriate outcomes though, they actually enforce the problem with people losing out of 0.002. The stats merely support judges behavior. If all judges are giving a weight of 1 vote. Then this statistic is no longer needed, there would be no need to be concerned with who is the pulling judge - this is something participants really don't want imo. And as a judge of the NGUnderDogs, I would not like to think, my score can outweigh for better or worse several other judges. The scoring statistics of tracks wasn't added to the run down either. The mean of the judges was, nobody asked for the average of the tracks.
And then you see why we are going in full circle. We are concerned with the averages of judges habits because they do impact the outcome. The judges shouldnt be in question, but they are because of the lack of equal influence.
Also, I like being able to see from a glance the highest and lowest average scores from each round. You're not going to be able to glean that kind of record from Y/N -- and it's also interesting to see how the line-up would have looked were we to go by each judge's individual perspective, something a Y/N will not give you either.
That's understandable, you get to see which tracks the judges favor, and potentially who is a threat. But, I'd argue fairness and equal judge influence outweighs the need to see scores. If anything, you could keep this as a happy medium. Maintain scores for this visual statistic. But judges influence is only 1 or 0 per group.
I also find that Y/N is like being on a jury. There's so much pressure to agree with everyone else, even though you know you aren't supposed to, just so you can go home.
Judges should be able to deliberate and make a decision. They do this already. Judges afterall do not adhere to strict criteria, they score as they see fit. If a judge cannot decide which is the stronger piece, even through existing criteria then there is a case of uncertainty. Your own decision can't be swayed if you have an even number of judges. It will always side one or another, it's up to you as a judge how you arrive at your decision.
The idea here is to yes, have the tracks evaluated, but your evaluation will not impact another judges decision, which is what's happening right now (not of your own doing of course). Judges decisions need to have the same amount of influence. It's creates a fair equalization among yourselves so when arrives at the end of the round, it's solid.