The randomness of evaluation
You may expect that, with all the advices I am giving out, I am a great expert at evaluating presentations… Well, recently I had another proof that the task is probably undecidable.
I was in a panel evaluating student presentations. My guts considered some of them as perfect as one can expect at that level; others as catastrophic as it can get; a couple of them average. But, instead of using guts, we had agreed on an evaluation grid, giving points for criteria like “quality of the slides”, “delivery”, “ability to answer questions”. Then, I had to admit that the delivery of the “perfect” presentations was maybe not so perfect, that the slides of the “catastrophic” ones were not so catastrophic… In short, the analytical approach tends to squeeze the distribution around the average, while the synthetic approach of the guts tends to divide in just “good” and “bad”.
Now, which approach to follow? There is no clear cut answer. In our super-bureaucratized universities, the analytical approach is almost compulsory: the other one would be regarded as too subjective. Moreover… hmmm… I don’t trust myself the guts of some of my colleagues, so why should they trust mine?
However, let me tell you a personal story which shows how the analytical approach can lead to monumental failures. It happened a few years ago.
I was assigned to evaluate some projects for a scientific competition of high school students. In one of the projects, the student had combined several notions and techniques of modern physics, in order to propose a new device. According to all the criteria I was asked to judge, the project was absolutely outstanding: “creativity”, “effort”, “understanding of the basics”, “quality of the report” etc. It suffered from a little problem: the whole proposal did not make any sense! The reason is that those notions and techniques just could not be combined in the same implementation. Unfortunately, “correctness” was NOT one of the criteria we were asked to assess and grade. I had to use the space left blank for “additional remarks” and write there “The whole proposal is flawed”.
This happened twice, evaluating the report first and a poster presentation later. Guess what? Bureaucrats had no time to read the additional remarks, they just summed up the points in the main table: the student won the very first prize with high honors and was invited to attend the international phase of the competition in the U.S.
This is enough to prove my point. However, for completeness, let me tell you how the story ends. Upon receiving notification of the triumph, the school contacted me with an enthusiastic e-mail, amounting at: “We heard that you were involved in the evaluation, now he has to go to the U.S. and we need to make sure that the proposal looks perfect there: would you mind helping us to fix all the remaining details?” To which, I replied in capital letters: THAT PROPOSAL IS FLAWED, followed by the list of flaws. After some days of silence, a much humbler e-mail popped up in my box: “Is there anything we can do to fix it?” Well, to fix the science, one could just replace electrons with neutrons. But since the whole device was meant for use in a computer, the applied part would then become a proposal to put sources of neutrons (= a nuclear reaction) in personal computers, which may not be deemed as practical or even desirable… Finally, the outburst of H1N1 solved the case: all the trips of students to the U.S. were canceled and the project was forgotten. The school and the student retained all the honors — which they both fully deserve, maybe not on the basis of that project, but for the ensemble of their achievements; of course, this is only my guts’ assessment 😉