Feedback Is Tricky

by AJ

Another great post over at Creating Passionate Users:

"So how can we hope to learn anything about what our students want and need if the very act of answering a question could change their answer? We have to get better at making inferences from what we observe without intervention. We have to get to the spirit of what we observe, rather than focusing on the specific details. We have to recognize that what they do says much more than what they say, especially when they're not saying anything at all."

Kathy Sierra calls this the "Quantum Mechanics of Users".... for in teaching, as in Physics, the act of observing/measuring something changes it.

Which goes a long way towards explaining the utter uselessness of traditional "teacher evaluations"..... as well as focus groups and other highly contrived attempts to gather student feedback. The problem: the act of administering these measures warps the students responses. In most cases, this has the effect of unconsciously prodding the students to tell you what they think you want to hear. You get the occasional suggestion... and a whole lot of vague nothing: "good class", etc.

Often, they even contradict their true feelings. In relaxed conversations with students... those that I have built rapport with... Ive never had one tell me "I wish we would learn more formal grammar rules". Ive also talked to my foreign friends (who are not in my classes) and asked them about how they learned English, what was wrong with their English classes, and what was right. In every case they complain about the focus on grammar rule memorization and translation.

Yet, if you give them a formal survey and direct them with questions such as "How important is grammar instruction?",... you will often get results that indicate a strong desire for formal grammar instruction. This can seem baffling.

But Sierra's post clears up much of the confusion. She's 100% right... the act of measuring and analyzing causes students to change their answers. Ie. No such thing as "objective" measures. In fact, those that are considered most "objective" (ie formal) may cause the most distortion.

Sierra calls for more naturalistic and anthropological means of gaining feedback... methods such as observation (ie. taping class), informal conversations, participant research (become a student yourself), careful listening between the lines, etc... These methods can also distort students' responses but they seem to do so in a less dramatic way.... and thus provide more useful information to teachers and admins.

So throw out the clunky surveys and forget the "focus groups". Become a student anthropologist.