One of my excuses for not posting very frequently lately is that I had been studying for a Microsoft certification exam that I took on Thursday (I passed!). I had earned an MCSE about six years ago (back when the current SQL Server version was 6.5, and Windows Server was NT 4.0) but it has since expired, and I figured that it was time to renew my certification to prove to potential employers that I have current knowledge and that my brain still works.

So, I’ve been thinking about tests and testing recently.

I had a junior high school geometry teacher who was otherwise very forgettable, but I remember that one day she sympathized with some students’ complaints about certain tests not reflecting their knowledge and she said that: “All a test can measure is how well you took that test.”

I liked that.

The Microsoft test was of the type where a fairly elaborate scenario would be described, a problem situation posed, and then a set of potential solutions offered. The test taker was supposed to choose the “best” answer (or sometimes all applicable answers). Often there was an unambiguously best choice (in my opinion), but at other times there were several reasonable choices and I thought it was at least controversial to call one of them “best”. I found that I was often trying to psycho-analyze the test writer; looking for clues in the scenario description for what he might have wanted to emphasize. Now, psycho-analyzing test writers from their questions might be a useful
skill, but it was not a skill that the test purported to measure.

Another problem I have with testing is that the tests are often closed-book, time pressured experiences. People with good memories and who are fast readers and decision-makers do better than others. But, in the real world (especially in modern technical fields) it’s unrealistic to expect people to remember all the facts they need or to come to a conclusion in a matter of seconds. The skill you should really be looking for is the ability to solve a problem with all of the resources available in a typical work environment, and in a reasonable length of time (not a matter of seconds). But, such testing would be more difficult to control. So, we end up measuring an approximation of the desired skills because measuring the actually desired skills is too
difficult. (Which reminds me of an I Love Lucy episode in which Lucy claimed to be looking for an item that she lost in another room “Because the light’s better here.”)

I don’t have a real solution to any of this.

I understand that it can be important to get an idea of a person’s skills. And, also, that it’s helpful for the individual to learn about which areas he could focus on to improve. And, imperfect testing is better for these things than no information at all.

But, I guess I hope that people are aware that test scores do not necessarily accurately reflect a person’s knowledge and capabilities. And, I hope that we can come up with better ways of judging these things in the future.