Today, I’m going to expound on two examples I have come across recently of researchers trying to measure properties that are very difficult to measure, and politically charged.
At SIGCSE, I saw a paper on sex and gender in CS presented, which was thoughtfully titled Cultural Representations of Gender Among U.S. Computer Science Undergraduates: Statistical and Data Mining Results. The results depend on running a bunch of (CS and non-CS) students through the Bem Sex Role Inventory and looking through the results for patterns.
The Bem inventory involves looking at a bunch of adjectives (“analytical”, “warm”, “adaptable”, …), and deciding how much that describes you. Each of the adjectives has a gender that it describes (“masculine” or “feminine”), and you get a score at the end. Of course, there’s a huge cultural bias involved in this score, but gender (as opposed to sex) is a social construct anyway, so off we go.
The results he found were very interesting, but not what I want to get into here. At the end of the author’s presentation, there was much consternation over the use of the Bem Inventory. One woman in the audience in particular had the view that he should have not done the study at all, rather than use this measure that offended her sensibilities.
Of course, the Bem Inventory isn’t perfect: it’s trying to measure an idea that’s moving target, that is not uniform across any significant population, and that is very specific to the U.S. But, it clearly measures something, and not something that there is (apparently) no better way to measure.
I ran across my second example when looking up the authors of the truly excellent book Why Beautiful People Have More Daughters. One of them has recently published an article
on the relationship between IQ and health. [Those with a decent University library can probably find the whole article online: Kanazawa, British Journal of Health Psychology, Volume 11, Number 4, November 2006, pp. 623-642(20).]
The article does confound the terms “general intelligence” and “IQ”. I think it’s pretty hard to argue that the thing we measure and call “IQ” is the platonic ideal measure of “intelligence”. That being said, it’s clear that IQ is a measure of something. It turns out that the “something” that IQ measures is strongly correlated with life expectancy (stronger than income inequality or economic development).
As a result, the author was accused of promoting eugenics. Now, maybe I didn’t read the paper carefully enough, but I didn’t see the “kill the dumb ones” part. The author didn’t even actually measure anything himself: the whole paper is a meta-analysis of other studies of IQ and health. All he did was grind out some stats.
Anyway, both of these studies have the same underlying issue: they rely on the measurement of something that isn’t possible to measure very well. In addition, the thing being “measured” is something that has a bunch of emotion attached to it. I don’t think the solution here it to just not study this stuff. Let’s just all recognize that correlation with the thing we’re trying to measure will do in a pinch.