Today, I’m going to expound on two examples I have come across recently of researchers trying to measure properties that are very difficult to measure, and politically charged.
At SIGCSE, I saw a paper on sex and gender in CS presented, which was thoughtfully titled Cultural Representations of Gender Among U.S. Computer Science Undergraduates: Statistical and Data Mining Results. The results depend on running a bunch of (CS and non-CS) students through the Bem Sex Role Inventory and looking through the results for patterns.
The Bem inventory involves looking at a bunch of adjectives (“analytical”, “warm”, “adaptable”, …), and deciding how much that describes you. Each of the adjectives has a gender that it describes (“masculine” or “feminine”), and you get a score at the end. Of course, there’s a huge cultural bias involved in this score, but gender (as opposed to sex) is a social construct anyway, so off we go.
The results he found were very interesting, but not what I want to get into here. At the end of the author’s presentation, there was much consternation over the use of the Bem Inventory. One woman in the audience in particular had the view that he should have not done the study at all, rather than use this measure that offended her sensibilities.
Of course, the Bem Inventory isn’t perfect: it’s trying to measure an idea that’s moving target, that is not uniform across any significant population, and that is very specific to the U.S. But, it clearly measures something, and not something that there is (apparently) no better way to measure.
I ran across my second example when looking up the authors of the truly excellent book Why Beautiful People Have More Daughters. One of them has recently published an article
on the relationship between IQ and health. [Those with a decent University library can probably find the whole article online: Kanazawa, British Journal of Health Psychology, Volume 11, Number 4, November 2006, pp. 623-642(20).]
The article does confound the terms “general intelligence” and “IQ”. I think it’s pretty hard to argue that the thing we measure and call “IQ” is the platonic ideal measure of “intelligence”. That being said, it’s clear that IQ is a measure of something. It turns out that the “something” that IQ measures is strongly correlated with life expectancy (stronger than income inequality or economic development).
As a result, the author was accused of promoting eugenics. Now, maybe I didn’t read the paper carefully enough, but I didn’t see the “kill the dumb ones” part. The author didn’t even actually measure anything himself: the whole paper is a meta-analysis of other studies of IQ and health. All he did was grind out some stats.
Anyway, both of these studies have the same underlying issue: they rely on the measurement of something that isn’t possible to measure very well. In addition, the thing being “measured” is something that has a bunch of emotion attached to it. I don’t think the solution here it to just not study this stuff. Let’s just all recognize that correlation with the thing we’re trying to measure will do in a pinch.
April 1st, 2008 at 9:46 am
It is always interesting when people do not want to tackle things that are important but difficult to measure and politically charged. The classic “I don’t like where this is going, so I think you shouldn’t study it” mentality.
Just because the truth may be uncomfortable does not make it wrong. My two cents….
Oh, if you’re looking for another example of this, check out Freakonomics and the chapter on the relationship between legal abortion and the crime rate. Boy did that one cause some outrage!
Paul
April 1st, 2008 at 5:13 pm
Funny, I was just thinking that there’s no way that IQ is positively correlated with life expectancy. If anything it’s probably an inverted U-shaped curve (Oli! It’s hormesis!).
Then I realized… the authors are comparing countries’ mean IQs to their mean life expectancies. Yes, I study individual variation, so I may be biased, but I’m pretty sure that doing the stats on the means doesn’t make any sense in this case. Within a country the people with high IQs could be dying early, and the people with lower IQs could be living longer. The people with mid-level IQs are probably the ones that are responsible for setting that country’s mean life expectancy.
I think the uncomfortable topics should be studied – just on the individual level – which would probably make them way more uncomfortable.
Okay, individual variation rant over now.
April 1st, 2008 at 9:58 pm
I don’t know where I stand on this issue.
I don’t ever want to say, “I don’t like where this is going so we shouldn’t study it”, but at the same time studying something and using the wrong terms of measurement just gives us false data. And as we’ve seen time and time again, data (even accurate data) can be used against (groups of) people.
So to say, “well we can’t really accurately measure x, but this measurement y is good enough” doesn’t really cut it for me. It has the very real potential to do more harm than good.
But where does that leave us? X still needs to be studied.