Big Data Is The New Phrenology
Have you ever heard of phrenology? It was, once upon a time, the “science” of measuring someone’s skull to understand their intellectual capabilities.
This sounds totally idiotic but was a huge fucking deal in the mid-1800’s, and really didn’t stop getting some credit until much later. I know that because I happen to own the 1911 edition of the Encyclopedia Britannica, which was written by the top scholars of the time but is now horribly and fascinatingly outdated.
For example, the entry for “Negro” is famously racist. Wikipedia has an excerpt: “Mentally the negro is inferior to the white… the arrest or even deterioration of mental development [after adolescence] is no doubt very largely due to the fact that after puberty sexual matters take the first place in the negro’s life and thoughts.”
But really that one line doesn’t tell the whole story. Here’s the whole thing, it’s long:
Pages 1 and 2
Pages 3 and 4
Pages 5 and 6
As you can see, they really go into it, with all sorts of data and speculative theories. But near the beginning there’s straight up racist phrenology:
From page 1
To be clear: this was produced by a culture that was using pseudo-scientific nonsense to validate an underlying toxic and racist mindset. There was nothing more to it, but because people become awed and confused around scientific facts and figures, it seemed to work as a validating argument in 1911.
Anyhoo, I thought this was an interesting back drop to the NPR story I wanted to share with you (hat tip Yves Smith) entitled Recruiting Better Talent With Brain Games And Big Data. You can read the transcript as well, you don’t have to listen. Basically the idea is you play video games and the machine takes note of how you play and the choices you make and comes back to you with a personality profile. That profile will help you get a job or will exclude you from a job if the company believes in the results. There’s been no scientific tests to see if or how this stuff works, we’re supposed to just believe in it because, you know, data is objective and everything.
Here’s the thing. What we’ve got is a new kind of awful pseudo-science, which replaces measurements of skulls with big data. There’s no reason to think this stuff is any less biased or discriminatory either: given that there’s no actual science behind it, we might simply be replicating a selection method to get people who we like and who remind us of ourselves. To be sure, it might not be as deliberate as what we saw above, but that doesn’t mean it’s not happening.
The NPR reporter who introduced this story did so by saying, “let’s start this hour with a look at an innovation in something that’s gone unchanged, it seems, forever.” That one sentence already gets it wrong, though. This is, unfortunately, not innovative. This is just the big data version of phrenology.
mathbabe 
I would insert one: maybe big data is the new Myers-Briggs? And in its day, Myers-Briggs was the new phrenology?
LikeLike
Fair!
LikeLike
“after puberty sexual matters take the first place in the negro’s life and thoughts.” Surely this is true for people of every race, at least for those critical first few years…
LikeLike
I fear that you’ve picked a target that is too easy to criticize. Despite the vast amount of time and effort devoted to thinking about this, we seem to know very little about what makes a business successful or who makes an ideal employee. Starting years ago from a naive belief that meritocracy was meaningful, I have come to see that there are really three dimensions we can assess when hiring:
(0) does the candidate have the (minimum) required technical skills
(1) how much does this person want the job (through direct and indirect signals)
(2) how likely is it that I (the hiring manager) will enjoy working with this person? this is usually simplified into “is this person similar to me?”
I’ve started numbering with 0 because I think this is really a rejection factor, i.e., people who don’t have the skills (should) get rejected, but being qualified over the minimum doesn’t help much. Instead you want people who want the job: motivation trumps prior ability.
Also, many jobs don’t actually have any truly required technical skills.
Factor (2) trumps (0) because eagerness to work together as a team is the sine qua non of a functioning team. Or, phrased differently, you can guarantee a failed team by putting together people who don’t want to work together, but the closest you can get a priori to a successful team is a group of people who want to be a successful team.
Ironically, in polite conversation, ability (0) is the only characteristic we are supposed to acknowledge having any weight. That leaves the HR industry to dress up the other two points, and all their associated biases, in fancy disguises.
I’d welcome an alternative grand unified theory of hiring. Or, any evidence that someone, somewhere is actually good at hiring.
LikeLike
There is nothing pseudoscientific about the idea of external noninvasive proxies for brain measurement.
Measuring babies’ head circumference at various ages is a routine part of medical care for children these days, with percentiles reported to the parents.
I know of no serious scientific objection to pursuing a Big Data reboot of phrenology, such as correlating noninvasive head measurements with internal data known from tomography.
LikeLike
But those head measurements are not to determine intelligence. They are a baseline for future growth, and there are problems, like hydrocephalus, that correlate with unusually large or rapidly increasing skull sizes.
LikeLike
Those measurements don’t “determine” hydrocephalus, any more than skull measurements can determine intelligence. They are used as a non-invasive and cheap 1-dimensional surrogate for more detailed 2 and 3 dimensional procedures (X rays, tomography, surgery). Since intelligence correlates with physical brain measurements, about as well (maybe better in some ways) as head measurements correlate with various medical diagnoses, I don’t see what difference you are suggesting exists in principle between the two situations.
LikeLike
When I was a medical student I was told the experiment to correlate brain size with intelligence had been done and the results showed it to be false, sure more intelligence is shown in species with bigger brains but the structure of that brain makes it impossible to really compare across species. Only within a species can we measure comparative brains and to a extent measure intelligence, no conclusive evidence was found that human brains are bigger in the heads of more intelligent humans.
LikeLike
https://xkcd.com/882/
LikeLike
Reblogged this on Error Statistics Philosophy and commented:
It happens I’ve been reading a lot lately about the assumption in social psychology and psychology in general that what they’re studying is measurable, quantifiable. Addressing the problem has been shelved to the back burner for decades thanks to some redefinitions of what it is to “measure” in psych (anything for which there’s a rule to pop out a number says Stevens–a operationalist in the naive positivist spirit). This at any rate is what I’m reading, thanks to papers sent by a colleague of Meehl’s. I think it’s time to reopen the question. The measures I see of “severity of moral judgment”, “degree of self-esteem” and much else in psychology appear to fall into this behavior in a very non-self critical manner.
LikeLike
Cathy, – Interesting to hear that you possess an 11th Britannica. And to read your assessment of it which is, of course, painfully accurate. But ownership of such a thing is very unusual these days. I know because I also have one. Funny though, I hope these books never entirely disappear, simply because they do provide a record of attitudes and biases from a specific and important time. I use mine occasionally too, mostly for a check on some of our current opinions and for other articles, some of which are actually correct and maybe enjoyable.
LikeLike
I too have a 1911 edition of the EB, which my mother and I and my children grew up with, fascinated by its highly academic, as well as often twisted, contrast with the simplified language of modern day pablum in popular encyclopedias. I seem to remember that the large entry on Quaternions was one of the last attempts to save them from the onslaught and fracturing into vastly simplified vectors, matrices, and tensors after Nature Magazine’s biased “war on quaternions”. Now in the era of digital computation (which is not at all boggled by their incredible complexity) quaternion concepts are returning in geometric and Clifford algebra applications not only in theoretical physics, but in the tracking, positioning and control of satellites, image processing, computer vision, artificial neural networks, Fourier transforms, position determination and aircraft navigation, electrical engineering, robotics, classical and differential geometry, education, and ‘effective’ algorithms and computations.
LikeLike
You seem to be suggesting that the people who do this data analysis are just inventing the results the same way phrenologists did. Is that actually true?
If you *wanted* to do this kind of thing right, aren’t there plenty of psychometric constructs with actual, well-tested validity that you could calibrate your game analysis against? E.g., if empathy is correlated with management performance (heh), then you calibrate your predictor against a well-tested inventory of empathy and it should be a reasonable predictor of performance as well.
Of course that’s not an *independent* scientific test, and I hope these startups eventually hire independent auditors to confirm their predictive validity–but unless they’re straight-out lying about their process, they’re already being a LOT more scientific than whatever crappy resume screen/interview process their clients previously had in place, where in all likelihood nobody even tried to calibrate the outcome to anything they cared about.
(And pymetrics, at least, claims to have tested their fit-rating algorithm to ensure non-discrimination. Don’t know about the others.)
LikeLike
Vacuous fields suffer frequent name changes. The rubes catch on quickly so we must constantly invent new terms to maintain a nice level of confusion. Before “Big Data” caught on we used data warehouse. Then someone noticed that data warehouses typically contained piles of mostly useless or misleading crap. Attempts to use the contents of data warehouses resulted in the creation of the more accurate term “Data Whorehouse.” Shortly after “Data Whorehouse” stormed the rhetorical barricades “Big Data” appeared. No doubt after phrenology catches on we will endure another change of terminology for what used to be called “garbage in garbage out.”
LikeLike
Good complement to Tim Harford’s “Big data: are we making a big mistake?”
Link = http://www.ft.com/intl/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html
LikeLike
Surely, you can’t conclude that Big Data is the new phrenology from this? All you can say is that “certain data analytics techniques count as pseudoscience”, right? Which hardly comes as a surprise.
LikeLike
I’d recommend both “The Mismeasure of Man” by Stephen Jay Gould, a good introduction to the problems inherent in the pseudo-science of IQ testing, and “The Cult of Personality Testing: How Personality Tests Are Leading Us to Miseducate Our Children, Mismanage Our Companies, and Misunderstand Ourselves” by Annie Murphy Pau
LikeLike
The original article (Recruiting Better Talent With Brain Games And Big Data.) has very little to do with Big Data. It is, in turn, using the new buzz-word to justify its pseudo-science. Big data is essentially data analytics: using large volumes of data (with outcomes) to classify new data. But this work is just the same old stuff of psychometric testing with a new label – to make it *actually* big data, there would need to be a large volume of actual results (how good was the employee? was it an appropriate hiring?) to correlate with the measurements. And (as tyaresun notes) you have to be really careful with correlation, particularly if you use p values (see also http://www.tylervigen.com). So: just because someone calls it big data doesn’t mean that it uses big data in an appropriate way. Bad science is bad science – there’s always been a lot of it about!
LikeLike