Palantir’s leaked documents and the concept of uncertainty
Did you hear about TechCrunch’s leaked documents detailing the client list of Palantir, the super secretive data mining contractor (hat tip Chris Wiggins)? Palantir, founded by uberlibertarian Peter Thiel, had clients as of 2013 including the LAPD, the CIA, DHS, NSA, the FBI, and CDC. Besides data mining for government agencies, they also work in the finance sector and the legal sector.
Here’s the scariest thing about the TechCrunch article:
Samuel Reading, a former Marine who works in Afghanistan for NEK Advanced Securities Group, a U.S. military contractor, was quoted in the document as saying It’s the combination of every analytical tool you could ever dream of. You will know every single bad guy in your area.”
That quote, if true, belies a lack of understanding of what data mining can actually do in terms of accuracy. No data mining tool can be both comprehensive and accurate – find all the bad guys with no accidental good guys getting caught in the net. It’s just not possible, unless you have DNA samples with markers for “bad guyness,” and even then DNA tests sometimes get mixed up.
It behooves an expensive and fancy consulting company to act like their tools are prophetic, however, even if that means false positives or false negatives happen all the time, which of course they do, with any algorithm.
It’s bad enough when stupid start-up companies claim big data solves everything, when what they’re doing is trying to solve a problem nobody cares about. It’s another thing altogether when it’s our military and military contractors and police and secret services, and when we don’t have any view into what it actually does. Scary stuff.
mathbabe 
Evidently “überlibertarian” means something like “my freedom über alles”.
LikeLike
Spot on.
LikeLike
I had similar feelings of fright reading the NY Times article “Why Reams of Data Did Not Thwart the Paris Attacks”.
If we expect perfection (that is to stop all attacks) we justify a police state (which still will be unsuccessful).
Should we look to security forces as the primary means of reducing attacks?
To the extent we do look to the security forces, do they benefit from widespread surveillance or would they be better with more targeted methods?
Perhaps it is BECAUSE of reams of data that they didn’t thwart the attacks.
LikeLike
really, they called it Palantir ? named for the intelligence artifacts of Mordor and Sauron ?
Well, at least we know what to expect from them.
Also, the story of the Palantir is of incomplete information and misleading intelligence – Saruman thought he saw overwhelming forces in Mordor and joined them, Sauron thought Pippin was the ringbearer and missed the real ringbearer infiltrating Mordor.
So I guess it’s actually a good name for what they do – evocative of both evil and incompetence – though perhaps not quite the image they had in mind.
LikeLike
The palantiri were made by the Eldar of Eressea, not Sauron! True about the misleading info, though you may be conflating Saruman and Denethor.
LikeLike
I fully encourage the path this conversation is taking, but can we continue in Elvish?
LikeLiked by 1 person
Quenya or Sindarin?
LikeLike
of course Sauron didn’t make the palantir, but he was able to use it for his fell purposes.. similarly Palantir the company uses data science for theirs..
ah, but in Elvish, Pedin i phith in aníron, a nin ú-cheniathagir..
LikeLike
Omg awesome
LikeLike
Pretty much any time anyone even uses the phrase “bad guys” in seriousness I immediately discount what they’re saying as bombastic bullshit.
LikeLike
+1
LikeLike
What might have thwarted the Paris attacks would have been not to kill hundreds of thousands of people by Western bombings in the Arab and Muslim worlds over the last 25 or 30 years. Or not to have supported the worst regimes there.
LikeLike
Very much related to the earlier discussion of Lewin and Birkhoff. “Bad guys” betrays a fixed mindset that is deeply problematic. I think your dna test comment was meant with heavy sarcasm, but it doesn’t quite come across in the post.
For the liberals, “bad guyness” fails to capture the fact that people who have committed, or have the potential to commit, bad acts (a) could be rehabilitated and (b) may also commit beautiful acts.
For conservatives, “bad guyness” fails to recognize that current non-enemies could become enemies in the future. It also suggests that the only way to deal with bad acts is to eliminate the actor.
Also, a possible strategic interpretation of Peter Thiel’s involvement: if the state is going to use these methods, best to be the one controling where the eye looks and what types of conclusions it draws. If I ran the shop, I would make sure every script had a header removing my personal data (or that for friends and family).
LikeLike
I couldn’t agree more, Josh.
LikeLike
Possible future blog topic, semi-buried in the following news article: is there any particular reason why murders, robberies, etc. fell during the NYPD police slowdown?
“During the slowdown, there was also a moderate decline in reports of murder, robbery and other serious crime, the data showed.”
http://www.reuters.com/article/2015/01/13/us-usa-police-slowdown-idUSKBN0KL28C20150113
LikeLike
Well as intelligence tools, its quite likely that palantir products score highly in some evaluation metric. They”ll point you to some (most?) targets while also outputing a smaller set of false positives and negatives. They are not long term solutions for a less fractured global society but a bandage that treats a symptom, not a cause
LikeLike