The smell test for big data
The other day I was chatting with a data scientist (who didn’t know me), and I asked him what he does. He said that he used social media graphs to see how we might influence people to lose weight.
Whaaaa? That doesn’t pass the smell test.
If I can imagine it happening in real life, between people, then I can imagine it happening in a social medium. If it doesn’t happen in real life, it doesn’t magically appear on the internet.
So if I have a huge crush on LeBron James (true), and if he tweets that I should go out and watch “Life of Pi” because it’s a great movie (true), then I’d do it, because I’d imagine he is here with me in my living room suggesting that I see that movie, and I’d do anything that man says if he’s in my living room, especially if he’s jamming with me.
Not actually my living room.
But if LeBron James tells me to lose weight while we’re hanging, then I just feel bad and weird. Because nobody can influence someone else to lose weight in person*.
Bottomline: there’s a smell test, and it states that real influence happening inside a social graph isn’t magical just because it’s mathematically formulated. It is at best an echo of the actual influence exerted in real life. I have yet to see a counter-example to that. If you have one, please challenge me on this.
Any data scientist going around claiming they’re going to surpass this smell test should stop right now, because it adds to the hype and adds to the noise around big data without adding to the conversation.
* I’ll make an exception if they’re a doctor wielding a surgical knife about to remove my stomach or something, which doesn’t translate well into social media, and might not always work long-term. And to be fair, you (or LeBron) can influence me to not eat a given thing on a given day, or even to go on a diet, but by now we should know that doesn’t have long term effects. There’s a reason Weight Watchers either doesn’t publish their results or relies on survivorship bias for fake results.
mathbabe 
This is why I have a crush on LeBron James.
LikeLike
Huh? Why would deciding what to eat have a different impact matrix than what movie to go to? Let’s imagine LeBron, after jamming, says “man, I’m hungry, let’s eat some greek yoghurt” Do you dispute that you’d be more likely to eat greek yoghurt? If you hang out with LeBron a lot over a distance of time, then you’ll eat lots more greek yoghurt, no? For long-term impacts, can LeBron change your actual taste in movies or just get you to see one movie the day he’s hanging out with you?
FoW
LikeLike
He can influence what I eat on one day I admit that.
LikeLike
…and that’s the basis for long-term changes in behavior. The smell test to me indicates that peer pressure can definitely have an impact (not the only impact but a significant impact) on behavior, both short-term and long-term. The only thing that makes food different from movies is that there’s a physical/metabolic element at play as well, so it’s analogous to, for example, using social networking/peer pressure to help someone quit smoking.
It’s an oversimplification and overstatement to say that peer pressure will make you quit smoking or change your diet or vote Republican or whatever, but it’s a significant understatement (and a statement that doesn’t pass the smell test) to say that peer pressure has nothing to do with those results…
LikeLike
But that’s not what I was saying. I am saying that social media doesn’t make magic things happen, that a given thing only happens via social media if it can happen in real life.
And I’m sorry but you can’t influence people to lose weight. You can’t, doctors can’t, Dr. Oz can’t, and Oprah can’t. I’ve got the data to back me up.
LikeLike
OK–I agree that social media reflect real life and not the other way around. I disagree that social influences (online or in real life) can have a cognitive impact and that what we eat is one of the things that can be impacted greatly by those influences both in the short and long term.
Weight is also a tricky one, because there’s a whole field of debate on what you “should” eat to manage your weight, and a lot of argument that well-intentioned advice might be just plain wrong (depending on who you’re listening to). Contrast with smoking where the outcome is simple: you smoke or you don’t smoke.
LikeLike
But what if he leans over, puts his hand on your shoulder, and says “Cathy, as you know, I am your trusted friend and also a professional athlete who takes great care of his body. I’m worried about your health Cathy, and I’d like to help you. Remember, I’m LeBron James. With your permission, I’d like to set you up with my fitness and nutrition guy/gal Kim, who’ll make a great, long-term plan for you that I know will make you feel and look great. What do you say?”
LikeLike
I’d say, “Do I get to work out with you, King James?” And I’d bat my eyelashes.
LikeLike
But being overweight IS the sort of thing that is subtly but measurably in-real-life influenced by your social connections. http://www.nytimes.com/2007/07/25/health/25cnd-fat.html?_r=0 It’s certainly not clear that this could carry over to online interactions. But I think it clears your smell test hurdle.
LikeLike
There’s nothing in that study that talks about causality. It just says you’re likely to be friends with fat people if you’re fat. It could be caused by being in the same social class and having the same kind of food environment, or anything else. In any case it’s not “influence” as it is commonly understood, i.e. intentional and controllable.
LikeLike
There is a strong relationship between modelers’ not understanding a smell test and their thinking that data mining is an unbiased process. In both, the idea is that the math speaks for itself. The thought goes, once the relationships are established with statistical rigor, such soft counterarguments as a smell test or biases underneath the search for a model form have no meaning.
As for the smell test, I see this issue all the time. I work a lot with data scientists who do demand and price forecasting. They use big datasets of potential drivers and use statistical analysis to identify the “best” drivers. Never mind that the model indicates that the price of chewing gum in China is a significant driver for computer demand in Brazil, that’s what the math “proves” (I exaggerate, but only slightly).
The idea of a mathematical proof outside of the purely mathematical realm is a category error.
LikeLike
There’s always a market for a new weight loss strategy, and there has been progress through time on weight loss methodologies for people.
Most businesses aren’t faced with profit gains so large that they need to understand all the science that underlies a predictor. If a business doesn’t fully understand the linkages in the patterns they’re seeing, then it’s just necessary to take that risk into account.
Scientifically, use of social graphs to influence weight loss might just prove anew things nutritionists have been advocating for decades. Financially though, I’d never bet against it.
Dieting it tough. People need help. Many of us can’t smoke a cancer stick every time hunger strikes and have it go unnoticed, like Barack Obama.
LikeLike
The thing about smell tests is that they rely on intuition. My sense of smell is the exact opposite of yours here. To me it seems totally reasonable to expect that your friends can influence you to lose weight — maybe not by direct harangue as in your thought experiment, but at least by example and other forms of indirect influence. Every health behavior change I can remember making in my own life was induced by people I knew.
(I concede that the extent to which behavior change can induce long-term weight loss at all is very much in question, but presumably the guy who studies how social media influences people to lose weight does, in fact, believe that it’s not inherently impossible for people to be influenced to lose weight. If he’s wrong about that, he’s wrong, but it’s not the Internet or big data he’s wrong about.)
LikeLike
But what you describe happens in real life and is subtle. What he was implying was that we could do something _momentous_ via social media data mining.
LikeLike
I’d say that a smell test may not be mere intuition but rather a first approximation of a theory, i.e. one that generally holds unless there are particular secondary effects that are larger than usual.
And by the way, I’m not suggesting social graphs are useless – they allow us to see information in a new and useful way. So for example they allow us to see that fat people are generally friends with fat people. But again, this doesn’t imply that anyone is wielding controlled and intentional influence over their friend’s weight.
LikeLike
There is a book called Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives — How Your Friends’ Friends’ Friends Affect Everything You Feel, Think, and Do, which describes the subtle ways in which influence spreads from people to people even when they are connected by more than one degree of separation.
You and the data scientist whom you spoke may be on to something. I say this because you are right in saying that we are not going to change our behavior simply because somebody is doing something that we must; however, as the book posits, influence spreads in subtle ways among network nodes, both strong and weak, in subtle ways that merit to be studied, or “smelled” to use your term. I think this is what the data scientist should be focusing on.
LikeLike
If I shared with you that I really enjoyed the movie titled, “Fat Head”, which was a rebuttal of the really silly movie about why we are all fat, called “Super Size Me”, would you go watch it? If added, carbohydrates are apparently the number one reason for increase in Diabetes and Heart Disease based on details in that movie, would you go watch it?
If I finally concluded that the nice “software engineer” who made the movie, a logical thinking person, since he writes software, ate McDonalds food, just like the dork in “Super Size Me”, and lost weight, improved his health by improving his cholesterol and reducing his blood pressure, would you watch the movie (it was on Netflix, last I looked)?
Social media exchanges are places where information can be shared, and providing the right information in the right form, can motivate people to change what they do.
BTW, stop eating potatoes! They are the ruin of the world! Did that motivate you?
LikeLike
Haha I haven’t eaten carbs in a year.
LikeLike
Convincing someone to read a book about losing weight or watch a movie about losing weight definitely passes the smell test btw.
LikeLike
Then what didn’t you like about what the data scientist said? Presumably, sharing information about books and movies is a natural mechanism by which social media could influence people to lose weight. And the people who write those books certainly think that wide adoption of their beliefs would have momentous effects on weight loss!
LikeLike
First of all, they don’t. And second of all, he didn’t say he was having data mining to see how people influence each other to read books (about losing weight). He said he is using data mining of social media to influence people to lose weight. Big diff!!
LikeLike
You know, Cathy, I think what’s going on is this — I default to assuming people are not idiots, so I have been imagining the most reasonable possible version of the argument the guy could have been making, and defending that; but what I now realize after some reflection is that if he’d been making that argument, you wouldn’t have gone off on him, and he probably was just an idiot delivering idiocy.
LikeLike
Just a reply to your challenge.:New Music.
It’s unlikely I would ever corner my nephew at a family gathering and ask what he’s got on his ipod, but thanks to Spotify, I already know. Most of it sucks, but some of it is pretty good. I’ve picked up a lot of good recommendations on pandora, which isn’t social networking in the same way, but it isn’t an echo of what’s happening in my life – the company I keep these days isn’t giving me any good recommendations.
In general there is a possibility for things to fail the echo test if the social network is quite a bit different. Have you visited patientslikeme.com? The idea is that you create a new network of people to whom you would never be connected in real life. This influence may be more effective.
LikeLike
I’m not convinced you understand my smell test. I’d say that music suggestions definitely pass the smell test, since I get music suggestions in real life too, and I influence people’s music tastes in person.
LikeLike
Yes, but you probably have a much more interesting real life social life than I do. My real life social network is extremely narrow compared to my online social network, which sees a much richer cross section.
LikeLike
OK, I see – Your smell test is that it has to work for someone, somewhere,at sometime in real life. Whether or not novel connections may have a more pronounced effect is a different question.
LikeLike
http://www.health.harvard.edu/blog/social-networks-can-affect-weight-happiness-201112163983
Quoting:
But if study participants had a friend who became obese, the chance the study participant would become obese rose by 57%. Sure, friends share some meals, but nothing like most meals.
Although scientists don’t fully understand how obesity spreads, they suspect that a social network influences what its members perceive as normal and acceptable. If people see their friends becoming heavier and heavier over time, they may accept weight gain as natural, even inevitable. Instead of exercising more or eating less when their weight begins to creep up, they may simply go with the flow and join the crowd.
Incidentally, there is evidence that the influence of social networks can also work in the other direction, and help people maintain a healthy weight.
LikeLike
“Suspect” is the operative word here. What they are seeing is something they can describe and maybe predict, but the ultimate test of causality is whether they can control it. They can’t.
LikeLike
Despite your belief about not being able to lose weight, it is possible. I have been a hanger-on with WW for 7 years. However, the quickest and most reliable way to lose weight is to just about eliminate sugar (substitute other, low-impact sweets) and reduce your starch intake. Gary Taubes is on the right track. I lost 55 pounds in six months in 1999, but was under pressures to abandon the low-carb life, so I gained it back. With WW, I got down 58 pounds in 2006-2007, although gained some of it back while we were dealing with my late mother’s end of life. I am back down to about 14 pounds from my goal weight by going no sugar and reduced starch. It can be done. See what Charlie Martin is doing. He started at about 300 pounds and is dropping weight. You can find him by searching.
LikeLike
Look, I’m not saying that nobody can lose weight. I’m saying, if it was just a choice one made, like reading a book or watching a movie, then it would be transferable to the realm of “influence”. But as it is, it’s a very individualized miniature miracle for those lucky people who manage it, and it’s definitely not something that data scientists can “solve” with a magic machine learning algorithm.
LikeLike
While I do appreciate the beauty of conservation of energy, I don’t think it deserves the “very individualized miniature miracle” label.
You like data, so just track your calories in and out, make an adjustment for metabolism (aka your body temperature) by tracking weight vs net calories for a little while, and then just tweak the in and out a little bit. If that doesn’t work, just figure out where it breaks and fix that. Given the popularity of all the fluff like carb/no-carb and the bazillion other diets, I would guess that most commonly people just can’t stick to it, making it a mental health issue, in which case I, personally, would consider professional help (assuming this aligns with my utility function).
For the record (to point out my potential biases), I don’t have weight issues, but have on occasion intentionally lost/gained weight for athletic performance reasons and I do monitor my weight pretty religiously.
LikeLike
I hope that last comment was a joke. Are you suggesting that 99.9% of the population is crazy? Harhar.
LikeLike
mental health issue != being crazy, just like regular health issue != being on the deathbed; not that “crazy” is a meaningful word in a mental health discussion
LikeLike
also, wtf @ the 99.9% – what exactly is that number supposed to represent?
LikeLike
Typically people mean “almost all people” when they say 99.9% of people. And that’s what I meant. It’s absolutely ridiculous to say that most people can’t stick to diets means they all must have mental health issues.
LikeLike
Ok, so seems like you meant 99.9% of people who can’t stick to their diets, and not 99.9% of the population. The later is what you wrote, but doesn’t seem like that’s what you meant.
It might be ridiculous to say that most people have mental health issues, but I don’t see the “absolute ridiculousness” of saying that the subset of people I highlighted (those who want to lose weight, have figured out necessary calorie intake, but can’t stick to that) do.
LikeLike
I am for any method that works and yes you are right on your post, you have to do it for yourself as LeBron may help for a day, he doesn’t hang out with you. Education is still the best and I have not had to deal with losing a lot of weight at one time, no more than 20 pounds let’s say but still I had to focus and work on it and do it for me. I refused to buy “fat” clothes so until I lost weight I was self punished with a very limited wardrobe. My weight was enough to where some things didn’t fit well or I couldn’t get the zipper up.
What does anger me though is when companies use the media to turn overweight people into evil twins when they blame our economy on the fact that too many people need to lose weight. If you lose weight as one person, you have to do it for yourself and you did not contribute to the economy problems, banks did it:) United Healthcare does that all the time with their studies and the language is horrible and they repeat it. Last time it was the sedentary people who were the evil twins. You can put put number crunches all one wants but you can leave off the twisting and taking out of context please. I called them on the last one. Now we get reports that say preventive care is a waste of money…oh well the other side with someone else and their numbers. Losing weight is a personal issue and yes having others egg you on that friends, social media people what ever your choice is is wonderful but I just hate with the media does with some of this as people start thinking that way after so many doses of the evil twin stuff.
I had to laugh at Governor Christie as he brought it to a head and admitted he needs to lose weight and has a sense of humor but when it got down to Barbara Walters asking him if his weight would stop him from running for President someday, enough. He doesn’t sit on his brain (grin).
Common sense has to enter here somewhere along the line and people have to be humans too and we need a balance and corporate USA sometimes tries to take that away from us and they do snow people with spun marketing. I keep that Charlie Siefe video on my blog when he talks about context. The video is great as he debunks some of the garbage out there, like there’s no “real” algorithm for the perfect butt. I like his book, Proofiness, the Dark Arts of Mathematical Deception and if not already maybe the two of you may cross paths one day.
Good stuff as always and your contributions are wonderful and bring back common sense with how to work with math and analytics as with anything else if you obsess, weird things happen:) 2 years ago I asked the question if data addiction and abuse was the next upcoming 12 step program in jest..anymore I still think it was humor but I don’t know myself:)
LikeLike
I am not an expert, but I can say the following with full confidence in the merits of the information. The most important issue about weight loss is, in fact, carbohydrates. No other form of food, is directly transferred to your body as FAT. Saturated fat is a blood system health management necessity. Not eating fat or other HDL laden foods, will keep your blood system from repairing itself. Then when you eat carbs, and too many at that, which most forms of “snacks” include, the conversion to fat by your body, causes the production of LDL cholesterol, and the tiny pellet kind at that. The result is that they imbed in the damaged tissue, and the next time you have some HDL laden food, the arterial system heals over that crud and creates narrowing.
The reason why diabetes and heart disease is on the upswing in recent times, is the commercialization of agriculture around carbs.
Again, go watch the movie “Fat Head”, and spread it amongst your family and friends. You will be amazed at what you don’t know.
LikeLike
corrected:
“The most important issue about weight loss is, in fact, *calories*.”
LikeLike
While calories are important, the types of calories are more important. If you eat carbs, they are going to be stored on your body as fat. If you eat meat and vegetables, which digest slower, they will be consumed by your calorie burning and less, if any will be stored on you body as fat. That’s what I’ve seen in countless, non-mainstream discussions.
http://www.three-peaks.net/annette/Together.htm
http://usatoday30.usatoday.com/news/health/story/2012-06-27/calories-low-carb-weight-loss/55843134/1
Certainly you need to pick a reasonable number of calories, per day, and high fiber carbs are fine. Potatoes have a high glycemic index and that’s the main issue for these types of carbs.
Carb choice will far and away make a big difference in your weight management and overall health.
LikeLike
Calorie “type” can only be important for side issues, like health. Energy conservation (and so your weight) unfortunately doesn’t care about your health. This is as elementary physics as it gets.
LikeLike
The subtle issue, is which calories in your body are accessed first. If you have calories in your blood stream, those are more accessible than calories stored as fat. The manner that carbohydrates are introduced into your blood stream, is a big deal.
Carbo loading by athletes is a great indicator of how easily calories in your blood stream (carbs go there directly), are used. When your blood stream is empty of spare calories, only then will you body start working on fat, and then you require huge amounts of water to aid in that conversion. People riding bikes or running long distances, crash late due to dehydration because they are starting to burn fat and are not drinking enough.
LikeLike
and here’s an example of energy conservation in action: http://www.cnn.com/2010/HEALTH/11/08/twinkie.diet.professor/index.html
LikeLike
Ed, most people don’t immediately go for a negative calorie intake, because that’s uncomfortable and more difficult to manage. The link you provide doesn’t show enough details of the total caloric content nor the fat, carb, fiber, sugar content to draw a conclusion about where the LDL and HDL improvements came from. Go watch the “Fat Head” video and you’ll see similar “that’s odd, I didn’t image that would work” moments there too.
My point is about speed of weight loss/gain. Carbs can provide a definitive indicator in that equation. Certainly conservation of energy is real. I’m not talking about “just” calories. I’m talking about total health, and in particular diabetes and heart disease. Carbs play big there, and things like the “Atkins Diet” are demonstrations of how to change your types of calories to create weight loss, without changing your total caloric intake, drastically.
LikeLike
We can argue about ease of things and health of diets, etc, but if you just focus on weight loss/gain, that’s irrelevant (assuming you don’t do smth radical like stop eating completely and die). It is also largely irrelevant for the speed of weight loss/gain.
And, yes, I agree that most of the time consumption of e.g. simple sugars (I really dislike your “carbs” generalization, as it promotes incorrect information) is bad for your health (though, I’d note that there are also times when it’s the best type of calorie to consume, e.g. during exercise), but that’s a separate conversation topic, largely unrelated to weight loss.
If you can do two things – understand data (not too hard) and track calories (pretty hard, mostly for mental and not for counting reasons, and is why the various prepared meals diets are so popular), then (slow) weight loss is very easy, as smth like decreasing your calorie intake by 5-10% requires very small lifestyle changes (the tracking your calories is a much larger lifestyle change).
LikeLike
But Ed, there is something that doesn’t happen with calorie reduction, which people want to see. What needs to happen, is that you need to burn fat to lose weight. If you have too many calories in your blood stream that are easily accessible, your body will burn those first. Reduction of carbs with high glycemic content like potatoes that have the biggest impact, creates a better chance for your blood system to support weight loss.
If you eat too few calories, overall, then your body can in fact burn all of the carb calories, and go after some of your fat too. But to lose fat the fastest, you need to focus your body on fat burning. The easiest way to do that, is to not provide your body with easy access to calories in your blood stream, and to instead cause your body to keep asking for fat to burn. Calorie from fat burning, in my experience (I am a bicycle rider) and learned knowledge from other sources, happens best, when you do a number of things differently, none of which involve, overall, a drastic reduction in calorie intake.
1) build muscles, they burn more calories, continuously.
2) don’t eat high glycemic carbs such as potatoes, white bread or sugar, more than once week.
3) get on your bike and ride 20 miles a couple of times a week (you should be planning to ride at 20mph in 10 miles in 30 mins and turn around and do rest for 5 mins and turn around and ride 10 miles in 25 mins.).
4) get on your bike and ride at least 40 miles on the weekend. Do this in 10, 20, 10 increments in no more than 2.5 hours. Rest 5 mins on the first stop and 5-10 mins on the second.
30 mins of bike riding at 20mph on flat terrain is going to burn around 200 calories, that’s all. It takes huge effort to burn carbs. 1000-1500 carb calories in a carb loading exercise will this let you ride for around 3hours before you get into a deficit and start burning fat.
LikeLike
See–this is exactly the debate I was talking about earlier in the comment thread. You’re both well-intentioned, both believe you’re educating the reader and both convinced you are right, but, in fact, one of you must be wrong.
From what little I know of this debate let me claim that “the answer is not known with convincing authority as of yet”. There are theories, but no final evidence either way. (I have my own viewpoint which is neither position stated, but I’ll keep it to myself)
So–what does this do to our modelling analysis? What impact does this have on measuring the effect of social pressures on weight management? How should we account for this additional fact in approaching the question?
Please limit answers to five blue book pages.
FoW
LikeLike
I have conservation of energy on my side. I’m pretty sure that has not failed to date.
There is no debate whatsoever that the cause of weight gain is positive energy imbalance, while of weight loss is negative energy imbalance. The only debate is about what actions humans should take to achieve the required imbalance.
LikeLike
Check this out re: conservation of energy:
http://www.economist.com/news/science-and-technology/21571119-having-wrong-gut-bacteria-can-cause-malnutrition-debugging-problem
If malnutrition is not about calories consumed, but rather the type of calories consumed, how can we be so sure about obesity? In fact there are documented cases of people with disrupted gut bacteria who are apparently thwarting the laws of thermodynamics… can’t find the study right now….
LikeLike
Wow, just how loudly can I say:
Q. E. D.
LikeLike
Do you also use have this attitude watching creationism/evolution discussions? Just use your brain instead of going “he said this, she said the opposite, therefore the answer is unknown”.
LikeLike
With all due respect, you should follow your own advice.
LikeLike
Just another confirmation of the conservation of energy – if your in-calories are less than your out-calories (because you don’t process them for whatever reason), you’ll lose weight. Nothing mysterious.
LikeLike
Can you freaking stop?
LikeLike
I agree with your smell test, but I don’t think it goes far enough. I often hear data scientists say they’re going to build a machine that predicts x or recommends y for a bunch of people.
My smell test is I imagine a human predictor that was exposed to all of the information that the machine has on a single individual and ask “could the predictor predict x or recommend y for that individual?” If no, then there’s no chance the machine can. For example human adjudication still out performs credit checks in predicting debt write offs.
Machines with data are great for providing almost human level advice to lots of people where it is not reasonable to provide an adviser for all of them. But when I a data scientist say they can build a machine that gives advice better than an individual with the same information I call hogwash.
LikeLike
As a social marketing professional, I assure all of you that @mathbabe is 100% right about how hard it is to change eating behaviors, and not just in social media.
As anyone (like me) who has worked on behavior modification efforts for Type II diabetes can tell you, there is no form of “messaging” that can significantly change behaviors that are programmed into the human brain, which definitely includes what you eat and how much. You couldn’t get someone off of heroin with peer pressure or social media either. The motivation must come from within, or from someone extremely close to you (like your kids) and that isn’t any kind of “media.”
Just to speculate, any social program would have to aim at nudging people just a little toward something really huge like a free weight-loss-theme-park-cruise-year-long-holiday, or something, that was the real motivator. You could also try to socially start a religion, because religions change people’s brain functions. Beyond that, take the poison foods off the shelves if you want people to lose weight.
LikeLike