The senseless war between business and IT/data
Last night I attended a NYC Data Business Meetup at Bloomberg, organized by Matt Turck of Bloomberg Ventures.
There were four startups talking about their analytics-for-big-data products. Most of the audience was on the entrepreneurial side of big data, and not themselves data scientists. Of the people on stage, there were four entrepreneur/marketing people and one data scientist.
I noticed, during the Q&A part at the end, that there was a weird vibe in relation to IT/data teams versus business teams. Not everyone present was involved, to be clear, but rather a consistent thread of the conversation.
There was a conflict, we were told, between business and data, and the goal of these analytics platforms seemed to be, to a large extent, a way of bypassing the need for letting data people own the data. The idea was to expedite the “handoff” between the data/IT people and the business people, so that the business people could do rapid, iterative data investigations (without interference, presumably, from pesky data people).
The discussion even went so far as to describe the IT/data team as “territorial” with the data, and there was a short discussion as to how to create processes so that control of the data is clearly spelled out and is in the hands of the business, rather than the data people.
All this left we wondering if I am crazy to believe that, as a data scientist, I am also a business person.
Are we in a war that I didn’t know about? Is it a war between the business side and the data side of the business? And are these analytics platforms the space on which the war is waged? Are they either going to make data people obsolete, by making it unnecessary to hire data scientists, or are they going to make business analytics people obsolete, by allowing data scientists to quickly iterate models?
Are there really such lines drawn, and are they necessary?
Personally, I didn’t leave research in academia so that I could be an mere implementer of a “business person”‘s idea. I left so that I could be part of the decision-making process in an agile business, so that I can be part of the process that figures out what questions to ask, and moreover how to answer them, using my quantitative background.
I don’t think this war is a good idea – instead, we should strive toward creating a scenario in which data scientists and domain experts work together towards forming the question and investigating a solution.
To silo a data person is to undervalue them – indeed my best guess as to why some business people see data people as belligerent is that they’ve been undervaluing their data people, and that tends to make people belligerent.
And to give a business analyst a button on a screen which says “clustering algorithm” is to give them tools they can perhaps use but very probably can’t interpret. It’s in nobody’s interest to do this, and it’s certainly not in the interest of the ambient business.
From now on, if someone asks me if they should accept an offer as a data scientist, I’ll suggest they find out if the place is engaged in an “IT/data versus business” war, and if they are, to run away quickly. It’s a mindset that spells trouble.
mathbabe 
This is a new instantiation of the ubiquitous (quant/geek vs business person) schism of the Wall Street 80s and 90s that to some extent, but not entirely, has faded away with the rise of algorithmic trading.
LikeLike
and just like wall street, it is the IT side of house eventually win. but not in the sense of IT operation, but the data science IT.
LikeLike
Well said. In contrast, if the company has a Chief Data Officer that’s a quite good sign that they’re on board with data science as an important and integral part of the business that requires deep expertise. Due diligence during the interview process still required.
LikeLike
Here’s my two cents – It is not that they are territorial, IMO, I think they are afraid. Many times I have had IT balk at me for requesting access to their master database (what if I screw it up OMG) well, 1st off, this isn’t my first rodeo – I have been doing this for years – Didn’t matter, they were still freaked out! I think for so long, IT has been “responsible” for managing the data, that they are even afraid to give access to their “own” people – At my last company, some developers were never given access to Master DB – only TEST and STAGING / DEV – Hopefully, now that this field has gotten some well deserved attention, they will see people like you and me as a resource and not “a threat” – I really enjoy reading your blog Cathy :o) Thanks!
LikeLike
The IT folks are correct in “freaking out” about giving you or anyone else access to a Master DB -regardless of how adept and knowledgeable you are about working with that data. Master DB data is the blood of a plethora of decision support systems or systems that need the utmost integrity in their data.
What bother me is that the conversation very often stops there, without the IT folks realizing that you may be asking for access to the data for legitimate business reasons, be it because you want to “explore” such data or because you simply need to run some simulations of what-ifs scenarios, etc… More companies should have a data policy of allowing trusted, outside-of-IT stakeholders access to copies of master database data for legitimate reasons. All this can be implemented via a robust Change Management process within an ITIL framework. This way, more people closer to the stakeholder can become better educated about what their data can or cannot reveal and have a leverage axis from which to influence IT-related conversations with their IT managers.
LikeLike
These types of “wars” between IT and the business side are perennial. Years ago it was about who owned what application; today it is about analytics and big data; tomorrow it will be about something else. Part of the problem is that, for better or worse, IT people are always perceived as lacking the business skills necessary to make business decisions. Likewise, IT folks tend to see business types as lacking the insightful understanding of the technologies needed to arrive at a meaningful outcome. This continues to happen as more business types are getting more educated in IT and more IT types are getting MBAs and other business-related certifications.
In my opinion, company culture and organizational structure are the key here. But again, many books and articles have been written about how to align the two, with mixed results.
LikeLike
That’s my experience too. I was an IT/management consultant and researcher for 30 years and MOST of my clients showed tensions between IT and, err, non-IT business people and depts. Sad but true.
LikeLike
I think this is a growing pain of data science in that companies are trying to figure out where it’s proper place is in the organization. Historically, data people may have been in IT because the business’ systems generated the data and so the data belongs to the systems. Additionally, the IT folks possessed the skills to do things with the data that the business people could not because programming isn’t a “core” business skill. This seems to be changing, as data scientists now possess sort of a blend of IT, mathematics, and business skills that blurs the lines of what is IT and what is business. I think companies will sort all this out, but change isn’t easy for people and different people adapt to change at different speeds. Until then, I think we’ll continue to see this sort of thing.
Great post!
LikeLike
If big data is mission-critical for the business, then it will be a threat to current people in the business. Best case is they acknowledge this is a disruptive technology and they need to ride the wave or else be replaced. A worse case is they are insecure and try to control the technology and the tech people. In which case they talk about tech data-sharing (so they can get tech-savvy), they will likely not want to share business data with you (they don’t want you to get business-savvy).
LikeLike
Sad but true.
LikeLike
Data is property and its control is power within a corporation. In my experience IT and Market Research feel that they are the only ones that should see the data. they will distribute extensively editied summaries. Often the market research agencies hold the raw data and charge handsomly for each and every table.
It’s a great idea from their perspective, because then they feel that they are responsible for any inference that’s drawn from it, and would lose influence if someone else were to find something that they didn’t originate, and couldn’t explain. God help you then, because that’s when the knives come out.
So to answer your question: yes there is a “war”, and its been going on for a long time. (Spoken as a 30 year vet, as a statistician in research.)
LikeLike
I once met a market-oriented guy that founded his own business and whose master plan for the continuity of the company was… seting up all his daughters with nerdy IT guys.
Why his daughters agreed to the plan is out of my understanding but they were gorgeous. Do you know those women with such motor precision that seem like they walk in the air? Like Charlize Theron in Prometheus.
LikeLike
Am I the only one stunned by the juxtaposition of “gorgeous women” and “motor precision” outside of a car ad?
LikeLike
wait, what?
LikeLike
The mission of these companies should not be get the data directly to business people, but to get the data directly to anyone who wants to use the data, period. If a “business person” wants direct access to data, then by all means give it to them, and make it reasonably easy for them to process. But I wouldn’t necessarily expect all business people to want to write up their own statistics code and spend their time sorting through hard-to-understand results, even if they are very statistically savvy. I’m guessing what the presenters meant by business people are senior decision makers who possess broad control over strategic and product direction, client engagement, budgets, etc. Such folks, regardless of their background and skills, often need to hear the high-level takeaway from data analysis and don’t have time to dig into the fine points without suffering from information overload.
LikeLike
A lot of the conflict emerges from the “searching-for-gold” style of the any analytics type enterprise. Business people, besides their capital, are basically useless until “gold” is found. In my start-up we have 4 quants (including myself) and one guy from a private equity firm who is going to flip our product when it is operational.
From what I understand, such ratios are typical in well-functioning early stage start-ups that sell organized data. Once “gold” has been found however, the data scientist only really needs to maintain some code and must go prospecting again; now selling the data is more important.
Since both actors are redundant at different stages of the process their mutual animosity is to be expected and so we observe futile attempts to create tools for business analytics people to do simple quant work quickly (in a futile attempt to render “overpaid” data scientists redundant).
@Professor Derman: While the jocks vs. quants divide hits home for me, I think the phenomenon Cathy describes differs from banking in that bankers – quants or jocks – work all the time and often have strong economic incentives to cooperate closely if possible.
LikeLike
When I started working in management consulting, a lot of people warned me about the friction between business and IT (not to mention the friction between Consultants and IT). I’ve seen many co-workers and clients run into issues meeting deadlines and completing projects as a result of the conflict with the IT department.
That being said, I’ve spearheaded large scale implementations at several well known companies in NYC and I’ve only had positive experiences collaborating with the IT department at the client sites.
Personally, I consider myself a business person and as a data scientist, I don’t think I would have had such success leading the Quantitative Analysis Committee in Rachel’s class without my business background.
LikeLike