Thomas C. Redman and Bill Sweeney have provided 7 questions to ask our "data geeks" on the HBR Blog. We have included the first four below, you can catch them all in this article though.
Using data to manage is nothing new. But using big data to manage IS new, offering unprecedented challenges, opportunity, and risk. Senior executives need to learn and learn quickly. They must be prepared to ask penetrating questions when their data scientists bring them a new idea. Those who ask the following questions will be better prepared to both exploit the valuable insights and avoid the disasters that can arise from big — but bad — data.
When your data scientists bring you an idea, ask them the following:
1. What problem are you trying to solve? It is far too easy for data scientists (and others for that matter) to go on extended "fishing expeditions," seeking "interesting insights" that aren't tethered to the business. While a certain amount of exploration is healthy, most innovation is of the small-scale, one improvement at a time, variety — even with data. Encourage your data scientists to focus initially on known issues and opportunities, as well as more tangible insights. As confidence grows, your data scientists should be less restrained. And you should develop a keen eye for the difference between "exploring a difficult path" and "wallowing around."
2. Do you have a deep understanding of what the data really mean? Too often people gather data without complete understanding of the wider context in which the data were created, and misunderstandings find ways to hide themselves until it is too late. All data, even well-known quantities like "force" are subtle and nuanced. NASA (which truly has "Rocket Scientists") crashed a Mars lander because one team used the English measurement "foot pounds " and another used the Metric measurement "Newtons." The potential for such problems only grows the less familiar the data — especially through social media, automatic measurement devices, etc. — and the more intermediaries that touch the data.
3. Should we trust the data? Untrustworthy, inaccurate data is all too common. Just as a car can be no better than its parts, so too analytics can be no better than the data. Some data is inherently inaccurate (GDP forecasts); other data becomes inaccurate through processing errors. All too often, data collection is just not up to snuff. For example, far too many credit reports contain inaccuracies. Unless there is a solid quality program in place, expect the data to be poor!
4. Are there "big factors," preconceived notions, hidden assumptions, or conflicting data that could compromise your analyses? There is a lot going on here. First, it's natural to expect a return from our investment in data and analytics, but there's a sneaky side effect. People will "find" what they think you want. Saying upfront that you expect a 10% uptick in revenue can cause people to find a short-term 10% growth that's not there for the long-term, to be so busy looking for the 10% that they'll miss a potential 100% gain, or miss negative correlations entirely.
Second, advanced data analytics involves considerable judgment. Data scientists may have included some data sets and excluded others, from their analyses. You need to make sure they've not done so in unfair ways. The clarity and completeness of the answer correlates with the weight you should give to their conclusions.
Third, analytics is essentially about developing a deeper understanding of how the world works. False assumptions are crippling. For example, the assumption that home prices were uncorrelated across markets was a major contributor to the financial crisis.