finding the middle path

, and November 25, 2016

IT and intuition work best when placed  next to each other

Rajesh Maurya


In a new digital age of ‘big data’ and quantitative analytics, does a manager’s intuition matter anymore? Big data is an umbrella term for solutions which were earlier referred to as data storage, analytics, prediction, and optimization. It has gained popularity today because of the volume, speed, and variety of data being generated from mobile devices and IoT. Organizations are evolving and responding to changes in the IT environment as new services and applications improve the market. Lost in all this hype is a key factor—the power of intuition.


trust your instincts

Relying on intuition is a practice that has driven business leaders to success. Intuition and experiential learning empower you to judge the competitive landscape, and understand what to look for in a product, what matters to customers, and what to do with the data. There is an interesting theory that intuition is data—that human intuition is processed in the ultimate black box algorithm.


big data and intuition not to be seen in silos

Surely, IT and intuition work best when placed not against, but next to each other. Intuition comes into play when managers speak from experience—insights an organization would miss if it spent its time staring at numbers. Organizations must continue to invest in big data and base crucial decisions on it. Meanwhile, the management can ensure IT and business executives are a part of the analytics process right from the beginning. Otherwise, there is a risk of useful insights staying hidden. The machines do not have all the answers, nor do the executives. Management can avoid arguments between numbers and experience by creating an analytical ecosystem which includes all the involved members.


finding a middle path

The management should understand the strengths and weaknesses of man and machine. Firstly, identify the exclusive benefits each one offers and how the other one can take it forward. For this, you must also understand the limitations each one comes with.  For instance, it is a known fact that algorithms work as tools, not as an end-to-end solution. Data science is not sophisticated enough to predict whether a product will be successful. Ultimately, the decision is left to instinct on how consumers will react to a new idea. To ask the right questions, decision-makers must have a intuitive understanding of the organization and its objectives. The trick lies in knowing when to dive into data. It requires an understanding of when and how to incorporate managerial instincts into data-driven decision-making. For this reason, even data-driven companies should cultivate intuition.


but you need a little security for big data

Organizations are increasingly relying on analytics to make real-time decisions. Unlike intuition, even small changes in big data can have a big impact. This brings to fore the importance of safeguarding it by determining data confidentiality levels, classifying sensitive data, deciding where critical data is to be located, and establishing secure access models for both the data and analysis.


who wins?

After all, “We start with the data, but the final call is always gut,” as Netflix’s CEO Reed Hastings said. Thus, the days of picking between big data and intuition have been replaced by the awareness of its merged benefits.



harness their combined power

Faisal Hoque

For now anyway, intuition is alive and well in the era of data analytics. Only about a third of executives in a 2014 PwC survey reported relying first and foremost on the data for their last big business decision. For the rest, the advice of their colleagues and good,
old-fashioned gut instinct both played major roles.

It is no wonder, is it? Leaders tend to get where they are at least partly on the basis of their good judgment, which has guided them to success time and again. These days, with a deluge of new data insights closer to hand than ever before, wading through them all can seem overwhelming or even distracting. We are still in the early days of big data—and of data science in business writ large—so it is probably not a surprise that subjective measures still play a predominant role.

That does not need to be a bad thing, but sometimes it is. Here is what you need to know in order to keep your gut instinct and the cold, hard numbers in good balance.


will intuition ever be obsolete?

It is worth asking whether we will eventually be able to remove gut instinct and intuition from our decision-making process altogether, even if we wanted to.

Netflix founder Reed Hastings, who worked as an artificial intelligence engineer before starting the video company, cautions that the human element is still critical. “We start with the data,” he recently told VentureBeat, “but the final call is always gut. It’s informed intuition.”

While experts have found that intuition can be powerful, it can often lead us astray. In general, even the most experienced decision-makers among us are often quite irrational, making choices that are not in our best interests, acting on impulse, and blind to unseen biases.

Success comes from connecting the dots between our emotional selves and systematic thinking that can be checked quantitatively. It is all about how you combine data, predictive analysis, and your time-honed intuition.


intuition helps us dig deeper

Getting that balance right starts with understanding which types of real-world situations demand more or less subjective versus quantitative evaluation.

In 2012, researchers Erik Dane of Rice University and Kevin W Rockmann of George Mason University found that people mostly rely on intuition when making broad evaluations, particularly in areas where they already have indepth knowledge of the subject—often called ‘domain expertise’.

Today, organizations may have huge amounts of data to analyze, but the challenge is to define what they can reasonably expect from any effort of data analysis. And intuition still comes into play when defining which questions to ask in order to churn out actionable insights. Here are a few:

  • what market to pursue
  • what products and services to develop
  • what business model to adopt
  • what partnerships to pursue

It is judgment and experience—often in the shape of gut instinct—that should and do cause business leaders to raise those questions at the right times. But it is data analysis that can help answer them in their particulars, like these:

  • market segmentation
  • customer and product portability
  • customer retention strategies
  • operations and performance management
  • resource requirements

To be sure, this is an intentionally broad and far-from-comprehensive breakdown. But distinguishing between the strategic and tactical questions can sometimes still be a challenge; up until quite recently, both were subject to basically equal doses of intuition. And while market research has long been a staple, it is only lately that data analysis has come to reshape how businesses approach the latter type of insight.


validation is mandatory

Today, a massive amount of data is being collected in real time, which makes the ‘human with machine’ (instead of ‘human versus machine’) formulation more important to get right. Algorithms and systems can process information more efficiently and accurately than ever, which significantly increases that information’s business relevance.

Where past decisions were based on historical precedent, like customer satisfaction surveys and sales patterns, machines can now create predictive models, patterns, and trends that incorporate what-if scenarios. Here is how author of Big Data: Using Smart Big Data, Analytics and Metrics To Make Better Decisions and Improve Performance Bernard Marr recently summed up that shift  on LinkedIn:

“U.S. retailer Target is now able to very accurately predict when one of their customers will expect a baby. Using big data, telecom companies can now better predict customer churn; Walmart can predict what products will sell, and car insurance companies understand how well their customers actually drive.”

But that hints at a reverse imperative too—which is easier to miss. Just as we can now check our intuition with more reliable metrics than ever, we also need to subject our predictive models—what data tells us ‘might’ happen—against our best-informed hunches. Information on what ‘is’ happening, or already has, may be a sure-fire baseline to square our instincts against, but when it comes to the future, data may be able to guide us ahead even though intuition (so far, at least) still largely leads the way.

It will not always be this way, of course. As data innovation speeds forward, the ways we combine our instincts with the numbers will continue to evolve.

But as they do, it is unlikely that the quantitative and the subjective realms will ever be totally mutually exclusive. The challenge and opportunity for organizations is—and will continue to be—how best to harness their combined power. It is a science, sure, but it is a human one.


2 3 4