Dec 12, 2011

What is keeping big data from being effectively utilized by business in their decision making?

Only 1/3 of companies are making their business decisions by informed use of Big Data, according to an EMC Data Science Study. EMC asked 500 business intelligence decisions and data scientists from across the world about the state of big data in the business environment, and these individuals cited a number of obstacles to big data being leveraged to its potential, the most often noted being lack of skills/training, insufficient budgeting, and incompatible organizational structure. The problem of incompatible organizational structure is one that I can agree with; it can be so challenging to even discuss topics with "superiors" when their idea of cutting edge technology is something like an Apple Newton or a Palm Pre. I am only exaggerating slightly. Are we generationally limited in our adoption of new technology until the previous generation of executives age out and are replaced in their positions and replacement by younger, more tech savvy execs? Or are we still at the leading edge of a big data revolution, and more time is needed for businesses to determine how to effectively leverage the flood of information?


I think part of it is that we are just past the starting line of a new marathon with Big Data.  There is always going to be a time while the use of a new tool becomes second nature instead of a novelty.  A major obstacle to the effective use of big data in corporate planning is the way publicly held American corporations are essentially forced to plan in 3 month cycles.  Long term planning is often PUNISHED by the stock market, which primarily rewards meeting short term earning expectations on a quarterly basis.  Failure to do so, even when it is for long term benefit will more likely result in Kramer screaming about your company being trash on his TV show than a round of applause for solid strategic planning.  An example, although not one showing use of big data, is Amazon's stock hit when earnings decreased (but sales increased) this quarter, mainly because of the Kindle Fire.  Never mind that Amazon is creating a huge install base of a captive product that is tethered to for-profit Amazon content for years to come.  It is essentially the same thing with fully utilizing big data - a company is more likely to be punished for effective planning if the ROI takes more than 3 months to see than it is to be rewarded for good corporate governance. 

I think you hit the nail on the head with your comment about older executives. Seriously, how many of them even know what big data is? And how many have ANY clue how it might apply to their company's business? Precious few would be my guess.

If big data is going to be successful inside of companies like these, then it's going to require a serious education effort on the part of the few folks who truly understand it. They are going to need to reach out and teach the higher ups why big data is useful and what must be done to take advantage of it.

Frankly, I would not look for any sort of short term solution to this problem. I hate to say that but I think it's probably true. It will most likely take a new generation of younger executives with a more progressive outlook on big data to take full advantage of it.
Answer this