Sep 26, 2011

How has big data demonstrated a shift from data mining?

For the past couple of decades, large corporations and the government were focused on data mining. Why is there such a push into big data now, and who is leading that research?


Big Data deals with the flow of information, and having the ability to analyze this data in real time (or near-real-time). Data mining is more of the traditional searching the haystack for needles. Having faster access to the results of the analysis means big money for large corporations like Walmart and Boeing, so it's often commercial entities which are doing the research and development that leads their customers to migrate from the old methodologies to the new ones. Surprisingly, fresh approaches to solving the issues of big data comes from corporations like Sybase and not so much from universities or government institutions. This was one of the surprise findings at this year's ACM SIGKDD (Special interest group - Knowledge Discovery and Data Mining) meeting in August. This is a change from the years when universities would discover new approaches and then partner with commercial entities to make the fruits of their scientific endeavors available to the public writ-large.

Answer this