Almost every day, I see articles, software solutions, skeptic criticisms, futurist nightmares and more - all extolling the opportunities of Big Data. If you haven't been reading up on it, you will be soon. Big Data is everywhere.
If you are interested in managing the huge amount of data created both within your enterprise and the unstructured data from sources such as social media, then the place to start is within your IT Organization.
Big Data Has 3 Elements of Data: Volume, Velocity, and Variety.
One Big Data Q&A session that I recently watched noted that companies "must fully exploit all data and content by leveraging an enterprise-class Big Data platform as part of a comprehensive information management foundation." Katharyn White, Vice President Marketing, IBM Global Business Services. Hmm, I guess that Big Blue has a software solution for managing Big Data, or should I say managing Big Blue Data.
In the March 2012 report, The Big Returns from Big Data, from Nucleus Research, the conclusion says it all:
Although Big Data may seem overhyped, technology buyers should set aside their skepticism and consider making investments that enable the analysis of large and complex data sets. When analytics capabilities are applied to large data sets, be they on the enterprise, the Web, or the partner ecosystem, employees become capable of insights they can't make by examining traditional data sources. The scouring of the Web for important shifts in customer sentiment, the use of acquired credit-ratings data for loan portfolio improvement, and the analysis of warranty databases for the detection of potential product failures are all examples of benefits from Big Data that have significant bottom-line impact, yet are unavailable from less mature analytics deployments.
So, there is definitely a need to manage Big Data. The next area of Big Data being talked about out there in the Twitterverse and beyond, that I want to mention is the predictions made from Big Data. In the post by Patrick Gray, Big Data: The Perils of Past Performance, he notes that Big Data, although almost instantaneous, is still looking back, trying to predict the future.
It's easy to assume there's a "right" answer to Big Data and to put an unflinching faith in the ability of the system to do what amounts to predicting the future.
Big Data can also put your focus in the wrong place. Most of the events that shape the future are external: a competitor may enter a market or launch a new product, an economic or social calamity may reshape markets, or anything from a war, demographic shift, or alien sighting could change the future overnight. Tools like Big Data focus on past performance. Even with Big Data's promise of near real-time analysis, you're still working with "old" information, even if it is only microseconds old.
Of course, all of this talk about Big Data naturally leads to the alarming scenario found in Facebook's Gen Y Nightmare, edited by Frederic Filloux. In the not so distant future a recent law school grad's resume has been automatically sorted to the top of the pile and is being reviewed by actual humans from Human Resources. Do they still have real humans in HR today? Well, this imagined future still has HR personnel?
So Alan, the head of HR says, "you can't take Ms Porter in the firm. Our Predictive Workforce Expenditure Model shows that she will cost you at least 15% more in lost productivity." OK, so there are some data models used to estimate risks and attempt to mitigate it though analysis of Big Data. Nobody said Big Data was nice.
However you see the future of Big Data within your organization, you need to start with a solid foundation. So, make sure your IT Organization is built on an ITIL out-of-the-box ready solution because Big Data is coming to your enterprise soon.