Last week I participated in a Big Data Insight panel at the monthly Technology Executive Networking Group (TENG) in Dallas and the discussion was lively among the 50+ attendees. What I learned from the discussion and the great questions from the audience is that there still seems to be a lot of confusion about what Big Data is and how different it is from “traditional” Data.
I started with an example that for me describes “what is Big Data”. Think about buying a vacation. You need hotel, rental car, airline ticket, theater tickets, etc. In the past you called travel agents, explained what you wanted and they assembled your vacation for you and you bought it. Today you do this in a completely new way. You search for places to go, then you check the weather pattern for that time of year, look at hotel ratings, shop for flights, bid on a rental car, and you do this over a period of time to check if prices change before you make your purchase online with a credit card. This is Big Data and this resonated very well with the audience.
Among my talking points about what Big Data is, I realized I was breaking some perceived myths about the definition of Big Data:
- Big Data is about taking large volumes of data and processing it in Hadoop. In reality many Big Data initiatives will not reach the data volumes that require processing in Hadoop. Smaller data sets extracted from multiple sources can be as game changing when used to answer business questions.
- Big Data is about performing analytics on data. Yes data exploration and analytics is a big part of Big Data initiatives, but it must be tied to performing an action based on the results. Too many analytic solutions leave you with a report or data visualization without giving you the ability to change a business process or create new records in another system based on the data.
- Big Data relates primarily to unstructured Data. Key new sources that became available like social data might be unstructured in nature but to make data actionable its needs to be as structured and factual as possible.
So, what is different from data as we used to know it and Big Data? One of the Panelists, Chris Boult, Vice President of Infrastructure & Data Services, Sabre Holdings, said it perfectly: “Traditionally we used to examine our transaction database to create business insight and derive new business initiatives. Now we look for the data that leads to the transaction data, for example how does people search and book airline tickets.” This is the essence of Big Data – leveraging data sets from internal and external sources to extract insights that you can act on.
Big Data is hard though. As data generating and data driven applications spread throughout companies and their business ecosystem, getting and distributing data has become the biggest integration nightmare ever. The explosion of data sources and the fact that the half-life of data (the period in which data has business value) has become shorter requires a different approach to dealing with data. An approach that enables organizations to:
- Quickly discover, acquire and combine data from internal and external sources
- Intuitively explore the data to figure out what to do with it
- Take data-driven action such as perform a business transaction or dynamically change prices based on competitors’ prices
- Create and package steps 1-3 into a smart business application that can be used directly by business consumers, and – this is important – do that in a matter of hours or days, rather than weeks or months
At the end of the session the panel was asked, “What is the biggest mistake you have seen around Big Data?” My clear answer was, “The biggest mistake it not to pursue Big Data”. Only by stepping over the line of how you used to work with data and get going with Big Data, will your company prosper.
A big thank you to BravoTECH and co-founder Andrew Jackson for hosting a successful event.