From Big Data to Skynet – Takeaways from the Gartner BI Summit
Earlier this week, I spent three great days attending Gartner’s Business Intelligence (BI) Summit in Los Angeles. All the usual suspects presented on a variety of topics ranging from Big Data to cloud computing to mobile. However, this year felt a bit different to me. There seemed to be a realization that BI (and related) technologies alone do not represent the road to perfection and information nirvana.
Instead, what I observed was organizations being careful about how they leverage the hot trends of the day. They seem to recognize that they must carefully watch the technology evolution, understand the associated risks and opportunities, and only then determine how to incorporate them into plans for supporting the business.
Here are some personal takeaways from the conference, in no particular order:
- Big Data means big noise. As companies start to analyze Big Data, the amount of noise grows exponentially. In fact, some studies estimate the noise level to be greater than 70%. Therefore, the challenge becomes how to efficiently and effectively process all the data while filtering out the noise. As Gartner analysts mentioned, organizations need to be very careful not to add bad data in their quest to leverage Big Data. I agree.
- Information is about connecting the dots. Once we’ve filtered out the noise, we have to connect the dots. Raw data by itself has marginal value. Connecting the dots enables us to convert data into information, adding tremendous amount of value along the way. For instance, having comprehensive data about suspected terrorists has nearly no value if we can’t intelligently connect the dots to unmask their network and predict the next move. Data Integration plays a key role as the first line of defense not only to integrate the myriad of sources of information, but also to do so in a timely fashion.
- The decision environment has evolved. Instead of only the strategic aspect, the decision environment now also includes the management and operational aspects. This results in new requirements in terms of velocity, variety and volumes of data. For instance, operational workers need near real-time data at the lowest level of detail while strategists may look at weekly, monthly, even yearly trends of aggregated data. No wonder Gartner predicts that by 2014, most organizations will not scale to meet the requirements of Big Data! Now think about what happens if you have underperforming data integration tools pushing transformations down to the database. Performance clearly has a huge impact across the entire organization.
- Balancing resources is a daunting, but critical task. Fortunately, this is not the case for Syncsort. At the event, Mark Beyer did a great job highlighting this challenge. In the era of Big Data, it’s more important than ever to balance resource utilization – that is CPU, memory, storage and I/O. Workloads are competing for all these resources and the variables are not static. Organizations have different workloads on different months, weeks, and days of the year. IT organizations are being forced to either “oversize” their systems or leave constant tuning/optimization cycles while living in constant fear of failure at “rush hour.” This is why having a highly scalable, self-tuning engine like DMExpress is so powerful.
- Volume is (not) a “20 mules problem.” This is a funny yet interesting analogy. Basically, it means that you could just throw more cores (mules) to parallelize a given data processing job and get it done. Of course, it’s not that simple. I would argue that with more mules also come more issues such as what to feed the mules, how to house them, clean them, keep them healthy, etc. Therefore, you might as well keep the number of mules (or cores) to a minimum! Again, this is another area where DMExpress is highly differentiated in the marketplace.
I’ve been working on shortening my blog posts. It seems that “attention span thing” always gets in the way of everything I want to share! However, if you’ve made it this far, surely you are wondering what Skynet has to do with any of this.
Well, for the first time at a BI conference (and I’ve been to many of them), I observed a subtle, but legitimate concern among attendees about how information – and more specifically algorithms and automated decision making – are shaping our lives and culture. To paraphrase Kevin Slavin, algorithms are shaping the way we live, what we read, what we write, what we consume.
Don’t believe me? Think about the last time you bought a product on Amazon, selected a movie from Netflix, or found a business through Google search.
What do you think? If you attended Gartner BI Summit, what were your takeaways? How is Big Data impacting you? Let’s keep the discussion going.