Friend or Foe: A Tale of Big Data and Data Integration (Part 2 of 2)
This is the second and last part of the “Friend or Foe” saga. For those who missed the first part, I focused on the opportunities and challenges presented by Big Data, including highlights from a recent study released last month by the McKinsey Global Institute (MGI).
Now it’s time to turn our attention, using a similar lens, to look specifically at data integration tools. Data integration tools are supposed to help IT implement their data strategies, which in turn should help organizations achieve strategic objectives such as increasing revenue, widening profit margins, improving value to customers, remaining competitive, etc.
As we explored in Part 1, this is especially important in light of Big Data. Therefore, a first step towards building a sound Big Data strategy is to look at the effectiveness of data integration tools today. In others words, are they successfully transforming data into value?
Well, apparently today’s data integration tools are not as effective as one would think for a relatively mature market. In fact, recent independent research from BeyeNETWORK (full disclosure that the study was commissioned by Syncsort), with more than 350 IT participants from US and Canada, shows data integration tools are generally failing to deliver on their promise. Key findings of the research include:
- 68% of respondents agree data integration tools are impeding (i.e. not helping) the organization’s ability to achieve strategic business objectives. This is probably the most striking finding of the survey, signaling a big problem for organizations that even today find it challenging to leverage their data, running the risk of falling behind the competitive race.
- Growing data and shrinking batch windows, pose significant challenges to over 70% of participants. This argument points to 2 key dimensions of Big Data: growing data volumes and increased velocity. As such, this statement is a clear indication that Big Data is already here and is already a challenge for the majority of organizations.
- 39% consider Total Cost of Ownership as the number one issue with their data integration tools. This seems to hit at the core of the problem. As data volumes grow and batch windows shrink, organizations turn to expensive and inefficient workarounds. This is how we end up with exponential costs, reduced business agility, and ultimately, failure to support the strategic objectives of the business.
Not surprisingly, data integration tools have become a foe for many IT organizations. However, it does not need to be this way. Data integration can be a friend that helps organizations cope with, and generate significant value, from Big Data.
For starters, organizations need to rethink their strategies. Traditional ETL tools struggle to cope with the demands of Big Data and organizations can no longer turn to inefficient workarounds. In fact, only 31% of respondents say transformations happen in their ETL engine; over 55% indicate their tools require significant tuning to achieve suitable performance; more than 45% state it takes them between 1 week and 1 month to develop reports with new data. Need more evidence than that to believe that today’s data integration tools are failing?
Clearly, a new approach is needed. One that delivers on the long overdue promises of ETL and data integration: fast, efficient, simple, cost-effective data integration, with no tuning required. It is only by reducing total cost of ownership through a lean, scalable data integration layer, that organizations will be able to leverage Big Data to its full potential. Only then, can your organization be one of the few to profit from the immense (Big Data) opportunities identified on the McKinsey report.
You can find the complete survey from BeyeNETWORK here. And while you read it, you may want to think about your strategy to turn Big Data and data integration into your best friend in order to compete and succeed over the next decade and beyond.