Big Data Q&A with ESG’s Julie Lockner
It has been a great week at Syncsort’s 2012 Global Sales Kickoff in Newport Beach, CA. As you might expect, there has been a lot of discussion about Big Data, Hadoop, ETL, data analytics, business intelligence, and more.
In addition to some great networking with colleagues from around the world, we have been fortunate to have several of our partners join us. During the event, we also announced the winners of our Partner of the Year Awards program.
However, the presentation that had everyone excited and talking was from Enterprise Strategy Group’s Julie Lockner. We’ve interacted regularly with Julie since she joined the ESG team a little more than a year ago, and find her to be really in touch with the end user community and the challenges they face. I had the chance to chat with Julie for a bit after her presentation and wanted to share a portion of the conversation. Thanks again to Julie for participating in our event!
KK: When you speak with end users, what do they tell you about the challenges they face with data integration?
JL: What is interesting is that everyone I speak with regarding data integration state that they have challenges. Aside from the common challenges associated data quality and data consistency, most talk how the development team has challenges responding to new requirements in a timely fashion. More data, shorter windows, and new applications to be integrated all contribute to the challenges.
KK: What do you see organizations doing to overcome data integration challenges?
JL: Unfortunately, according to recent ESG research, many are still investing in more hardware and software to address the performance challenges. What is interesting, however, is more than 1/5th of the research respondents are using technology classified as data integration accelerators to complement their ETL strategies.
KK: Are you hearing more about data integration accelerators?
JL: As I mentioned in your previous question, this is absolutely becoming a viable option. Data integration accelerators can play an important role in data transformations – especially when you consider that 80% of the tasks involve simple to complex data flows with sorts being a large part of what consumes hardware and compute resources. Organizations I speak with that are using solutions such as DMExpress love the fact that it is fairly straightforward to implement and with little effort. They see significant performance improvements in their ETL jobs.
KK: What are you seeing in terms of Hadoop and the profiles of early adopters?
JL: Hadoop has a lot of promise. There is a lot of excitement because it is new and with that comes hype. What I am seeing, and what is being confirmed by ESG research, is that many are still trying to figure out how to leverage Hadoop in their environment. This year seems to be the year of experimentation with Hadoop. Those that are using Hadoop had an obvious need for something like Hadoop to deal with a seriously big data challenge – and Hadoop made economical sense. The time and effort associated with coming up to speed with Hadoop and deploying a dedicated compute cluster needs to be vetted out in a business case before taking the plunge. In the meantime, I like the idea that there are a lot of “skunks work” projects out there. This will give the technical team a head start when the business case arrives at their door.
KK: Syncsort is based in New Jersey (home of the New York Giants), but has a number of employees (like myself) that either live in New England or have roots there. Do you want to go on the record with your Super Bowl prediction?
JL: I would love to see the New England Patriots win the Super Bowl this year. With these two great teams going head to head, it will be a great game.