A Dispatch from the Hadoop Summit
And we’re back to San Jose for another exciting week at the Hadoop Summit. Seems like we have the biggest Summit so far with over 3,200 attendees representing 1,000 different organizations. And among all the excitement, there seems to be a couple of very common themes that I’ll explore here.
First, to quote one of the keynote speakers, “Hadoop’s momentum is unstoppable”. Hadoop is moving very quickly from contained departmental solutions to enterprise-wide architectures. Yes, Hadoop and Big Data are happening now. Merv Adrian, Gartner analyst and also keynote speaker at the Summit, illustrated this point with stats from the Gartner website, where users submit over 1,000 searches per month on the terms “Big Data + Hadoop.” Another more tangible proof point is the fact that traditional organizations are adopting this technology. AT&T, for instance, collects and analyzes over 30 billion data points per hour with Hadoop, to monitor the customer experience across its cellular network. A leading healthcare organization is reducing costs by offloading ELT workloads from the data warehouse into Hadoop. The cool thing about what these companies are doing is that, in this process, organizations are shifting the focus from the technologies – Hadoop, Big Data – to innovation itself. And that is the biggest proof that Hadoop is being quickly assimilated. More importantly, these organizations are becoming active participants and contributors to a new data-driven economy.
The second theme is probably a consequence of the first. As more organizations adopt Hadoop as the standard, they are demanding key capabilities characteristic of enterprise-caliber platforms. One of them is the ability to leverage existing skills within their teams. For instance, many IT teams have dozens of ETL developers that recognize the value of Hadoop, but they don’t necessarily have the skills they need – Hive, Pig, Java – to develop ETL jobs in Hadoop. Another key area is security, something that will only become more important as financial, healthcare, telco and other organizations who manage very sensitive data make the leap into Hadoop.
As always, you can feel the excitement and if you listen closely, you’ll hear lots of different themes and interesting projects revolving around Hadoop. Hadoop 2.0 and the emergence of YARN as the framework that is transforming Hadoop to enable adoption of broader use cases is definitely one of them. Another that really caught my attention Docker, which when combined with Hadoop could bring an even more disruptive force to the enterprise.
In the end, the message is clear, borrowing Merv Adrian’s words: “If you are still waiting for Hadoop, it’s time to stop waiting!” And this week, Hadoop is happening at the Hadoop Summit!