Real World Big Data: Challenges and Opportunities
First blog focused on some of the major drivers behind Hadoop adoption in the Enterprise.
Many organizations that were previously pushing transformations to the database such as Teradata, have reached both the technical and the economic limitations of that approach. Many are doing what we call “Extract, Load and Transform” (ELT) in the database, something that vendors like Teradata have encouraged. As a result, these data warehouses have become overloaded and 60-70% of their capacity is being spent on ELT and not analytics. That is pushing our customers up against the architecture limits of legacy data warehouse, such as Netezza or Teradata. The result is a very real adoption of Big Data technologies, in particular Hadoop, to move beyond these economic and architectural limitations and offload expensive workloads to Hadoop.
The economics and cost is a big part this of course. The ability to use commodity hardware helped drive the adoption of the Hadoop as a platform, first by Internet companies and now by enterprises.
Second blog focused on the challenges in the adoption, skills gap and security being among many. The eco system is acknowledging these challenges and vendors are providing new and improved capabilities to increase adoption of Hadoop in the Enterprise. Hadoop 2 / YARN is certainly is evolving as a data processing platform, with a lot of enterprise strength features in the recent releases from Cloudera, Hortonworks and MapR.
As the technology matures, there is a lot of opportunity for start-ups to focus on optimizing the experience for the business user. Syncsort’s Hadoop products, not only provide improved performance due to its native integration and ability to move data in a parallel fashion within the cluster, but also help with these challenges and eliminate the need for manual coding with an easy to use graphical interface and light-weight deployments whether it is on premise or on cloud.