What are the Challenges Related to Hadoop Adoption? (And, How Can You Overcome?)
For every article you read about the advantages and potential associated with Hadoop adoption, you read one warning of the downsides. The lack of Hadoop talent, the difficulty in migrating data from the mainframe to Hadoop, the security problems, and of course, the potential for costs to spiral out of control. Yet many businesses are doing it, and doing so with stellar success. In fact, the latest reports prove that companies leveraging big data are indeed outpacing their competitors, especially in the realms of marketing and operational intelligence.
So, what exactly should you expect to face during a Hadoop migration, and how can you overcome those challenges for ultimate success (without busting the budget)?
Overcoming Migration and Integration Issues
The right Hadoop vendor and products won’t leave you hanging off a cliff.
Hadoop doesn’t use the same computer languages as the mainframe (and other infrastructures). Plus, data isn’t in the right format for a simple transfer. The offload process and loading to Hadoop requires the skills of both a mainframer (or someone familiar with your architecture) plus someone skilled in Hadoop. This combination is rarely found in the same individual, though most any skilled programmer can pick up on Hadoop with some time and practice.
Plan to contend with the filtering, aggregation, and language rationalization, as well as FTP transfer and loading to Hadoop. Fortunately, there are a variety of useful tools and products available to make this process go smoother, easier, and faster.
Overcoming the Skills Barrier
Don’t have the right skills to take on Hadoop? Develop your own Hadoop ninja from your current base of exceptional employees.
As mentioned, you’ll either have to get someone with the Hadoop skills you need or give your in-house programmers the training and time they need to develop those skills. Aside from the programming aspect, you’ll also need a good data scientist at your disposal, or the data won’t do much for you once it’s in Hadoop. The news will try to terrify you that such a data scientist is more rare than finding a million dollars floating down the gutter in front of your home. But good talent is often grown, not found. Many businesses are developing their own in-house talent and making quite fine data scientists out of them.
Overcoming the Cost Barrier
As a free, open source solution, Hadoop sure takes a lot of flack for being expensive. Indeed, if you aren’t careful about issues like data storage, hiring outside contractors, and the tools you choose, you can run up an impressive tab trying to get a Hadoop operation up and off the ground. But carefully choosing the right Hadoop products, starting slow and allowing the initiative to grow, and managing these costs ahead of time will allow you to keep those costs under control.
Overcoming the Security Barrier
Finally, you’ll need to address the security issues that comes with any means of storing and transferring data. Hadoop security has come a long way recently, even with marked improvements during this year. Just as with any plan for storing and processing data, you’ll need to use solid encryption, carefully screen your third-party vendors, and select the right security tools and solutions.
The bottom line: Hadoop is doable, and offers a great deal of reward for those willing to undertake it. Be aware of the challenges, make plans to overcome, choose the right tools and solutions, and your Hadoop project will ultimately be a success.