Removing Barriers for Hadoop Adoption
A recent post, ” What are the Challenges Related to Hadoop Adoption? (And, How Can You Overcome?)” mentioned several barriers to Hadoop Adoption, and briefly mentioned ways to overcome those barriers. Since you now know what challenges you face, let’s take a little deeper look into overcoming those barriers so that your Hadoop initiatives (from the very beginning) will be successful.
Removing Migration and Integration Barriers
Though each IT environment is unique, there are solutions available to migrate data from most any source into Hadoop.
Even companies sold on the idea of Hadoop get baffled by exactly how to get all their data off the mainframe (or other infrastructure) and into Hadoop clusters. Granted, without the right tools, this one is tricky. However, there is an entire line of proven products to help offload from any infrastructure into Hadoop. These data transfer products allow you to offload from the mainframe, a data warehouse, etc. You can even try it for free before making a decision.
Removing the Skills Barrier
Can’t find anyone qualified to come take over your Hadoop operations? Grow your own talent.
News of the looming tech labor shortage, particularly when it comes to big data science and analysis, has companies scared to delve into Hadoop. For sure, it takes skills that most programmers or systems admins don’t currently have. But hiring an expensive (think $200,000 per year and up) talent isn’t your only option. Home-growing this talent is not only possible, but could be preferable over hiring from the outside.
Most experienced programmers can pick up on Hadoop skills within a few months. In fact, the time it takes them to become proficient in Hadoop is roughly the length of time it will take your company to test Hadoop and see if it’s viable in your organization. Give your programmers the time and tools to learn Hadoop on their own, and assign a Hadoop test operation for them to undertake.
This process might seem slow, but the benefits are that it keeps your adoption costs at a minimum, and you end up with a team that not only has the Hadoop skills you need, but also has an intimate knowledge of your business and IT department, which you’ll never obtain by hiring from the outside. Just be sure you’re paying them well and keeping them happy. You don’t want to become a training ground for your workers to springboard into better jobs elsewhere.
Removing the Cost Barrier
The cost of undertaking this “free” software can rack up quickly and steeply. But there are ways to manage these costs and end up with a Hadoop infrastructure that actually is able to pay for itself. One common mistake is to try to buy enough servers to manage the peak workloads, since resource allocation is a big issue with Hadoop. Yet this usually means you’re stuck with an enormous set of expensive hardware that sits idly most of the time. A better option is to invest only in the amount of hardware needed for normal operations, and manage the workloads better. another option is to migrate your Hadoop operations to the cloud, leasing only the space and computing power you need.
Removing the Security Barriers
Hadoop does offer security and access controls, but these lack the enterprise-grade security measures most businesses need today. There are several options for improving the security of Hadoop, and a multi-layer approach that includes several or all of these measures is ideal. Firewalls are essential, but again won’t be enough. Not only are you protecting the data you use for Hadoop analysis, but the analysis produces even more sets of data, which is usually more valuable anyway. Encrypt data at the records level when it is in storage or in transit. It’s also a good idea not to even offload data that isn’t necessary, because it introduces a risk without providing any reward.