Data infrastructure optimization, availability & security software
Data integration & quality software
The Next Wave of technology & innovation

Solutions for Offloading From Mainframe to Hadoop

Looking to initiate a big data project or begin using big data for operational intelligence, business intelligence, or marketing endeavors? If so, you’re in the majority. While mainframes remain a solid solution for storing and running business processes, Hadoop is the go-to solution for big data. This post explains why Hadoop is a great solution, what challenges you can expect to face, and how to overcome these challenges for big data success.

The Advantages of Offloading Mainframe Data Into Hadoop

data integration
When it comes to handling big data, Hadoop is putting the foot on the competition.

First, why would you even need to bother offloading data from the mainframe to Hadoop?

  • Mainframes are great with structured data, but aren’t so good when it comes to managing unstructured data. In the era of big data, Hadoop is ideally suited to produce the data analytics necessary to leverage the insight, predictions, and intelligence that Hadoop can derive from big data.
  • When you move data from the mainframe to Hadoop, you won’t necessarily need all of the software you’ve been keeping on the mainframe. This can reduce the overhead you’re paying for those software licenses (not to mention free up needed space on the mainframe).
  • The primary cost associated with the mainframe is purchasing additional memory, power, or computers to keep up with growing data sets and workloads. By moving your big data operations to Hadoop, you eliminate the need to continually upgrade the mainframe, lowering growth costs substantially.
  • Hadoop is open source, meaning that it’s a low-cost startup and has lots of support from communities and third-party vendors. Most businesses that choose to undertake big data initiatives select Hadoop due to this affordability and support.

The Challenges You’ll Face Offloading Mainframe Data Into Hadoop

Though Hadoop is a great solution, it is by no means a walk in the park. There are challenges, but as you’ll see, each problem indeed has a solution.

  • Challenges with Integration – The data has to be filtered, aggregated, and put into a language which Hadoop can understand. The code (generally COBOL) has to be rationalized. Then you have to oversee FTP transfer and the loading process to Hadoop.
  • Challenges with Skills – Most mainframers don’t yet have the Hadoop skills needed to start up and run such an operation. Your current mainframers will need to acquire the Hadoop skills (which is definitely possible!), or you’ll need to hire or contract with someone who has the Hadoop knowledge you need.
  • Challenges with Security – As the data integration process is underway, and even after it is offloaded to Hadoop, you’ll need to maintain the integrity and security of the data. Hadoop security, however, has come a long way in the past year.

Also, Hadoop skills and big data analytics skills are not necessarily the same thing. You’ll need someone with strong analytics skills to get that data to sing and dance for you once it’s in Hadoop and ready to go to work.

The Solutions for Offloading From Mainframe to Hadoop

data integration

Once you’ve addressed the challenges, the road ahead is clear and bright.

Now for your solutions! First, you’ll need to determine what, exactly, you’ll be offloading from the mainframe to Hadoop. Many companies find that it’s best to start small, learn their way, and then gradually take on more as their skill sets, experience, and appetite for big data grow.

For a powerful, easy-to-use, all-in-one tool for ingesting, translating, processing, and distributing mainframe data with Hadoop, turn to Syncsort’s DMX-h. This product is a happy marriage of mainframe plus Hadoop for offloading heavy mainframe work to Hadoop.   Cognizant, Cloudera & Syncsort  have partnered to create BIGFrame, an end-to-end solution to simplify the offload of mainframe workloads & data into Hadoop leveraging DMX-h to collect, transform and distribute the data  to Cloudera as the scalable, low cost, high performance Hadoop hub.

Related Posts