Archiving Your Data for Regulatory Compliance with Hadoop and DMX-h

Archiving Your Data for Regulatory Compliance with Hadoop and DMX-h

You know that smart use of data can help drive your business and earn you money. But do you realize that poor data management can also cost lots of money in regulatory fees and audit fines? Here’s what you need to do with your data to satisfy regulatory compliance requirements.

The Cost of Poor Compliance

Let’s start this discussion by emphasizing just how much is at stake when it comes to big data and regulatory compliance. It’s not just about pleasing federal authorities sitting in an office so that they don’t write bad reports about it. Responsible data management has a significant and direct impact on your reputation, your relationship with customers and your bottom line.

While the price tag for violation of compliance policies varies on a case-by-case basis, the fines that the government imposes add up to more than pocket change. You don’t want to make the same mistakes as two financial industry companies, for example, who were fined $1.8M and $1.3M in for reporting violations the last two years.

Download our free eBook -- Bringing Big Data to Life: Overcoming the Challenge of Legacy Data in Hadoop

The cost of poor compliance doesn’t end with regulatory fines. Reports of mismanaged data damage your company’s image and undercut your relationship with customers. These factors also cut into your bottom line.

The Challenge of Regulatory Compliance

For most organizations, assuring compliance with regulatory policies that govern how data is stored and accessed is challenging. That’s because the regulations that apply to the use of data, such as the Security and Exchange Commission’s Order Audit Trail System (OATS) policy, present a two-fold challenge. They require that you both store data securely and keep it accessible.

Achieving secure data storage while maintaining accessibility to data is hard. If you only had to do one or the other your life would be much easier.

You could store data in a way or place where no one could ever access it, or you could make it accessible to everyone without storing it securely. But, alas, this is not an either/or choice. You have to achieve both goals in order to stay compliant today.

Using Hadoop to Stay Compliant

Fortunately, platforms like Hadoop can help you meet this challenge. When properly managed, Hadoop provides an environment where you can both secure your data and keep it available to people with the proper privileges to access it.

In this respect, your data stored on Hadoop is much more compliance-friendly than it is if you leave it in a legacy environment, like your mainframe. Mainframes were not designed with modern compliance and access-control requirements in mind, and they are a poor solution for meeting regulatory needs.

Securing your Hadoop instance is beyond the scope of this article, but suffice it to say that there are plenty of resources online to help you doing this. Following the official documentation about running Hadoop in Secure Mode is a good place to start. How-tos like this one take Hadoop security a step further.

Using Hadoop to meet regulatory compliance

Moving Data into Hadoop

The only other challenge you have to solve in order to take advantage of Hadoop to meet regulatory needs is to transfer your data into Hadoop. If you try to do this by hand, you’ll likely find it difficult. Mainframe data exists in diverse forms, is often stored on legacy hardware like tape drives, and can’t be accessed by Hadoop directly.

If you take advantage of tools like DMX-h in order to ingest your data automatically into Hadoop, you can avoid obstacles. DMX-h handles the tedious part of Hadoop ingest for you, allowing you to move data seamlessly from legacy environments to Hadoop.

If you think compliance requirements are just too complicated or difficult to meet, think again. With Hadoop and DMX-h, you can be compliant – meaning you can protect your bottom line and reputation while achieving better data analytics results at the same time.

Legacy data in Hadoop causing unwanted roadblocks? Don’t miss opportunities to maximize the breadth of your data lake – Download our latest eBook, Bringing Big Data to Life, to learn trending insights on integrating mainframe data into Hadoop.

Download DMX-h for free today!

Christopher Tozzi

Authored by Christopher Tozzi

Christopher Tozzi has written about emerging technologies for a decade. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, is forthcoming with MIT Press in July 2017.
  1. This is a very serious business topic that is often overlooked within IT budgets. Compliance is a cost center. No one wants to allocate additional budget to a cost center. However these fines, and many others like them in the trading industry around the world, prove that it is a business issue that must be thought through.

    FINRA requires the storage of “books and records”. These files require that all order transactions be recorded and maintained for a period of 6 years.

    This isn’t limited to stocks and bonds either. The CFTC requires a 5 year retention period.

    What does it cost to fail in this area? Each regulator and exchange has their own process for determining the fine but here are some past examples.

    $2.8M – Oct. 2016 FINRA Fines Merrill Lynch $2.8 Million for Systemic Reporting, Books and Records, and Related Supervisory Violations. source:

    $14.4M – Dec 2016. FINRA Fines 12 Firms a Total of $14.4 Million for Failing to Protect Records From Alteration source:

    $9.5 – July 2013: FINRA Joins Exchanges in Fining Newedge USA, LLC $9.5 Million for Supervisory, Regulation SHO, and Books and Records Violations. source

    $906K – Dec 2016 CFTC Charges Brett G. Hartshorn of Sarasota, Florida with Soliciting at Least $906,000 in an Off-Exchange Foreign Currency Fraud Scheme (and for failure to have books and records) source:

    $1M – June 2013 CFTC Orders ABN AMRO Clearing Chicago LLC to Pay $1 Million to Settle Charges of Segregated and Secured Fund Deficiencies, a Minimum Net Capital Violation, Books and Records Violation, and Supervision Failures. Source:

    BTW. I have no affiliation with syncsort. I am just passionate about this topic.

    1. Michael Kornspan
      Michael Kornspan May 1, 2017 at 9:17 am

      Thanks for adding a good deal of very valuable information on this topic Tayloe!

Leave a Comment