How Big Data is Transforming the World of Finance
The global financial crisis of 2007-8 demolished the idea that financial markets and institutions could regulate themselves. Broad procedural and institutional reforms have followed in an effort to prevent future financial meltdowns. But questions have continued to plague policymakers. Why didn’t we see the crisis coming? How will we know if another crisis is imminent?
Congress created the Office of Financial Research to help predict and prevent future financial crises.
It was with these questions in mind that Congress created the Office of Financial Research as part of 2010’s Dodd – Frank Wall Street Reform and Consumer Protection Act. The OFR is a special branch of the U.S. Treasury Department. Its mission, according to its official website, is “to improve the quality of financial data available to policymakers and to facilitate more robust and sophisticated analysis of the financial system.” In short, the office is responsible for monitoring the markets and raising an alert if it detects an oncoming financial crisis.
“The financial crisis revealed serious deficiencies in our understanding of vulnerabilities in the financial system and the financial data needed to measure them,” says OFR director Richard Berner. OFR’s mission is to fill in the gaps in data and analysis.
Financial systems are notoriously complex. Monitoring the status of the U.S. economy is a classic job for big data. Researchers at OFR must track countless bits of information and extract meaningful trends and high-level analyses from them. This extraction of information from data is precisely what big data is all about.
It seemed at first that OFR and its counterparts around the world would collaborate smoothly in gathering and sharing worldwide financial data for the benefit of all. But then nationalist tensions compromised financial planners’ commitment to sharing data. When the U.S. government’s pursuit of terrorists led it to subpoena confidential financial data from the Brussels-based Swift banking system, European countries reacted with distrust. Later, the Edward Snowden revelations regarding international data collection by the National Security Agency further eroded the international community’s willingness to cooperate. The goal of gathering all relevant data into a single, shared big-data resource seems far from a reality.
The Private Sector Steps In
International relations have not prevented the private sector from implementing big-data approaches in finance, however. In addition to predicting and preventing crises, financial institutions are motivated by increased government regulatory pressure, the need to provide better service to customers, and cost-control concerns. Big data — extracted from structured, semi-structured, and unstructured data sources — can help finance companies develop new products and services, reach new customers, cross-sell products to existing customers, and manage risk. In short, big data makes good financial sense.
Financial firms were among the first to appreciate the benefits of big data in smoothing operations and boosting profitability.
Take a relatively simple issue like fraud detection. Effective detection requires the integration of transaction and customer data from multiple sources. These data sources are often incompatible information-management services that operate across state and national borders. Before the advent of big data systems, fraud often wasn’t detected until major damage had been done. Integrated big-data solutions can detect fraudsters’ activities as they are preparing to act, however, in time for financial institutions to shut them down before they strike.
Government regulation is another factor driving the adoption of big-data solutions in finance. Global banks are now required to provide risk-management data to regulators as often as several times per day. The data for these reports comes from different trading arms in multiple countries, each with its own DBMS and data-structure conventions. New information is generated at the rate of terabytes per day. This challenge is custom-tailored to big data. It is only through application of big-data resources and techniques that banks can hope to meet the regulatory burden.
Many financial institutions are also involved in insurance, a field in which data is stored in millions or billions of hand-typed claim forms. This data is largely unstructured, but it is of key importance to decision makers within financial services organizations. Extracting the information from those terabytes of data is what big data is all about.
Remember annual reports? Once a year, financial institutions used to gather and collate the operating figures for all of their business operations and present them in a form that management and investors could use to support decision-making. But the pace of business in today’s world does not support once-a-year decision-making. To be competitive, financial institutions must be able to provide management reports based on actual data on a much more frequent basis — in real-time, if possible. Only big-data approaches can make that goal a reality.
The Rush to Big Data
Then there are risk analytics, trading analytics, fiduciary management, legal compliance issues…all of these are driving financial institutions to implement big-data solutions as quickly as they can.
How quickly is that? A 2013 survey of Wall Street firms by NewVantage Partners found that 96 percent of surveyed executives had big data initiatives in the works.
The federal government’s efforts to create a crisis-aversion big-data system may not yield fruit in the near future. But within the industry, financial institutions are embracing big data as a key to remaining competitive in a rapidly changing world.