How Big Data & Hadoop are Revolutionizing the Insurance Industry
The insurance industry doesn’t change often, and it doesn’t change quickly. For most of the history of this industry, insurance actuaries have depended on a body of internal data, primarily loss history information, to set rates and develop new products. Then, insurance companies began to incorporate third-party data into their analytics — most notably, the use of credit scores to determine which of the insured would be safe drivers. As it happens, there is a correlation between people who pay their bills in a timely fashion and those who drive safely. This use of data outside a customer’s own driving record has caused no shortage of controversy in consumer groups.
With the innovation of Hadoop and the ready availability of ever-growing sources of data, the insurance industry can now move far beyond their own historical data. These new data sources and analytical tools empower insurers to glean deeper insight into all sorts of customers (life insurance, property and casualty, health, etc.), and even helps them determine what new products will be both popular and lucrative. Big data and Hadoop can also help actuaries determine suitable rates for those products. Here’s what these developments bring to the insurance industry’s table.
Beyond Internal Actuarial Data
When internal historical data on claims is combined with data from the government, medical industry, and other external sources, actuaries get a much deeper, more accurate picture of their insured and liabilities.
There is a growing body of resources for data that can be used to predict things like illnesses, accidents, and outcomes. For example, the government, the healthcare industry, educational organizations, the manufacturing industry, the energy sector, business, and others are collecting growing bodies of data continually. Within these data sets, insurers are finding new correlations that can be used both to assess current risk and to predict future risks.
For example, how many floods or fires will an insurer have to cover within a given geographical region? What are the chances that a 44-year-old white male industrial worker with no family history will develop Parkinson’s disease? How do things like dental care, pet ownership, number of marriages, and a preference for coffee over tea affect a person’s driving record or life expectancy? New correlations are emerging all the time, powered by data that is readily available from social media, government records, mobile devices, online search histories, and other consumer and business sensors and data sources.
Better Tools for Analyzing Data & Finding Correlations
Of course, the ready availability of data doesn’t alone make for better prediction of liability or improved product development. Hadoop is an open-source framework for big data analytics, and quickly became synonymous with big data and analytics. While Hadoop began as a rather complex and daunting platform, new technologies have made it easier to use, empowering insurance companies to make use of it for all sorts of analysis and predictive modeling.
For example, insurance companies have long depended on mainframe computers for storing and processing data. Now, many are faced with the rather daunting challenge of offloading this transactional, customer, historical, and other data into Hadoop for analytical purposes. For example, Guardian Insurance uses Syncsort DMX-h to access data from their mainframes and integrate with data from other sources to create a “data supply chain” and conduct powerful analytics using Hadoop.
Other Uses Insurance Companies Have for Big Data and Hadoop
Big data is useful for setting premiums and developing products for property and casualty, life insurance, health insurance, worker’s comp policies and much more.
Aside from setting premiums and developing new products, what are insurance companies using big data and Hadoop to do?
- Fight fraud
- Provide assistance to their insured by educating and empowering them to live healthier lifestyles and make smarter choices
- Improve the customer experience
- Improving things like workplace safety by educating employers on conditions that lead to increased or decreased worker’s compensation claims
- Marketing and sales, much like retailers and e-commerce businesses do