Putting Quality Data in Perspective
Syncsort released the results of its latest Data Quality survey this week and it’s an interesting piece to put in perspective. Around fifteen years ago, I was consulting with a Fortune 500 company engaged in addressing data quality during the rollout of a new ERP system. During interviews with their operations team, the view was very clear that there were no significant data quality issues – goods were ordered, shipped, and delivered, and payments for those goods were received. The operational systems worked, they worked in a timely fashion, and they ensured that revenue goals were met. Yet, in interviews with their analytics team, a whole different perspective emerged. Data quality issues were pervasive and directly embedded in the operational systems which fed the downstream data warehouse. Two perspectives in one organization, both correct yet reflecting a larger issue where the data quality issues were strangling the organization’s ability to innovate and transform.
I have a similar sense while looking through the Data Quality survey: there are mixed perspectives here, some positive, some negative, but ultimately resulting in a lot of wasted time (and ultimately money), ineffective business decisions, and customer dissatisfaction.
While nearly 70 percent of our Data Quality survey participants felt that their business leaders had enough insights to inform business decisions, other recent industry statistics suggest that only 35% of senior executives have a high level of trust in the accuracy of their Big Data Analytics. If you consider that nearly 50 percent of our respondents indicated both that: 1) their organizations lack a standard data profiling or data cataloging tool; and 2) that they personally had previously experienced un-trustworthy or inaccurate insights from analytics due to lack of quality, then there appears to either be a disconnect or a difference in perspectives around organizational data quality.
More telling, I think, is that 75% of our respondents cite data quality as a high or growing priority in their organizations. This aligns with other industry reports that 84% of CEOs are concerned about the quality of the data they’re basing decisions on. With greater emphasis placed on the ability to respond quickly to customers, to rapidly innovate, and to gain new competitive insights, just having good quality operational data is no longer good enough.
The top challenges are neither new nor surprising: many varied sources of data (70%), applying governance processes to measure and monitor data quality (50%) and volume of data (48%) are the top three. Industry expert Michael Stonebraker noted the first as the “800-pound gorilla in the room” at this year’s Enterprise Data World conference. And 75% of our respondents noted large data volume as a barrier to data profiling to gain insight into the data quality issues and subsequently to ensure the quality of the data being used. Whether stored in the data lake or in the Cloud, roughly 20% of the participants cited the quality of that data as “Fair” or “Poor”. Without the ability to gain effective understanding or insight, or to address data quality, it’s no wonder that the recent study by Dimensional Research reports that nearly 80% of AI initiatives have stalled due to data quality issues.
If organizations wish to position themselves to take advantage of advanced analytics, machine learning, and AI in order to stay competitive and provide the optimal customer experience, it’s imperative to move past differing and often siloed organizational perspectives, and accept that effective data governance with the appropriate resources, processes, and tools to support high quality data that is well understood and effectively leveraged, regardless of where its stored, how varied it is, or how large the volume has become.
Download our report for highlights from the survey as well as a deeper look at the full results.