A Dream of Great Big Data Riches – Harvesting Mainframe Log Data
Z/OS system components, subsystems, applications and management tools continually issue messages, alerts and status and completion data, and write them to log files or make streams available via APIs. We are talking hundreds of thousands of data items every day, much more from big systems. This “log data” generally comes under the heading of unstructured or semi-structured data and has not traditionally been seen as a resource of great value. In some cases it is archived for later manual research if required, in many cases it just disappears! In the case of SMF records it has traditionally been consumed by expensive mainframe based reporting products that unlock the value, but at great cost and you still need special expertise to do anything with it.
What if all this potentially valuable data could be collected painlessly in real-time, made usable by a simple query language and presented in easy to read visualizations for use by operational teams? This sounds like a fantasy dream, but it is what Syncsort and Splunk have achieved through their partnership and products.
Nuggets and gemstones
Of all the data sources we are talking about, SMF (System Management Facility) records are the wealthiest trove with over 150 different record types that can be collected. SMF provides valuable security and compliance data that can be used for intrusion detection, tracking of account usage, data movement tracking and data access pattern analysis. SMF also provides an abundance of availability and performance data for the z/OS operating system, applications, web servers, DB2, CICS, Websphere and the MQ sub-system.
But there is much additional information in feeds like SYSLOG, RMF (Resource Management Facility) and Log4J. And there are the more open ended sources that could be considered log data, like the SYSOUT reports from batch jobs.
The gem collector and now Lapidarist too
Syncsort’s solution for the collection of mainframe log data is called Ironstream and it is a super-efficient pipeline to get data into Splunk Enterprise or Splunk Cloud. Designed from the start to be lightweight with minimum CPU overhead, Ironstream is a data forwarder that converts log data into JSON field/value pairs for easy ingestion. We built it in direct response to Splunk customers who wanted to complete their Enterprise IT picture with critical mainframe data to complete an end-to-end, 3600 view.
In addition to all the data sources listed above, Ironstream offers access to any sequential file and USS files. This gives very comprehensive coverage to any source of log data that an organization might be producing from an application. But in addition we offer an Ironstream API that can be used by any application to send data directly to Splunk if it’s not already writing it out somewhere.
Of course something has to be too good to be true here doesn’t it? Well yes, one potential issue is the sheer volume of data that is available and the cost of storing it. While all of it could be valuable, most companies are going to want to selectively focus on the items that are most valuable to them now. To address this requirement, our Ironstream engineers became digital Lapidarists. In the non-digital world, Lapidarists are expert artisans, who refine precious gemstones into wearable works of art. With the latest release of Ironstream, we now offer a filtering facility that allows you to refine large the large volumes of mainframe data by selecting individual fields from records, discarding the rest. By customer request, we have on our roadmap an even more powerful “WHERE” select clause that will allow you to select data elements across records based upon subject or content.
Why didn’t I know this?
There is a fast moving disruption happening in the world of IT management and not everyone wants you to know it. Open source solutions and new analytical tools are changing everything.
For the last 40 years complex, point-management tools have been used by highly skilled mainframe personnel to keep mainframes running efficiently. Critical status messages are intercepted on their way to SYSLOG and trigger automation to assist the operational staff. All this infrastructure has made most of this log data unnecessary for operations and mainly of archival interest if of any interest at all. The most valuable SMF data usable for capacity planning, chargeback and other use cases has been kept in expensive mainframe databases and processed by expensive reporting tools.
In parallel to the disruption that is being driven by emerging technologies there is a special skill crisis in the mainframe world; the experts that have been managing these systems for 40-50 years are retiring and there are not enough people being trained to replace them.
Fortunately in the confluence of these two trends a solution is born. By leveraging this new ability to process mainframe log data in platforms like Splunk and Hadoop, a new generation of IT workers can assist “Mainframe IT” by proactively seeing problems emerge and assisting in their resolution. In the first wave of adoption this will help offset the reduced availability of mainframe skills, but it won’t obviate the need for them completely and it won’t replace the old point management tools. Yet.
As this technology matures, and machine learning solutions become proven and trusted, we will see emerge a new generation of tools. Based on deep learning, these will replace both the old mainframe tools and the personnel who used them, but now want to be left in peace by the lake. My prediction is that as this comes to be a reality, we will also see a move of analytics technology back onto the mainframe platform. The old dream of “autonomic computing” will become a reality and a new mainframe will in effect evolve; one that tunes and self-heals itself.
Syncsort plans to be there, in fact we are leading the way there. We offer the keys to the treasure chest for anyone who wants to follow our map to find the dream of great riches!