Data infrastructure optimization software
Data integration and quality software
Data availability and security software
Cloud solutions

Why Customers Want to Get Real-Time Mainframe Big Data into Splunk

Splunk Enterprise is the leading software platform for real-time operational intelligence.   Its easy-to-use searches, dashboards and alerts improve the responsiveness of system administrators responsible for security and application monitoring.  Many organizations rely on Splunk Enterprise to keep their systems running smoothly.

Unfortunately, for those environments where mainframe applications are a critical component of an enterprise IT operation, large quantities of log data generated on those systems are inaccessible to Splunk.  This leaves these organizations with several klugey and insufficient options.  They can use and maintain separate tools and go back and forth between them or they can use one tool, and somehow extract data from the mainframe at the end of the day and then live with greatly diminished value because the latency of the mainframe data is so large.

Recently Splunk and Syncsort announced a technical alliance that has resulted in Ironstream™, a new product created by Syncsort. Ironstream collects, transforms and forwards mainframe log data (SMF, syslog, Log4j data, etc.) in real time directly into Splunk Enterprise.  This eliminates the problems of getting mainframe data in real time into Splunk.  Now customers can extend the benefits of Splunk to their entire IT infrastructure.  Comprehensive security and application monitoring is now easily achievable across the entire enterprise, correlating data across platforms and creating 360 degree views of the system wide applications.

Ironstream Use Cases 

Early adopters of Ironstream all have similar goals but their exact use cases differ.  One customer has a critical application that spans multiple environments.  It is crucial to address problems quickly, identify the cause and fix it.  Waiting until the end of the day to try to understand what happened is not acceptable any more.  The company has invested in an elaborate logging system to keep track of the progress of transactions.   This application runs on multiple different platforms including the mainframe.

Transactions move on and off of this platform.  For the web services and the distributed environments Splunk is used to monitor transactions as they make their way through the various systems but once the process transitions to the mainframe this monitor loses visibility.  Now they have to switch to a different monitor.  This is error prone and more time consuming then it would be if the entire process was managed by one monitor.

After several unsuccessful attempts to get the data off the mainframe, this organization tried Ironstream.  As a result, they were able to get their critical log data off the mainframe and into Splunk.  The data was delivered real-time and the MIPS impact was negligible.  Now this customer is able to get an end-to-end view of their transactions from one central point and is able to deliver better service to end users.

Another customer wanted to extend the use of high powered real time analytics app on Splunk to mainframe log data.  Generally, this type of analytics is extremely CPU intensive and therefore not a good candidate to be run on the mainframe.  The analytics are most needed precisely during periods of peak activity when mainframe CPU cycles are at a premium.  This type of processing is ideally suited to be run off the mainframe, i.e. with Splunk.  The problem is how to get the data into Splunk quick enough that it can be analyzed and acted upon.

Once again, Ironstream was the answer.  Its light-weight MIPS footprint allows it to intercept and stream records into Splunk while not impacting the existing mainframe workload.   The small data latency makes the results, alerts, findings all actionable.   Ironstream opens the door for many mainframe logs to be analyzed in this fashion.

These are just two of the use cases that we have discussed with customers.  There are many more.  Ironstream is used to bridge the gap between real time mainframe data and the power of Splunk and the apps that are part of Splunk.  Whether you require a unified view of an application or need to get data into an environment where advanced analytics are more suited to run, Ironstream is a critical tool in this process.

The good news is we just announced that Ironstream will be Generally Available on October 10.   Learn more about Ironstream.

2 comments
  • […] examples of what is possible with Ironstream, check out this video or read Syncsort’s blog. Also, don’t miss our Fifth Annual Worldwide Splunk Users’ Conference  in Las Vegas, Oct. 6-9, […]

  • Demetri Zavala — October 1, 2014 at 10:32 pm

    would like to see this Tool in action, even though we have other tools in house (and having to do Extracts of SMF data and research the Record Format in order so Sort the records before composing a Report), would like to see the Guide on this Product for the Mainframe utilizing CICS, DB2, MQ.

    Thanks

Leave a Comment

Related Posts