Splunk Best Practices
Real time operational intelligence — Splunk® puts power in your hands if you know how to use it right to get the most out of it. A few simple practices can make a huge difference when it comes time to audit, analyze, or debug. Here are your best practices, along with how you can use Syncsort with Splunk to get real time information even if you’re using a mainframe.
Make Everything You Can Human Readable
Making it human-readable will save you searching for info later.
Splunk features the capability of naming your events for you, but you shouldn’t let it. Creating human readable events eliminates all those lookups that waste time and effort. In addition to events, you’ll want to create human readable timestamps. Place the timestamps at the beginning of the line, or as close to the beginning as possible. Also, include a time zone marker, and keep timestamps in the milliseconds. This way, if your event becomes separated from its original source file at any point in the future, it’s easy to determine exactly where it’s supposed to go. Finally, try not to use XML and JSON when there is any way around it. Not only are these not readable by humans, it takes more time to parse this way.
Use Detailed and Unique Identifiers
Creating unique identifiers keeps you from getting lost in the data.
Identifiers, such as transaction IDs and user IDs, need to be as unique as possible. This helps out during the debugging process, as well as when it comes time to gather analytics. Unique identifiers allow you to point to the exact transaction instead of just a range of transactions.
Pay Attention to Logging Practices
When logging, use text format instead of binary. Using binary requires you to do lots of decoding, and it won’t segment. Also, don’t just log the debugging events. Include the semantic meaning in order to maximize what you can get out of your data. For example, log:
- audit trails
- information on the timing
- what the user is doing at the time
- any other information that could be valuable during analysis, charting, or aggregation
Get the Goodies Out of Splunk, Even if You Use Mainframe
Until recently, mainframe users were severely limited in their ability to make the most out of Splunk capabilities. Mainframe users had to either use separate tools, switching continually back and forth among them, or turn to a single tool, which meant waiting until the end of the day to extract data and losing the value of real-time operational intelligence. This changed with the innovation of Ironstream™. Ironstream is the product of a joint venture between Splunk and Syncsort. It allows you to extract large amounts of log data off the mainframe, transform it into usable form, and use it in real time. This includes SMF (>30 types), syslog, DB2 and Log4 data.
Ironstream lets you utilize the power of Splunk across your entire infrastructure, including security monitoring, application monitoring, and correlating data across different platforms. It gives you a 360-degree view of your applications across the system. Visit Syncsort today to see what Ironstream can do for your mainframe environment.