The Ultimate Guide to Mainframe Machine Data: Part 1
There are a number of different data sources that are available within the IBM z/OS® mainframe that can be leveraged to provide insight into the operational health of the system and applications as well as providing visibility into security and compliance issues.
The primary data source is the System Management Facility (SMF) on z/OS, a component which collects and records a large amount of real-time and historical information on performance, security, and technical operations. An abundance of very useful information, providing a wealth of insights can be recorded daily. Virtually every operational event that occurs on the mainframe — from a simple log-in attempt at a particular workstation to a potential breach of system security — is captured and recorded in one or more SMF record types.
There are other data sources available which can also provide valuable insight into performance, availability, health, and security of the system and its underlying applications. With all the data sources and volume of data available on the IBM z/OS mainframe, each use case becomes an exercise in identifying the right set of information needed to meet the needs of the organization. Organizations definitely understand and see the value of incorporating mainframe logs into their analytics processes. Let’s take a look at the different data sources and how they can be used for both operational intelligence and to address security challenges.
SMF Data: The Mother of All Data Sources
What is SMF?
The System Management Facility (SMF) is a logging capability provided by the IBM z/OS mainframe operating system to capture detailed information about every activity happening within the system. This includes system-level information, application information, security information and events, transaction information, database information, and virtual anything related to the system’s operating environment. SMF data is probably the biggest single source of operational and security information on the mainframe. But it is also the most complex making it a challenge to extract information from.
What Does SMF Data Look Like?
SMF data is very complex! Every aspect of the system operating environment generates a unique record type which is self-describing and can contain thousands of unique fields. Most records typically contain multiple “parts” referred to as subtypes. Understanding how to decompose an SMF record for use typically requires a mainframe subject matter expert. However, once the record has been decomposed fields can be easily named, identified, and used by anyone familiar with an analytics platform like Splunk.
How is SMF Data Used?
TSMF Data is primarily used to provide an organization with IT Operational Analytics (ITOA) and address requirements related to Security Information and Event Management (SIEM).
What is Syslog?
Syslog on the IBM z/OS mainframe is very similar to system logging facilities on other platforms. System components, applications, and workloads write text-based messages to Syslog to record different events. These can include both normal and abnormal operational issues.
How is Syslog Used?
Analysis of Syslog messages can be used to look for issues that impact the operating environment of the system as well as to detect security issues and threats. Since Syslog is a text-based message logging facility it contains less detailed metrics than what would be provided in an SMF record, however, analysis of the messages within Syslog can still address some of the ITOA and SIEM use cases addressed with SMF data.
UNIX System Services (USS) Files
What is USS?
The UNIX® System Services element of z/OS, sometimes referred to as “z/OS UNIX”, is tightly integrated into the operating system to provide UNIX capabilities within the IBM z/OS operating environment. UNIX System Services allows UNIX applications from other platforms to run on IBM System z mainframes running z/OS. UNIX System Services is a key element of IBM’s open and distributed computing strategy. USS is a certified UNIX operating system implementation optimized for mainframe architecture, and an integral element of z/OS.
What are USS Files?
Non-UNIX z/OS workloads utilize standard mainframe datasets. However, z/OS UNIX provides a hierarchical file system (HFS) familiar to UNIX users. In addition to the HFS, the IBM z/OS operating system also provides support for another UNIX filesystem — the z/OS® Distributed File Service (DFS™) zSeries® File System (zFS). Much like HFS, zFS contains files and directories that can be accessed with APIs, as well as be mounted into the z/OS UNIX hierarchy along with other local or remote file systems types such as HFS, TFS, and NFS.
How are USS Files Used?
USS files can contain information from Java applications, C++ programs, and other UNIX-based applications running within the USS environment on z/OS. Incorporation and analysis of USS files can address a variety of use cases centered around the applications which write and read information from USS files. USS files can be used to investigate application activity, performance, availability and virtually any operational issue related to the application. Data generated by an application into a USS files could also be used for other analytics such as understanding business performance and customer buying patterns. Use cases based on USS files are virtually unlimited. IBM Security Key Lifecycle Manager (SKLM): Provides centralized and automated encryption key management processes on z/OS including key storage, key services, and key life-cycle management functions. IBM SKLM includes support for IBM and non-IBM storage solutions using the OASIS Key Management Interoperability Protocol (KMIP).
Check back for part 2 where we’ll discuss: Log4j, syslogd, SYSOUT, and RMF data.
Learn more about Ironstream including why it’s the industry’s most comprehensive automatic forwarder of IBM i and IBM z machine and log data to analytics platforms.