Sort: Gets No Respect but Wields Mainframe Speed Secrets
Quiz: Name the computing process that:
- Consumes as much as 20-25% of all CPU cycles but is generally ignored when optimizing mainframe resources
- Is one of the most-covered topics in introductory computer science classes, but tends to be ignored once school is out…and
- (A cheap clue in case you haven’t guessed it yet) includes processing types referred to as injection, histogram, merge, quick, radix, bin, bubble
It’s a computing process that, as Rodney Dangerfield less artfully but pointedly put it, “Can’t get no respect.”
Times up. We’re speaking of sort processing – one of the most widely relied-upon components in software engineering. While you’re refreshing yourself on the difference between a quick sort and an insertion sort, it’s worth considering the first of these points. While solid research on the resource utilization of sort is hard to come by, it’s self-evident that sort algorithms gobble-up resources in modern operating systems, utilities and applications.
Sort as Silent Partner
Sort often operates behind the scenes as a silent partner. The connection between sort and other functions, such as compression, continues to capture the interest of optimizers. For example, Matt Allington, working on a Microsoft stack with SQL Server and PowerPivot, looked at the benefits of sorting data on load.
An IBM note on improving DB2 for z/OS LOAD performance offers similar tips, e.g., “present data to LOAD in the optimal order: sort the data in cluster order to avoid needing to reorganize it after loading” and “If you are loading a single table that has, at most, one foreign key or one index key, sort the data in key sequence.” And so on.
IBM’s IMS Performance and Tuning Guide Redbook contains a section on multi-streaming batch processes (9.2.10.). The guidance invokes the sort magic dust several times:
“If you use flat files as drivers for your batch process, then ensure that you sort and split the files in the sequence of the database. This means that if you are PHIDAM, then you would sort and split the files using the Partition Selection Exit or high keys as specified in the RECON in ascending sequence. If you are using DEDBs, then you would sort and split the files in randomizer sequence” (p. 178).
Back in 2010, Willie Favero writing in Enterprise Systems Media introduced the topic of DB2 sorting by admitting that “sort invokes many thoughts and emotions. And “most in the “necessary evil” category,” he quipped. Like it or not, Favero advised that in “SQL, sort is one of those things you do whatever you can to avoid, though that’s an almost impossible task.”
Favero’s focus was DB2’s relational RDS sort, but DB2 sort performance is an issue not lost on sort specialists like Syncsort. In DB2, the DB2 Query feature allows Syncsort MFX SORT or COPY operations to directly retrieve data from a DB2 database based on a query specified by an SQL SELECT statement.
BMC Software’s CHECK PLUS for DB2 offers this tuning tip pertaining to memory use: “To improve memory usage, consider the following information: Specifying a value greater than 0 for SORTNUM (with no SORTWK data sets specified in the JCL) allows CHECK PLUS to perform the maximum multitasking of sort processes based on available memory.”
SAS Software users may have noticed that as much as 30% of all SAS processing is consumed by sort processes. That was a level of utilization that Syncsort MFX PROCSort and PROCSortIE were designed to alleviate.
Sort is so omnipresent that it’s often taken for granted. User interfaces regularly call for resorting of columnar displays and bar charts whose back end executions usually entail sorting. Compuware’s mainframe tool Topaz, for example, allows users to “filter, sort, export and display recorded events in table form.
Sort’s Impact on Performance
The Big Data era has garnered attention for new products, new startups and new applications, but that’s not the whole story. A survey conducted by Enterprise Systems Media last year suggested that Big Data would have an impact on mainframe processing, too. Over 89 percent of respondents anticipated that mainframes would play a significant role in their Big Data processing plans – through CICS, DB2 and WebSphere. Many also see mainframes as an opportunity to provide data to Big Data platforms such as IDAA, Netezza, Splunk, Spark, Oracle, Teradata and Hadoop.
The Syncsort incarnation of the venerable sort can also benefit from its use of IBM’s zIIP processors, which can crank processing speeds for sort tasks by as much as 90%.
Because of its central, albeit behind-the-scenes importance, research into sort techniques is ongoing. Intel Labs’ Throughput Computing group reported success at increasing radix sort performance by 2.2X on the Intel Core i7 CPU.
In-memory and load-time sort have proven critical to memory-resident business intelligence tools like QlikView that do not require data warehouse planning and deployment. As data volumes swell, mainframe hardware resources capable of both high speed I/O processes and high speed in-memory can produce similar benefits with even more vast amounts of data.
Fast sort will be a quiet, secret enabler.
Exploiting the Mainframe’s Powerhouse I/O
The current release of Syncsort MFX® for z/OS allows users to optimize on elapsed time or on CPU time. It also leverages IBM’s System z High Performance Fiber Connection (zHPF), the hardware maker’s most comprehensive set of I/O architecture and firmware improvements. As IBM Fellow Harry Yudenfriend explained:
“The zHPF open standard has been adopted by all enterprise-class vendors providing DASD storage for the mainframe. In its essence, zHPF leverages hardware assists provided by host bus adapter vendors for the fiber channel protocol transport while preserving enterprise class qualities of service supported by Fiber Connection (FICON).”
Beginning in late 2011, the zHPF standard on z/OS added its muscle to QSAM, BSAM and BPAM methods. IBM’s DFSORT and merge apps now use BSAM to exploit zHPF. Measurements made by Syncsort with its release of MFX 2.1 suggest that using zHPF can cut as much as 15% off elapsed times.
Speed over Respect
This explanation gets pretty technical, but the payoff is straightforward and transparent: using a zIIP to sort, with most sort steps taking place on that specialized mainframe-class iron, is typically easy to use.
Keep in mind that the sort game is far from over. In 2011, Bertasi et. al introduced a fast sort called psort in the Journal of Experimental Algorithmics. And a just-published 2016 article by Matai, Richmond, Lee et al. proposes a framework that generates sorting architectures for different computing needs.
Speed will come. Respect? That’s a concern of lightweights.
You’ll garner respect for your glistening new Mainframe DevOps Chief job title instead.