How it works: Improve workload throughput with HiperDispatch on System z10 (Part 1)
from CCR2, Issue 4 - 2008
By Laurence Hart
Software Developer
IBM Tivoli System z software
On February 26, IBM announced a new server, the IBM System z10 Enterprise Class. The platform incorporates significant advances in processor technology, capacity and performance, including the introduction of 4.4 GHz, quad-core chips, up to 64 processing units and 1.5 terabytes of processor memory. On the IBM System z10 running z/OS Versions 1.9, 1.8 or 1.7 with maintenance, a new feature called HiperDispatch can improve workload throughput by optimizing processor level 2 cache utilization.
This first article in a two-part series will introduce you to the HiperDispatch technology. A follow-on article will describe how IBM Tivoli OMEGAMON XE on z/OS, which delivers Day 1 support for the System z10 platform, can help you better understand the actions of HiperDispatch under z/OS.
|
z/OS on the IBM System z10 implements a new approach to dispatching work to address IBM customers’ increasing workload demand for processor cycles and high speed memory access. HiperDispatch throughput improvements have been achieved by making z/OS aware of the underlying physical topology of configured processors.
z/OS can use this awareness to attempt to re-dispatch a unit of work repeatedly on the same physical CPU, or collection of physically adjacent CPUs (an affinity node), to increase the chances of obtaining data from processor L1, L1.5 or L2 cache, instead of incurring a time delay by going to main memory.
Prior to HiperDispatch, each System z processor was given equal access to memory through the symmetric multi-processing protocol (SMP). Under SMP, workloads are dispatched on the first available processor, regardless of where the workload was sent previously. The SMP approach results in memory access latency times caused by inefficient use of processor caches and more frequent cache discards.
Both the System z9 and System z10 platforms employ a book architecture where groups of processors – two (on z9), three or four (on z10) to a chip – are mounted on multi-chip modules (MCMs), each of which shares a Level 2 cache among the module’s processors. Level 2 caches on each MCM are connected, either through a two-way ring on the System z9 or a more efficient point-to-point star configuration on the System z10.
Under SMP, in addition to making no attempt to re-dispatch a workload on the same processor, the workload might be dispatched on a processor on a different book. This approach can result in Level 2 cache searches that extend across two books (on a System z9 with four books) – introducing significant memory latency overhead, given today’s high CPU processing speeds.
What is HiperDispatch?
HiperDispatch helps to maximize processor cache utilization by dispatching workloads within an LPAR on a set of logical central processors (LCPs), some of which are mapped one-for-one with physical processors. The number of LCPs in an LPAR that are mapped to a physical processor depends on the LPAR’s weight as a percentage of the weight of all other LPARs in the complex and the number of shared physical processors in the complex.
The store system information (STSI) instruction on System z10 hardware has a variation that allows the Processor Resource/Systems Manager (PR/SM) hypervisor to determine the topology of the physical processors, including the book placements of the various types of CPUs. This capability allows PR/SM to dispatch an LPAR’s LCPs to a specific physical processor, instead of the next-available processor.
HiperDispatch designates an assortment of high-, medium- or low- share LCPs, based on an LPAR’s relative share of the pool of shared physical processors.
Each LCP designated as high-share (high LCP) is mapped one-for-one with a physical processor and effectively gets full access to its own dedicated physical processor. Other LCPs, which receive less than full access are designated medium-share (medium LCPs) or low-share LCPs (low LCPs) and divide the remaining physical processing resources available to the LPAR.
Calculating LPAR logical processor share and percentage
The following example illustrates how HiperDispatch designates the priorities for various LCPs within an LPAR, and their associated share of physical processing resources.
 This example assumes a System z10 model E40 fully configured with 40 physical processors – 32 standard central processors (CPs), two IBM System z Integrated Information Processor engines (zIIPs) and six IBM System z Application Assist Processor engines (zAAPs). Calculations are shown for one LPAR (LPAR01) in a complex (see Figure 1). LPAR01 weights are based on hypothetical user-configured values for this LPAR.
Figure 1: Calculation of HiperDispatch physical processor allocation
These calculations yield the following LCP priorities and percentages for LPAR01:
| Processor type |
High LCPs |
High share % |
Medium LCPs |
Medium share % |
Low LCPs |
Low share % |
| Standard CP |
3 |
100 |
2 |
66 + 66 |
11 |
0 |
| zAAP |
1 |
100 |
1 |
100 |
2 |
0 |
| zIIP |
0 |
100 |
1 |
60 |
1 |
0 |
In the above example, the high and low LCPs receive percentage shares of 100 percent and zero percent, respectively.
HiperDispatch may designate two LCPs as medium, if the fractional value resulting from the calculations dips below 50 percent. In that case, an LCP initially designated as high will be re-assigned to medium with the combined percentages of the two LCPs shared equally between the two.
Parked and unparked logical processors
Work is dispatched only on the high and medium LCPs while overall utilization for this processor group remains at or within the optimally efficient capacity. Under these conditions, even though they are online, low LCPs are maintained in parked status. Low LCPs are considered a discretionary CPU resource within the LPAR, having a logical share of zero percent when they are not considered for dispatching work.
When the resource demands of an LPAR exceed the optimal utilization of its high and medium LCPs, yet the resource demands of other LPARs in the complex remain at less than their entitled share, HiperDispatch will unpark low LCPs. Conversely, when the resource demands of an LPAR decrease such that high and medium LCPs, and any unparked LCPs, lag below their optimally efficient capacities, HiperDispatch will park low LCPs.
When low CPs are unparked, they are entitled to a portion of the assigned CPU resource assigned to medium LCPs, and the total medium LCP share will be divided equally among the medium and unparked low LCPs.
Like high and medium LCPs, unparked low LCPs are driven to their most optimally efficient utilizations before unparking additional low LCPs. This mechanism conforms to the design criteria intended for HiperDispatch by maximizing the workload dispatching on the same physical processor or physically adjacent group of processors at the maximum, cache-efficient level of utilization. This approach assures the best chance of data hits in the Level 2 processor caches and minimizes memory latencies due to cross-book Level 2 cache searches.
Performance expectations
Switching on HiperDispatch on an LPAR will typically result in processor performance improvements of zero to 10 percent. Very large z/OS images configured on groups of physical processors spanning multiple books and involving memory operations with intensive reference patterns are likely to achieve the most significant improvements. Configurations of up to 64 LCPs to a single z/OS image on System z10 can achieve significant performance benefits.
In comparison, batch jobs running in a z/OS configuration where the configured physical processors fit on a single book are unlikely to gain significant performance improvements. Such workloads tend to dispatch long-running tasks that are dedicated to just one LCP at a time.
In the middle, transactional workloads with high throughput rates where many of the functions are shared among transactions are likely to achieve mid-range improvements.
On the low end, configurations where the logical-to-physical processor over-commit ratio is high may yield reduced benefits from HiperDispatch, because an active Intelligent Resource Director (IRD) reduces the number of LCPs a workload runs on.
In addition, capacity provisioning management (CPM), introduced on the System z10 platform, presents an additional variable. In this case, physical processors can be dynamically provisioned to, and deprovisioned from, a System z10 complex based on an installation policy, with or without operator intervention.
As CPM adjusts the number of physical processors based on workload demands for an entire complex, calculation results for the numbers of designated high and medium LCPs in an LPAR with HiperDispatch fluctuate. Depending on an LPAR’s relative weight, the resulting increase or decrease in dedicated physical CPU resource may affect the contributions HiperDispatch can make to performance.
At a minimum, carefully evaluate your workloads and performance with and without the HiperDispatch feature.
Deploying HiperDispatch
HiperDispatch can be deployed during initial program load (IPL) through a new SYS1.PARMLIB IEAOPT keyword parameter:
HIPERDISPATCH=YES | NO
HiperDispatch can also be deployed dynamically after IPL by issuing the following console command:
SET OPT=xx where xx is the suffix of SYS1.PARMLIB(IEAOPTxx)
In addition, you can specify HIPERDISPATCH=YES in the IEAOPTxx member to switch HiperDispatch on, or HIPERDISPATCH=NO to turn HiperDispatch off.
Summary
HiperDispatch is a new performance-enhancing feature available on the IBM System z10 platform. The feature can be deployed on one or more LPARs in a complex, either through SYS1.PARMLIB controls or dynamically through an operator command.
Performance improvements are achieved by maximizing processor caching efficiency through cooperation between the System z10 hardware, PR/SM and z/OS. This cooperation takes advantage of the availability of processor physical topology information exposed through store system information (STSI) enhancements on the IBM System z10 platform.
A CPU performance boost as high as 10 percent is possible depending on a workload’s characteristics. Very large, memory intensive workloads running on large numbers of physical processors (16 to 54) are most likely to achieve the highest performance gains.
IBM Tivoli OMEGAMON XE on z/OS version 4.1 with maintenance provides extensive statistics related to HiperDispatch through both the IBM Tivoli Enterprise Portal and 3270-based classic interfaces.
|
