International Business Machines

Computer systems are becoming increasingly complex both from hardware and software perspectives. Simulation is the defacto method of performance analysis in the microprocessor and computer system design community, however, often takes days, weeks and sometimes months to simulate one configuration. Analytical performance modeling, synthetic benchmarks/traces all traditionally lack credibility since it is extremely hard to create and validate models for complex machines/complex workloads. Detailed full-system simulation with real-world workloads has been able to achieve credibility, however, the very large simulation times taken by this methodology has started to prohibit good design space exploration. More research into alternate performance evaluation methodologies needs to be done for effective design and evaluation of next generation computing systems.

The proposed research seeks to investigate mathematical and formal techniques that can help the performance evaluation of next generation computers. It involves exploring the existing body of knowledge in mathematical and statistical theory to manage the complexity of evaluation of complex future systems and adapting theories and techniques and developing new techniques to effectively handle evaluation of emerging and future systems.

Typically tomorrow's computers are designed on the basis of evaluations with benchmarks of today, which are programs from yesterday. While this has been recognized as a problem, it has been hard to solve the problem. Finding benchmarks that are representative of existing applications is hard enough a problem, that developing benchmarks that represent future workloads that have not yet arrived is next to impossible. Representativeness has been hard to quantify and establish.

Simulation is the defacto method of performance analysis in microprocessor design, but detailed microarchitectural simulators and full system simulators are taking prohibitively large simulation times (days, weeks and sometimes months). This limits the design choices that can be evaluated. Hence it is important to be able to capture important features of the workloads and create manageable benchmark suites. If benchmarks were short snippets of programs or traces, elaborate design space exploration can be performed, whereas long-running benchmarks will limit the choices that can be studied. . It will be useful and interesting to obtain miniature traces with the same performance and power characteristics as TPC or SPEC traces.

The objective of the proposed project is three-fold. Firstly, we will develop a methodology to generate reduced traces/benchmarks which are representative of current benchmarks. We will focus on finding small synthetic workloads for power studies and performance evaluation, with the primary purpose of increasing ability to perform extensive design space explorations. Secondly, we will focus on identifying redundancy/similarity in benchmark suites and will generate reduced suites. This will also improve our ability to perform increased power and performance evaluations. Thirdly, we will develop a methodology to speculatively generate/synthesize workloads of the future based on the applications that are just emerging.

IBM provided equipment to support a cooperative research program between IBM's eServer xSeries development organization and the Electrical and Computer Engineering Department at the University of Texas at Austin. The research will use an IBM xSeries 440 4-way SMP as the host for both measurement of execution behavior of workloads at various levels of resolution and for execution of simulators of computer system architectures.

The measurements will be conducted using several different and complementary measurement packages including those that may be supplied by IBM and/or their collaborators. Compilers available in the public domain or those supplied by IBM collaborators will be used. The workloads characterized will include technical as well as commercial workloads.

Typically tomorrow's computers are designed on the basis of evaluations with benchmarks of today, which are programs from yesterday. While this has been recognized as a problem, it has been hard to solve the problem. Finding benchmarks that are representative of existing applications is hard enough a problem, that developing benchmarks that represent future workloads that have not yet arrived is next to impossible. Representativeness has been hard to quantify and establish.

The objective of the proposed project is to develop a methodology to speculatively generate/synthesize workloads of the future based on the applications that are just emerging. For instance, the e-business field is constantly evolving. If we do not predict these workloads and understand their features and characteristics, computer system designers could be caught by surprise after spending millions or billions of dollars on designing the next generation computers.

The basic methodology that can be applied to find a solution to this problem involves understanding the parameters that define a workload, understanding the values of these parameters for modern workloads and then extrapolating into workload predictions of the future. The generated future workloads need not be completely synthesized and artificial; they can be real workloads with adjustable 'knobs' with which many of the workload characteristics can be modified according to predictions.

The objective of the proposed project will be to determine the usefulness of aggressive out-of-order scheduling in the IBM PPC processors. The study will utilize the RS6400 II, PowerPC 604e and PowerIII processors. The performance counters in the respective processors will be used to conduct the measurements. The three aforementioned processors cover the spectrum of PPC machines from in-order to aggressively out of order and speculative. In particular, effort would focus on using the performance counters to identify hardware or software inefficiencies, measuring and identifying components of processor performance (CPI), and studying the effects of speculative and out-of-order execution on resource usage and overall performance. Another objective of the research would be to comment on the performance monitoring facility of the processors. The deliverable from the first phase of the project would be a report on the impact of aggressive dynamic scheduling on web server workloads.

The path to improved computer architectures is via monitoring and characterizing existing architectures, identifying bottlenecks and alleviating the bottlenecks. Performance Monitoring of real world systems running full-blown real world applications can yield a variety of valuable information that simulation studies cannot yield. Simulation is very slow (often 100 to 1000 times) and hence limits the size of applications studied. Moreover, contemporary applications involving Java and database systems are extremely difficult to run on simulation environments.

This project consists of performance monitoring counter measurements on the IBM PowerPC processors by professors at the University of Texas at Austin and UT San Antonio. Profiling of program execution will be done for studying performance and energy consumption issues. The machines will be used for research, graduate and undergraduate course projects, and senior design projects.