Lone Eagle Systems: Evaluating Performance of Complex Systems

Evaluating Performance of Complex Systems

The demands on today's computing infrastructure is complex and rapidly-changing. Applications are now developed on `Internet time,' and the ways in which they are deployed and used change at an astonishing rate. The underlying computing systems that are developed and tuned for one service may quickly be pressed into use by another -- sometimes with disastrous consequences. The results of changing workloads make national news, with frequent Web site outages being due to quickly-changing demands and a lack of capacity to handle them.

The combination of rapid application deployment and significant consequences of inadequate capacity makes the need for performance characterization, improvement, and capacity planning ever more acute. Several benefits accrue to forward-looking organizations that establish performance characterization as part of their operations:

Application developers can use benchmarks to ensure that each software release achieves desired performance goals, as well as to provide a standard against which performance improvement efforts can be tracked.
System integrators and end users can use performance measurements in procurement decisions, capacity planning, improvement efforts, and in ensuring that completed configurations meet or exceed customer needs.
Hardware vendors find performance characterizations essential for differentiating systems within their own product lines, for tailoring configurations for specific applications, and of course for comparisons against competing vendors.

Using Representative Model Workloads

The key factor determining whether performance characterization efforts produce useful information or irrelevant data is whether the benchmarks are representative of the actual work that the system is to accomplish. Accurate performance characterizations depend on measuring a model workload that is derived from a system's real workload. This provides a measure of confidence that the performance of the performance benchmark will be a good predictor of real workload performance.

Insofar as standard benchmarks -- such as those developed by SPEC, TPC, and various third-party vendors -- are representative of real workloads, they can be used as predictors of real-world performance. More often than not, however, standard benchmarks fall short of being representative of workloads in actual use today. When the performance of complex systems must be characterized, there is no substitute for specific benchmarks that utilize representative workload models.

Performance Characterization Experience

Steve Gaede, Software Scientist with Lone Eagle^TM Systems, has been developing benchmarks based on representative model workloads for more than two decades. With a research background in workload characterization and modeling, combined with years of industry experience, Steve has a track record in narrowing in on the exactly the metrics to obtain, whether the issue is procurement, characterization, or performance improvement. Lone Eagle Systems has completed numerous performance evaluation and improvement and capacity planning projects that have enabled its clients to make business decisions founded on real performance data. Some example projects include:

One of the more ambitious studies conducted by Lone Eagle Systems was a performance characterization of digital video streaming over a trial broadband cable network. This project involved creating a harness which could invoke the load of hundreds of home customers with digital set-top boxes and monitor the quality of still and streaming video that the cable plant could produce. This project helped locate numerous bugs and bottlenecks that impaired system performance, and it resulted in a precise characterization of system response under different workloads -- enabling an accurate cost per user calculation for the video servers to be deployed.
Read more about Multimedia Server Performance Evaluation

Using standard benchmarks for X Window System Server performance, Lone Eagle Systems assisted a major UNIX workstation vendor in characterizing performance between two different system architectures, pinpointing areas where software improvements could reduce the differences between the two architectures, allowing customers more freedom to choose platforms for reasons other than graphics performance.

Some of Software Scientist Steve Gaede's earliest - and most well-known work - was developing a benchmark that measures the capacity of the UNIX® operating system to scale onto various platforms. Developed at Bell Laboratories in 1979, the benchmark applies a model workload at increasing levels of concurrency in order to quantify a system's maximum throughput, as well as obtaining qualitative measures of a system's stability. The benchmark has been so popular throughout the industry that it was included in the SPEC suite of benchmarks in 1991 as the Software Development Environment Throughput (SDET) benchmark. All of the major UNIX operating system vendors still use this benchmark in their nightly builds and regression tests; in its capacity as an Affiliate of SPEC, Lone Eagle Systems has worked with SPEC members on updated benchmarks using the same time-proven technique.
Read Perspectives on the SPEC SDET Benchmark.

The two decades of SDET use is testimony to the quality workload characterization, modeling, benchmark development, and characterization services that are available from Lone Eagle Systems. The usefulness of a 1979 software development workload model to evaluate performance of modern systems is limited, however, and Lone Eagle Systems advises using only up-to-date, representative workloads for evaluating the performance of today's complex systems.