12
for class on Tuesday Oct. 27, 1998
Please re-read Sections 2.6 through 2.9 of the Patterson & Hennessy text, together with the attached material from SPEC and be prepared to discuss the following:
The Standard Performance Evaluation Corporation (a.k.a. SPEC) is a group of computer manufacturers that agree on a set of standard benchmarks and on measurement methods for evaluating their machines. Execution times are compared with the times on a ``reference machine,'' which is now a certain SPARC workstation. Before this organization was formed (in the late 1980's) competing companies often used different benchmarks to tout their machines, and/or suspicious figures of merit like MIPS (not the company, the metric) and/or questionable ways of combining a set of benchmark numbers.
The SPEC Website, http://www.spec.org, contains a great deal of information and links, including benchmark results for a large number of machines. Attached to this sheet (if there's a staple in the upper lefthand corner) is an excerpt from a SPEC FAQ about the CPU95 benchmarks, together with four results pages from four different machines that were tested in 1998.
Manufacturers and customers rely on these figures to understand computer performance. What do you think of the SPEC approach to measuring and reporting performance? If you were a customer, how would you use the SPEC reports? If you were a manufacturer, could you take advantage of the SPEC evaluation to make your products look better? If you were in charge of SPEC, would you urge any changes to make SPEC's work more accurate and helpful?