| Running the Tests |
| ================= |
| |
| All the tests are executed using the "Run" script in the top-level directory. |
| |
| The simplest way to generate results is with the commmand: |
| ./Run |
| |
| This will run a standard "index" test (see "The BYTE Index" below), and |
| save the report in the "results" directory, with a filename like |
| hostname-2007-09-23-01 |
| An HTML version is also saved. |
| |
| If you want to generate both the basic system index and the graphics index, |
| then do: |
| ./Run gindex |
| |
| If your system has more than one CPU, the tests will be run twice -- once |
| with a single copy of each test running at once, and once with N copies, |
| where N is the number of CPUs. Some categories of tests, however (currently |
| the graphics tests) will only run with a single copy. |
| |
| Since the tests are based on constant time (variable work), a "system" |
| run usually takes about 29 minutes; the "graphics" part about 18 minutes. |
| A "gindex" run on a dual-core machine will do 2 "system" passes (single- |
| and dual-processing) and one "graphics" run, for a total around one and |
| a quarter hours. |
| |
| ============================================================================ |
| |
| Detailed Usage |
| ============== |
| |
| The Run script takes a number of options which you can use to customise a |
| test, and you can specify the names of the tests to run. The full usage |
| is: |
| |
| Run [ -q | -v ] [-i <n> ] [-c <n> [-c <n> ...]] [test ...] |
| |
| The option flags are: |
| |
| -q Run in quiet mode. |
| -v Run in verbose mode. |
| -i <count> Run <count> iterations for each test -- slower tests |
| use <count> / 3, but at least 1. Defaults to 10 (3 for |
| slow tests). |
| -c <n> Run <n> copies of each test in parallel. |
| |
| The -c option can be given multiple times; for example: |
| |
| ./Run -c 1 -c 4 |
| |
| will run a single-streamed pass, then a 4-streamed pass. Note that some |
| tests (currently the graphics tests) will only run in a single-streamed pass. |
| |
| The remaining non-flag arguments are taken to be the names of tests to run. |
| The default is to run "index". See "Tests" below. |
| |
| When running the tests, I do *not* recommend switching to single-user mode |
| ("init 1"). This seems to change the results in ways I don't understand, |
| and it's not realistic (unless your system will actually be running in this |
| mode, of course). However, if using a windowing system, you may want to |
| switch to a minimal window setup (for example, log in to a "twm" session), |
| so that randomly-churning background processes don't randomise the results |
| too much. This is particularly true for the graphics tests. |
| |
| |
| ============================================================================ |
| |
| Tests |
| ===== |
| |
| The available tests are organised into categories; when generating index |
| scores (see "The BYTE Index" below) the results for each category are |
| produced separately. The categories are: |
| |
| system The original Unix system tests (not all are actually |
| in the index) |
| 2d 2D graphics tests (not all are actually in the index) |
| 3d 3D graphics tests |
| misc Various non-indexed tests |
| |
| The following individual tests are available: |
| |
| system: |
| dhry2reg Dhrystone 2 using register variables |
| whetstone-double Double-Precision Whetstone |
| syscall System Call Overhead |
| pipe Pipe Throughput |
| context1 Pipe-based Context Switching |
| spawn Process Creation |
| execl Execl Throughput |
| fstime-w File Write 1024 bufsize 2000 maxblocks |
| fstime-r File Read 1024 bufsize 2000 maxblocks |
| fstime File Copy 1024 bufsize 2000 maxblocks |
| fsbuffer-w File Write 256 bufsize 500 maxblocks |
| fsbuffer-r File Read 256 bufsize 500 maxblocks |
| fsbuffer File Copy 256 bufsize 500 maxblocks |
| fsdisk-w File Write 4096 bufsize 8000 maxblocks |
| fsdisk-r File Read 4096 bufsize 8000 maxblocks |
| fsdisk File Copy 4096 bufsize 8000 maxblocks |
| shell1 Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1") |
| shell8 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8") |
| shell16 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16") |
| |
| 2d: |
| 2d-rects 2D graphics: rectangles |
| 2d-lines 2D graphics: lines |
| 2d-circle 2D graphics: circles |
| 2d-ellipse 2D graphics: ellipses |
| 2d-shapes 2D graphics: polygons |
| 2d-aashapes 2D graphics: aa polygons |
| 2d-polys 2D graphics: complex polygons |
| 2d-text 2D graphics: text |
| 2d-blit 2D graphics: images and blits |
| 2d-window 2D graphics: windows |
| |
| 3d: |
| ubgears 3D graphics: gears |
| |
| misc: |
| C C Compiler Throughput ("looper 60 $cCompiler cctest.c") |
| arithoh Arithoh (huh?) |
| short Arithmetic Test (short) (this is arith.c configured for |
| "short" variables; ditto for the ones below) |
| int Arithmetic Test (int) |
| long Arithmetic Test (long) |
| float Arithmetic Test (float) |
| double Arithmetic Test (double) |
| dc Dc: sqrt(2) to 99 decimal places (runs |
| "looper 30 dc < dc.dat", using your system's copy of "dc") |
| hanoi Recursion Test -- Tower of Hanoi |
| grep Grep for a string in a large file, using your system's |
| copy of "grep" |
| sysexec Exercise fork() and exec(). |
| |
| The following pseudo-test names are aliases for combinations of other |
| tests: |
| |
| arithmetic Runs arithoh, short, int, long, float, double, |
| and whetstone-double |
| dhry Alias for dhry2reg |
| dhrystone Alias for dhry2reg |
| whets Alias for whetstone-double |
| whetstone Alias for whetstone-double |
| load Runs shell1, shell8, and shell16 |
| misc Runs C, dc, and hanoi |
| speed Runs the arithmetic and system groups |
| oldsystem Runs execl, fstime, fsbuffer, fsdisk, pipe, context1, |
| spawn, and syscall |
| system Runs oldsystem plus shell1, shell8, and shell16 |
| fs Runs fstime-w, fstime-r, fstime, fsbuffer-w, |
| fsbuffer-r, fsbuffer, fsdisk-w, fsdisk-r, and fsdisk |
| shell Runs shell1, shell8, and shell16 |
| |
| index Runs the tests which constitute the official index: |
| the oldsystem group, plus dhry2reg, whetstone-double, |
| shell1, and shell8 |
| See "The BYTE Index" below for more information. |
| graphics Runs the tests which constitute the graphics index: |
| 2d-rects, 2d-ellipse, 2d-aashapes, 2d-text, 2d-blit, |
| 2d-window, and ubgears |
| gindex Runs the index and graphics groups, to generate both |
| sets of index results |
| |
| all Runs all tests |
| |
| |
| ============================================================================ |
| |
| The BYTE Index |
| ============== |
| |
| The purpose of this test is to provide a basic indicator of the performance |
| of a Unix-like system; hence, multiple tests are used to test various |
| aspects of the system's performance. These test results are then compared |
| to the scores from a baseline system to produce an index value, which is |
| generally easier to handle than the raw sores. The entire set of index |
| values is then combined to make an overall index for the system. |
| |
| Since 1995, the baseline system has been "George", a SPARCstation 20-61 |
| with 128 MB RAM, a SPARC Storage Array, and Solaris 2.3, whose ratings |
| were set at 10.0. (So a system which scores 520 is 52 times faster than |
| this machine.) Since the numbers are really only useful in a relative |
| sense, there's no particular reason to update the base system, so for the |
| sake of consistency it's probably best to leave it alone. George's scores |
| are in the file "pgms/index.base"; this file is used to calculate the |
| index scores for any particular run. |
| |
| Over the years, various changes have been made to the set of tests in the |
| index. Although there is a desire for a consistent baseline, various tests |
| have been determined to be misleading, and have been removed; and a few |
| alternatives have been added. These changes are detailed in the README, |
| and should be born in mind when looking at old scores. |
| |
| A number of tests are included in the benchmark suite which are not part of |
| the index, for various reasons; these tests can of course be run manually. |
| See "Tests" above. |
| |
| |
| ============================================================================ |
| |
| Graphics Tests |
| ============== |
| |
| As of version 5.1, UnixBench now contains some graphics benchmarks. These |
| are intended to give a rough idea of the general graphics performance of |
| a system. |
| |
| The graphics tests are in categories "2d" and "3d", so the index scores |
| for these tests are separate from the basic system index. This seems |
| like a sensible division, since the graphics performance of a system |
| depends largely on the graphics adaptor. |
| |
| The tests currently consist of some 2D "x11perf" tests and "ubgears". |
| |
| * The 2D tests are a selection of the x11perf tests, using the host |
| system's x11perf command (which must be installed and in the search |
| path). Only a few of the x11perf tests are used, in the interests |
| of completing a test run in a reasonable time; if you want to do |
| detailed diagnosis of an X server or graphics chip, then use x11perf |
| directly. |
| |
| * The 3D test is "ubgears", a modified version of the familiar "glxgears". |
| This version runs for 5 seconds to "warm up", then performs a timed |
| run and displays the average frames-per-second. |
| |
| On multi-CPU systems, the graphics tests will only run in single-processing |
| mode. This is because the meaning of running two copies of a test at once |
| is dubious; and the test windows tend to overlay each other, meaning that |
| the window behind isn't actually doing any work. |
| |
| |
| ============================================================================ |
| |
| Multiple CPUs |
| ============= |
| |
| If your system has multiple CPUs, the default behaviour is to run the selected |
| tests twice -- once with one copy of each test program running at a time, |
| and once with N copies, where N is the number of CPUs. (You can override |
| this with the "-c" option; see "Detailed Usage" above.) This is designed to |
| allow you to assess: |
| |
| - the performance of your system when running a single task |
| - the performance of your system when running multiple tasks |
| - the gain from your system's implementation of parallel processing |
| |
| The results, however, need to be handled with care. Here are the results |
| of two runs on a dual-processor system, one in single-processing mode, one |
| dual-processing: |
| |
| Test Single Dual Gain |
| -------------------- ------ ------ ---- |
| Dhrystone 2 562.5 1110.3 97% |
| Double Whetstone 320.0 640.4 100% |
| Execl Throughput 450.4 880.3 95% |
| File Copy 1024 759.4 595.9 -22% |
| File Copy 256 535.8 438.8 -18% |
| File Copy 4096 1261.8 1043.4 -17% |
| Pipe Throughput 481.0 979.3 104% |
| Pipe-based Switching 326.8 1229.0 276% |
| Process Creation 917.2 1714.1 87% |
| Shell Scripts (1) 1064.9 1566.3 47% |
| Shell Scripts (8) 1567.7 1709.9 9% |
| System Call Overhead 944.2 1445.5 53% |
| -------------------- ------ ------ ---- |
| Index Score: 678.2 1026.2 51% |
| |
| As expected, the heavily CPU-dependent tasks -- dhrystone, whetstone, |
| execl, pipe throughput, process creation -- show close to 100% gain when |
| running 2 copies in parallel. |
| |
| The Pipe-based Context Switching test measures context switching overhead |
| by sending messages back and forth between 2 processes. I don't know why |
| it shows such a huge gain with 2 copies (ie. 4 processes total) running, |
| but it seems to be consistent on my system. I think this may be an issue |
| with the SMP implementation. |
| |
| The System Call Overhead shows a lesser gain, presumably because it uses a |
| lot of CPU time in single-threaded kernel code. The shell scripts test with |
| 8 concurrent processes shows no gain -- because the test itself runs 8 |
| scripts in parallel, it's already using both CPUs, even when the benchmark |
| is run in single-stream mode. The same test with one process per copy |
| shows a real gain. |
| |
| The filesystem throughput tests show a loss, instead of a gain, when |
| multi-processing. That there's no gain is to be expected, since the tests |
| are presumably constrained by the throughput of the I/O subsystem and the |
| disk drive itself; the drop in performance is presumably down to the |
| increased contention for resources, and perhaps greater disk head movement. |
| |
| So what tests should you use, how many copies should you run, and how should |
| you interpret the results? Well, that's up to you, since it depends on |
| what it is you're trying to measure. |
| |
| Implementation |
| -------------- |
| |
| The multi-processing mode is implemented at the level of test iterations. |
| During each iteration of a test, N slave processes are started using fork(). |
| Each of these slaves executes the test program using fork() and exec(), |
| reads and stores the entire output, times the run, and prints all the |
| results to a pipe. The Run script reads the pipes for each of the slaves |
| in turn to get the results and times. The scores are added, and the times |
| averaged. |
| |
| The result is that each test program has N copies running at once. They |
| should all finish at around the same time, since they run for constant time. |
| |
| If a test program itself starts off K multiple processes (as with the shell8 |
| test), then the effect will be that there are N * K processes running at |
| once. This is probably not very useful for testing multi-CPU performance. |
| |
| |
| ============================================================================ |
| |
| The Language Setting |
| ==================== |
| |
| The $LANG environment variable determines how programs abnd library |
| routines interpret text. This can have a big impact on the test results. |
| |
| If $LANG is set to POSIX, or is left unset, text is treated as ASCII; if |
| it is set to en_US.UTF-8, foir example, then text is treated as being |
| encoded in UTF-8, which is more complex and therefore slower. Setting |
| it to other languages can have varying results. |
| |
| To ensure consistency between test runs, the Run script now (as of version |
| 5.1.1) sets $LANG to "en_US.utf8". |
| |
| This setting which is configured with the variable "$language". You |
| should not change this if you want to share your results to allow |
| comparisons between systems; however, you may want to change it to see |
| how different language settings affect performance. |
| |
| Each test report now includes the language settings in use. The reported |
| language is what is set in $LANG, and is not necessarily supported by the |
| system; but we also report the character mapping and collation order which |
| are actually in use (as reported by "locale"). |
| |
| |
| ============================================================================ |
| |
| Interpreting the Results |
| ======================== |
| |
| Interpreting the results of these tests is tricky, and totally depends on |
| what you're trying to measure. |
| |
| For example, are you trying to measure how fast your CPU is? Or how good |
| your compiler is? Because these tests are all recompiled using your host |
| system's compiler, the performance of the compiler will inevitably impact |
| the performance of the tests. Is this a problem? If you're choosing a |
| system, you probably care about its overall speed, which may well depend |
| on how good its compiler is; so including that in the test results may be |
| the right answer. But you may want to ensure that the right compiler is |
| used to build the tests. |
| |
| On the other hand, with the vast majority of Unix systems being x86 / PC |
| compatibles, running Linux and the GNU C compiler, the results will tend |
| to be more dependent on the hardware; but the versions of the compiler and |
| OS can make a big difference. (I measured a 50% gain between SUSE 10.1 |
| and OpenSUSE 10.2 on the same machine.) So you may want to make sure that |
| all your test systems are running the same version of the OS; or at least |
| publish the OS and compuiler versions with your results. Then again, it may |
| be compiler performance that you're interested in. |
| |
| The C test is very dubious -- it tests the speed of compilation. If you're |
| running the exact same compiler on each system, OK; but otherwise, the |
| results should probably be discarded. A slower compilation doesn't say |
| anything about the speed of your system, since the compiler may simply be |
| spending more time to super-optimise the code, which would actually make it |
| faster. |
| |
| This will be particularly true on architectures like IA-64 (Itanium etc.) |
| where the compiler spends huge amounts of effort scheduling instructions |
| to run in parallel, with a resultant significant gain in execution speed. |
| |
| Some tests are even more dubious in terms of host-dependency -- for example, |
| the "dc" test uses the host's version of dc (a calculator program). The |
| version of this which is available can make a huge difference to the score, |
| which is why it's not in the index group. Read through the release notes |
| for more on these kinds of issues. |
| |
| Another age-old issue is that of the benchmarks being too trivial to be |
| meaningful. With compilers getting ever smarter, and performing more |
| wide-ranging flow path analyses, the danger of parts of the benchmarks |
| simply being optimised out of existance is always present. |
| |
| All in all, the "index" and "gindex" tests (see above) are designed to |
| give a reasonable measure of overall system performance; but the results |
| of any test run should always be used with care. |
| |
