Giter Club home page Giter Club logo

test-profiles's Introduction

Phoronix Test Suite 10.8.5

Phoronix Test Suite

The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available for Linux, Solaris, macOS, Windows, and BSD operating systems. The Phoronix Test Suite allows for carrying out tests in a fully automated manner from test installation to execution and reporting. All tests are meant to be easily reproducible, easy-to-use, and support fully automated execution. The Phoronix Test Suite is open-source under the GNU GPLv3 license and is developed by Phoronix Media in cooperation with partners.

Phoronix Test Suite Documentation

The Phoronix Test Suite itself is an open-source framework for conducting automated tests along with reporting of test results, detection of installed system software/hardware, and other features. This framework is designed to be an extensible architecture so that new test profiles and suites can be easily added to represent performance benchmarks, unit tests, and other quantitative and qualitative (e.g. image quality comparison and pass/fail) measurements. Available through OpenBenchmarking.org, a collaborative storage platform developed in conjunction with the Phoronix Test Suite, are more than 600 individual test profiles and more than 200 test suites available by default from the Phoronix Test Suite. Independent users are also able to upload their test results, profiles, and suites to OpenBenchmarking.org.

A test profile is a single test that can be executed by the Phoronix Test Suite -- with a series of options possible within every test -- and a test suite is a seamless collection of test profiles and/or additional test suites. A test profile consists of a set of Bash/shell scripts and XML files while a test suite is a single XML file. Modules for the Phoronix Test Suite also allow for integration with git-bisect and other revision control systems for per-commit regression testing, system sensor monitoring, and other extras.

OpenBenchmarking.org also allows for conducting side-by-side result comparisons, a central location for storing and sharing test results, and collaborating over test data. Phoromatic is a complementary platform to OpenBenchmarking.org and the Phoronix Test Suite for interfacing with Phoronix Test Suite client(s) to automatically execute test runs on a timed, per-commit, or other trigger-driven basis. Phoromatic is designed for enterprise and allows for the easy management of multiple networked systems running Phoronix Test Suite clients via a single web-based interface.

Professional support and custom engineering for the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org is available by contacting https://www.phoronix-test-suite.com/.

Full details on the Phoronix Test Suite setup and usage is available from the included HTML/PDF documentation within the phoronix-test-suite package and from the Phoronix Test Suite documentation directory.

Installation & Setup

The Phoronix Test Suite is supported on Linux, *BSD, Solaris, macOS, and Windows systems. However, the most full-featured and well supported operating system for conducting the tests is Linux with some non-basic functionality not being available under all platforms. The Phoronix Test Suite software/framework is compatible with all major CPU architectures (e.g. i686, x86_64, ARM / AArch64, RISC-V, POWER 64-bit), but not all of the test profiles/suites are compatible with all architectures.

The Phoronix Test Suite can be installed for system-wide usage or run locally without installation from the extracted tar.gz/zip package. The only hard dependency on the Phoronix Test Suite is having command-line support for PHP (PHP 5.3+) installed. A complete PHP stack (e.g. with web server) is not needed, but merely the PHP command-line support, which is widely available from operating system package managers under the name php, php7-cli, php5-cli, php7 or php8.

Usage

The process to download, install/setup, execute, and report the results of a benchmark can be as simple as a command such as phoronix-test-suite benchmark smallpt to run a simple CPU test profile. If wishing to simply install a test, it's a matter of running phoronix-test-suite install <test or suite name> and to run it's phoronix-test-suite run <test or suite name>. There's also a batch mode for non-interactive benchmarking by first running phoronix-test-suite batch-setup and then using the batch-run sub-command rather than run. For more guidance see the Phoronix Test Suite documentation.

Viewing installed system hardware and software is available via phoronix-test-suite system-info.

Facilitating a result comparison from OpenBenchmarking.org can be done by running, for example, phoronix-test-suite benchmark 1204293-BY-PHORONIX357 if wishing to compare the results of the https://openbenchmarking.org/result/1204293-BY-PHORONIX357 result file.

test-profiles's People

Contributors

ardickerson64 avatar chenming1986 avatar dborquez avatar ewouth avatar hakzsam avatar jwyatt-feral avatar kagami avatar kimwalisch avatar liam-middlebrook avatar loader-bsd avatar m87carlson avatar markkrj avatar michaellarabel avatar nealeu avatar swarth100 avatar victorrodriguez avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

konserw feralinteractive fenrus75 hellvesper simar0at tkuennen nealeu lsandov1 luctrudeau rossburton kimwalisch busykai dborquez aduenasd loader-bsd m87carlson liam-middlebrook kagami ossdev07 ewouth spacefish fjudith stefanos1316 chenming1986 dapeng-mi cheeyanglee rakeshmadivi ardickerson64 guowangy adam-dej jjayl mayujsw dalejung sbakir kangshan0910 victorrodriguez guangli0510 garwynn mo5h deepfire colmharr gggk jkwmoore oneoa jonsneyers wubian6688 zaunerc nihui natusameer wozard andrepuschmann zorro-fr24 yadro-kns zmike kangshan1157 forrest-ly skeevy420 lfsu-ali nhubbard 0lvin adeptg denkertm thaddeusc1 daandemeyer nettrino filipecosta90 kotarou3 jiebinn intelmy lwandrebeck tad4 colinianking lucic71 adrianheine dbernhard-0x7cd casualwind sm9cc

test-profiles's Issues

pts/x265 -- Current verison of source tar file won't compile in POWER9 systems

Using current tar ball for benchmark:

(base) ubuntu@opzaius1:~/.phoronix-test-suite/installed-tests/local/x265-1.0.0/x265_2.8/build$ make -j 8
Scanning dependencies of target encoder
Scanning dependencies of target common
[  6%] Building CXX object encoder/CMakeFiles/encoder.dir/motion.cpp.o
[  6%] Building CXX object encoder/CMakeFiles/encoder.dir/analysis.cpp.o
...
[ 65%] Building CXX object common/CMakeFiles/common.dir/param.cpp.o
/home/ubuntu/.phoronix-test-suite/installed-tests/local/x265-1.0.0/x265_2.8/source/common/param.cpp: In function âint x265_param_parse(x265_param*, const char*, const char*)â:
/home/ubuntu/.phoronix-test-suite/installed-tests/local/x265-1.0.0/x265_2.8/source/common/param.cpp:636:50: error: too few arguments to function âint x265::parseCpuName(const char*, bool&, bool)â
             p->cpuid = parseCpuName(value, bError);
                                                  ^
In file included from /home/ubuntu/.phoronix-test-suite/installed-tests/local/x265-1.0.0/x265_2.8/source/common/param.cpp:28:0:
/home/ubuntu/.phoronix-test-suite/installed-tests/local/x265-1.0.0/x265_2.8/source/common/param.h:36:7: note: declared here
 int   parseCpuName(const char *value, bool& bError, bool bEnableavx512);
       ^~~~~~~~~~~~
common/CMakeFiles/common.dir/build.make:614: recipe for target 'common/CMakeFiles/common.dir/param.cpp.o' failed
make[2]: *** [common/CMakeFiles/common.dir/param.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....

Chaging to latest stable release (x265_3.0.tar.gz) from https://bitbucket.org/multicoreware/x265/downloads/ seems to circumvent issue.

pts/ramspeed-1.4.2: PowerPC compilation fails

In a POWER9 system I see the following:

pts/ramspeed-1.4.2: PowerPC compilation fails 
    Test Installation 39 of 39
    1 File Needed [0.08 MB / 1 Minute]
    Downloading: ramsmp-3.5.0.tar.gz                                                                           [0.08MB]
    Estimated Download Time: 1m .......................................................................................
    Installation Size: 0.72 MB                                                                                         
    Installing Test @ 14:54:46                                                                                         
        The installer exited with a non-zero exit status.                                                              
        ERROR: gcc: error: unrecognized command line option â-march=nativeâ; did you mean â-mcpu=nativeâ?
        LOG: ~/.phoronix-test-suite/installed-tests/pts/ramspeed-1.4.2/install-failed.log

On PowerPC -mcpu=native -mtune=native must be used instead.
This is related to phoronix-test-suite/phoronix-test-suite#333

Possible Tests To Explore For Packaging For PTS

(Just moving the list here rather than in Git tree for more exposure and easier to maintain.)

POSSIBLE NEW TEST PROFILES (random order)

pts/unigine-super-1.0.4 fails installation because of checksum mismatch

I believe that Unigine may have silently changed the files they're hosting at http://sp-assets.unigine.com/d/Unigine_Superposition-1.0.run (which gets redirected to a mirror like m11-assets.unigine.com).

The test profile contains the following checksums for Unigine_Superposition-1.0.run:

  • MD5: 51f6fd0adf388d180dd3fede476f4b04
  • SHA256: c46c87b74dae78cfcd6b4cdb14c53077ea3d3c6d88774b67abaa4994295b936d

However if I manually download the Unigine_Superposition-1.0.run from the link included in the test profile I get:

  • MD5: 991127f472facfd3f75a25c968138974
  • SHA256: ba828e48ae1082dc5734dc584df7e2d91f56e1c48edb9708cd24c42f6e5d8a32

For reference the internal sums listed within the .run file are:

  • CRC: 3004177712
  • MD5: 5f003973696172ef6df518d33e26f149

pts/tensorflow: missing dependency

test-profiles/pts/tensorflow-1.0.0/install.sh

Line 11 in 0cb9662

pip3 install tensorflow

python3-pip is not a required dependency, so it is not installed (Ubuntu 16.04). In addition, the output of pip3 install tensorflow is not checked, so it silently fails, but the test is considered installed, so one must (1) install python3-pip, (2) remove-installed-test tensorflow, (3) install tensorflow again, which is a hassle.

The command "phoronix-test-suite dump-tests-to-git" should be run automatically after an (official) change to test profiles on OpenBenchmarking.org

The desired behaviour is that this git repository is kept in sync with OpenBenchmarking.org as good as possible.
This is currently not the observed behaviour (for proof, see below), that's why I created this issue report.
The solution to this problem is provided in the title of this issue report.

Currently this git repository is out of sync with OpenBenchmarking.org by at least 26 days.
Proof:
This issue report was written on December 21st 2017.
According to https://openbenchmarking.org/test/pts/supertuxkart, the latest version of the supertuxkart test profile is "supertuxkart-1.4.0", added on November 25th 2017.
The last commit at the moment this issue report was written was 0cb9662: November 7th 2017, which means "supertuxkart-1.4.0" is not in the repository on that branch, this can be verified: https://github.com/phoronix-test-suite/test-profiles/tree/0cb966241d99bf2040ec166b9190152be17280d2/pts/supertuxkart-1.4.0 says "Page not found"

This should be fixed in the way as mentioned in the title (or better), i.e., this process must be done automatically, not manually.
This will prevent other people from frustrations in the future.

glmark2-1.0.2 can not be installed on Ubuntu 17.04

when installing glmark2-1.0.2 on Ubuntu 17.04 a number of errors occur.

These Issues I manually corrected in my local glmark2-rev276.tar.bz2 and then the installation was successful:
check C libraries using the C compiler in wscript
glmark2/glmark2@88bbc83

Support libpng 1.6 in wscript (libpng12-0 v1.2.54-1ubuntu1 on my machine was not detected)
glmark2/glmark2@499aa81

Replace the use of the deprecated auto_ptr with unique_ptr in src/image-reader.cpp, src/model.cpp, src/scene-jellyfish.cpp and src/libmatrix/shader-source.cc
glmark2/glmark2@3137e5c

Make sure to include cstring for 'memcpy' declaration in src/image-reader.cpp
glmark2/glmark2@5584822

C++11 requires a space between literal and string macro
in src/native-state-x11.cpp
glmark2/glmark2@1ac261a
in src/native-state-mir.cpp
glmark2/glmark2@a8d0f41

pts/openssl -- Using upstream code shows up to 75% improvement in POWER9 system 

Phoronix Test Suite v8.2.0m1

    Installed:     pts/openssl-1.10.0

System Information

  PROCESSOR:          POWER9 altivec supported @ 3.80GHz
    Core Count:       96                                
    Scaling Driver:   powernv-cpufreq performance       

  GRAPHICS:           ASPEED ASPEED Family
    Screen:           1024x768            

  MOTHERBOARD:        PowerNV 0000000000000000
    Network:          Broadcom Limited NetXtreme BCM5719 Gigabit PCIe

  MEMORY:             262144MB

  DISK:               960GB MZ7LM960HCHP-000
    File-System:      ext4                  
    Mount Options:    data=ordered errors=remount-ro relatime rw
    Disk Scheduler:   CFQ                                       

  OPERATING SYSTEM:   Ubuntu 18.04
    Kernel:           4.15.0-33-generic (ppc64le)
    Compiler:         GCC 8.0.1 20180414

    Would you like to save these test results (Y/n): n

OpenSSL 1.1.0f:
    pts/openssl-1.10.0
    Test 1 of 1
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 1 Minute [13:33 CDT]
        Started Run 1 @ 13:32:39
        Started Run 2 @ 13:33:01
        Started Run 3 @ 13:33:22

    RSA 4096-bit Performance:
        1977.5
        1977.4
        1989.6

    Average: 1981.50 Signs Per Second
    Deviation: 0.35%

Compiling latest code from git repository , https://github.com/openssl/openssl.git

OpenSSL 1.1.2-dev  xx XXX xxxx
built on: Tue Sep 11 18:38:52 2018 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DAES_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG
                  sign    verify    sign/s verify/s
rsa 4096 bits 0.000286s 0.000004s   3496.8 249940.2

Issues with pts/parboil-1.2.0 on Arch Linux

Hello,

Hopefully this is the right place to report this!

I've attempted to run some of the pts/parboil-1.2.0 tests but I keep experiencing errors.

I've run: phoronix-test-suite debug-benchmark parboil

and it generates the following command:

cd /home//.phoronix-test-suite/installed-tests/pts/parboil-1.2.0/ && ./parboil lbm omp_cpu long 2>&1

It then errors with the following after appearing to do a successful run

Traceback (most recent call last):
  File "tools/compare-output", line 8, in <module>
    import filecompare as fc
  File "/home/<login>/.phoronix-test-suite/installed-tests/pts/parboil-1.2.0/parboil-2.5-tree/common/python/filecompare.py", line 11
    raise TypeError, "Not a CompareMonad instance"
                   ^
SyntaxError: invalid syntax
Parboil parallel benchmark suite, version 0.2

Output checking tool detected a mismatch

The test quit with a non-zero exit status.


E: SyntaxError: invalid syntax

I'm wondering if this is some sort of python 2 vs 3 issue as with some of the other tests I get:

  File "tools/compare-output", line 12
    print "Pass"
               ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Pass")?
Parboil parallel benchmark suite, version 0.2

Output checking tool detected a mismatch

The test quit with a non-zero exit status.


E: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Pass")?

On my system this is how python seems to resolve:

➜  .phoronix-test-suite python2 --version
Python 2.7.15
➜  .phoronix-test-suite python3 --version
Python 3.7.1
➜  .phoronix-test-suite python --version
Python 3.7.1

iperf produce incorrect result

location: ~/.phoronix-test-suite/test-profiles/pts/iperf-1.0.2/results-definition.xml

iperf can not produce a correct result for these following options.

<Description>Duration: 10 Seconds - Test: TCP - Parallel: 10</Description>

<Description>Duration: 10 Seconds - Test: UDP - Parallel: 5</Description>

<Description>Duration: 10 Seconds - Test: UDP - Parallel: 10</Description>

<Description>Duration: 10 Seconds - Test: UDP - Parallel: 20</Description>

<Description>Duration: 10 Seconds - Test: UDP - 100Mbit Objective - Parallel: 5</Description>

<Description>Duration: 10 Seconds - Test: UDP - 100Mbit Objective - Parallel: 10</Description>

The only way to get the correnct result of 6 indicators is to delete the first two options in ~/.phoronix-test-suite/test-profiles/pts/iperf-1.0.2/results-definition.xml

pts/mysqlslap

Both mysql_install_db and mysqld_safe should have --no-defaults as the first argument to prevent any installed mysql configuration file settings taking effect for the test.

minor: 10.3.9 is out too

mkl-dnn: wrong parameters

Hi! First of all, thanks for packaging this. I noticed a few issues:

  1. There is a bug in tests-definitions.xml for the mkl-dnn profile: the resulting benchdnn command line is incorrect.

Currently produced command line:

benchdnn --conv --batch=inputs/conv_alexnet --cfg=u8s8s32

this will actually run f32 configuration because parameters are positional.

Correct command line is

benchdnn --conv --cfg=u8s8s32 --batch=inputs/conv_alexnet

This is one of the reasons why in your Skylake vs Cascadelake testing you see no difference between the configurations.

  1. Affinity is not set up in install.sh. The most portable setting for Linux compilers is probably via
export OMP_PLACES=cores
export OMP_PROC_BIND=close

Additionally, it would probably best to test just a single socket, but this is up to you guys to decide.

iperf version 3.1.3 can't figure out the maximum UDP throughput when Linux kernel adopts fair-queue Qdisc policy

Currently phoronix adopts the iperf 3.1.3 version to run the iperf test. While there is an issue in iperf 3.1.3 version. Iperf 3.1.3 would apply the application-level (user space) and fair-queuing-based (kernel space) pacing rate simultaneously base on the configured pumping rate (argument -b) for UDP stream.

Iperf 3.1.3 would call the syscall "setsockopt(s, SOL_SOCKET, SO_MAX_PACING_RATE, &rate, sizeof(rate))" to set the pacing rate into kernel fair-queue scheduler. fair-queue scheduler would calculate the maximum transmitting packets limit per second base on this pacing rate. While the calculated packets limit is a little bit smaller than the idle value because the division is used and the decimal fraction is lost in the calculation.

A example running 100M pumping rate UDP stream test on 1000M Ethernet interface. The test distro is Clear Linux and the default qdisc policy is fair-queue (fq).

$ sysctl net.core.default_qdisc
net.core.default_qdisc = fq

iperf 3.1.3 test results:

dmi@dmi-i9-test~/.phoronix-test-suite/installed-tests/pts/iperf-1.0.3/iperf-install/bin $ ./iperf3 -c 10.239.159.127 -p 8000 -t 10 -u -b 100M -P 1 -f m -i 0 -l 8192
Connecting to host 10.239.159.127, port 8000
[ 4] local 10.239.159.149 port 37766 connected to 10.239.159.127 port 8000
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-10.00 sec 115 MBytes 96.6 Mbits/sec 14736

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 115 MBytes 96.6 Mbits/sec 0.102 ms 0/14726 (0%)
[ 4] Sent 14726 datagrams

iperf Done.

we can see the UDP throughput is 96.6Mbps which is a little bit lower than the pumping rate 100Mbps.

Iperf version 3.1.5 has fixed the issue and won't set the fair-queuing-based (kernel space) pacing rate any more base on the configured pumping rate.

iperf 3.1.5 test result on same test case:

dmi@dmi-i9-test~/Downloads/iperf-3.1.5/install/bin $ ./iperf3 -c 10.239.159.127 -p 8000 -t 10 -u -b 100M -P 1 -f m -i 0 -l 8192
Warning: UDP block size 8192 exceeds TCP MSS 1448, may result in fragmentation / drops
Connecting to host 10.239.159.127, port 8000
[ 4] local 10.239.159.149 port 49987 connected to 10.239.159.127 port 8000
[ ID] Interval Transfer Bandwidth Total Datagrams
[ 4] 0.00-10.00 sec 118 MBytes 99.1 Mbits/sec 15123

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 4] 0.00-10.00 sec 118 MBytes 99.1 Mbits/sec 0.103 ms 0/15123 (0%)
[ 4] Sent 15123 datagrams

iperf Done.

we can see iperf 3.1.5 can figure out the more accurate ethernet bandwidth. Iperf 3.1.3 is a quite old release version (almost 3 years ago), the latest iperf release version is 3.6. Is there any plan to upgrade iperf to 3.1.5 or later version? Thanks.

Deus Ex, Dirt, ... preferences should include flag for "unsupported distro" nag bypass 

   When running on an "unsupported" distro such as Arch, Feral games will show a nag screen with their launcher asking you to "continue anyway".

It would be nice to set the following in the preferences files to bypass it:

             <key name="SpecificationAlerts">
                        <value name="LnxDistributionUnsupported" type="integer">1</value>
                    </key>

I've done it for Deus Ex MD, can do it for other titles as desired, but I'm not sure where to send the patch.

Thanks

pts/vpxenc-2.3.0 Increase performance on multi-threaded machines

vpxenc is being called with threads and tile-columns, but this limits the maximum threading to only the columns of each frame which for 1080p is 4 threads. Since version 1.7 there is another option --row-mt=1 which should be enabled that allows for additional parallel processing. This option is also safe for systems that have less cores than tile-columns just won't provide as much of a significance in performance boost. This allows all 16 cores to be utilized in this test.

pts/apitrace install fail due to redeclipse-1-trace.7z not found

When trying to install pts/apitrace for Phoronix Test Suite 7.8.0on on a Kabylake NUC(NUC7i5DNB) which is running Ubuntu 17.10, I always get same failure:
{noformat}
Download Failed: http://www.phoronix-test-suite.com/benchmark-files/redeclipse-1-trace.7z
Downloading of needed test files failed.
{noformat}

It turns out that the file is not available at http://www.phoronix-test-suite.com/benchmark-files/.
Is this an issue supposed to be fixed?

Detailed logs and system information attached.
apitrace_install_fail.log

pts/apache-siege requires /usr/local/var

pts/apache-siege requires /usr/local/var to create the siege.log file.
If it does not exists, the test crash. Maybe it should be created on install?

This directory does not exists at least on Arch Linux.

LOG FILE: /usr/local/var/siege.log
You can disable this log file notification by editing
/var/lib/phoronix-test-suite/installed-tests/pts/apache-siege-1.0.2//.siege/siege.conf and changing 'show-logfile' to false.
[error] unable to create log file: /usr/local/var/siege.log: No such file or directory

Creating the missing var directory solves the problem.

.mov is not supported by vpxenc

test-profiles/pts/vpxenc-2.0.0/downloads.xml

Line 12 in 572a2b0

<URL>http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_720p_h264.mov, https://archive.org/download/big_buck_bunny_720p_h264/big_buck_bunny_720p_h264.mov, http://www.peach.themazzone.com/big_buck_bunny_720p_h264.mov</URL>

vpxenc requires yuv or y4m input. The mov file is being treated as yuv. Playing the video back will just show random snow.

pts/enconde-mp3 -- Issue with power9-fix.patch

Looks like the patch has an issue:

lame-3.100/vc_solution/vc9_lame_dshow.vcproj
patching file configure.in
Hunk #1 FAILED at 94.
1 out of 2 hunks FAILED -- saving rejects to file configure.in.rej
checking build system type... powerpc64le-unknown-linux-gnu

Diffing my patch vs. power9-fix.patch I see the following:

1,2c1,2
< --- configure.in.old  2017-08-15 11:16:31.000000000 -0400
< +++ configure.in      2018-09-23 19:41:40.012250933 -0400
---
> --- configure.in      2018-08-02 15:07:41.944163387 -0500
> +++ configure.in.jjm  2018-08-02 15:10:14.318522611 -0500
5c5
<       if test "" = "yes"; then
---
>       if test "${HAVE_GCC}" = "yes"; then
7,9c7,9
< -             GCC_version="` --version | sed -n '1s/^[[^ ]]* (.*) //;s/ .*$//;1p'`"
< +             GCC_version="` --version | head -n1 | awk '{print }' `"
<               case "" in 
---
> -             GCC_version="`${CC} --version | sed -n '1s/^[[^ ]]* (.*) //;s/ .*$//;1p'`"
> +             GCC_version="`${CC} --version | head -n1 | awk '{print $3}' `"
>               case "${GCC_version}" in 
12c12
<                       AC_MSG_RESULT()
---
>                       AC_MSG_RESULT(${GCC_version})

pts/cuda-mini-nbody

I would like to propose a change to the cuda mini-nbody as it is currently overall execution time (sys.time), rather than actually using the performance information provided by the benchmark itself.

PREMISE:
To get this information I suggest to allow the seconds per iteration metric to be exposed from within the individual cuda sources. In a simple hacky way, it could be done as follows:

#ifndef SHMOO
    printf("Iteration %d: %.3f seconds\n", iter, tElapsed);
#endif

to:

    printf("Iteration %d: %.3f seconds\n", iter, tElapsed);

Now lets look at what running the shmoo-cuda-nbody-orig.sh outputs with this change:

NVIDIA TITAN XP
nbody-orig
[...]
Iteration 1: 1.254 seconds
Iteration 2: 1.261 seconds
Iteration 3: 1.258 seconds
Iteration 4: 1.259 seconds
Iteration 5: 1.259 seconds
Iteration 6: 1.245 seconds
Iteration 7: 1.258 seconds
Iteration 8: 1.250 seconds
Iteration 9: 1.259 seconds
Iteration 10: 1.259 seconds
524288, 218.758

NVIDIA TESLA V100
nbody-orig
Iteration 1: 0.561 seconds
Iteration 2: 0.584 seconds
Iteration 3: 0.564 seconds
Iteration 4: 0.584 seconds
Iteration 5: 0.604 seconds
Iteration 6: 0.584 seconds
Iteration 7: 0.564 seconds
Iteration 8: 0.564 seconds
Iteration 9: 0.563 seconds
Iteration 10: 0.564 seconds
524288, 477.949

There are 2 metrics which are more significant than overall execution time:

  1. The time it takes to perform an iteration with the larges nBody (524288) is significant.
  2. The result on the final (nBodies * nBodies / avgTime) for the largest nBody (524288) is also a lot more significant than execution time.

Why am I stating this?
Well, when I simply run the test as it currently is:

TITAN XP
$ phoronix-test-suite run pts/cuda-mini-nbody

[...]
    Average: 17.89 Seconds
    Deviation: 0.49%

TESLA V100
$ phoronix-test-suite run pts/cuda-mini-nbody
[...]
    Average: 26.91 Seconds
    Deviation: 3.25%

We see that even though the V100 tecnically is 2 times faster at executing the test, it seems to be outperformed by the TitanXP

Hence the simple fix is to change the results-definition.xml for this test to:

<?xml version="1.0"?>
<!--Phoronix Test Suite v6.4.0-->
<PhoronixTestSuite>
  <ResultsParser>
    <OutputTemplate>524288, #_RESULT_# </OutputTemplate>
    <LineHint>524288,</LineHint>
    <ResultScale>(NBody^2)/s</ResultScale>
    <ResultProportion>LIB</ResultProportion>
  </ResultsParser>
</PhoronixTestSuite>

build-llvm broken on Windows

Hi, it seems that installing the build-llvm profile is not working on Windows. I get:

phoronix-test-suite install build-llvm


Phoronix Test Suite v8.2.0m0

    To Install:    pts/build-llvm-1.0.1

    Determining File Requirements ..................................................................................
    Searching Download Caches ......................................................................................

    1 Test To Install
        1 File To Download [20.09MB]

    pts/build-llvm-1.0.1:
        Test Installation 1 of 1
        1 File Needed [20.09 MB / 1 Minute]
        Downloading: llvm-4.0.1.src.tar.xz                                                                 [20.09MB]
        Estimated Download Time: 1m ................................................................................
        Installing Test @ 16:22:55
The system cannot find the path specified.
The system cannot find the path specified.

The system cannot find the path specified.

The directory contents:

07/07/2018  04:25 PM    <DIR>          .
07/07/2018  04:25 PM    <DIR>          ..
07/07/2018  04:23 PM                79 build-llvm
07/07/2018  04:22 PM        21,065,652 llvm-4.0.1.src.tar.xz
07/07/2018  04:23 PM               996 pts-install.xml

Let me know if you need any further information.

pts/compress-xz

was doing system profiling while running the benchmark suite and noticed that the pts/compress-xz test was not using all the cores on my system. After testing options by hand it appears that level 9 only uses 5 cores maximum, and level 8 only 9. It isn't until I drop down to level 7 that all cores on the system were utilized.

Is this the expected results of the test? It seems like using the default compression level would be the most real world benchmark since that is what most users would invoke.

[pts/m-queens] update request

Based on your newest 128 Thread EPYC tests I noticed a bottleneck in m-queens for high thread counts.

Please update the profile to use https://github.com/sudden6/m-queens/releases/tag/v1.2

Note: This release also contains some other optimizations so that the performance is not really comparable.

Additionally, would you mind running a comparison for this program between some ARM or PowerPC system and x86_64?

deus-exmd: Downloading preferences

While I'm creating this issue related with DeusEx MD test profile, I think this should apply to any other other test that downloads preferences files.

  1. When overwriting the user's preferences files, please backup original file before starting the test and restore it at the end (preferably, even if ctrl-c is pressed).
  2. If the game includes a popup, then the preference to skip that popup should be enabled.
  3. While taking 1. into account, why not use 'sed' or even a 'cat >' to modify user's preferences file? This would make the test self-contained and independent of an online file that may change over time completely unrelated with the test's version.

Thanks in advance.

pts/enconde-mp3 -- Fixing configuration file shows up to 4x improvement in POWER9 system

Before running autoconf, I changed the following in the configure.in file:

@@ -94,9 +94,9 @@
 
 	if test "${HAVE_GCC}" = "yes"; then
 		AC_MSG_CHECKING(version of GCC)
-		GCC_version="`${CC} --version | sed -n '1s/^[[^ ]]* (.*) //;s/ .*$//;1p'`"
+		GCC_version="`${CC} --version | head -n1 | awk '{print $3}' `"
 		case "${GCC_version}" in 
-		[0-9]*[0-9]*)
+		[[0-9]]*[[0-9]]*)
 			AC_MSG_RESULT(${GCC_version})
 			;;
 		*)
@@ -701,7 +701,7 @@
 	AC_DEFINE(TAKEHIRO_IEEE754_HACK, 1, IEEE754 compatible machine)
 	AC_DEFINE(USE_FAST_LOG, 1, faster log implementation with less but enough precission)
 	;;
-powerpc)
+powerpc*|ppc64*)
 	CPUTYPE="no"
 
 	# use internal knowledge of the IEEE 754 layout

powerpc -- python benchmarks benefit from conda binaries installation

Installing https://repo.anaconda.com/archive/Anaconda2-5.3.0-Linux-ppc64le.sh and using the binaries by including them in path, i.e PATH=/root/anaconda2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin, improves perfomance in several benchmarks.

An example below using pts/scikit-learn

System Information

  PROCESSOR:          POWER9 altivec supported @ 3.80GHz
    Core Count:       94
    Scaling Driver:   powernv-cpufreq performance

  GRAPHICS:           ASPEED ASPEED Family
    OpenGL:           3.3 Mesa 18.0.5 (LLVM 6.0 128 bits)
    Screen:           1024x768

  MOTHERBOARD:        PowerNV 0000000000000000
    Network:          Broadcom Limited NetXtreme BCM5719 Gigabit PCIe

  MEMORY:             262144MB

  DISK:               960GB MZ7LM960HCHP-000
    File-System:      ext4
    Mount Options:    data=ordered errors=remount-ro relatime rw
    Disk Scheduler:   CFQ

  OPERATING SYSTEM:   Ubuntu 18.04
    Kernel:           4.15.0-33-generic (ppc64le)
    Compiler:         GCC 7.3.0

    Would you like to save these test results (Y/n): n

==== run without CONDA ===
Scikit-Learn 0.17.1:
    pts/scikit-learn-1.0.1
    Test 1 of 1
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 5 Minutes [12:45 CDT]
        Started Run 1 @ 12:41:52
        Started Run 2 @ 12:46:02
        Started Run 3 @ 12:50:14

    Test Results:
        249.3496940136
        251.49449396133
        251.18054199219

    Average: 250.67 Seconds
    Deviation: 0.46%

==== run with CONDA ===
Scikit-Learn 0.17.1:
    pts/scikit-learn-1.0.1
    Test 1 of 1
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 2 Minutes [14:11 CDT]
        Started Run 1 @ 14:09:47
        Started Run 2 @ 14:10:12
        Started Run 3 @ 14:10:37

    Test Results:
        24.453907012939
        23.821007966995
        23.303120851517

    Average: 23.86 Seconds
    Deviation: 2.42%

`pts/pgbench` missing dependency on `bc`

While trying to benchmark servers using pts/pgbench I noticed that the test failed silently on my servers.

Running with phoronix-test-suite debug-benchmark pts/pgbench yield this output, where I noticed the script depends on bc which is not installed on Ubuntu by default.

========================================                                                                                                                                                                           
PostgreSQL pgbench (Run 1 of 1)                                                                                                                                                                                    
========================================                                                                                                                                                                           
                                                                                                                                                                                                                   
                                                                                                                                                                                                                   
Test Run Command: cd /home/sysop/.phoronix-test-suite/installed-tests/pts/pgbench-1.8.4/ && ./pgbench ON_DISK NORMAL_LOAD READ_WRITE 2>&1                                                                          
                                                                                                                                                                                                                   
waiting for server to start....2018-04-14 22:09:43.972 CEST [279846] LOG:  listening on IPv4 address "127.0.0.1", port 7777
2018-04-14 22:09:43.972 CEST [279846] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2018-04-14 22:09:43.972 CEST [279846] HINT:  Is another postmaster already running on port 7777? If not, wait a few seconds and retry.
2018-04-14 22:09:43.989 CEST [279846] LOG:  listening on Unix socket "/tmp/.s.PGSQL.7777"
2018-04-14 22:09:44.022 CEST [279847] LOG:  database system was shut down at 2018-04-14 03:27:36 CEST
2018-04-14 22:09:44.032 CEST [279846] LOG:  database system is ready to accept connections                                               
 done                    
server started     
./pgbench: 1: ./pgbench: bc: not found
invalid scaling factor: "pgbench"
2018-04-14 22:09:54.336 CEST [279983] ERROR:  relation "pgbench_branches" does not exist at character 22
2018-04-14 22:09:54.336 CEST [279983] STATEMENT:  select count(*) from pgbench_branches
waiting for server to shut down....2018-04-14 22:09:54.447 CEST [279846] LOG:  received fast shutdown request
2018-04-14 22:09:54.457 CEST [279846] LOG:  aborting any active transactions
2018-04-14 22:09:54.458 CEST [279846] LOG:  worker process: logical replication launcher (PID 279852) exited with exit code 1
2018-04-14 22:09:54.458 CEST [279848] LOG:  shutting down
2018-04-14 22:09:54.510 CEST [279846] LOG:  database system is shut down
 done
server stopped

Result Key: #_RESULT_#


Template Line: tps = #_RESULT_# (excluding connections establishing) TPS


Search Key: excluding connections establishing


Result Parsing Search Key: "excluding connections establishing"


Result Line:


No Test Results


The test run did not produce a result.

Update SVT-AV1 with live preset

With the latest commit SVT-AV1 got higher presets for faster encodes. Aside from the currently used -enc-mode 3, a test run with -enc-mode 7 would be great to test live encoding. Soon 1080p 30fps will be possible on high-end consumer hardware.

clpeak: multiple platforms

Hello.

The clpeak benchmark does not generate correct results on my machine. I have 3 OpenCL platforms, each platform has a single device. Running clpeak in PTS unexpectedly records the performance of the last platform, not the first platform.

In order to product correct results, PTS needs to additionally pass -p 0 to clpeak to select the first platform:

clpeak -p 0 -d 0 --compute-sp

Currently, only -d 0 is passed to clpeak, generating the following (incorrect) results:

$ ./clpeak -d 0 --compute-sp
Platform: AMD Accelerated Parallel Processing
  Device: Hawaii
    Single-precision compute (GFLOPS)
      float   : 5057.77
      float2  : 5051.59
      float4  : 5034.63
      float8  : 5012.79
      float16 : 4959.53

Platform: NVIDIA CUDA
  Device: GeForce GTX 750
    Single-precision compute (GFLOPS)
      float   : 714.02
      float2  : 1104.40
      float4  : 1171.74
      float8  : 1104.74
      float16 : 1161.95

Platform: Intel(R) OpenCL
  Device: AMD Ryzen 5 1600 Six-Core Processor            
    Single-precision compute (GFLOPS)
      float   : 133.67
      float2  : 265.19
      float4  : 328.40
      float8  : 133.27
      float16 : 239.27

pts/mysqlslap : mysqld does not run due to missing rights on directory

mysqld would not run :

181123 06:27:30 mysqld_safe Starting mysqld daemon with databases from /opt/phoronix-test-suite/installed-tests/pts/mysqlslap-1.0.4//mysql_/.data
/opt/phoronix-test-suite/installed-tests/pts/mysqlslap-1.0.4//mysql_/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)'

When redirecting logs to syslog I see :

Nov 23 06:36:55 vs-int-prd-bch-fr-301 mysqld: 2018-11-23  6:36:55 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
Nov 23 06:36:55 vs-int-prd-bch-fr-301 mysqld: 2018-11-23  6:36:55 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.

a quick and dirty chown/chmod did the trick :

chmod -R 777 installed-tests/pts/mysqlslap-1.0.4/mysql_/data
chown -R mysql:mysql installed-tests/pts/mysqlslap-1.0.4/mysql_/data
chmod -R 777 installed-tests/pts/mysqlslap-1.0.4/mysql_/.data
chown -R mysql:mysql installed-tests/pts/mysqlslap-1.0.4/mysql_/.data

system/blender(-1.0.2) cpu profiles still use gpu based files

When a system/blender test is run the process list shows that the gpu file is being run. The GPU file contains a tile size of 256x256 which for a single frame JPEG render limits the number of CPUs being used.

The cpu blend files in cycles_benchmark_20160228 contain a 32x32 tile size and is configured for CPU.

https://openbenchmarking.org/test/pts/blender version 1.4.1 contains a BLEND_ARGS=${BLEND_ARGS/_gpu/_cpu} fix in the generated blender when a CPU profile is selected.

pts/mysqlslap 1.0.5 - cannot download sources

Hi,

pts/mysqlslap cannot be installed, broken links to sources:

Downloading: mariadb-10.3.8.tar.gz                         [68.81MB]
Estimated Download Time: 5m ........................................
File Not Found: http://ftp.osuosl.org/pub/mariadb/mariadb-10.3.8/source/mariadb-10.3.8.tar.gz
Attempting to download from alternate mirror.
Downloading: mariadb-10.3.8.tar.gz                         [68.81MB]
Estimated Download Time: 5m ........................................
Download Failed: ftp://ftp.ulak.net.tr/pub/MariaDB/mariadb-10.3.8/source/mariadb-10.3.8.tar.gz
Downloading of needed test files failed.

Possible fix:

switch to latest 10.3 version and use URL - https://downloads.mariadb.org/f/mariadb-10.3.13/source/mariadb-10.3.13.tar.gz

Thanks in advance

glibc benchmarks are unreliable and produce incorrect results

The glibc benchmarks (glibc-benchmarks-1.tar.gz) are unreliable and are generating incorrect results because their aren't compiled with USE_CLOCK_GETTIME=1.

Incorrect result for bench-ffs (glibc-benchmarks-1.tar.gz, Ryzen 5 1600):

"ffs": {
 "": {
  "duration": 3.18057e+09,
  "iterations": 4.975e+08,
  "max": 26.368,
  "min": 6.048,
  "mean": 6.3931
 }
}

Correct result for bench-ffs (USE_CLOCK_GETTIME=1, glibc 2.27, Ryzen 5 1600):

"ffs": {
 "": {
  "duration": 9.3078e+09,
  "iterations": 4.3072e+09,
  "max": 8.67,
  "min": 2.04,
  "mean": 2.16099
 }
}

Suggested resolution: Compile glibc benchmarks via make USE_CLOCK_GETTIME=1 bench-build, package it as glibc-benchmarks-2.tar.gz and create new version glibc-bench-1.6 (version numbers may vary).

pts/mbw-1.0.0: PowerPC compilation fails

On a POWER9 system I see the following:

pts/mbw-1.0.0:
    Test Installation 2 of 39
    1 File Needed [0.06 MB]  
    Downloading: mbw-20180908.tar.xz                                                                           [0.06MB]
    Downloading .......................................................................................................
    Installation Size: 1 MB                                                                                            
    Installing Test @ 14:33:43                                                                                         
        The installer exited with a non-zero exit status.                                                              
        ERROR: gcc: error: unrecognized command line option â-march=nativeâ; did you mean â-mcpu=nativeâ?              
        LOG: ~/.phoronix-test-suite/installed-tests/pts/mbw-1.0.0/install-failed.log

On PowerPC -mcpu=native -mtune=native must be used instead.
This is related to phoronix-test-suite/phoronix-test-suite#333

system/gimp-1.1.0: eval: unbound variable: batch-unsharp-mask

system/gimp-1.1.10 failed to run any of the gimp sub-tests because GIMP V2.10 was unable to find the bench.scm script on the respective phoronix local gimp dir scripts:
phoronix-test-suite/installed-tests/system/gimp-1.1.0/.config/GIMP/2.10/scripts.

pts/stream: PowerPC compilation fails

In a POWER8 system I see the following:

ubuntu@oprom8:~$ phoronix-test-suite debug-install pts/stream

To Install:    pts/stream-1.3.1


========================================
Installing Test: pts/stream-1.3.1
========================================


========================================
Downloading Files: pts/stream-1.3.1
========================================

gcc: error: unrecognized command line option â-march=nativeâ; did you mean â-mcpu=nativeâ?

On PowerPC we usually use the following:
gcc -m64 -O3 -fopenmp -DSTREAM_ARRAY_SIZE=536895856 -DNTIMES=20 -mcmodel=large stream.c -o stream-bin
This is related to phoronix-test-suite/phoronix-test-suite#333

Measure CPU usage in video encoding tests, maybe others

Being able to see the average CPU usage of video encoder tests could be very useful for judging how well an encoder scales across multiple cores. This would be especially useful for libvpx and libaom that currently don't scale as well as, say, x264 and x265.

I'm not sure how to best measure the average CPU usage of a process on all the platforms supported by PTS, but on Linux you can prepend a command with /usr/bin/time -f "%P" and the average CPU usage of the appended command will be printed after it has run. Note, /usr/bin/time is required because time invokes a different, built-in time command of most shells.

This could also be useful in other benchmarks, although average CPU usage across all cores/threads is quite a simplistic metric.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.