1{ 2 "cells": [ 3 { 4 "cell_type": "markdown", 5 "metadata": { 6 "deletable": true, 7 "editable": true 8 }, 9 "source": [ 10 "# Geekbench benchmark on Android\n", 11 "\n", 12 "Geekbench4 is an app offering several benchmarks to run on android smartphones. The one used in this notebook is the '**CPU**' benchmark, which runs several workloads that follow the lines of what is commonly run by smartphones (AES, JPEG codec, FFT, and so on). The benchmark runs all the tests in '**Single-Core**' mode as well as in '**Multi-Core**' in order to compare the single-thread and multi-thread performances of the device.\n", 13 "\n", 14 "**Do note that the benchmark will attempt to upload its results, which includes some hardware information**" 15 ] 16 }, 17 { 18 "cell_type": "code", 19 "execution_count": 1, 20 "metadata": { 21 "collapsed": false, 22 "deletable": true, 23 "editable": true 24 }, 25 "outputs": [ 26 { 27 "name": "stderr", 28 "output_type": "stream", 29 "text": [ 30 "2017-05-03 10:54:02,800 INFO : root : Using LISA logging configuration:\n", 31 "2017-05-03 10:54:02,801 INFO : root : /home/vagrant/lisa/logging.conf\n" 32 ] 33 } 34 ], 35 "source": [ 36 "from conf import LisaLogging\n", 37 "LisaLogging.setup()" 38 ] 39 }, 40 { 41 "cell_type": "code", 42 "execution_count": 2, 43 "metadata": { 44 "collapsed": false, 45 "deletable": true, 46 "editable": true 47 }, 48 "outputs": [ 49 { 50 "name": "stdout", 51 "output_type": "stream", 52 "text": [ 53 "Populating the interactive namespace from numpy and matplotlib\n" 54 ] 55 }, 56 { 57 "name": "stderr", 58 "output_type": "stream", 59 "text": [ 60 "2017-05-03 10:54:03,346 WARNING : EnergyModel : Unusual max capacity (1023), overriding capacity_scale\n" 61 ] 62 } 63 ], 64 "source": [ 65 "%pylab inline\n", 66 "\n", 67 "import json\n", 68 "import os\n", 69 "\n", 70 "# Support to access the remote target\n", 71 "import devlib\n", 72 "from env import TestEnv\n", 73 "\n", 74 "# Import support for Android devices\n", 75 "from android import Screen, Workload\n", 76 "\n", 77 "# Support for trace events analysis\n", 78 "from trace import Trace\n", 79 "\n", 80 "# Suport for FTrace events parsing and visualization\n", 81 "import trappy\n", 82 "\n", 83 "import pandas as pd" 84 ] 85 }, 86 { 87 "cell_type": "markdown", 88 "metadata": { 89 "deletable": true, 90 "editable": true 91 }, 92 "source": [ 93 "## Support Functions" 94 ] 95 }, 96 { 97 "cell_type": "markdown", 98 "metadata": { 99 "deletable": true, 100 "editable": true 101 }, 102 "source": [ 103 "This function helps us run our experiments:" 104 ] 105 }, 106 { 107 "cell_type": "code", 108 "execution_count": 3, 109 "metadata": { 110 "collapsed": true, 111 "deletable": true, 112 "editable": true 113 }, 114 "outputs": [], 115 "source": [ 116 "def experiment():\n", 117 " \n", 118 " # Configure governor\n", 119 " target.cpufreq.set_all_governors('sched')\n", 120 " \n", 121 " # Get workload\n", 122 " wload = Workload.getInstance(te, 'Geekbench')\n", 123 " \n", 124 " # Run Geekbench workload\n", 125 " wload.run(te.res_dir, test_name='CPU', collect='ftrace')\n", 126 " \n", 127 " # Dump platform descriptor\n", 128 " te.platform_dump(te.res_dir)" 129 ] 130 }, 131 { 132 "cell_type": "markdown", 133 "metadata": { 134 "deletable": true, 135 "editable": true 136 }, 137 "source": [ 138 "## Test environment setup\n", 139 "For more details on this please check out **examples/utils/testenv_example.ipynb**." 140 ] 141 }, 142 { 143 "cell_type": "markdown", 144 "metadata": { 145 "deletable": true, 146 "editable": true 147 }, 148 "source": [ 149 "**devlib** requires the ANDROID_HOME environment variable configured to point to your local installation of the Android SDK. If you have not this variable configured in the shell used to start the notebook server, you need to run a cell to define where your Android SDK is installed or specify the ANDROID_HOME in your target configuration." 150 ] 151 }, 152 { 153 "cell_type": "markdown", 154 "metadata": { 155 "deletable": true, 156 "editable": true 157 }, 158 "source": [ 159 "In case more than one Android device are conencted to the host, you must specify the ID of the device you want to target in **my_target_conf**. Run **adb devices** on your host to get the ID." 160 ] 161 }, 162 { 163 "cell_type": "code", 164 "execution_count": 4, 165 "metadata": { 166 "collapsed": false, 167 "deletable": true, 168 "editable": true 169 }, 170 "outputs": [], 171 "source": [ 172 "# Setup target configuration\n", 173 "my_conf = {\n", 174 "\n", 175 " # Target platform and board\n", 176 " \"platform\" : 'android',\n", 177 " \"board\" : 'pixel',\n", 178 " \n", 179 " # Device\n", 180 " \"device\" : \"0123456789ABCDEF\",\n", 181 " \n", 182 " # Android home\n", 183 " \"ANDROID_HOME\" : \"/home/vagrant/lisa/tools/android-sdk-linux/\",\n", 184 "\n", 185 " # Folder where all the results will be collected\n", 186 " \"results_dir\" : datetime.datetime.now()\\\n", 187 " .strftime(\"Geekbench_example_\" + '%Y%m%d_%H%M%S'),\n", 188 "\n", 189 " # Define devlib modules to load\n", 190 " \"modules\" : [\n", 191 " 'cpufreq' # enable CPUFreq support\n", 192 " ],\n", 193 "\n", 194 " # FTrace events to collect for all the tests configuration which have\n", 195 " # the \"ftrace\" flag enabled\n", 196 " \"ftrace\" : {\n", 197 " \"events\" : [\n", 198 " \"sched_switch\",\n", 199 " \"sched_wakeup\",\n", 200 " \"sched_wakeup_new\",\n", 201 " \"sched_overutilized\",\n", 202 " \"sched_load_avg_cpu\",\n", 203 " \"sched_load_avg_task\",\n", 204 " \"cpu_capacity\",\n", 205 " \"cpu_frequency\",\n", 206 " ],\n", 207 " \"buffsize\" : 100 * 1024,\n", 208 " },\n", 209 "\n", 210 " # Tools required by the experiments\n", 211 " \"tools\" : [ 'trace-cmd', 'taskset'],\n", 212 "}" 213 ] 214 }, 215 { 216 "cell_type": "code", 217 "execution_count": 5, 218 "metadata": { 219 "collapsed": false, 220 "deletable": true, 221 "editable": true, 222 "scrolled": false 223 }, 224 "outputs": [ 225 { 226 "name": "stderr", 227 "output_type": "stream", 228 "text": [ 229 "2017-05-03 10:54:03,551 INFO : TestEnv : Using base path: /home/vagrant/lisa\n", 230 "2017-05-03 10:54:03,551 INFO : TestEnv : Loading custom (inline) target configuration\n", 231 "2017-05-03 10:54:03,552 INFO : TestEnv : External tools using:\n", 232 "2017-05-03 10:54:03,553 INFO : TestEnv : ANDROID_HOME: /home/vagrant/lisa/tools/\n", 233 "2017-05-03 10:54:03,554 INFO : TestEnv : CATAPULT_HOME: /home/vagrant/lisa/tools//platform-tools/systrace/catapult\n", 234 "2017-05-03 10:54:03,555 INFO : TestEnv : Devlib modules to load: ['bl', 'cpufreq']\n", 235 "2017-05-03 10:54:03,556 INFO : TestEnv : Connecting Android target [DEFAULT]\n", 236 "2017-05-03 10:54:03,557 INFO : TestEnv : Connection settings:\n", 237 "2017-05-03 10:54:03,558 INFO : TestEnv : None\n", 238 "2017-05-03 10:54:03,761 INFO : android : ls command is set to ls -1\n", 239 "2017-05-03 10:54:04,637 INFO : TestEnv : Initializing target workdir:\n", 240 "2017-05-03 10:54:04,638 INFO : TestEnv : /data/local/tmp/devlib-target\n", 241 "2017-05-03 10:54:06,555 INFO : TestEnv : Topology:\n", 242 "2017-05-03 10:54:06,557 INFO : TestEnv : [[0, 1], [2, 3]]\n", 243 "2017-05-03 10:54:06,748 INFO : TestEnv : Loading default EM:\n", 244 "2017-05-03 10:54:06,749 INFO : TestEnv : /home/vagrant/lisa/libs/utils/platforms/pixel.json\n", 245 "2017-05-03 10:54:07,301 INFO : TestEnv : Enabled tracepoints:\n", 246 "2017-05-03 10:54:07,302 INFO : TestEnv : sched_switch\n", 247 "2017-05-03 10:54:07,303 INFO : TestEnv : sched_wakeup\n", 248 "2017-05-03 10:54:07,304 INFO : TestEnv : sched_wakeup_new\n", 249 "2017-05-03 10:54:07,305 INFO : TestEnv : sched_overutilized\n", 250 "2017-05-03 10:54:07,306 INFO : TestEnv : sched_load_avg_cpu\n", 251 "2017-05-03 10:54:07,307 INFO : TestEnv : sched_load_avg_task\n", 252 "2017-05-03 10:54:07,308 INFO : TestEnv : cpu_capacity\n", 253 "2017-05-03 10:54:07,309 INFO : TestEnv : cpu_frequency\n", 254 "2017-05-03 10:54:07,310 INFO : TestEnv : Set results folder to:\n", 255 "2017-05-03 10:54:07,311 INFO : TestEnv : /home/vagrant/lisa/results/Geekbench_example_20170503_105403\n", 256 "2017-05-03 10:54:07,312 INFO : TestEnv : Experiment results available also in:\n", 257 "2017-05-03 10:54:07,313 INFO : TestEnv : /home/vagrant/lisa/results_latest\n" 258 ] 259 } 260 ], 261 "source": [ 262 "# Initialize a test environment using:\n", 263 "te = TestEnv(my_conf, wipe=False)\n", 264 "target = te.target" 265 ] 266 }, 267 { 268 "cell_type": "markdown", 269 "metadata": { 270 "deletable": true, 271 "editable": true 272 }, 273 "source": [ 274 "## Workloads execution\n", 275 "\n", 276 "This is done using the **experiment** helper function defined above which is configured to run a **Geekbench - CPU** experiment." 277 ] 278 }, 279 { 280 "cell_type": "code", 281 "execution_count": 6, 282 "metadata": { 283 "collapsed": false, 284 "deletable": true, 285 "editable": true 286 }, 287 "outputs": [ 288 { 289 "name": "stderr", 290 "output_type": "stream", 291 "text": [ 292 "2017-05-03 10:54:07,990 INFO : Workload : Supported workloads available on target:\n", 293 "2017-05-03 10:54:07,992 INFO : Workload : gmaps, youtube, jankbench, geekbench\n", 294 "2017-05-03 10:54:10,644 INFO : Screen : Force manual orientation\n", 295 "2017-05-03 10:54:10,645 INFO : Screen : Set orientation: PORTRAIT\n", 296 "2017-05-03 10:54:12,496 INFO : Screen : Set brightness: 0%\n", 297 "2017-05-03 10:54:14,950 INFO : Geekbench : adb -s HT67M0300128 logcat ActivityManager:* System.out:I *:S GEEKBENCH_RESULT:*\n", 298 "2017-05-03 10:54:17,071 INFO : Geekbench : FTrace START\n", 299 "2017-05-03 10:58:23,429 INFO : Geekbench : FTrace STOP\n", 300 "2017-05-03 10:58:33,944 INFO : Screen : Set orientation: AUTO\n", 301 "2017-05-03 10:58:35,526 INFO : Screen : Set brightness: AUTO\n" 302 ] 303 } 304 ], 305 "source": [ 306 "# Initialize Workloads for this test environment\n", 307 "results = experiment()" 308 ] 309 }, 310 { 311 "cell_type": "markdown", 312 "metadata": { 313 "deletable": true, 314 "editable": true 315 }, 316 "source": [ 317 "## Results analysis" 318 ] 319 }, 320 { 321 "cell_type": "markdown", 322 "metadata": { 323 "collapsed": true, 324 "deletable": true, 325 "editable": true 326 }, 327 "source": [ 328 "Geekbench4 uses a baseline score of 4000, which is the benchmark score of an Intel Core i7-6600U. Higher scores are better, with double the score indicating double the performance. You can have a look at the results for several android phones here https://browser.primatelabs.com/android-benchmarks" 329 ] 330 }, 331 { 332 "cell_type": "code", 333 "execution_count": 7, 334 "metadata": { 335 "collapsed": false, 336 "deletable": true, 337 "editable": true 338 }, 339 "outputs": [], 340 "source": [ 341 "class Geekbench(object):\n", 342 " \"\"\"\n", 343 " Geekbench json results parsing class\n", 344 " \"\"\"\n", 345 " def __init__(self, filepath):\n", 346 " with open(filepath) as fd:\n", 347 " self.__json = json.loads(fd.read())\n", 348 " \n", 349 " self.benchmarks = {}\n", 350 " for section in self.__json[\"sections\"]:\n", 351 " self.benchmarks[section[\"name\"]] = section\n", 352 " for workload in section[\"workloads\"]:\n", 353 " self.benchmarks[section[\"name\"]][workload[\"name\"]] = workload \n", 354 " \n", 355 " def name(self):\n", 356 " \"\"\"Get a human-readable name for the geekbench run\n", 357 " \"\"\"\n", 358 " gov = \"\"\n", 359 " build = \"\"\n", 360 " for metric in self.__json[\"metrics\"]:\n", 361 " if metric[\"name\"] == \"Governor\":\n", 362 " gov = metric[\"value\"]\n", 363 " elif metric[\"name\"] == \"Build\":\n", 364 " build = metric[\"value\"]\n", 365 "\n", 366 " return \"[build]=\\\"{}\\\" [governor]=\\\"{}\\\"\".format(build, gov)\n", 367 " \n", 368 " def benchmarks_names(self):\n", 369 " \"\"\"Get a list of benchmarks (e.g. Single-Core, Multi-Core) found in the run results \n", 370 " \"\"\"\n", 371 " return [section[\"name\"] for section in self.__json[\"sections\"]]\n", 372 " \n", 373 " def workloads_names(self):\n", 374 " \"\"\"Get a list of unique workloads (e.g. EAS, Dijkstra) found in the run results\n", 375 " \"\"\"\n", 376 " return [workload[\"name\"] for workload in self.benchmarks.values()[0][\"workloads\"]]\n", 377 " \n", 378 " def global_scores(self):\n", 379 " \"\"\"Get the overall scores of each benchmark\n", 380 " \"\"\"\n", 381 " data = {}\n", 382 " for benchmark in self.benchmarks_names():\n", 383 " data[benchmark] = self.benchmarks[benchmark][\"score\"]\n", 384 " return data\n", 385 " \n", 386 " def detailed_scores(self):\n", 387 " \"\"\"Get the detailed workload scores of each benchmark\n", 388 " \"\"\"\n", 389 " benchmark_fields = [\"score\", \"runtime_mean\", \"rate_string\"]\n", 390 " benches = {}\n", 391 " benchmarks = self.benchmarks_names()\n", 392 " workloads = self.workloads_names() \n", 393 " \n", 394 " for benchmark in benchmarks:\n", 395 " data = {}\n", 396 " for workload in workloads:\n", 397 " data[workload] = {}\n", 398 " for field in benchmark_fields:\n", 399 " data[workload][field] = self.benchmarks[benchmark][workload][field] \n", 400 " benches[benchmark] = data\n", 401 " \n", 402 " return benches" 403 ] 404 }, 405 { 406 "cell_type": "code", 407 "execution_count": 8, 408 "metadata": { 409 "collapsed": false, 410 "deletable": true, 411 "editable": true 412 }, 413 "outputs": [], 414 "source": [ 415 "def display_bench_results(geekbench, detailed=False):\n", 416 " print \"===== Global results =====\"\n", 417 " \n", 418 " scores = geekbench.global_scores()\n", 419 " \n", 420 " # Build dataframe for display\n", 421 " row = []\n", 422 " for bench_type, score in scores.iteritems():\n", 423 " row.append(score)\n", 424 " \n", 425 " df = pd.DataFrame(data=row, index=scores.keys(), columns=[\"Global score\"])\n", 426 " display(df)\n", 427 " \n", 428 " if not detailed:\n", 429 " return\n", 430 " \n", 431 " print \"===== Detailed results =====\"\n", 432 " \n", 433 " scores = geekbench.detailed_scores()\n", 434 " \n", 435 " for benchmark, results in geekbench.detailed_scores().iteritems():\n", 436 " print \"----- {} benchmark -----\".format(benchmark)\n", 437 " # Build dataframe for display\n", 438 " data = []\n", 439 " idx = []\n", 440 " columns = results.values()[0].keys()\n", 441 " for workload, fields in results.iteritems():\n", 442 " data.append(tuple(fields.values()))\n", 443 " idx.append(workload)\n", 444 " display (pd.DataFrame(data=data, index=idx, columns=columns))" 445 ] 446 }, 447 { 448 "cell_type": "code", 449 "execution_count": 9, 450 "metadata": { 451 "collapsed": false, 452 "deletable": true, 453 "editable": true, 454 "scrolled": false 455 }, 456 "outputs": [ 457 { 458 "name": "stdout", 459 "output_type": "stream", 460 "text": [ 461 "Analysing geekbench [build]=\"sailfishf-userdebug 7.1.1 NMF26P 3525730 dev-keys\" [governor]=\"sched\"\n", 462 "===== Global results =====\n" 463 ] 464 }, 465 { 466 "data": { 467 "text/html": [ 468 "<div>\n", 469 "<table border=\"1\" class=\"dataframe\">\n", 470 " <thead>\n", 471 " <tr style=\"text-align: right;\">\n", 472 " <th></th>\n", 473 " <th>Global score</th>\n", 474 " </tr>\n", 475 " </thead>\n", 476 " <tbody>\n", 477 " <tr>\n", 478 " <th>Single-Core</th>\n", 479 " <td>1594</td>\n", 480 " </tr>\n", 481 " <tr>\n", 482 " <th>Multi-Core</th>\n", 483 " <td>4170</td>\n", 484 " </tr>\n", 485 " </tbody>\n", 486 "</table>\n", 487 "</div>" 488 ], 489 "text/plain": [ 490 " Global score\n", 491 "Single-Core 1594\n", 492 "Multi-Core 4170" 493 ] 494 }, 495 "metadata": {}, 496 "output_type": "display_data" 497 }, 498 { 499 "name": "stdout", 500 "output_type": "stream", 501 "text": [ 502 "===== Detailed results =====\n", 503 "----- Single-Core benchmark -----\n" 504 ] 505 }, 506 { 507 "data": { 508 "text/html": [ 509 "<div>\n", 510 "<table border=\"1\" class=\"dataframe\">\n", 511 " <thead>\n", 512 " <tr style=\"text-align: right;\">\n", 513 " <th></th>\n", 514 " <th>score</th>\n", 515 " <th>runtime_mean</th>\n", 516 " <th>rate_string</th>\n", 517 " </tr>\n", 518 " </thead>\n", 519 " <tbody>\n", 520 " <tr>\n", 521 " <th>AES</th>\n", 522 " <td>649</td>\n", 523 " <td>0.261358</td>\n", 524 " <td>501.0 MB/sec</td>\n", 525 " </tr>\n", 526 " <tr>\n", 527 " <th>HDR</th>\n", 528 " <td>3050</td>\n", 529 " <td>0.394782</td>\n", 530 " <td>11.1 Mpixels/sec</td>\n", 531 " </tr>\n", 532 " <tr>\n", 533 " <th>Rigid Body Physics</th>\n", 534 " <td>2147</td>\n", 535 " <td>0.239238</td>\n", 536 " <td>6285.5 FPS</td>\n", 537 " </tr>\n", 538 " <tr>\n", 539 " <th>HTML5 Parse</th>\n", 540 " <td>1565</td>\n", 541 " <td>0.173668</td>\n", 542 " <td>7.11 MB/sec</td>\n", 543 " </tr>\n", 544 " <tr>\n", 545 " <th>Lua</th>\n", 546 " <td>1220</td>\n", 547 " <td>0.254486</td>\n", 548 " <td>1.25 MB/sec</td>\n", 549 " </tr>\n", 550 " <tr>\n", 551 " <th>Camera</th>\n", 552 " <td>2605</td>\n", 553 " <td>0.148324</td>\n", 554 " <td>7.22 images/sec</td>\n", 555 " </tr>\n", 556 " <tr>\n", 557 " <th>Histogram Equalization</th>\n", 558 " <td>1425</td>\n", 559 " <td>0.245729</td>\n", 560 " <td>44.6 Mpixels/sec</td>\n", 561 " </tr>\n", 562 " <tr>\n", 563 " <th>SQLite</th>\n", 564 " <td>1189</td>\n", 565 " <td>0.612012</td>\n", 566 " <td>33.0 Krows/sec</td>\n", 567 " </tr>\n", 568 " <tr>\n", 569 " <th>Face Detection</th>\n", 570 " <td>2106</td>\n", 571 " <td>0.123343</td>\n", 572 " <td>615.2 Ksubwindows/sec</td>\n", 573 " </tr>\n", 574 " <tr>\n", 575 " <th>Memory Copy</th>\n", 576 " <td>2691</td>\n", 577 " <td>0.269374</td>\n", 578 " <td>7.46 GB/sec</td>\n", 579 " </tr>\n", 580 " <tr>\n", 581 " <th>Memory Latency</th>\n", 582 " <td>917</td>\n", 583 " <td>1.001564</td>\n", 584 " <td>471.8 ns</td>\n", 585 " </tr>\n", 586 " <tr>\n", 587 " <th>Canny</th>\n", 588 " <td>2191</td>\n", 589 " <td>0.167685</td>\n", 590 " <td>30.4 Mpixels/sec</td>\n", 591 " </tr>\n", 592 " <tr>\n", 593 " <th>PDF Rendering</th>\n", 594 " <td>2137</td>\n", 595 " <td>0.452174</td>\n", 596 " <td>56.8 Mpixels/sec</td>\n", 597 " </tr>\n", 598 " <tr>\n", 599 " <th>Gaussian Blur</th>\n", 600 " <td>2313</td>\n", 601 " <td>0.126649</td>\n", 602 " <td>40.5 Mpixels/sec</td>\n", 603 " </tr>\n", 604 " <tr>\n", 605 " <th>Speech Recognition</th>\n", 606 " <td>1263</td>\n", 607 " <td>1.126760</td>\n", 608 " <td>10.8 Words/sec</td>\n", 609 " </tr>\n", 610 " <tr>\n", 611 " <th>LLVM</th>\n", 612 " <td>1817</td>\n", 613 " <td>1.057730</td>\n", 614 " <td>124.9 functions/sec</td>\n", 615 " </tr>\n", 616 " <tr>\n", 617 " <th>Ray Tracing</th>\n", 618 " <td>1818</td>\n", 619 " <td>0.339196</td>\n", 620 " <td>265.5 Kpixels/sec</td>\n", 621 " </tr>\n", 622 " <tr>\n", 623 " <th>JPEG</th>\n", 624 " <td>2605</td>\n", 625 " <td>0.248902</td>\n", 626 " <td>21.0 Mpixels/sec</td>\n", 627 " </tr>\n", 628 " <tr>\n", 629 " <th>SGEMM</th>\n", 630 " <td>551</td>\n", 631 " <td>0.790329</td>\n", 632 " <td>11.7 Gflops</td>\n", 633 " </tr>\n", 634 " <tr>\n", 635 " <th>LZMA</th>\n", 636 " <td>1211</td>\n", 637 " <td>0.495195</td>\n", 638 " <td>1.89 MB/sec</td>\n", 639 " </tr>\n", 640 " <tr>\n", 641 " <th>SFFT</th>\n", 642 " <td>1130</td>\n", 643 " <td>0.169216</td>\n", 644 " <td>2.82 Gflops</td>\n", 645 " </tr>\n", 646 " <tr>\n", 647 " <th>Memory Bandwidth</th>\n", 648 " <td>2420</td>\n", 649 " <td>0.100460</td>\n", 650 " <td>12.9 GB/sec</td>\n", 651 " </tr>\n", 652 " <tr>\n", 653 " <th>N-Body Physics</th>\n", 654 " <td>1445</td>\n", 655 " <td>0.256496</td>\n", 656 " <td>1.08 Mpairs/sec</td>\n", 657 " </tr>\n", 658 " <tr>\n", 659 " <th>Dijkstra</th>\n", 660 " <td>2063</td>\n", 661 " <td>0.374775</td>\n", 662 " <td>1.40 MTE/sec</td>\n", 663 " </tr>\n", 664 " <tr>\n", 665 " <th>HTML5 DOM</th>\n", 666 " <td>613</td>\n", 667 " <td>0.913885</td>\n", 668 " <td>556.2 KElements/sec</td>\n", 669 " </tr>\n", 670 " </tbody>\n", 671 "</table>\n", 672 "</div>" 673 ], 674 "text/plain": [ 675 " score runtime_mean rate_string\n", 676 "AES 649 0.261358 501.0 MB/sec\n", 677 "HDR 3050 0.394782 11.1 Mpixels/sec\n", 678 "Rigid Body Physics 2147 0.239238 6285.5 FPS\n", 679 "HTML5 Parse 1565 0.173668 7.11 MB/sec\n", 680 "Lua 1220 0.254486 1.25 MB/sec\n", 681 "Camera 2605 0.148324 7.22 images/sec\n", 682 "Histogram Equalization 1425 0.245729 44.6 Mpixels/sec\n", 683 "SQLite 1189 0.612012 33.0 Krows/sec\n", 684 "Face Detection 2106 0.123343 615.2 Ksubwindows/sec\n", 685 "Memory Copy 2691 0.269374 7.46 GB/sec\n", 686 "Memory Latency 917 1.001564 471.8 ns\n", 687 "Canny 2191 0.167685 30.4 Mpixels/sec\n", 688 "PDF Rendering 2137 0.452174 56.8 Mpixels/sec\n", 689 "Gaussian Blur 2313 0.126649 40.5 Mpixels/sec\n", 690 "Speech Recognition 1263 1.126760 10.8 Words/sec\n", 691 "LLVM 1817 1.057730 124.9 functions/sec\n", 692 "Ray Tracing 1818 0.339196 265.5 Kpixels/sec\n", 693 "JPEG 2605 0.248902 21.0 Mpixels/sec\n", 694 "SGEMM 551 0.790329 11.7 Gflops\n", 695 "LZMA 1211 0.495195 1.89 MB/sec\n", 696 "SFFT 1130 0.169216 2.82 Gflops\n", 697 "Memory Bandwidth 2420 0.100460 12.9 GB/sec\n", 698 "N-Body Physics 1445 0.256496 1.08 Mpairs/sec\n", 699 "Dijkstra 2063 0.374775 1.40 MTE/sec\n", 700 "HTML5 DOM 613 0.913885 556.2 KElements/sec" 701 ] 702 }, 703 "metadata": {}, 704 "output_type": "display_data" 705 }, 706 { 707 "name": "stdout", 708 "output_type": "stream", 709 "text": [ 710 "----- Multi-Core benchmark -----\n" 711 ] 712 }, 713 { 714 "data": { 715 "text/html": [ 716 "<div>\n", 717 "<table border=\"1\" class=\"dataframe\">\n", 718 " <thead>\n", 719 " <tr style=\"text-align: right;\">\n", 720 " <th></th>\n", 721 " <th>score</th>\n", 722 " <th>runtime_mean</th>\n", 723 " <th>rate_string</th>\n", 724 " </tr>\n", 725 " </thead>\n", 726 " <tbody>\n", 727 " <tr>\n", 728 " <th>AES</th>\n", 729 " <td>2086</td>\n", 730 " <td>0.336575</td>\n", 731 " <td>1.57 GB/sec</td>\n", 732 " </tr>\n", 733 " <tr>\n", 734 " <th>HDR</th>\n", 735 " <td>8435</td>\n", 736 " <td>0.588926</td>\n", 737 " <td>30.6 Mpixels/sec</td>\n", 738 " </tr>\n", 739 " <tr>\n", 740 " <th>Rigid Body Physics</th>\n", 741 " <td>5976</td>\n", 742 " <td>0.363937</td>\n", 743 " <td>17496.3 FPS</td>\n", 744 " </tr>\n", 745 " <tr>\n", 746 " <th>HTML5 Parse</th>\n", 747 " <td>4747</td>\n", 748 " <td>0.269989</td>\n", 749 " <td>21.6 MB/sec</td>\n", 750 " </tr>\n", 751 " <tr>\n", 752 " <th>Lua</th>\n", 753 " <td>3475</td>\n", 754 " <td>0.385767</td>\n", 755 " <td>3.57 MB/sec</td>\n", 756 " </tr>\n", 757 " <tr>\n", 758 " <th>Camera</th>\n", 759 " <td>7272</td>\n", 760 " <td>0.203743</td>\n", 761 " <td>20.2 images/sec</td>\n", 762 " </tr>\n", 763 " <tr>\n", 764 " <th>Histogram Equalization</th>\n", 765 " <td>3785</td>\n", 766 " <td>0.368308</td>\n", 767 " <td>118.3 Mpixels/sec</td>\n", 768 " </tr>\n", 769 " <tr>\n", 770 " <th>SQLite</th>\n", 771 " <td>3287</td>\n", 772 " <td>1.166350</td>\n", 773 " <td>91.1 Krows/sec</td>\n", 774 " </tr>\n", 775 " <tr>\n", 776 " <th>Face Detection</th>\n", 777 " <td>5702</td>\n", 778 " <td>0.188087</td>\n", 779 " <td>1.67 Msubwindows/sec</td>\n", 780 " </tr>\n", 781 " <tr>\n", 782 " <th>Memory Copy</th>\n", 783 " <td>3974</td>\n", 784 " <td>0.378651</td>\n", 785 " <td>11.0 GB/sec</td>\n", 786 " </tr>\n", 787 " <tr>\n", 788 " <th>Memory Latency</th>\n", 789 " <td>1865</td>\n", 790 " <td>0.705881</td>\n", 791 " <td>232.1 ns</td>\n", 792 " </tr>\n", 793 " <tr>\n", 794 " <th>Canny</th>\n", 795 " <td>6648</td>\n", 796 " <td>0.265390</td>\n", 797 " <td>92.2 Mpixels/sec</td>\n", 798 " </tr>\n", 799 " <tr>\n", 800 " <th>PDF Rendering</th>\n", 801 " <td>5910</td>\n", 802 " <td>0.663689</td>\n", 803 " <td>157.0 Mpixels/sec</td>\n", 804 " </tr>\n", 805 " <tr>\n", 806 " <th>Gaussian Blur</th>\n", 807 " <td>6175</td>\n", 808 " <td>0.200005</td>\n", 809 " <td>108.2 Mpixels/sec</td>\n", 810 " </tr>\n", 811 " <tr>\n", 812 " <th>Speech Recognition</th>\n", 813 " <td>3755</td>\n", 814 " <td>1.520384</td>\n", 815 " <td>32.1 Words/sec</td>\n", 816 " </tr>\n", 817 " <tr>\n", 818 " <th>LLVM</th>\n", 819 " <td>6785</td>\n", 820 " <td>1.141085</td>\n", 821 " <td>466.6 functions/sec</td>\n", 822 " </tr>\n", 823 " <tr>\n", 824 " <th>Ray Tracing</th>\n", 825 " <td>5002</td>\n", 826 " <td>0.498715</td>\n", 827 " <td>730.4 Kpixels/sec</td>\n", 828 " </tr>\n", 829 " <tr>\n", 830 " <th>JPEG</th>\n", 831 " <td>7192</td>\n", 832 " <td>0.363033</td>\n", 833 " <td>57.9 Mpixels/sec</td>\n", 834 " </tr>\n", 835 " <tr>\n", 836 " <th>SGEMM</th>\n", 837 " <td>1697</td>\n", 838 " <td>1.057814</td>\n", 839 " <td>35.9 Gflops</td>\n", 840 " </tr>\n", 841 " <tr>\n", 842 " <th>LZMA</th>\n", 843 " <td>3977</td>\n", 844 " <td>0.587279</td>\n", 845 " <td>6.21 MB/sec</td>\n", 846 " </tr>\n", 847 " <tr>\n", 848 " <th>SFFT</th>\n", 849 " <td>3443</td>\n", 850 " <td>0.250660</td>\n", 851 " <td>8.58 Gflops</td>\n", 852 " </tr>\n", 853 " <tr>\n", 854 " <th>Memory Bandwidth</th>\n", 855 " <td>3375</td>\n", 856 " <td>0.142451</td>\n", 857 " <td>18.0 GB/sec</td>\n", 858 " </tr>\n", 859 " <tr>\n", 860 " <th>N-Body Physics</th>\n", 861 " <td>3891</td>\n", 862 " <td>0.398617</td>\n", 863 " <td>2.91 Mpairs/sec</td>\n", 864 " </tr>\n", 865 " <tr>\n", 866 " <th>Dijkstra</th>\n", 867 " <td>5168</td>\n", 868 " <td>0.601840</td>\n", 869 " <td>3.50 MTE/sec</td>\n", 870 " </tr>\n", 871 " <tr>\n", 872 " <th>HTML5 DOM</th>\n", 873 " <td>2158</td>\n", 874 " <td>1.031061</td>\n", 875 " <td>1.96 MElements/sec</td>\n", 876 " </tr>\n", 877 " </tbody>\n", 878 "</table>\n", 879 "</div>" 880 ], 881 "text/plain": [ 882 " score runtime_mean rate_string\n", 883 "AES 2086 0.336575 1.57 GB/sec\n", 884 "HDR 8435 0.588926 30.6 Mpixels/sec\n", 885 "Rigid Body Physics 5976 0.363937 17496.3 FPS\n", 886 "HTML5 Parse 4747 0.269989 21.6 MB/sec\n", 887 "Lua 3475 0.385767 3.57 MB/sec\n", 888 "Camera 7272 0.203743 20.2 images/sec\n", 889 "Histogram Equalization 3785 0.368308 118.3 Mpixels/sec\n", 890 "SQLite 3287 1.166350 91.1 Krows/sec\n", 891 "Face Detection 5702 0.188087 1.67 Msubwindows/sec\n", 892 "Memory Copy 3974 0.378651 11.0 GB/sec\n", 893 "Memory Latency 1865 0.705881 232.1 ns\n", 894 "Canny 6648 0.265390 92.2 Mpixels/sec\n", 895 "PDF Rendering 5910 0.663689 157.0 Mpixels/sec\n", 896 "Gaussian Blur 6175 0.200005 108.2 Mpixels/sec\n", 897 "Speech Recognition 3755 1.520384 32.1 Words/sec\n", 898 "LLVM 6785 1.141085 466.6 functions/sec\n", 899 "Ray Tracing 5002 0.498715 730.4 Kpixels/sec\n", 900 "JPEG 7192 0.363033 57.9 Mpixels/sec\n", 901 "SGEMM 1697 1.057814 35.9 Gflops\n", 902 "LZMA 3977 0.587279 6.21 MB/sec\n", 903 "SFFT 3443 0.250660 8.58 Gflops\n", 904 "Memory Bandwidth 3375 0.142451 18.0 GB/sec\n", 905 "N-Body Physics 3891 0.398617 2.91 Mpairs/sec\n", 906 "Dijkstra 5168 0.601840 3.50 MTE/sec\n", 907 "HTML5 DOM 2158 1.031061 1.96 MElements/sec" 908 ] 909 }, 910 "metadata": {}, 911 "output_type": "display_data" 912 } 913 ], 914 "source": [ 915 "for f in os.listdir(te.res_dir):\n", 916 " if f.endswith(\".gb4\"):\n", 917 " geekbench = Geekbench(te.res_dir + \"/\" + f)\n", 918 " \n", 919 " print \"Analysing geekbench {}\".format(geekbench.name())\n", 920 " display_bench_results(geekbench, True)" 921 ] 922 }, 923 { 924 "cell_type": "markdown", 925 "metadata": { 926 "deletable": true, 927 "editable": true 928 }, 929 "source": [ 930 "## Analysing several runs" 931 ] 932 }, 933 { 934 "cell_type": "markdown", 935 "metadata": { 936 "deletable": true, 937 "editable": true 938 }, 939 "source": [ 940 "It can be interesting to compare Geekbench results with different parameters (kernel, drivers) and even different devices to gauge the impact of these parameters. As Geekbench results can vary a bit from one run to another, having a set of repeated runs is preferable.\n", 941 "\n", 942 "The following section will grab the results of all the **Geekbench\\_exemple\\_\\*** results found in the LISA results directory" 943 ] 944 }, 945 { 946 "cell_type": "code", 947 "execution_count": 10, 948 "metadata": { 949 "collapsed": false, 950 "deletable": true, 951 "editable": true 952 }, 953 "outputs": [], 954 "source": [ 955 "import glob\n", 956 "\n", 957 "def fetch_results():\n", 958 " results_path = os.path.join(te.LISA_HOME, \"results\")\n", 959 " \n", 960 " results_dirs = [results_path + \"/\" + d for d in os.listdir(results_path) if d.startswith(\"Geekbench_example_\")]\n", 961 " \n", 962 " res = []\n", 963 " \n", 964 " for d in results_dirs:\n", 965 " bench_file = glob.glob(\"{}/*.gb4\".format(d))[0]\n", 966 " res.append(Geekbench(bench_file))\n", 967 " \n", 968 " return res\n", 969 "\n", 970 "def compare_runs():\n", 971 " geekbenches = fetch_results()\n", 972 " \n", 973 " # Pick one run to build a baseline template\n", 974 " benchmarks = geekbenches[0].benchmarks_names()\n", 975 " workloads = geekbenches[0].workloads_names()\n", 976 " \n", 977 " stats = [\"avg\", \"min\", \"max\"]\n", 978 " count = len(geekbenches)\n", 979 " \n", 980 " print \"Parsing {} runs\".format(count)\n", 981 "\n", 982 " \n", 983 " # Initialize stats\n", 984 " results = {benchmark : \n", 985 " {\"min\" : sys.maxint, \"max\" : 0, \"avg\" : 0} \n", 986 " for benchmark in benchmarks}\n", 987 " \n", 988 " # Get all the data\n", 989 " for benchmark in results.iterkeys():\n", 990 " for bench in geekbenches:\n", 991 " score = bench.global_scores()[benchmark]\n", 992 " \n", 993 " if score > results[benchmark][\"max\"]:\n", 994 " results[benchmark][\"max\"] = score\n", 995 " \n", 996 " if score < results[benchmark][\"min\"]:\n", 997 " results[benchmark][\"min\"] = score\n", 998 " \n", 999 " results[benchmark][\"avg\"] += score\n", 1000 " \n", 1001 " results[benchmark][\"avg\"] /= count\n", 1002 " \n", 1003 " # Convert data to Dataframe\n", 1004 " data = []\n", 1005 "\n", 1006 " for benchmark in results.iterkeys():\n", 1007 " row = []\n", 1008 " for stat in stats:\n", 1009 " row.append(results[benchmark][stat])\n", 1010 " data.append(tuple(row))\n", 1011 " \n", 1012 " df = pd.DataFrame(data, index=results.iterkeys(), columns=stats)\n", 1013 " \n", 1014 " return df" 1015 ] 1016 }, 1017 { 1018 "cell_type": "code", 1019 "execution_count": 11, 1020 "metadata": { 1021 "collapsed": false, 1022 "deletable": true, 1023 "editable": true 1024 }, 1025 "outputs": [ 1026 { 1027 "name": "stdout", 1028 "output_type": "stream", 1029 "text": [ 1030 "Parsing 2 runs\n" 1031 ] 1032 }, 1033 { 1034 "data": { 1035 "text/html": [ 1036 "<div>\n", 1037 "<table border=\"1\" class=\"dataframe\">\n", 1038 " <thead>\n", 1039 " <tr style=\"text-align: right;\">\n", 1040 " <th></th>\n", 1041 " <th>avg</th>\n", 1042 " <th>min</th>\n", 1043 " <th>max</th>\n", 1044 " </tr>\n", 1045 " </thead>\n", 1046 " <tbody>\n", 1047 " <tr>\n", 1048 " <th>Single-Core</th>\n", 1049 " <td>1602</td>\n", 1050 " <td>1594</td>\n", 1051 " <td>1610</td>\n", 1052 " </tr>\n", 1053 " <tr>\n", 1054 " <th>Multi-Core</th>\n", 1055 " <td>4176</td>\n", 1056 " <td>4170</td>\n", 1057 " <td>4182</td>\n", 1058 " </tr>\n", 1059 " </tbody>\n", 1060 "</table>\n", 1061 "</div>" 1062 ], 1063 "text/plain": [ 1064 " avg min max\n", 1065 "Single-Core 1602 1594 1610\n", 1066 "Multi-Core 4176 4170 4182" 1067 ] 1068 }, 1069 "metadata": {}, 1070 "output_type": "display_data" 1071 } 1072 ], 1073 "source": [ 1074 "display(compare_runs())" 1075 ] 1076 } 1077 ], 1078 "metadata": { 1079 "kernelspec": { 1080 "display_name": "Python 2", 1081 "language": "python", 1082 "name": "python2" 1083 }, 1084 "language_info": { 1085 "codemirror_mode": { 1086 "name": "ipython", 1087 "version": 2 1088 }, 1089 "file_extension": ".py", 1090 "mimetype": "text/x-python", 1091 "name": "python", 1092 "nbconvert_exporter": "python", 1093 "pygments_lexer": "ipython2", 1094 "version": "2.7.6" 1095 }, 1096 "toc": { 1097 "toc_cell": false, 1098 "toc_number_sections": true, 1099 "toc_threshold": 6, 1100 "toc_window_display": false 1101 } 1102 }, 1103 "nbformat": 4, 1104 "nbformat_minor": 0 1105} 1106