{ "cells": [ { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "# Run Generic Automated EAS tests\n", "This is a starting-point notebook for running tests from the generic EAS suite in `tests/eas/generic.py`. The test classes that are imported here provide helper methods to aid analysis of the cause of failure. You can use Python's `help` built-in to find those methods (or you can just read the docstrings in the code).\n", "\n", "These tests make estimation of the energy efficiency of task placements, without directly examining the behaviour of cpufreq or cpuidle. Several test classes are provided, the only difference between them being the workload that is used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Setup" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:17,847 INFO : root : Using LISA logging configuration:\n", "2017-01-23 18:50:17,847 INFO : root : /home/brendan/sources/lisa/logging.conf\n" ] } ], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline\n", "\n", "import logging\n", "from conf import LisaLogging\n", "LisaLogging.setup()#level=logging.WARNING)\n", "\n", "import pandas as pd\n", "\n", "from perf_analysis import PerfAnalysis\n", "\n", "import trappy\n", "from trappy import ILinePlot\n", "from trappy.stats.grammar import Parser" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run test workload\n", "\n", "If you simply want to run all the tests and get pass/fail results, use this command in the LISA shell: `lisa-test tests/eas/generic.py`. This notebook is intended as a starting point for analysing what scheduler behaviour was judged to be faulty.\n", "\n", "Target configuration is taken from `$LISA_HOME/target.config` - you'll need to edit that file to provide connection details for the target you want to test." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from tests.eas.generic import TwoBigTasks, TwoBigThreeSmall, RampUp, RampDown, EnergyModelWakeMigration, OneSmallTask" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default we'll run the EnergyModelWakeMigration test, which runs a workload alternating between high and low-intensity. All the other test classes shown above have the same interface, but run different workloads. To run the tests on different workloads, change this line below:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", " Test EAS for tasks alternating beetween 10% and 50%\n", " \n" ] } ], "source": [ "t = EnergyModelWakeMigration(methodName=\"test_task_placement\")\n", "print t.__doc__" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:18,176 INFO : LisaTest : Setup tests execution engine...\n", "2017-01-23 18:50:18,178 INFO : TestEnv : Using base path: /home/brejac01/sources/lisa\n", "2017-01-23 18:50:18,178 INFO : TestEnv : Loading default (file) target configuration\n", "2017-01-23 18:50:18,179 INFO : TestEnv : Loading target configuration [/home/brejac01/sources/lisa/target.config]...\n", "2017-01-23 18:50:18,181 INFO : TestEnv : Loading custom (inline) test configuration\n", "2017-01-23 18:50:18,182 INFO : TestEnv : Devlib modules to load: ['bl', u'cpuidle', 'cgroups', 'cpufreq']\n", "2017-01-23 18:50:18,183 INFO : TestEnv : Connecting linux target:\n", "2017-01-23 18:50:18,184 INFO : TestEnv : username : brendan\n", "2017-01-23 18:50:18,185 INFO : TestEnv : host : 192.168.2.2\n", "2017-01-23 18:50:18,186 INFO : TestEnv : password : password\n", "2017-01-23 18:50:18,187 INFO : TestEnv : Connection settings:\n", "2017-01-23 18:50:18,188 INFO : TestEnv : {'username': u'brendan', 'host': u'192.168.2.2', 'password': u'password'}\n", "2017-01-23 18:50:25,803 INFO : TestEnv : Initializing target workdir:\n", "2017-01-23 18:50:25,805 INFO : TestEnv : /home/brendan/devlib-target\n", "2017-01-23 18:50:29,349 INFO : CGroups : Available controllers:\n", "2017-01-23 18:50:31,076 INFO : CGroups : schedtune : /home/brendan/devlib-target/cgroups/devlib_cgh2\n", "2017-01-23 18:50:31,749 INFO : TestEnv : Topology:\n", "2017-01-23 18:50:31,751 INFO : TestEnv : [[0, 3, 4, 5], [1, 2]]\n", "2017-01-23 18:50:34,373 INFO : TestEnv : Loading default EM:\n", "2017-01-23 18:50:34,374 INFO : TestEnv : /home/brejac01/sources/lisa/libs/utils/platforms/juno.json\n", "2017-01-23 18:50:36,104 INFO : TestEnv : Enabled tracepoints:\n", "2017-01-23 18:50:36,106 INFO : TestEnv : sched_overutilized\n", "2017-01-23 18:50:36,107 INFO : TestEnv : sched_energy_diff\n", "2017-01-23 18:50:36,108 INFO : TestEnv : sched_load_avg_task\n", "2017-01-23 18:50:36,109 INFO : TestEnv : sched_load_avg_cpu\n", "2017-01-23 18:50:36,110 INFO : TestEnv : sched_migrate_task\n", "2017-01-23 18:50:36,111 INFO : TestEnv : sched_switch\n", "2017-01-23 18:50:36,112 INFO : TestEnv : cpu_frequency\n", "2017-01-23 18:50:36,114 INFO : EnergyMeter : HWMON module not enabled\n", "2017-01-23 18:50:36,115 WARNING : EnergyMeter : Energy sampling disabled by configuration\n", "2017-01-23 18:50:36,116 INFO : TestEnv : Set results folder to:\n", "2017-01-23 18:50:36,117 INFO : TestEnv : /home/brejac01/sources/lisa/results/20170123_185036\n", "2017-01-23 18:50:36,118 INFO : TestEnv : Experiment results available also in:\n", "2017-01-23 18:50:36,119 INFO : TestEnv : /home/brejac01/sources/lisa/results_latest\n", "2017-01-23 18:50:36,121 INFO : Executor : Loading custom (inline) test configuration\n", "2017-01-23 18:50:36,121 INFO : Executor : \n", "2017-01-23 18:50:36,123 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:36,124 INFO : Executor : Experiments configuration\n", "2017-01-23 18:50:36,125 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:36,126 INFO : Executor : Configured to run:\n", "2017-01-23 18:50:36,127 INFO : Executor : 1 target configurations:\n", "2017-01-23 18:50:36,128 INFO : Executor : energy_aware\n", "2017-01-23 18:50:36,129 INFO : Executor : 1 workloads (1 iterations each)\n", "2017-01-23 18:50:36,130 INFO : Executor : wake_migration\n", "2017-01-23 18:50:36,131 INFO : Executor : Total: 1 experiments\n", "2017-01-23 18:50:36,132 INFO : Executor : Results will be collected under:\n", "2017-01-23 18:50:36,133 INFO : Executor : /home/brejac01/sources/lisa/results/20170123_185036\n", "2017-01-23 18:50:36,134 INFO : Executor : rt-app workloads found, installing tool on target\n", "2017-01-23 18:50:38,252 INFO : LisaTest : Experiments execution...\n", "2017-01-23 18:50:38,254 INFO : Executor : \n", "2017-01-23 18:50:38,255 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:38,256 INFO : Executor : Experiments execution\n", "2017-01-23 18:50:38,257 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:38,258 INFO : Executor : \n", "2017-01-23 18:50:38,259 INFO : Executor : ================================================================================\n", "2017-01-23 18:50:38,260 INFO : Executor : configuring target for [energy_aware] experiments\n", "2017-01-23 18:50:39,976 INFO : Executor : Set scheduler feature: ENERGY_AWARE\n", "2017-01-23 18:50:40,881 WARNING : Executor : cpufreq governor not specified, using currently configured governor\n", "2017-01-23 18:50:40,883 INFO : Workload : Setup new workload wake_migration\n", "2017-01-23 18:50:40,884 INFO : Workload : Workload duration defined by longest task\n", "2017-01-23 18:50:40,885 INFO : Workload : Default policy: SCHED_OTHER\n", "2017-01-23 18:50:40,886 INFO : Workload : ------------------------\n", "2017-01-23 18:50:40,888 INFO : Workload : task [task_wmig0], sched: using default policy\n", "2017-01-23 18:50:40,889 INFO : Workload : | calibration CPU: 1\n", "2017-01-23 18:50:40,890 INFO : Workload : | loops count: 2\n", "2017-01-23 18:50:40,891 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", "2017-01-23 18:50:40,892 INFO : Workload : | period 100000 [us], duty_cycle 10 %\n", "2017-01-23 18:50:40,893 INFO : Workload : | run_time 10000 [us], sleep_time 90000 [us]\n", "2017-01-23 18:50:40,894 INFO : Workload : + phase_000002: duration 1.000000 [s] (10 loops)\n", "2017-01-23 18:50:40,895 INFO : Workload : | period 100000 [us], duty_cycle 50 %\n", "2017-01-23 18:50:40,896 INFO : Workload : | run_time 50000 [us], sleep_time 50000 [us]\n", "2017-01-23 18:50:40,897 INFO : Workload : ------------------------\n", "2017-01-23 18:50:40,897 INFO : Workload : task [task_wmig1], sched: using default policy\n", "2017-01-23 18:50:40,898 INFO : Workload : | calibration CPU: 1\n", "2017-01-23 18:50:40,899 INFO : Workload : | loops count: 2\n", "2017-01-23 18:50:40,900 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n", "2017-01-23 18:50:40,901 INFO : Workload : | period 100000 [us], duty_cycle 10 %\n", "2017-01-23 18:50:40,902 INFO : Workload : | run_time 10000 [us], sleep_time 90000 [us]\n", "2017-01-23 18:50:40,903 INFO : Workload : + phase_000002: duration 1.000000 [s] (10 loops)\n", "2017-01-23 18:50:40,904 INFO : Workload : | period 100000 [us], duty_cycle 50 %\n", "2017-01-23 18:50:40,905 INFO : Workload : | run_time 50000 [us], sleep_time 50000 [us]\n", "2017-01-23 18:50:41,966 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n", "2017-01-23 18:50:41,968 INFO : Executor : Experiment 0/1, [energy_aware:wake_migration] 1/1\n", "2017-01-23 18:50:41,976 WARNING : Executor : FTrace events collection enabled\n", "2017-01-23 18:50:48,009 INFO : Workload : Workload execution START:\n", "2017-01-23 18:50:48,011 INFO : Workload : /home/brendan/devlib-target/bin/rt-app /home/brendan/devlib-target/run_dir/wake_migration_00.json 2>&1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:57,357 INFO : Executor : Collected FTrace binary trace:\n", "2017-01-23 18:50:57,359 INFO : Executor : /rtapp:energy_aware:wake_migration/1/trace.dat\n", "2017-01-23 18:50:57,360 INFO : Executor : Collected FTrace function profiling:\n", "2017-01-23 18:50:57,361 INFO : Executor : /rtapp:energy_aware:wake_migration/1/trace_stat.json\n", "2017-01-23 18:50:57,362 INFO : Executor : --------------------------------------------------------------------------------\n", "2017-01-23 18:50:57,363 INFO : Executor : \n", "2017-01-23 18:50:57,365 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:57,366 INFO : Executor : Experiments execution completed\n", "2017-01-23 18:50:57,367 INFO : Executor : ################################################################################\n", "2017-01-23 18:50:57,368 INFO : Executor : Results available in:\n", "2017-01-23 18:50:57,369 INFO : Executor : /home/brejac01/sources/lisa/results/20170123_185036\n" ] } ], "source": [ "t.setUpClass()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "experiment = t.executor.experiments[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Examine trace\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`get_power_df` and `get_expected_power_df` look at the ftrace results from the workload estimation and judge the energy efficiency of the system, considering *only task placement* (assuming perfect load-tracking/prediction, cpuidle, and cpufreq systems). The energy estimation doesn't take every single wakeup and idle period into account, but simply estimates an average power usage over the time that each task spent attached to each CPU during each phase of the rt-app workload.\n", "\n", "These return DataFrames estimating the energy usage of the system under each task placement. `estimated_power` will represent this estimation for the scheduling pattern that we actually observed, while `expected_power` will represent our estimation of how much power an **optimal** scheduling pattern would use.\n", "\n", "Check the docstrings for these functions (and other functions in the test class) for more detail." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [], "source": [ "# print t.get_power_df.__doc__\n", "estimated_power = t.get_power_df(experiment)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:58,352 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,355 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:58,356 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,361 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:58,363 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,367 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:58,369 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,374 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:58,375 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,378 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:58,380 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:58,381 INFO : EnergyModel : EnergyModel - Done\n" ] } ], "source": [ "# print t.get_expected_power_df.__doc__\n", "expected_power = t.get_expected_power_df(experiment)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot Schedule" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "trace = t.get_trace(experiment)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "scrolled": false }, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n", "\n", "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "trappy.plotter.plot_trace(trace.ftrace)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot estimated ideal and estimated power usage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This plot shows how the power estimation for the observed scheduling pattern varies from the estimated power for an ideal schedule.\n", "\n", "Where the plotted value for the observed power is higher than the plotted ideal power, the system was wasting power (e.g. a low-intensity task was unnecessarily placed on a high-power CPU). Where the observed value is *lower* than the ideal value, this means the system was *too* efficient (e.g. a high-intensity task was placed on a low-power CPU that could not accomadate its compute requirements)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
\n", "\n", "\n", "\n", "\n", "\n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "df = pd.concat([\n", " expected_power.sum(axis=1), estimated_power.sum(axis=1)], \n", " axis=1, keys=['ideal_power', 'observed_power']).fillna(method='ffill')\n", "\n", "ILinePlot(df, column=df.columns.tolist(), drawstyle='steps-post').view()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot CPU frequency" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:59,019 WARNING : Analysis : Events [cpu_frequency] not found, plot DISABLED!\n" ] } ], "source": [ "trace.analysis.frequency.plotClusterFrequencies()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Assertions\n", "These are the assertions used to generate pass/fail results s. They aren't very useful in this interactive context - it's much more interesting to examine plots like the one above and see whether the behaviour was desirable or not. These are intended for automated regression testing. Nonetheless, let's see what the results would be for this run." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`test_slack` checks the \"slack\" reported by the rt-app workload. If this slack was negative, this means the workload didn't receive enough CPU capacity. In a real system this would represent lacking interactive performance." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "test_slack failed:\n", "task task_wmig0 missed 10.0% of activations\n", "\tCheck trace file: /home/brejac01/sources/lisa/results/20170123_185036/rtapp:energy_aware:wake_migration/1/trace.dat\n" ] } ], "source": [ "try:\n", " t.test_slack()\n", "except AssertionError as e:\n", " print \"test_slack failed:\"\n", " print e\n", "else:\n", " print \"test_slack passed\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`test_task_placement` checks that the task placement was energy efficient, taking advantage of lower-power CPUs whenever possible." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2017-01-23 18:50:59,076 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,079 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:59,080 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,083 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:59,084 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,086 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:59,087 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,090 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:59,092 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,094 INFO : EnergyModel : EnergyModel - Done\n", "2017-01-23 18:50:59,095 INFO : EnergyModel : EnergyModel - Searching 36 configurations for optimal task placement...\n", "2017-01-23 18:50:59,096 INFO : EnergyModel : EnergyModel - Done\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "test_task_placement passed\n" ] } ], "source": [ "try:\n", " t.test_task_placement()\n", "except AssertionError as e:\n", " print \"test_task_placement failed:\"\n", " print e\n", "else:\n", " print \"test_task_placement passed\"" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 1 }