{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Executor API - Executor\n",
"\n",
" A tests executor is a module which supports the execution of a configured set of experiments.
\n",
" Each experiment is composed by:\n",
" - a target configuration\n",
" - a workload to execute\n",
"\n",
"The executor module can be configured to run a set of workloads (wloads) in each different target configuration of a specified set (confs). These wloads and confs can be specified by the \"experiments_conf\" input dictionary which is described below at **Experiments Configuration**.
\n",
"All the results generated by each experiment will be collected in a results folder. The format and content fo the results forlder is detailed in the last cell of **Tests execution** below.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-08 11:58:22,693 INFO : root : Using LISA logging configuration:\n",
"2016-12-08 11:58:22,694 INFO : root : /home/vagrant/lisa/logging.conf\n"
]
}
],
"source": [
"import logging\n",
"\n",
"from conf import LisaLogging\n",
"LisaLogging.setup()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import os\n",
"import json\n",
"\n",
"from env import TestEnv\n",
"from executor import Executor"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Target Configuration\n",
"The target configuration it's used to describe and configure your test environment.\n",
"You can find more details in **examples/utils/testenv_example.ipynb**."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Setup a target configuration\n",
"my_target_conf = {\n",
" \n",
" # Target platform and board\n",
" \"platform\" : 'linux',\n",
" \"board\" : 'juno',\n",
" \n",
" # Target board IP/MAC address\n",
" \"host\" : '192.168.0.1',\n",
" \n",
" # Login credentials\n",
" \"username\" : 'root',\n",
" \"password\" : 'juno',\n",
"\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiments Configuration\n",
"\n",
"The experiments configuration defines the software setups that we need on our hardware target.
\n",
"This can be given as an argument to an Executor instance or to a TestEnv one.
\n",
"Elements of the experiments configuration:\n",
" - **confs**: **mandatory** platform configurations to be tested.\n",
" - tag: relevant string to identify your configuration.\n",
" - flags: ftrace (to enable ftrace events) is the only one supported at the moment.\n",
" - sched_features: features to be added to /sys/kernel/debug/sched_features.\n",
" - cpufreq: CpuFreq governor and tunables.\n",
" - cgroups: CGroups configuration (controller). The default CGroup will be used otherwise.\n",
" - **wloads**: **mandatory** workloads to run on each platform configuration.\n",
" - **iterations**: number of iterations for each workload.\n",
" - **tools**: binary tools (available under ./tools/$ARCH/) to install by default; these will be merged with the ones in the target configuration.\n",
" - **ftrace**: FTrace events to collect for all the experiments configurations which have the \"ftrace\" flag enabled.\n",
" - **modules**: modules required by the experiments resulted from the experiments configurations.\n",
" - **exclude_modules** - modules to be disabled.\n",
" - **results_dir**: results directory - experiments configuration results directory overrides target one."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [],
"source": [
"my_experiments_conf = {\n",
"\n",
" # Folder where all the results will be collected\n",
" \"results_dir\" : \"ExecutorExample\",\n",
"\n",
" # Platform configurations to test: you can specify any number of configurations\n",
" \"confs\" : [\n",
" { \n",
" \"tag\" : \"base\", # Relevant string to identify configuration\n",
" \"flags\" : [\"ftrace\", \"freeze_userspace\"], # Enable FTrace events, freeze userspace while running \n",
" \"sched_features\" : \"NO_ENERGY_AWARE\", # Disable EAS\n",
" \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n",
" \"governor\" : \"performance\",\n",
" },\n",
" },\n",
" {\n",
" \"tag\" : \"eas\", # Relevant string to identify configuration\n",
" \"flags\" : [\"ftrace\", \"freeze_userspace\"], # Enable FTrace events, freeze userspace while running\n",
" \"sched_features\" : \"ENERGY_AWARE\", # Enable EAS\n",
" \"cpufreq\" : { # Use PERFORMANCE CpuFreq\n",
" \"governor\" : \"performance\",\n",
" },\n",
" },\n",
" ],\n",
" \n",
" # Workloads to run (on each platform configuration)\n",
" \"wloads\" : {\n",
" # Run hackbench with 1 group using pipes\n",
" \"perf\" : {\n",
" \"type\" : \"perf_bench\",\n",
" \"conf\" : {\n",
" \"class\" : \"messaging\",\n",
" \"params\" : {\n",
" \"group\" : 1,\n",
" \"loop\" : 10,\n",
" \"pipe\" : True,\n",
" \"thread\": True,\n",
" }\n",
" }\n",
" },\n",
" # Run a 20% duty-cycle periodic task\n",
" \"rta\" : {\n",
" \"type\" : \"rt-app\",\n",
" \"loadref\" : \"big\",\n",
" \"conf\" : {\n",
" \"class\" : \"profile\",\n",
" \"params\" : {\n",
" \"p20\" : {\n",
" \"kind\" : \"Periodic\",\n",
" \"params\" : {\n",
" \"duty_cycle_pct\" : 20,\n",
" },\n",
" },\n",
" },\n",
" },\n",
" },\n",
" },\n",
" \n",
" # Number of iterations for each workloaditerations\n",
" \"iterations\" : 1,\n",
"}\n",
" \n",
"my_test_conf = {\n",
" # FTrace events to collect for all the tests configuration which have\n",
" # the \"ftrace\" flag enabled\n",
" \"ftrace\" : {\n",
" \"events\" : [\n",
" \"sched_switch\",\n",
" \"sched_wakeup\",\n",
" \"sched_wakeup_new\",\n",
" \"cpu_frequency\",\n",
" ],\n",
" \"buffsize\" : 80 * 1024,\n",
" },\n",
" \n",
" # Tools required by the experiments\n",
" \"tools\" : [ 'trace-cmd', 'perf' ],\n",
" \n",
" # Modules required by these experiments\n",
" \"modules\" : [ 'bl', 'cpufreq', 'cgroups' ],\n",
"\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tests execution"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-07 10:17:28,037 INFO : TestEnv : Using base path: /home/vagrant/lisa\n",
"2016-12-07 10:17:28,039 INFO : TestEnv : Loading custom (inline) target configuration\n",
"2016-12-07 10:17:28,039 INFO : TestEnv : Devlib modules to load: ['bl', 'hwmon', 'cpufreq']\n",
"2016-12-07 10:17:28,040 INFO : TestEnv : Connecting linux target:\n",
"2016-12-07 10:17:28,040 INFO : TestEnv : username : root\n",
"2016-12-07 10:17:28,041 INFO : TestEnv : host : 192.168.0.1\n",
"2016-12-07 10:17:28,041 INFO : TestEnv : password : juno\n",
"2016-12-07 10:17:28,041 INFO : TestEnv : Connection settings:\n",
"2016-12-07 10:17:28,042 INFO : TestEnv : {'username': 'root', 'host': '192.168.0.1', 'password': 'juno'}\n",
"2016-12-07 10:17:45,282 INFO : TestEnv : Initializing target workdir:\n",
"2016-12-07 10:17:45,283 INFO : TestEnv : /root/devlib-target\n",
"2016-12-07 10:17:51,006 INFO : TestEnv : Topology:\n",
"2016-12-07 10:17:51,007 INFO : TestEnv : [[0, 3, 4, 5], [1, 2]]\n",
"2016-12-07 10:17:52,248 INFO : EnergyMeter : Scanning for HWMON channels, may take some time...\n",
"2016-12-07 10:17:52,250 INFO : EnergyMeter : Channels selected for energy sampling:\n",
"2016-12-07 10:17:52,250 INFO : EnergyMeter : BOARDBIG_energy\n",
"2016-12-07 10:17:52,251 INFO : EnergyMeter : BOARDLITTLE_energy\n",
"2016-12-07 10:17:52,251 INFO : TestEnv : Set results folder to:\n",
"2016-12-07 10:17:52,251 INFO : TestEnv : /home/vagrant/lisa/results/20161207_101752\n",
"2016-12-07 10:17:52,252 INFO : TestEnv : Experiment results available also in:\n",
"2016-12-07 10:17:52,252 INFO : TestEnv : /home/vagrant/lisa/results_latest\n",
"2016-12-07 10:17:52,253 INFO : Executor : Loading custom (inline) test configuration\n",
"2016-12-07 10:17:52,253 INFO : Executor : \n",
"2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n",
"2016-12-07 10:17:52,254 INFO : Executor : Experiments configuration\n",
"2016-12-07 10:17:52,254 INFO : Executor : ################################################################################\n",
"2016-12-07 10:17:52,255 INFO : Executor : Configured to run:\n",
"2016-12-07 10:17:52,255 INFO : Executor : 2 target configurations:\n",
"2016-12-07 10:17:52,256 INFO : Executor : base, eas\n",
"2016-12-07 10:17:52,256 INFO : Executor : 2 workloads (1 iterations each)\n",
"2016-12-07 10:17:52,257 INFO : Executor : rta, perf\n",
"2016-12-07 10:17:52,257 INFO : Executor : Total: 4 experiments\n",
"2016-12-07 10:17:52,257 INFO : Executor : Results will be collected under:\n",
"2016-12-07 10:17:52,258 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n"
]
}
],
"source": [
"executor = Executor(TestEnv(target_conf=my_target_conf, test_conf=my_test_conf), my_experiments_conf)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2016-12-07 10:17:59,239 INFO : Executor : \n",
"2016-12-07 10:17:59,239 INFO : Executor : ################################################################################\n",
"2016-12-07 10:17:59,240 INFO : Executor : Experiments execution\n",
"2016-12-07 10:17:59,240 INFO : Executor : ################################################################################\n",
"2016-12-07 10:17:59,241 INFO : Executor : \n",
"2016-12-07 10:17:59,241 INFO : Executor : ================================================================================\n",
"2016-12-07 10:17:59,241 INFO : Executor : configuring target for [base] experiments\n",
"2016-12-07 10:18:00,663 INFO : Executor : Set scheduler feature: NO_ENERGY_AWARE\n",
"2016-12-07 10:18:01,469 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n",
"2016-12-07 10:18:02,274 INFO : Workload : Setup new workload rta\n",
"2016-12-07 10:18:02,274 INFO : Workload : Workload duration defined by longest task\n",
"2016-12-07 10:18:02,275 INFO : Workload : Default policy: SCHED_OTHER\n",
"2016-12-07 10:18:02,275 INFO : Workload : ------------------------\n",
"2016-12-07 10:18:02,276 INFO : Workload : task [task_p200], sched: using default policy\n",
"2016-12-07 10:18:02,276 INFO : Workload : | calibration CPU: 1\n",
"2016-12-07 10:18:02,277 INFO : Workload : | loops count: 1\n",
"2016-12-07 10:18:02,277 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n",
"2016-12-07 10:18:02,278 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n",
"2016-12-07 10:18:02,278 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n",
"2016-12-07 10:18:05,732 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
"2016-12-07 10:18:05,734 INFO : Executor : Experiment 0/4, [base:rta] 1/1\n",
"2016-12-07 10:18:06,361 INFO : Workload : Workload execution START:\n",
"2016-12-07 10:18:06,363 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n",
"2016-12-07 10:18:13,384 INFO : Executor : --------------------------------------------------------------------------------\n",
"2016-12-07 10:18:13,386 INFO : Workload : Setup new workload perf\n",
"2016-12-07 10:18:13,712 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
"2016-12-07 10:18:13,712 INFO : Executor : Experiment 1/4, [base:perf] 1/1\n",
"2016-12-07 10:18:14,337 INFO : Workload : Workload execution START:\n",
"2016-12-07 10:18:14,338 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n",
"2016-12-07 10:18:14,664 INFO : perf_bench : PerfBench - Completion time: 0.010000, Performance 100.000000\n",
"2016-12-07 10:18:15,286 INFO : Executor : --------------------------------------------------------------------------------\n",
"2016-12-07 10:18:15,288 INFO : Executor : \n",
"2016-12-07 10:18:15,290 INFO : Executor : ================================================================================\n",
"2016-12-07 10:18:15,292 INFO : Executor : configuring target for [eas] experiments\n",
"2016-12-07 10:18:16,713 INFO : Executor : Set scheduler feature: ENERGY_AWARE\n",
"2016-12-07 10:18:17,519 INFO : Executor : Configuring all CPUs to use [performance] cpufreq governor\n",
"2016-12-07 10:18:18,325 INFO : Workload : Setup new workload rta\n",
"2016-12-07 10:18:18,326 INFO : Workload : Workload duration defined by longest task\n",
"2016-12-07 10:18:18,327 INFO : Workload : Default policy: SCHED_OTHER\n",
"2016-12-07 10:18:18,329 INFO : Workload : ------------------------\n",
"2016-12-07 10:18:18,330 INFO : Workload : task [task_p200], sched: using default policy\n",
"2016-12-07 10:18:18,331 INFO : Workload : | calibration CPU: 1\n",
"2016-12-07 10:18:18,332 INFO : Workload : | loops count: 1\n",
"2016-12-07 10:18:18,332 INFO : Workload : + phase_000001: duration 1.000000 [s] (10 loops)\n",
"2016-12-07 10:18:18,333 INFO : Workload : | period 100000 [us], duty_cycle 20 %\n",
"2016-12-07 10:18:18,333 INFO : Workload : | run_time 20000 [us], sleep_time 80000 [us]\n",
"2016-12-07 10:18:21,414 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
"2016-12-07 10:18:21,415 INFO : Executor : Experiment 2/4, [eas:rta] 1/1\n",
"2016-12-07 10:18:22,043 INFO : Workload : Workload execution START:\n",
"2016-12-07 10:18:22,045 INFO : Workload : /root/devlib-target/bin/rt-app /root/devlib-target/run_dir/rta_00.json 2>&1\n",
"2016-12-07 10:18:28,802 INFO : Executor : --------------------------------------------------------------------------------\n",
"2016-12-07 10:18:28,806 INFO : Workload : Setup new workload perf\n",
"2016-12-07 10:18:29,134 INFO : Executor : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
"2016-12-07 10:18:29,135 INFO : Executor : Experiment 3/4, [eas:perf] 1/1\n",
"2016-12-07 10:18:29,760 INFO : Workload : Workload execution START:\n",
"2016-12-07 10:18:29,761 INFO : Workload : /root/devlib-target/bin/perf bench sched messaging --pipe --thread --group 1 --loop 10\n",
"2016-12-07 10:18:30,089 INFO : perf_bench : PerfBench - Completion time: 0.009000, Performance 111.111111\n",
"2016-12-07 10:18:30,710 INFO : Executor : --------------------------------------------------------------------------------\n",
"2016-12-07 10:18:30,712 INFO : Executor : \n",
"2016-12-07 10:18:30,714 INFO : Executor : ################################################################################\n",
"2016-12-07 10:18:30,715 INFO : Executor : Experiments execution completed\n",
"2016-12-07 10:18:30,716 INFO : Executor : ################################################################################\n",
"2016-12-07 10:18:30,716 INFO : Executor : Results available in:\n",
"2016-12-07 10:18:30,717 INFO : Executor : /home/vagrant/lisa/results/20161207_101752\n"
]
}
],
"source": [
"executor.run()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/vagrant/lisa/results/20161207_101752\r\n",
"├── perf_bench_messaging:base:perf\r\n",
"│ ├── 1\r\n",
"│ │ ├── energy.json\r\n",
"│ │ ├── output.log\r\n",
"│ │ └── performance.json\r\n",
"│ ├── kernel.config\r\n",
"│ ├── kernel.version\r\n",
"│ └── platform.json\r\n",
"├── perf_bench_messaging:eas:perf\r\n",
"│ ├── 1\r\n",
"│ │ ├── energy.json\r\n",
"│ │ ├── output.log\r\n",
"│ │ └── performance.json\r\n",
"│ ├── kernel.config\r\n",
"│ ├── kernel.version\r\n",
"│ └── platform.json\r\n",
"├── rtapp:base:rta\r\n",
"│ ├── 1\r\n",
"│ │ ├── energy.json\r\n",
"│ │ ├── output.log\r\n",
"│ │ ├── rta_00.json\r\n",
"│ │ └── rt-app-task_p200-0.log\r\n",
"│ ├── kernel.config\r\n",
"│ ├── kernel.version\r\n",
"│ └── platform.json\r\n",
"└── rtapp:eas:rta\r\n",
" ├── 1\r\n",
" │ ├── energy.json\r\n",
" │ ├── output.log\r\n",
" │ ├── rta_00.json\r\n",
" │ └── rt-app-task_p200-0.log\r\n",
" ├── kernel.config\r\n",
" ├── kernel.version\r\n",
" └── platform.json\r\n",
"\r\n",
"8 directories, 26 files\r\n"
]
}
],
"source": [
"!tree {executor.te.res_dir}"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}