1# TFLite Model Benchmark Tool with Android Apk 2 3## Description 4 5This Android benchmark app is a simple wrapper around the TensorFlow Lite 6[command-line benchmark utility](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark). 7 8Pushing and executing binaries directly on an Android device is a valid approach 9to benchmarking, but it can result in subtle (but observable) differences in 10performance relative to execution within an actual Android app. In particular, 11Android's scheduler tailors behavior based on thread and process priorities, 12which differ between a foreground Activity/Application and a regular background 13binary executed via `adb shell ...`. This tailored behavior is most evident when 14enabling multi-threaded CPU execution with TensorFlow Lite. 15 16To that end, this app offers perhaps a more faithful view of runtime performance 17that developers can expect when deploying TensorFlow Lite with their 18application. 19 20## To build/install/run 21 22(0) Refer to 23https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/android/test 24to edit the `WORKSPACE` to configure the android NDK/SDK. 25 26(1) Build for your specific platform, e.g.: 27 28``` 29bazel build -c opt \ 30 --config=android_arm64 \ 31 tensorflow/lite/tools/benchmark/android:benchmark_model 32``` 33 34(Optional) To enable Hexagon delegate with `--use_hexagon=true` option, you can 35download and install the libraries as the guided in [hexagon delegate] 36(https://www.tensorflow.org/lite/performance/hexagon_delegate#step_2_add_hexagon_libraries_to_your_android_app) 37page. For example, if you installed the libraries at third_party/hexagon_nn_skel 38and created third_party/hexagon_nn_skel/BUILD with a build target, 39 40``` 41filegroup( 42 name = "libhexagon_nn_skel", 43 srcs = glob(["*.so"]), 44) 45``` 46 47you need to modify tflite_hexagon_nn_skel_libraries macro in 48tensorflow/lite/special_rules.bzl to specify the build target. 49 50``` 51return ["//third_party/hexagon_nn_skel:libhexagon_nn_skel"] 52``` 53 54(2) Connect your phone. Install the benchmark APK to your phone with adb: 55 56``` 57adb install -r -d -g bazel-bin/tensorflow/lite/tools/benchmark/android/benchmark_model.apk 58``` 59 60Note: Make sure to install with "-g" option to grant the permission for reading 61external storage. 62 63(3) Push the compute graph that you need to test. 64 65``` 66adb push mobilenet_quant_v1_224.tflite /data/local/tmp 67``` 68 69(4) Run the benchmark. Additional command-line flags are documented 70[here](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark/README.md) 71and can be appended to the `args` string alongside the required `--graph` flag 72(note that all args must be nested in the single quoted string that follows the 73args key). 74 75``` 76adb shell am start -S \ 77 -n org.tensorflow.lite.benchmark/.BenchmarkModelActivity \ 78 --es args '"--graph=/data/local/tmp/mobilenet_quant_v1_224.tflite \ 79 --num_threads=4"' 80``` 81 82(5) The results will be available in Android's logcat, e.g.: 83 84``` 85adb logcat | grep "Inference timings in us" 86 87... tflite : Inference timings in us: Init: 1007529, First inference: 4098, Warmup (avg): 1686.59, Inference (avg): 1687.92 88``` 89 90## To trace Tensorflow Lite internals including operator invocation 91 92The steps described here follows the method of 93https://developer.android.com/topic/performance/tracing/on-device. Refer to the 94page for more detailed information. 95 96(0)-(3) Follow the steps (0)-(3) of [build/install/run](#to-buildinstallrun) 97section. 98 99(4) Enable platform tracing. 100 101``` 102adb shell setprop debug.tflite.trace 1 103``` 104 105(5) Set up Quick Settings tile for System Tracing app on your device. Follow the 106[instruction](https://developer.android.com/topic/performance/tracing/on-device#set-up-tile). 107The System Tracing tile will be added to the Quick Settings panel. 108 109Optionally, you can set up other configurations for tracing from the app menu. 110Refer to the 111[guide](https://developer.android.com/topic/performance/tracing/on-device#app-menu) 112for more information. 113 114(6) Tap the System Tracing tile, which has the label "Record trace". The tile 115becomes enabled, and a persistent notification appears to notify you that the 116system is now recording a trace. 117 118(7) Run the benchmark with platform tracing enabled. 119 120``` 121adb shell am start -S \ 122 -n org.tensorflow.lite.benchmark/.BenchmarkModelActivity \ 123 --es args '"--graph=/data/local/tmp/mobilenet_quant_v1_224.tflite \ 124 --num_threads=4"' 125``` 126 127(8) Wait until the benchmark finishes. It can be checked from Android log 128messages, e.g., 129 130``` 131adb logcat | grep "Inference timings in us" 132 133... tflite : Inference timings in us: Init: 1007529, First inference: 4098, Warmup (avg): 1686.59, Inference (avg): 1687.92 134``` 135 136(9) Stop tracing by tapping either the System Tracing tile in the Quick Settings 137panel or on the System Tracing notification. The system displays a new 138notification that contains the message "Saving trace". When saving is complete, 139the system dismisses the notification and displays a third notification "Trace 140saved", confirming that your trace has been saved and that you're ready to share 141the system trace. 142 143(10) 144[Share](https://developer.android.com/topic/performance/tracing/on-device#share-trace) 145a trace file, 146[convert](https://developer.android.com/topic/performance/tracing/on-device#converting_between_trace_formats) 147between tracing formats and 148[create](https://developer.android.com/topic/performance/tracing/on-device#create-html-report) 149an HTML report. Note that, the captured tracing file format is either in 150Perfetto format or in Systrace format depending on the Android version of your 151device. Select the appropriate method to handle the generated file. 152 153(11) Disable platform tracing. 154 155``` 156adb shell setprop debug.tflite.trace 0 157``` 158