1==================== 2XRay Instrumentation 3==================== 4 5:Version: 1 as of 2016-11-08 6 7.. contents:: 8 :local: 9 10 11Introduction 12============ 13 14XRay is a function call tracing system which combines compiler-inserted 15instrumentation points and a runtime library that can dynamically enable and 16disable the instrumentation. 17 18More high level information about XRay can be found in the `XRay whitepaper`_. 19 20This document describes how to use XRay as implemented in LLVM. 21 22XRay in LLVM 23============ 24 25XRay consists of three main parts: 26 27- Compiler-inserted instrumentation points. 28- A runtime library for enabling/disabling tracing at runtime. 29- A suite of tools for analysing the traces. 30 31 **NOTE:** As of July 25, 2018 , XRay is only available for the following 32 architectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le, 33 mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 and 34 OpenBSD: x86_64. 35 36The compiler-inserted instrumentation points come in the form of nop-sleds in 37the final generated binary, and an ELF section named ``xray_instr_map`` which 38contains entries pointing to these instrumentation points. The runtime library 39relies on being able to access the entries of the ``xray_instr_map``, and 40overwrite the instrumentation points at runtime. 41 42Using XRay 43========== 44 45You can use XRay in a couple of ways: 46 47- Instrumenting your C/C++/Objective-C/Objective-C++ application. 48- Generating LLVM IR with the correct function attributes. 49 50The rest of this section covers these main ways and later on how to customise 51what XRay does in an XRay-instrumented binary. 52 53Instrumenting your C/C++/Objective-C Application 54------------------------------------------------ 55 56The easiest way of getting XRay instrumentation for your application is by 57enabling the ``-fxray-instrument`` flag in your clang invocation. 58 59For example: 60 61:: 62 63 clang -fxray-instrument ... 64 65By default, functions that have at least 200 instructions will get XRay 66instrumentation points. You can tweak that number through the 67``-fxray-instruction-threshold=`` flag: 68 69:: 70 71 clang -fxray-instrument -fxray-instruction-threshold=1 ... 72 73You can also specifically instrument functions in your binary to either always 74or never be instrumented using source-level attributes. You can do it using the 75GCC-style attributes or C++11-style attributes. 76 77.. code-block:: c++ 78 79 [[clang::xray_always_instrument]] void always_instrumented(); 80 81 [[clang::xray_never_instrument]] void never_instrumented(); 82 83 void alt_always_instrumented() __attribute__((xray_always_instrument)); 84 85 void alt_never_instrumented() __attribute__((xray_never_instrument)); 86 87When linking a binary, you can either manually link in the `XRay Runtime 88Library`_ or use ``clang`` to link it in automatically with the 89``-fxray-instrument`` flag. Alternatively, you can statically link-in the XRay 90runtime library from compiler-rt -- those archive files will take the name of 91`libclang_rt.xray-{arch}` where `{arch}` is the mnemonic supported by clang 92(x86_64, arm7, etc.). 93 94LLVM Function Attribute 95----------------------- 96 97If you're using LLVM IR directly, you can add the ``function-instrument`` 98string attribute to your functions, to get the similar effect that the 99C/C++/Objective-C source-level attributes would get: 100 101.. code-block:: llvm 102 103 define i32 @always_instrument() uwtable "function-instrument"="xray-always" { 104 ; ... 105 } 106 107 define i32 @never_instrument() uwtable "function-instrument"="xray-never" { 108 ; ... 109 } 110 111You can also set the ``xray-instruction-threshold`` attribute and provide a 112numeric string value for how many instructions should be in the function before 113it gets instrumented. 114 115.. code-block:: llvm 116 117 define i32 @maybe_instrument() uwtable "xray-instruction-threshold"="2" { 118 ; ... 119 } 120 121Special Case File 122----------------- 123 124Attributes can be imbued through the use of special case files instead of 125adding them to the original source files. You can use this to mark certain 126functions and classes to be never, always, or instrumented with first-argument 127logging from a file. The file's format is described below: 128 129.. code-block:: bash 130 131 # Comments are supported 132 [always] 133 fun:always_instrument 134 fun:log_arg1=arg1 # Log the first argument for the function 135 136 [never] 137 fun:never_instrument 138 139These files can be provided through the ``-fxray-attr-list=`` flag to clang. 140You may have multiple files loaded through multiple instances of the flag. 141 142XRay Runtime Library 143-------------------- 144 145The XRay Runtime Library is part of the compiler-rt project, which implements 146the runtime components that perform the patching and unpatching of inserted 147instrumentation points. When you use ``clang`` to link your binaries and the 148``-fxray-instrument`` flag, it will automatically link in the XRay runtime. 149 150The default implementation of the XRay runtime will enable XRay instrumentation 151before ``main`` starts, which works for applications that have a short 152lifetime. This implementation also records all function entry and exit events 153which may result in a lot of records in the resulting trace. 154 155Also by default the filename of the XRay trace is ``xray-log.XXXXXX`` where the 156``XXXXXX`` part is randomly generated. 157 158These options can be controlled through the ``XRAY_OPTIONS`` environment 159variable, where we list down the options and their defaults below. 160 161+-------------------+-----------------+---------------+------------------------+ 162| Option | Type | Default | Description | 163+===================+=================+===============+========================+ 164| patch_premain | ``bool`` | ``false`` | Whether to patch | 165| | | | instrumentation points | 166| | | | before main. | 167+-------------------+-----------------+---------------+------------------------+ 168| xray_mode | ``const char*`` | ``""`` | Default mode to | 169| | | | install and initialize | 170| | | | before ``main``. | 171+-------------------+-----------------+---------------+------------------------+ 172| xray_logfile_base | ``const char*`` | ``xray-log.`` | Filename base for the | 173| | | | XRay logfile. | 174+-------------------+-----------------+---------------+------------------------+ 175| verbosity | ``int`` | ``0`` | Runtime verbosity | 176| | | | level. | 177+-------------------+-----------------+---------------+------------------------+ 178 179 180If you choose to not use the default logging implementation that comes with the 181XRay runtime and/or control when/how the XRay instrumentation runs, you may use 182the XRay APIs directly for doing so. To do this, you'll need to include the 183``xray_log_interface.h`` from the compiler-rt ``xray`` directory. The important API 184functions we list below: 185 186- ``__xray_log_register_mode(...)``: Register a logging implementation against 187 a string Mode identifier. The implementation is an instance of 188 ``XRayLogImpl`` defined in ``xray/xray_log_interface.h``. 189- ``__xray_log_select_mode(...)``: Select the mode to install, associated with 190 a string Mode identifier. Only implementations registered with 191 ``__xray_log_register_mode(...)`` can be chosen with this function. 192- ``__xray_log_init_mode(...)``: This function allows for initializing and 193 re-initializing an installed logging implementation. See 194 ``xray/xray_log_interface.h`` for details, part of the XRay compiler-rt 195 installation. 196 197Once a logging implementation has been initialized, it can be "stopped" by 198finalizing the implementation through the ``__xray_log_finalize()`` function. 199The finalization routine is the opposite of the initialization. When finalized, 200an implementation's data can be cleared out through the 201``__xray_log_flushLog()`` function. For implementations that support in-memory 202processing, these should register an iterator function to provide access to the 203data via the ``__xray_log_set_buffer_iterator(...)`` which allows code calling 204the ``__xray_log_process_buffers(...)`` function to deal with the data in 205memory. 206 207All of this is better explained in the ``xray/xray_log_interface.h`` header. 208 209Basic Mode 210---------- 211 212XRay supports a basic logging mode which will trace the application's 213execution, and periodically append to a single log. This mode can be 214installed/enabled by setting ``xray_mode=xray-basic`` in the ``XRAY_OPTIONS`` 215environment variable. Combined with ``patch_premain=true`` this can allow for 216tracing applications from start to end. 217 218Like all the other modes installed through ``__xray_log_select_mode(...)``, the 219implementation can be configured through the ``__xray_log_init_mode(...)`` 220function, providing the mode string and the flag options. Basic-mode specific 221defaults can be provided in the ``XRAY_BASIC_OPTIONS`` environment variable. 222 223Flight Data Recorder Mode 224------------------------- 225 226XRay supports a logging mode which allows the application to only capture a 227fixed amount of memory's worth of events. Flight Data Recorder (FDR) mode works 228very much like a plane's "black box" which keeps recording data to memory in a 229fixed-size circular queue of buffers, and have the data available 230programmatically until the buffers are finalized and flushed. To use FDR mode 231on your application, you may set the ``xray_mode`` variable to ``xray-fdr`` in 232the ``XRAY_OPTIONS`` environment variable. Additional options to the FDR mode 233implementation can be provided in the ``XRAY_FDR_OPTIONS`` environment 234variable. Programmatic configuration can be done by calling 235``__xray_log_init_mode("xray-fdr", <configuration string>)`` once it has been 236selected/installed. 237 238When the buffers are flushed to disk, the result is a binary trace format 239described by `XRay FDR format <XRayFDRFormat.html>`_ 240 241When FDR mode is on, it will keep writing and recycling memory buffers until 242the logging implementation is finalized -- at which point it can be flushed and 243re-initialised later. To do this programmatically, we follow the workflow 244provided below: 245 246.. code-block:: c++ 247 248 // Patch the sleds, if we haven't yet. 249 auto patch_status = __xray_patch(); 250 251 // Maybe handle the patch_status errors. 252 253 // When we want to flush the log, we need to finalize it first, to give 254 // threads a chance to return buffers to the queue. 255 auto finalize_status = __xray_log_finalize(); 256 if (finalize_status != XRAY_LOG_FINALIZED) { 257 // maybe retry, or bail out. 258 } 259 260 // At this point, we are sure that the log is finalized, so we may try 261 // flushing the log. 262 auto flush_status = __xray_log_flushLog(); 263 if (flush_status != XRAY_LOG_FLUSHED) { 264 // maybe retry, or bail out. 265 } 266 267The default settings for the FDR mode implementation will create logs named 268similarly to the basic log implementation, but will have a different log 269format. All the trace analysis tools (and the trace reading library) will 270support all versions of the FDR mode format as we add more functionality and 271record types in the future. 272 273 **NOTE:** We do not promise perpetual support for when we update the log 274 versions we support going forward. Deprecation of the formats will be 275 announced and discussed on the developers mailing list. 276 277Trace Analysis Tools 278-------------------- 279 280We currently have the beginnings of a trace analysis tool in LLVM, which can be 281found in the ``tools/llvm-xray`` directory. The ``llvm-xray`` tool currently 282supports the following subcommands: 283 284- ``extract``: Extract the instrumentation map from a binary, and return it as 285 YAML. 286- ``account``: Performs basic function call accounting statistics with various 287 options for sorting, and output formats (supports CSV, YAML, and 288 console-friendly TEXT). 289- ``convert``: Converts an XRay log file from one format to another. We can 290 convert from binary XRay traces (both basic and FDR mode) to YAML, 291 `flame-graph <https://github.com/brendangregg/FlameGraph>`_ friendly text 292 formats, as well as `Chrome Trace Viewer (catapult) 293 <https://github.com/catapult-project/catapult>` formats. 294- ``graph``: Generates a DOT graph of the function call relationships between 295 functions found in an XRay trace. 296- ``stack``: Reconstructs function call stacks from a timeline of function 297 calls in an XRay trace. 298 299These subcommands use various library components found as part of the XRay 300libraries, distributed with the LLVM distribution. These are: 301 302- ``llvm/XRay/Trace.h`` : A trace reading library for conveniently loading 303 an XRay trace of supported forms, into a convenient in-memory representation. 304 All the analysis tools that deal with traces use this implementation. 305- ``llvm/XRay/Graph.h`` : A semi-generic graph type used by the graph 306 subcommand to conveniently represent a function call graph with statistics 307 associated with edges and vertices. 308- ``llvm/XRay/InstrumentationMap.h``: A convenient tool for analyzing the 309 instrumentation map in XRay-instrumented object files and binaries. The 310 ``extract`` and ``stack`` subcommands uses this particular library. 311 312Future Work 313=========== 314 315There are a number of ongoing efforts for expanding the toolset building around 316the XRay instrumentation system. 317 318Trace Analysis Tools 319-------------------- 320 321- Work is in progress to integrate with or develop tools to visualize findings 322 from an XRay trace. Particularly, the ``stack`` tool is being expanded to 323 output formats that allow graphing and exploring the duration of time in each 324 call stack. 325- With a large instrumented binary, the size of generated XRay traces can 326 quickly become unwieldy. We are working on integrating pruning techniques and 327 heuristics for the analysis tools to sift through the traces and surface only 328 relevant information. 329 330More Platforms 331-------------- 332 333We're looking forward to contributions to port XRay to more architectures and 334operating systems. 335 336.. References... 337 338.. _`XRay whitepaper`: http://research.google.com/pubs/pub45287.html 339 340