1.. _showcase-sense-tutorial-crash-handler: 2 3===================================== 415. Handle crashes and view snapshots 5===================================== 6When a system crashes it can be difficult to debug the cause of the 7crash. To help with this, :ref:`pw_system <module-pw_system>` provides 8a crash handler built on :ref:`module-pw_cpu_exception` which is invoked 9when a CPU exception is triggered. 10 11The :ref:`pw_system <module-pw_system>` crash handler will automatically 12create a crash snapshot on exception and reboot the system. This snapshot can 13then be downloaded onto a host system for analysis. 14 15 16.. _showcase-sense-tutorial-crash-handler-crash: 17 18---------------- 19Generate a crash 20---------------- 21#. If you're not already connected to your Pico via ``pw_console``, do so now: 22 23 .. tab-set:: 24 25 .. tab-item:: VS Code 26 :sync: vsc 27 28 .. tab-set:: 29 30 .. tab-item:: Pico 1 & 1W (RP2040) 31 :sync: rp2040 32 33 In **Bazel Targets** expand **//apps/production**, then right-click 34 **:rp2040_console (native binary)**, then select **Run target**. 35 36 .. tab-item:: Pico 2 & 2W (RP2350) 37 :sync: rp2350 38 39 In **Bazel Targets** expand **//apps/production**, then right-click 40 **:rp2350_console (native binary)**, then select **Run target**. 41 42 .. tab-item:: CLI 43 :sync: cli 44 45 Run the terminal-based console: 46 47 .. tab-set:: 48 49 .. tab-item:: Pico 1 & 1W (RP2040) 50 51 .. code-block:: console 52 53 bazelisk run //apps/production:rp2040_console 54 55 .. tab-item:: Pico 2 & 2W (RP2350) 56 57 .. code-block:: console 58 59 bazelisk run //apps/production:rp2350_console 60 61``pw_system`` provides an RPC to crash the system by triggering a 62`HardFault <https://developer.arm.com/documentation/107706/0100/System-exceptions/Fault-exceptions-and-their-causes>`_. 63To invoke this RPC, type the following into the **Python Repl** (bottom-left pane) of ``pw_console``: 64 65.. code-block:: pycon 66 67 >>> device.rpcs.pw.system.proto.DeviceService.Crash() 68 69If you're using the :ref:`full setup <showcase-sense-tutorial-hardware>`, 70on system restart the presence of a crash snapshot will be detected, and 71the following will be in the system logs: 72 73.. code-block:: text 74 75 INF pw_system RpcDevice 00:00:00.000 pw_system main 76 ERR pw_system RpcDevice 00:00:00.000 ========================== 77 ERR pw_system RpcDevice 00:00:00.000 ======CRASH DETECTED====== 78 ERR pw_system RpcDevice 00:00:00.000 ========================== 79 ERR pw_system RpcDevice 00:00:00.000 Crash snapshots available. 80 ERR pw_system RpcDevice 00:00:00.000 Run `device.get_crash_snapshots()` to download and clear the snapshots. 81 INF pw_system RpcDevice 00:00:00.000 System init 82 83 84On a :ref:`"Pico and Enviro+" or "Pico only" <showcase-sense-tutorial-hardware>` 85setup, the snapshot will still be generated, but logs won't be visible in the console. When 86the system restarts, the connection to the console is broken so logs 87will not be displayed. After invoking the ``Crash()`` RPC, exit the console 88and :ref:`start it again <showcase-sense-tutorial-pico-rpc-interact>` to 89re-establish the connection. 90 91.. _showcase-sense-tutorial-crash-handler-view: 92 93--------------------- 94View a crash snapshot 95--------------------- 96 97The crash snapshot contains relevant information to debug crashes, such 98as register state thread backtraces and un-flushed logs. If there is a 99crash snapshot on the system, it can be downloaded to the host with the 100following RPC. 101 102.. code-block:: pycon 103 104 >>> device.get_crash_snapshots() 105 106This RPC will download the snapshot, decode it and save it in a 107temporary directory, the location of which will be printed to the 108console as follows: 109 110.. code-block:: text 111 112 INF Wrote crash snapshot to: /var/folders/2j/sjk9390d5rxc3c9ycwcf3mdh0103lh/T/crash_0.txt 113 114It's also possible to specify the path as part of the RPC call: 115 116.. code-block:: pycon 117 118 >>> device.get_crash_snapshots("/path/") 119 120The decoded text file should look similar to this truncated example: 121 122.. code-block:: 123 124 Device crash cause: 125 pw_system/device_service_pwpb.cc:38 Crash: RPC triggered crash 126 127 Reason token: 0x735f7770 128 CPU Arch: ARMV8M 129 130 Exception caused by a usage fault. 131 132 Active Crash Fault Status Register (CFSR) fields: 133 UNDEFINSTR Undefined Instruction UsageFault. 134 The processor has attempted to execute an undefined 135 instruction. When this bit is set to 1, the PC value stacked 136 for the exception return points to the undefined instruction. 137 An undefined instruction is an instruction that the processor 138 cannot decode. 139 140 All registers: 141 pc 0x10000f0a pw::system::DeviceServicePwpb::Crash(pw::system::proto::pwpb::CrashRequest::Message const&, pw::system::proto::pwpb::CrashResponse::Message&) (/b/pw_system/device_service_pwpb.cc:38) 142 lr 0x10012787 pw::StringBuilder::FormatVaList(char const*, std::__va_list) (/build/pw_string/string_builder.cc:102) 143 psr 0x41000000 144 msp 0x20081fe0 __scratch_y_end__ (??:?) 145 psp 0x2000a100 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 146 exc_return 0xfffffffd 147 cfsr 0x00010000 148 msplim 0x00000000 149 psplim 0x20002288 150 mmfar 0xe000ed34 __scratch_y_end__ (??:?) 151 bfar 0xe000ed38 __scratch_y_end__ (??:?) 152 icsr 0x00400806 153 hfsr 0x00000000 154 shcsr 0x00070008 155 control 0x00000000 156 r0 0x2000a0e0 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 157 r1 0x0000003e pw_assert_basic_HandleFailure (/b/pw_assert_basic/basic_handler.cc:74) 158 r2 0x0000002b pw_assert_basic_HandleFailure (/b/pw_assert_basic/basic_handler.cc:74) 159 r3 0x2000a100 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 160 r4 0x10019596 161 r5 0x2000a178 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 162 r6 0x10019eec pw::system::proto::pw_rpc::pwpb::DeviceService::Service<pw::system::DeviceServicePwpb>::kPwRpcMethods (??:?) 163 r7 0x2000a108 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 164 r8 0x2000a118 pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 165 r9 0x2000a16e pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 166 r10 0x2000b4f0 pw::system::(anonymous namespace)::server (hdlc_rpc_server.cc:0) 167 r11 0x2000a22c pw::system::rpc_thread_context (freertos_target_hooks.cc:0) 168 r12 0x00000008 pw_assert_HandleFailure (/b/pw_assert_basic/assert_basic.cc:20) 169 170 Thread State 171 6 threads running, RpcThread active at the time of capture. 172 ~~~~~~~~~ 173 174 Thread (RUNNING): RpcThread <-- [ACTIVE] 175 Est CPU usage: unknown 176 Stack info 177 Current usage: 0x2000a288 - 0x2000a100 (392 bytes, 1.20%) 178 Est peak usage: 944 bytes, 2.88% 179 Stack limits: 0x2000a288 - 0x2000228c (32764 bytes) 180 Stack Trace (most recent call first): 181 1: at void pw::rpc::internal::PwpbMethod::CallSynchronousUnary<pw::system::proto::pwpb::RebootRequest::Message, pw::system::proto::pwpb::RebootResponse::Message>(pw::rpc::internal::CallContext const&, pw::rpc::internal::Packet const&, pw::system::proto::pwpb::RebootRequest::Message&, pw::system::proto::pwpb::RebootResponse::Message&) const (0x10000F59) 182 in /build/pw_rpc/pwpb/public/pw_rpc/pwpb/internal/method.h:258 183 2: at void pw::rpc::internal::PwpbMethod::CallSynchronousUnary<pw::system::proto::pwpb::CrashRequest::Message, pw::system::proto::pwpb::CrashResponse::Message>(pw::rpc::internal::CallContext const&, pw::rpc::internal::Packet const&, pw::system::proto::pwpb::CrashRequest::Message&, pw::system::proto::pwpb::CrashResponse::Message&) const (0x10001137) 184 in /build/pw_rpc/pwpb/public/pw_rpc/pwpb/internal/method.h:267 185 3: at xQueueSemaphoreTake (0x10013049) 186 in /build/external/freertos+/queue.c:1555 187 4: at void pw::rpc::internal::PwpbMethod::SynchronousUnaryInvoker<pw::system::proto::pwpb::CrashRequest::Message, pw::system::proto::pwpb::CrashResponse::Message>(pw::rpc::internal::CallContext const&, pw::rpc::internal::Packet const&) (0x10000F4F) 188 in /build/pw_rpc/pwpb/public/pw_rpc/pwpb/internal/method.h:322 189 5: at pw::rpc::Server::ProcessPacket(pw::rpc::internal::Packet) (0x1000EA9D) 190 in /build/pw_rpc/public/pw_rpc/internal/method.h:0 191 6: at pw::rpc::Server::ProcessPacket(pw::span<std::byte const, 4294967295u>) (0x1000E9CD) 192 in /build/pw_rpc/server.cc:40 193 7: at pw::system::RpcDispatchThread::Run() (0x10008625) 194 in /build/pw_system/hdlc_rpc_server.cc:127 195 8: at pw::thread::freertos::Context::ThreadEntryPoint(void*) (0x1000EFA5) 196 in /build/third_party/fuchsia/repo/sdk/lib/fit/include/lib/fit/internal/function.h:362 197 9: at prvTaskExitError (0x100137C9) 198 in /build/external/freertos+/portable/GCC/ARM_CM33_NTZ/non_secure/port.c:634 199 200 ... 201 202 Device Logs: 203 [RpcDevice] pw_system 0 pw_system main targets/rp2040/boot.cc:56 204 [RpcDevice] pw_system 0 System init pw_system/init.cc:65 205 [RpcDevice] pw_system 0 Registering RPC services pw_system/init.cc:75 206 207 ... 208 209 210.. _showcase-sense-tutorial-crash-handler-summary: 211 212------- 213Summary 214------- 215On this page, we met ``pw_cpu_exception``, the CPU exception handler entry point. 216We also learned how to generate crashes and download the resulting crash snapshot. 217 218Next, head over to :ref:`showcase-sense-tutorial-outro` to wrap up your 219tour of Pigweed. 220