1.. _gdb: 2 3========================================================= 4Debugging C API extensions and CPython Internals with GDB 5========================================================= 6 7.. highlight:: none 8 9This document explains how the Python GDB extension, ``python-gdb.py``, can 10be used with the GDB debugger to debug CPython extensions and the 11CPython interpreter itself. 12 13When debugging low-level problems such as crashes or deadlocks, a low-level 14debugger, such as GDB, is useful to diagnose and correct the issue. 15By default, GDB (or any of its front-ends) doesn't support high-level 16information specific to the CPython interpreter. 17 18The ``python-gdb.py`` extension adds CPython interpreter information to GDB. 19The extension helps introspect the stack of currently executing Python functions. 20Given a Python object represented by a :c:expr:`PyObject *` pointer, 21the extension surfaces the type and value of the object. 22 23Developers who are working on CPython extensions or tinkering with parts 24of CPython that are written in C can use this document to learn how to use the 25``python-gdb.py`` extension with GDB. 26 27.. note:: 28 29 This document assumes that you are familiar with the basics of GDB and the 30 CPython C API. It consolidates guidance from the 31 `devguide <https://devguide.python.org>`_ and the 32 `Python wiki <https://wiki.python.org/moin/DebuggingWithGdb>`_. 33 34 35Prerequisites 36============= 37 38You need to have: 39 40- GDB 7 or later. (For earlier versions of GDB, see ``Misc/gdbinit`` in the 41 sources of Python 3.11 or earlier.) 42- GDB-compatible debugging information for Python and any extension you are 43 debugging. 44- The ``python-gdb.py`` extension. 45 46The extension is built with Python, but might be distributed separately or 47not at all. Below, we include tips for a few common systems as examples. 48Note that even if the instructions match your system, they might be outdated. 49 50 51Setup with Python built from source 52----------------------------------- 53 54When you build CPython from source, debugging information should be available, 55and the build should add a ``python-gdb.py`` file to the root directory of 56your repository. 57 58To activate support, you must add the directory containing ``python-gdb.py`` 59to GDB's "auto-load-safe-path". 60If you haven't done this, recent versions of GDB will print out a warning 61with instructions on how to do this. 62 63.. note:: 64 65 If you do not see instructions for your version of GDB, put this in your 66 configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``):: 67 68 add-auto-load-safe-path /path/to/cpython 69 70 You can also add multiple paths, separated by ``:``. 71 72 73Setup for Python from a Linux distro 74------------------------------------ 75 76Most Linux systems provide debug information for the system Python 77in a package called ``python-debuginfo``, ``python-dbg`` or similar. 78For example: 79 80- Fedora: 81 82 .. code-block:: shell 83 84 sudo dnf install gdb 85 sudo dnf debuginfo-install python3 86 87- Ubuntu: 88 89 .. code-block:: shell 90 91 sudo apt install gdb python3-dbg 92 93On several recent Linux systems, GDB can download debugging symbols 94automatically using *debuginfod*. 95However, this will not install the ``python-gdb.py`` extension; 96you generally do need to install the debug info package separately. 97 98 99Using the Debug build and Development mode 100========================================== 101 102For easier debugging, you might want to: 103 104- Use a :ref:`debug build <debug-build>` of Python. (When building from source, 105 use ``configure --with-pydebug``. On Linux distros, install and run a package 106 like ``python-debug`` or ``python-dbg``, if available.) 107- Use the runtime :ref:`development mode <devmode>` (``-X dev``). 108 109Both enable extra assertions and disable some optimizations. 110Sometimes this hides the bug you are trying to find, but in most cases they 111make the process easier. 112 113 114Using the ``python-gdb`` extension 115================================== 116 117When the extension is loaded, it provides two main features: 118pretty printers for Python values, and additional commands. 119 120Pretty-printers 121--------------- 122 123This is what a GDB backtrace looks like (truncated) when this extension is 124enabled:: 125 126 #0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8 127 ) at Objects/obmalloc.c:748 128 #1 0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445 129 #2 0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412 130 #3 0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346 131 #4 0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed= 132 0x0) at Objects/unicodeobject.c:2531 133 #5 0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0) 134 at Objects/unicodeobject.c:2495 135 #6 0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11) 136 at Objects/unicodeobject.c:551 137 #7 0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569 138 #8 0x0000000000584abd in PyDict_GetItemString (v= 139 {'Yuck': <type at remote 0xad4730>, '__builtins__': <module at remote 0x7ffff7fd5ee8>, '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': <Yuck(i=0) at remote 0xaacd80>, 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': <Yuck(i=0) at remote 0xaace60>, '__doc__': None}, key= 140 0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171 141 142Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed 143as its ``repr()``, rather than an opaque ``PyObject *`` pointer. 144 145The extension works by supplying a custom printing routine for values of type 146``PyObject *``. If you need to access lower-level details of an object, then 147cast the value to a pointer of the appropriate type. For example:: 148 149 (gdb) p globals 150 $1 = {'__builtins__': <module at remote 0x7ffff7fb1868>, '__name__': 151 '__main__', 'ctypes': <module at remote 0x7ffff7f14360>, '__doc__': None, 152 '__package__': None} 153 154 (gdb) p *(PyDictObject*)globals 155 $2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5, 156 ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70 157 <lookdict_string>, ma_smalltable = {{me_hash = 7065186196740147912, 158 me_key = '__builtins__', me_value = <module at remote 0x7ffff7fb1868>}, 159 {me_hash = -368181376027291943, me_key = '__name__', 160 me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0}, 161 {me_hash = 0, me_key = 0x0, me_value = 0x0}, 162 {me_hash = -9177857982131165996, me_key = 'ctypes', 163 me_value = <module at remote 0x7ffff7f14360>}, 164 {me_hash = -8518757509529533123, me_key = '__doc__', me_value = None}, 165 {me_hash = 0, me_key = 0x0, me_value = 0x0}, { 166 me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}} 167 168Note that the pretty-printers do not actually call ``repr()``. 169For basic types, they try to match its result closely. 170 171An area that can be confusing is that the custom printer for some types look a 172lot like GDB's built-in printer for standard types. For example, the 173pretty-printer for a Python ``int`` (:c:expr:`PyLongObject *`) 174gives a representation that is not distinguishable from one of a 175regular machine-level integer:: 176 177 (gdb) p some_machine_integer 178 $3 = 42 179 180 (gdb) p some_python_integer 181 $4 = 42 182 183The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`: 184 185 (gdb) p *(PyLongObject*)some_python_integer 186 $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1}, 187 ob_digit = {42}} 188 189A similar confusion can arise with the ``str`` type, where the output looks a 190lot like gdb's built-in printer for ``char *``:: 191 192 (gdb) p ptr_to_python_str 193 $6 = '__builtins__' 194 195The pretty-printer for ``str`` instances defaults to using single-quotes (as 196does Python's ``repr`` for strings) whereas the standard printer for ``char *`` 197values uses double-quotes and contains a hexadecimal address:: 198 199 (gdb) p ptr_to_char_star 200 $7 = 0x6d72c0 "hello world" 201 202Again, the implementation details can be revealed with a cast to 203:c:expr:`PyUnicodeObject *`:: 204 205 (gdb) p *(PyUnicodeObject*)$6 206 $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12, 207 str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0} 208 209``py-list`` 210----------- 211 212 The extension adds a ``py-list`` command, which 213 lists the Python source code (if any) for the current frame in the selected 214 thread. The current line is marked with a ">":: 215 216 (gdb) py-list 217 901 if options.profile: 218 902 options.profile = False 219 903 profile_me() 220 904 return 221 905 222 >906 u = UI() 223 907 if not u.quit: 224 908 try: 225 909 gtk.main() 226 910 except KeyboardInterrupt: 227 911 # properly quit on a keyboard interrupt... 228 229 Use ``py-list START`` to list at a different line number within the Python 230 source, and ``py-list START,END`` to list a specific range of lines within 231 the Python source. 232 233``py-up`` and ``py-down`` 234------------------------- 235 236 The ``py-up`` and ``py-down`` commands are analogous to GDB's regular ``up`` 237 and ``down`` commands, but try to move at the level of CPython frames, rather 238 than C frames. 239 240 GDB is not always able to read the relevant frame information, depending on 241 the optimization level with which CPython was compiled. Internally, the 242 commands look for C frames that are executing the default frame evaluation 243 function (that is, the core bytecode interpreter loop within CPython) and 244 look up the value of the related ``PyFrameObject *``. 245 246 They emit the frame number (at the C level) within the thread. 247 248 For example:: 249 250 (gdb) py-up 251 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/ 252 gnome_sudoku/main.py, line 906, in start_game () 253 u = UI() 254 (gdb) py-up 255 #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/ 256 gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=<module at remote 0xb771b7f4>) 257 main.start_game() 258 (gdb) py-up 259 Unable to find an older python frame 260 261 so we're at the top of the Python stack. 262 263 The frame numbers correspond to those displayed by GDB's standard 264 ``backtrace`` command. 265 The command skips C frames which are not executing Python code. 266 267 Going back down:: 268 269 (gdb) py-down 270 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () 271 u = UI() 272 (gdb) py-down 273 #34 (unable to read python frame information) 274 (gdb) py-down 275 #23 (unable to read python frame information) 276 (gdb) py-down 277 #19 (unable to read python frame information) 278 (gdb) py-down 279 #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated) 280 swallower.run_dialog(self.dialog) 281 (gdb) py-down 282 #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>) 283 gtk.main() 284 (gdb) py-down 285 #8 (unable to read python frame information) 286 (gdb) py-down 287 Unable to find a newer python frame 288 289 and we're at the bottom of the Python stack. 290 291 Note that in Python 3.12 and newer, the same C stack frame can be used for 292 multiple Python stack frames. This means that ``py-up`` and ``py-down`` 293 may move multiple Python frames at once. For example:: 294 295 (gdb) py-up 296 #6 Frame 0x7ffff7fb62b0, for file /tmp/rec.py, line 5, in recursive_function (n=0) 297 time.sleep(5) 298 #6 Frame 0x7ffff7fb6240, for file /tmp/rec.py, line 7, in recursive_function (n=1) 299 recursive_function(n-1) 300 #6 Frame 0x7ffff7fb61d0, for file /tmp/rec.py, line 7, in recursive_function (n=2) 301 recursive_function(n-1) 302 #6 Frame 0x7ffff7fb6160, for file /tmp/rec.py, line 7, in recursive_function (n=3) 303 recursive_function(n-1) 304 #6 Frame 0x7ffff7fb60f0, for file /tmp/rec.py, line 7, in recursive_function (n=4) 305 recursive_function(n-1) 306 #6 Frame 0x7ffff7fb6080, for file /tmp/rec.py, line 7, in recursive_function (n=5) 307 recursive_function(n-1) 308 #6 Frame 0x7ffff7fb6020, for file /tmp/rec.py, line 9, in <module> () 309 recursive_function(5) 310 (gdb) py-up 311 Unable to find an older python frame 312 313 314``py-bt`` 315--------- 316 317 The ``py-bt`` command attempts to display a Python-level backtrace of the 318 current thread. 319 320 For example:: 321 322 (gdb) py-bt 323 #8 (unable to read python frame information) 324 #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>) 325 gtk.main() 326 #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated) 327 swallower.run_dialog(self.dialog) 328 #19 (unable to read python frame information) 329 #23 (unable to read python frame information) 330 #34 (unable to read python frame information) 331 #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () 332 u = UI() 333 #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>) 334 main.start_game() 335 336 The frame numbers correspond to those displayed by GDB's standard 337 ``backtrace`` command. 338 339``py-print`` 340------------ 341 342 The ``py-print`` command looks up a Python name and tries to print it. 343 It looks in locals within the current thread, then globals, then finally 344 builtins:: 345 346 (gdb) py-print self 347 local 'self' = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, 348 main_page=0) at remote 0x98fa6e4> 349 (gdb) py-print __name__ 350 global '__name__' = 'gnome_sudoku.dialog_swallower' 351 (gdb) py-print len 352 builtin 'len' = <built-in function len> 353 (gdb) py-print scarlet_pimpernel 354 'scarlet_pimpernel' not found 355 356 If the current C frame corresponds to multiple Python frames, ``py-print`` 357 only considers the first one. 358 359``py-locals`` 360------------- 361 362 The ``py-locals`` command looks up all Python locals within the current 363 Python frame in the selected thread, and prints their representations:: 364 365 (gdb) py-locals 366 self = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, 367 main_page=0) at remote 0x98fa6e4> 368 d = <gtk.Dialog at remote 0x98faaa4> 369 370 If the current C frame corresponds to multiple Python frames, locals from 371 all of them will be shown:: 372 373 (gdb) py-locals 374 Locals for recursive_function 375 n = 0 376 Locals for recursive_function 377 n = 1 378 Locals for recursive_function 379 n = 2 380 Locals for recursive_function 381 n = 3 382 Locals for recursive_function 383 n = 4 384 Locals for recursive_function 385 n = 5 386 Locals for <module> 387 388 389Use with GDB commands 390===================== 391 392The extension commands complement GDB's built-in commands. 393For example, you can use a frame numbers shown by ``py-bt`` with the ``frame`` 394command to go a specific frame within the selected thread, like this:: 395 396 (gdb) py-bt 397 (output snipped) 398 #68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> () 399 main() 400 (gdb) frame 68 401 #68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> (), throwflag=0) at Python/ceval.c:2665 402 2665 x = call_function(&sp, oparg); 403 (gdb) py-list 404 1543 # Run the tests in a context manager that temporary changes the CWD to a 405 1544 # temporary and writable directory. If it's not possible to create or 406 1545 # change the CWD, the original CWD will be used. The original CWD is 407 1546 # available from test_support.SAVEDCWD. 408 1547 with test_support.temp_cwd(TESTCWD, quiet=True): 409 >1548 main() 410 411The ``info threads`` command will give you a list of the threads within the 412process, and you can use the ``thread`` command to select a different one:: 413 414 (gdb) info threads 415 105 Thread 0x7fffefa18710 (LWP 10260) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 416 104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 417 * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82 418 419You can use ``thread apply all COMMAND`` or (``t a a COMMAND`` for short) to run 420a command on all threads. With ``py-bt``, this lets you see what every 421thread is doing at the Python level:: 422 423 (gdb) t a a py-bt 424 425 Thread 105 (Thread 0x7fffefa18710 (LWP 10260)): 426 #5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528) 427 self.__block.acquire() 428 #8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858a90>, saved_state=(1, 140737213728528)) 429 self._acquire_restore(saved_state) 430 #12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () 431 cond.wait() 432 #16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528) 433 f() 434 435 Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)): 436 #5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272) 437 self.__block.acquire() 438 #8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858860>, saved_state=(1, 140736940992272)) 439 self._acquire_restore(saved_state) 440 #12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () 441 cond.wait() 442 #16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272) 443 f() 444 445 Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)): 446 #5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait () 447 time.sleep(0.01) 448 #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=<ConditionTests(_testMethodName='test_notify', _resultForDoCleanups=<TestResult(_original_stdout=<cStringIO.StringO at remote 0xc191e0>, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=<file at remote 0x7ffff7fc6340>, _stdout_buffer=<cStringIO.StringO at remote 0xc9c7f8>, _stderr_buffer=<cStringIO.StringO at remote 0xc9c790>, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=<type at remote 0x928310>, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={<type at remote 0x7eba00>: <instancemethod at remote 0xd750e0>, <type at remote 0x7e7820>: <instancemethod at remote 0xd75160>, <type at remote 0x7e30e0>: <instancemethod at remote 0xd75060>, <type at remote 0x7e7d20>: <instancemethod at remote 0xd751e0>, <type at remote 0x7f19e0...(truncated) 449 _wait() 450