1This package contains debug symbols that can be useful for debugging 2applications that use grpc pre-compiled binary gems. 3 4An example of a pre-compiled binary gem is `grpc-1.58.0-x86_64-linux.gem` 5(as opposed to a source-built gem like `grpc-1.58.0.gem`). 6 7`grpc-native-debug` gems contain debug symbols which complement the 8native libraries in these grpc binary gems. After fetching and unpacking a 9proper `grpc-native-debug` gem, one can load the correct `.dbg` symbol file to 10debug their grpc application. 11 12# Background 13 14grpc-ruby pre-compiled binary gems are *released with debug symbols stripped*. 15As a consequence, if you are to examine a grpc stack trace in a debugger 16for example, a lot of information will initially be missing. 17 18# Using grpc-native-debug 19 20## Finding the correct grpc-native-debug gem 21 22Each `grpc-native-debug` gem is *one-to-one* with a `grpc` gem. Specifically: 23 24- The version of a `grpc-native-debug` gem **must match the version** of the `grpc` 25 gem. 26 27- The ruby platform of a `grpc-native-debug` gem **must match the ruby platform** of 28 the `grpc` gem. 29 30So for example, if you are debugging `grpc-1.60.1-x86_64-linux.gem`, then you 31need to fetch `grpc-native-debug-1.60.1-x86_64-linux.gem`. 32 33## Finding the correct .dbg symbol file 34 35Each `grpc-native-debug` gem has a top-level `symbols` directory containing 36symbol files ending in `.dbg`. 37 38`grpc` binary gems are shipped with multiple native libraries. There is one 39native library for each supported *minor version* of ruby. As such, 40`grpc-native-debug` gems have exactly one `.dbg` file for each native library 41in the corresponding `grpc` gem. 42 43If you unpack a `grpc-native-debug` gem and look at the `symbols` 44directory, you might see something like this: 45 46``` 47grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg 48grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-2.7.dbg 49grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.2.dbg 50grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.1.dbg 51``` 52 53In each of these `.dbg` files, the `ruby-<major>-<minor>` portion of the string 54indicates which ruby version it's supposed to be used with. 55 56So for example, if you are debugging `grpc-1.60.1-x86_64-linux.gem` on ruby-3.0, then you 57need to use symbol file 58`grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg`. 59 60## Putting symbols into action (example gdb workflow) 61 62There are a variety of ways to use these symbols. 63 64As a toy example, suppose we are running an application under gdb using: 65 66- ruby-3.0 67 68- grpc-1.60.1.x86_64-linux.gem 69 70At first, in gdb we might dump a grpc-ruby stack trace looking 71something like this: 72 73``` 74(gdb) bt 75#0 0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 76#1 0x00007ffff383eb9e in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 77#2 0x00007ffff355e002 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 78#3 0x00007ffff38466e2 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 79#4 0x00007ffff35ba2ea in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 80#5 0x00007ffff34abf6b in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 81#6 0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=<optimized out>) at thread.c:1669 82#7 0x00007ffff34ab110 in ?? () from /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 83#8 0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769 84#9 thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822 85#10 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994 86#11 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444 87#12 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 88``` 89 90We could take the following steps to get more debug info: 91 92<h3>1) Fetch the correct grpc-native-debug gem</h3> 93 94``` 95cd /home 96gem fetch grpc-native-debug-1.60.1.x86_64-linux.gem 97gem unpack grpc-native-debug-1.60.1.x86_64-linux.gem 98``` 99 100(note again the version and platform of `grpc-native-debug` must match the `grpc` gem) 101 102<h3>2) Load debug symbols (for ruby-3.0)</h3> 103 104``` 105(gdb) info sharedlibrary 106From To Syms Read Shared Object Library 107... 1080x00007ffff3497450 0x00007ffff3a61912 Yes (*) /home/.rvm/gems/ruby-3.0.0/gems/grpc-1.60.1-x86_64-linux/src/ruby/lib/grpc/3.0/grpc_c.so 1090x00007ffff3e78730 0x00007ffff3ea60df Yes (*) /home/.rvm/gems/ruby-3.0.0/gems/google-protobuf-3.24.4-x86_64-linux/lib/google/3.0/protobuf_c.so 110(*): Shared library is missing debugging information. 111(gdb) add-symbol-file /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg 0x00007ffff3497450 112add symbol table from file "/home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg" at 113 .text_addr = 0x7ffff3497450 114(y or n) y 115Reading symbols from /home/grpc-native-debug-1.60.1-x86_64-linux/symbols/grpc-1.60.1-x86_64-linux-ruby-3.0.dbg... 116(gdb) 117``` 118 119Our stack trace might look more like this now: 120 121``` 122(gdb) bt 123#0 0x00007ffff7926e56 in epoll_wait (epfd=5, events=0x7ffff3cb4144, maxevents=100, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 124#1 0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723 125#2 pollset_work (ps=0x555555ad1690, worker_hdl=0x0, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:1038 126#3 0x00007ffff355e002 in pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/ev_posix.cc:249 127#4 0x00007ffff38466e2 in grpc_pollset_work (pollset=<optimized out>, worker=<optimized out>, deadline=...) at src/core/lib/iomgr/pollset.cc:48 128#5 0x00007ffff35ba2ea in cq_next (cq=0x555555ad1510, deadline=..., reserved=<optimized out>) at src/core/lib/surface/completion_queue.cc:1043 129#6 0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663 130#7 0x00007ffff7c67ca7 in rb_nogvl (func=0x7ffff34abed3 <run_poll_channels_loop_no_gil>, data1=0x0, ubf=<optimized out>, data2=<optimized out>, flags=flags@entry=0) at thread.c:1669 131#8 0x00007ffff7c68138 in rb_thread_call_without_gvl (func=<optimized out>, data1=<optimized out>, ubf=<optimized out>, data2=<optimized out>) at thread.c:1785 132#9 0x00007ffff34ab110 in run_poll_channels_loop (arg=<optimized out>) at ../../../../src/ruby/ext/grpc/rb_channel.c:734 133#10 0x00007ffff7c6780c in thread_do_start (th=0x555555ad16e0) at thread.c:769 134#11 thread_start_func_2 (th=th@entry=0x555555ad16e0, stack_start=<optimized out>) at thread.c:822 135#12 0x00007ffff7c679a6 in thread_start_func_1 (th_ptr=<optimized out>) at /home/.rvm/src/ruby-3.0.0/thread_pthread.c:994 136#13 0x00007ffff78a63ec in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444 137#14 0x00007ffff7926a4c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 138``` 139 140This is better, but if we try to examine a frame closely we'll notice 141that source file information is still missing: 142 143``` 144(gdb) up 145#1 0x00007ffff383eb9e in do_epoll_wait (ps=0x555555ad1690, deadline=...) at src/core/lib/iomgr/ev_epoll1_linux.cc:723 146723 src/core/lib/iomgr/ev_epoll1_linux.cc: No such file or directory. 147(gdb) list 148718 in src/core/lib/iomgr/ev_epoll1_linux.cc 149(gdb) 150``` 151 152<h3>3) Resolve source files</h3> 153 154First, we fetch the *source* `grpc` gem at the **exact same version** of our binary 155`grpc` gem: 156 157``` 158cd /home 159gem fetch grpc-1.60.1.gem 160gem unpack grpc-1.60.1.gem 161``` 162 163Now we can load those sources in gdb: 164 165``` 166(gdb) dir /home/grpc-1.60.1 167Source directories searched: /home/grpc-1.60.1:$cdir:$cwd 168(gdb) 169``` 170 171Our stack frame will might look more like this now: 172 173``` 174(gdb) list 175warning: Source file is more recent than executable. 176718 int timeout = poll_deadline_to_millis_timeout(deadline); 177719 if (timeout != 0) { 178720 GRPC_SCHEDULING_START_BLOCKING_REGION; 179721 } 180722 do { 181723 r = epoll_wait(g_epoll_set.epfd, g_epoll_set.events, MAX_EPOLL_EVENTS, 182724 timeout); 183725 } while (r < 0 && errno == EINTR); 184726 if (timeout != 0) { 185727 GRPC_SCHEDULING_END_BLOCKING_REGION; 186(gdb) 187``` 188 189But if we move up a few stack frames we might *still* be missing 190some source information: 191 192``` 193(gdb) up 194#6 0x00007ffff34abf6b in run_poll_channels_loop_no_gil (arg=arg@entry=0x0) at ../../../../src/ruby/ext/grpc/rb_channel.c:663 195663 ../../../../src/ruby/ext/grpc/rb_channel.c: No such file or directory. 196``` 197 198A portion of the grpc-ruby native extension is built from a sub-directory: 199`src/ruby/ext/grpc`. So we also need to add that sub-directory, to fix this: 200 201``` 202(gdb) dir /home/grpc-1.60.1/src/ruby/ext/grpc 203Source directories searched: /home/grpc-1.60.1/src/ruby/ext/grpc:/home/grpc-1.60.1:$cdir:$cwd 204``` 205 206Note the additional info: 207 208``` 209(gdb) list 210warning: Source file is more recent than executable. 211658 gpr_mu_lock(&global_connection_polling_mu); 212659 gpr_cv_broadcast(&global_connection_polling_cv); 213660 gpr_mu_unlock(&global_connection_polling_mu); 214661 215662 for (;;) { 216663 event = grpc_completion_queue_next( 217664 g_channel_polling_cq, gpr_inf_future(GPR_CLOCK_REALTIME), NULL); 218665 if (event.type == GRPC_QUEUE_SHUTDOWN) { 219666 break; 220667 } 221(gdb) 222``` 223 224# Support 225 226grpc-native-debug currently only supports: 227 228- ruby platforms: `x86_64-linux` and `x86-linux` 229 230- grpc >= 1.60.0 231