ELF section retrieval with debuginfod

It's been almost a year since our last blog regarding debuginfod, an HTTP file server that provides debugging resources to debugger-like tools. Since then we have been busy working on improvements to the server itself, as well as the debuginfod clients built into tools such as GDB, Valgrind and Systemtap. One feature that we recently added is the ability to download specific ELF sections from ELF binaries available from the server. First I'll give brief summaries of how debuginfod works and what ELF sections are. For more information you can check out our other debuginfod blog posts.

What is debuginfod?

The debuginfod server indexes executable binaries, DWARF debuginfo and source files by build-id, unique hashes embedded in executable and debuginfo files. Tools such as debuggers, profilers and tracers can use debuginfod's client library, libdebuginfod, to query these servers for the executables, debuginfo and source files matching given build-ids and file names. This lets tools present higher quality and more detailed information without having to manually install debuginfo packages for all of the executables and shared libraries you need to debug, profile or trace.

Since 2021, Fedora comes configured with debuginfod already enabled. Tools with debuginfod support are able to automatically query the official Fedora debuginfod server for any executables, debuginfo and source files contained in any package available on Fedora 32+. In the following examples, GDB and valgrind automatically download any missing debuginfo and source files. Detailed Backtraces and source file information is then available to the user.

$ gdb -q /usr/bin/will_segfault
  Reading symbols from /usr/bin/will_segfault...
  Downloading separate debug info for /usr/bin/will_segfault...
  Reading symbols from /home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/debuginfo...
  Downloading separate debug info for /home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/debuginfo...
  (gdb) start
  Downloading source file /usr/src/debug/will-crash-0.13.3-6.fc36.x86_64/redhat-linux-build/../src/will_segfault.c...
  [...]
  Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe638) at ../src/will_segfault.c:78
  78 {
  (gdb) list
  73    _r_debug.r_map->l_next = (struct link_map *)0x1337BEEF;
  74    _r_debug.r_map->l_name = "invalid";
  75 }
  76
  77 int main(int argc, char *argv[])
  78 {
  79    if (argc >= 2 && 0 == strcmp(argv[1], "--break-link-map"))
  80    {
  81        break_link_map();
  82    }

  $ valgrind -v /usr/bin/will_stackoverflow
  [...]
  ==80841== Downloading debug info for /usr/bin/will_stackoverflow...
  --80841--   Considering /home/amerey/.cache/debuginfod_client/feb830c7c1e323da74937eb480bc611b5c95fafc/debuginfo ..
  --80841--   .. CRC is valid
  ==80841== Successfully downloaded debug file for /usr/bin/will_stackoverflow
  [...]
  ==80841== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
  ==80841==
  ==80841== Process terminating with default action of signal 11 (SIGSEGV): dumping core
  ==80841==  Access not within mapped region at address 0x1FFE801FF8
  ==80841== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
  ==80841== at 0x10918D: f (will_stackoverflow.c:7)

What are ELF sections?

Briefly, ELF is a common format for executable and debuginfo files. The binary contents of these files are organized into sections containing various types of data needed to link, run and debug a program. For example the “.text” section contains the executable code needed to run a program and the “.dynamic” section contains information regarding shared libraries used by the program. Sections like .text and .dynamic contain information needed for the operating system to properly run an executable, and they are copied into a process's memory at runtime.

Other sections aren't required to run an executable but rather exist for the benefit of debugger-like tools.  Some of these sections are part of the DWARF debugging information specification. The “.debug_info” section is one of the main components of DWARF debuginfo files. .debug_info consists of debugging information entries (DIEs) for all of the variables, functions, types and other kinds of constructs found in a program's source code. Debuggers can use this information to connect specific addresses within a running process’s memory with variables, functions, source files and line numbers.

In the following example, I download the will_segfault program’s debuginfo using the debuginfod-find command line tool and view the ELF/DWARF sections it contains using the eu-readelf tool from elfutils.

$ eu-readelf -S $(debuginfod-find debuginfo /usr/bin/will_segfault)
  There are 43 section headers, starting at offset 0x5198:

  Section Headers:
  [Nr] Name             Type     Addr         Off  Size ES Flags Lk Inf Al
  [ 0]                  NULL     0000000000000000 00000000 00000000  0    0   0  0
  [ 1] .interp          NOBITS   0000000000000318 00000318 0000001c  0 A  0   0  1
  [ 2] .note.gnu.property     NOTE     0000000000000338 00000318 00000050  0 A  0   0  8
  [ 3] .note.gnu.build-id     NOTE     0000000000000388 00000368 00000024  0 A  0   0  4
  [ 4] .note.package    NOTE     00000000000003ac 0000038c 00000090  0 A  0   0  4
  [ 5] .note.ABI-tag    NOTE     000000000000043c 0000041c 00000020  0 A  0   0  4
  [ 6] .gnu.hash        NOBITS   0000000000000460 00000440 00000024  0 A  7   0  8
  [ 7] .dynsym          NOBITS   0000000000000488 00000440 00000108 24 A  8   1  8
  [ 8] .dynstr          NOBITS   0000000000000590 00000440 00000101  0 A  0   0  1
  [ 9] .gnu.version     NOBITS   0000000000000692 00000440 00000016  2 A  7   0  2
  [10] .gnu.version_r   NOBITS   00000000000006a8 00000440 00000060  0 A  8   2  8
  [11] .rela.dyn        NOBITS   0000000000000708 00000440 000000d8 24 A  7   0  8
  [12] .rela.plt        NOBITS   00000000000007e0 00000440 00000078 24 AI 7  25  8
  [13] .init            NOBITS   0000000000001000 00000440 0000001b  0 AX 0   0  4
  [14] .plt             NOBITS   0000000000001020 00000440 00000060 16 AX 0   0 16
  [15] .plt.sec         NOBITS   0000000000001080 00000440 00000050 16 AX 0   0 16
  [16] .text            NOBITS   00000000000010d0 00000440 000002d4  0 AX 0   0 16
  [17] .fini            NOBITS   00000000000013a4 00000440 0000000d  0 AX 0   0  4
  [18] .rodata          NOBITS   0000000000002000 00000440 0000005a  0 A  0   0  4
  [19] .eh_frame_hdr    NOBITS   000000000000205c 00000440 00000054  0 A  0   0  4
  [20] .eh_frame        NOBITS   00000000000020b0 00000440 0000011c  0 A  0   0  8
  [21] .init_array      NOBITS   0000000000003d60 00000440 00000008  8 WA 0   0  8
  [22] .fini_array      NOBITS   0000000000003d68 00000440 00000008  8 WA 0   0  8
  [23] .data.rel.ro     NOBITS   0000000000003d70 00000440 00000008  0 WA 0   0  8
  [24] .dynamic         NOBITS   0000000000003d78 00000440 00000220 16 WA 8   0  8
  [25] .got             NOBITS   0000000000003f98 00000440 00000068  8 WA 0   0  8
  [26] .data            NOBITS   0000000000004000 00000440 00000004  0 WA 0   0  1
  [27] .bss             NOBITS   0000000000004008 00000440 00000030  0 WA 0   0  8
  [28] .comment         PROGBITS 0000000000000000 00000440 0000005c  1 MS 0   0  1
  [29] .gnu.build.attributes  NOTE     0000000000006038 0000049c 00000238  0 L 16   0  4
  [30] .debug_aranges   PROGBITS 0000000000000000 000006d4 00000040  0    0   0  1
  [31] .debug_info      PROGBITS 0000000000000000 00000714 00000545  0    0   0  1
  [32] .debug_abbrev    PROGBITS 0000000000000000 00000c59 0000038d  0    0   0  1
  [33] .debug_line      PROGBITS 0000000000000000 00000fe6 000001e5  0    0   0  1
  [34] .debug_str       PROGBITS 0000000000000000 000011cb 00000111  1 MS 0   0  1
  [35] .debug_line_str  PROGBITS 0000000000000000 000012dc 00000130  1 MS 0   0  1
  [36] .debug_loclists  PROGBITS 0000000000000000 0000140c 000000e2  0    0   0  1
  [37] .debug_rnglists  PROGBITS 0000000000000000 000014ee 00000058  0    0   0  1
  [38] .gdb_index       PROGBITS 0000000000000000 00001546 00002238  0    0   0  1
  [39] .gnu_debugaltlink PROGBITS 0000000000000000 0000377e 0000003f  0    0   0  1
  [40] .symtab          SYMTAB   0000000000000000 000037c0 00001008 24   41 148  8
  [41] .strtab          STRTAB   0000000000000000 000047c8 000007f5  0    0   0  1
  [42] .shstrtab        STRTAB   0000000000000000 00004fbd 000001d4  0    0   0  1

Downloading ELF sections with debuginfod

Until now, debuginfod has been able to provide three main types of resources for debugger tools. Executables/shared libraries, debuginfo and source code. As of elfutils version 0.188 debuginfod can now provide the raw binary contents of individual ELF/DWARF sections from executable and debuginfo files matching a given build-id. Debuginfod will attempt to retrieve a requested section from the debuginfo file corresponding to a given build-id. If the server has not indexed the debuginfo file or the section ought to be in the executable file corresponding to the given build-id, then the server will attempt to retrieve the section from the executable.

$ debuginfod-find section /usr/bin/will_segfault .gdb_index

/home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/section-.gdb_index

The contents and format of all the different ELF and DWARF sections is a rather complex topic, so I won't go into it any more in this post. But for those wishing to learn more about how a debugger uses debugging information checkout Keith Seitz's blog post on the topic. For more information about the ELF format, Wikipedia has a useful introduction.

Benefits of ELF section retrieval

This feature lets a tool download only the specific parts of debuginfo or executable files that may be of interest to it. Entire debuginfo files can get quite large. For example, the size of Firefox's libxul debuginfo is approximately 3.4 GB, and may take some time to download in its entirety. Currently GDB will attempt to read debuginfo for all shared libraries linked to an executable at the beginning of a debugging session, whether or not that debuginfo ends up actually being needed for that session. If debuginfod is enabled, GDB will attempt to download all missing debuginfo files including those that don’t end up being used.

We are working on improvements to GDB to help address this. The ELF section .gdb_index, which is typically found in debuginfo from Fedora packages, contains a concise list of all the symbols found in the corresponding debuginfo. The .gdb_index sections tend to be around just 7% of the size of an entire debuginfo file. GDB can use this index in place of the full debuginfo for shared libraries in order to successfully perform many user commands. By downloading just the index instead of the full debuginfo, GDB can potentially avoid downloading large quantities of debuginfo that otherwise wouldn’t be needed. Full debuginfo can be downloaded when an index no longer suffices for a given command and GDB requires the full debuginfo.

By using our experimental build of GDB that includes this feature, we can run some tests to see examples of the possible time and space saved by downloading indices instead of debuginfo. I tested Firefox, qemu-kvm and GDB itself by running them under our test build of GDB with .gdb_index download support. From the beginning of each of these program’s main functions, I stepped through the code until the program terminated or waited for user input. In the table below I compare the total amount of data GDB downloaded when .gdb_index support is enabled to the total amount that a standard build of GDB downloads.

To step through qemu-kvm and GDB, we end up downloading significantly more data by downloading indices and using them for as long as possible. Firefox is an interesting case because the total size of all of its shared library debuginfo is dominated by the size of libxul. To step through Firefox for this experiment, GDB ends up requiring libxul. In this case the total amount of data downloaded is almost the same as a standard build of GDB. However, if we only step through Firefox up until libxul debuginfo is needed, we see huge benefits from downloading just the libxul index.

This proof of concept demonstrates the potential benefits of using debuginfo ELF section extraction to facilitate efficient downloading of debugging resources. If .gdb_index sections tend to be roughly 7% of the size of the debuginfo file it indexes, then in the best case we avoid spending time and storage downloading the vast majority of all the shared library debuginfo for a program. And in the absolute worst case scenario (download all .gdb_index then require and download full debuginfo for each index) will on average result in downloading 107% of the size of all shared library debuginfo. Except for special cases, the benefits significantly outweigh the costs. Look forward to this feature in future versions of GDB!

If you have any questions or comments about debuginfod or elfutils feel free to contact us at elfutils-devel@sourceware.org or on the Libera.Chat IRC channel #elfutils.