It's been almost a year since our last blog regarding debuginfod, an HTTP file server that provides debugging resources to debugger-like tools. Since then we have been busy working on improvements to the server itself, as well as the debuginfod clients built into tools such as GDB, Valgrind and Systemtap. One feature that we recently added is the ability to download specific ELF sections from ELF binaries available from the server. First I'll give brief summaries of how debuginfod works and what ELF sections are. For more information you can check out our other debuginfod blog posts.
What is debuginfod?
The debuginfod server indexes executable binaries, DWARF debuginfo and source files by build-id, unique hashes embedded in executable and debuginfo files. Tools such as debuggers, profilers and tracers can use debuginfod's client library, libdebuginfod, to query these servers for the executables, debuginfo and source files matching given build-ids and file names. This lets tools present higher quality and more detailed information without having to manually install debuginfo packages for all of the executables and shared libraries you need to debug, profile or trace.
Since 2021, Fedora comes configured with debuginfod already enabled. Tools with debuginfod support are able to automatically query the official Fedora debuginfod server for any executables, debuginfo and source files contained in any package available on Fedora 32+. In the following examples, GDB and valgrind automatically download any missing debuginfo and source files. Detailed Backtraces and source file information is then available to the user.
$ gdb -q /usr/bin/will_segfault Reading symbols from /usr/bin/will_segfault... Downloading separate debug info for /usr/bin/will_segfault... Reading symbols from /home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/debuginfo... Downloading separate debug info for /home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/debuginfo... (gdb) start Downloading source file /usr/src/debug/will-crash-0.13.3-6.fc36.x86_64/redhat-linux-build/../src/will_segfault.c... [...] Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe638) at ../src/will_segfault.c:78 78 { (gdb) list 73 _r_debug.r_map->l_next = (struct link_map *)0x1337BEEF; 74 _r_debug.r_map->l_name = "invalid"; 75 } 76 77 int main(int argc, char *argv[]) 78 { 79 if (argc >= 2 && 0 == strcmp(argv[1], "--break-link-map")) 80 { 81 break_link_map(); 82 } $ valgrind -v /usr/bin/will_stackoverflow [...] ==80841== Downloading debug info for /usr/bin/will_stackoverflow... --80841-- Considering /home/amerey/.cache/debuginfod_client/feb830c7c1e323da74937eb480bc611b5c95fafc/debuginfo .. --80841-- .. CRC is valid ==80841== Successfully downloaded debug file for /usr/bin/will_stackoverflow [...] ==80841== Stack overflow in thread #1: can't grow stack to 0x1ffe801000 ==80841== ==80841== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==80841== Access not within mapped region at address 0x1FFE801FF8 ==80841== Stack overflow in thread #1: can't grow stack to 0x1ffe801000 ==80841== at 0x10918D: f (will_stackoverflow.c:7)
What are ELF sections?
Briefly, ELF is a common format for executable and debuginfo files. The binary contents of these files are organized into sections containing various types of data needed to link, run and debug a program. For example the “.text” section contains the executable code needed to run a program and the “.dynamic” section contains information regarding shared libraries used by the program. Sections like .text and .dynamic contain information needed for the operating system to properly run an executable, and they are copied into a process's memory at runtime.
Other sections aren't required to run an executable but rather exist for the benefit of debugger-like tools. Some of these sections are part of the DWARF debugging information specification. The “.debug_info” section is one of the main components of DWARF debuginfo files. .debug_info consists of debugging information entries (DIEs) for all of the variables, functions, types and other kinds of constructs found in a program's source code. Debuggers can use this information to connect specific addresses within a running process’s memory with variables, functions, source files and line numbers.
In the following example, I download the will_segfault program’s debuginfo using the debuginfod-find command line tool and view the ELF/DWARF sections it contains using the eu-readelf tool from elfutils.
$ eu-readelf -S $(debuginfod-find debuginfo /usr/bin/will_segfault) There are 43 section headers, starting at offset 0x5198: Section Headers: [Nr] Name Type Addr Off Size ES Flags Lk Inf Al [ 0] NULL 0000000000000000 00000000 00000000 0 0 0 0 [ 1] .interp NOBITS 0000000000000318 00000318 0000001c 0 A 0 0 1 [ 2] .note.gnu.property NOTE 0000000000000338 00000318 00000050 0 A 0 0 8 [ 3] .note.gnu.build-id NOTE 0000000000000388 00000368 00000024 0 A 0 0 4 [ 4] .note.package NOTE 00000000000003ac 0000038c 00000090 0 A 0 0 4 [ 5] .note.ABI-tag NOTE 000000000000043c 0000041c 00000020 0 A 0 0 4 [ 6] .gnu.hash NOBITS 0000000000000460 00000440 00000024 0 A 7 0 8 [ 7] .dynsym NOBITS 0000000000000488 00000440 00000108 24 A 8 1 8 [ 8] .dynstr NOBITS 0000000000000590 00000440 00000101 0 A 0 0 1 [ 9] .gnu.version NOBITS 0000000000000692 00000440 00000016 2 A 7 0 2 [10] .gnu.version_r NOBITS 00000000000006a8 00000440 00000060 0 A 8 2 8 [11] .rela.dyn NOBITS 0000000000000708 00000440 000000d8 24 A 7 0 8 [12] .rela.plt NOBITS 00000000000007e0 00000440 00000078 24 AI 7 25 8 [13] .init NOBITS 0000000000001000 00000440 0000001b 0 AX 0 0 4 [14] .plt NOBITS 0000000000001020 00000440 00000060 16 AX 0 0 16 [15] .plt.sec NOBITS 0000000000001080 00000440 00000050 16 AX 0 0 16 [16] .text NOBITS 00000000000010d0 00000440 000002d4 0 AX 0 0 16 [17] .fini NOBITS 00000000000013a4 00000440 0000000d 0 AX 0 0 4 [18] .rodata NOBITS 0000000000002000 00000440 0000005a 0 A 0 0 4 [19] .eh_frame_hdr NOBITS 000000000000205c 00000440 00000054 0 A 0 0 4 [20] .eh_frame NOBITS 00000000000020b0 00000440 0000011c 0 A 0 0 8 [21] .init_array NOBITS 0000000000003d60 00000440 00000008 8 WA 0 0 8 [22] .fini_array NOBITS 0000000000003d68 00000440 00000008 8 WA 0 0 8 [23] .data.rel.ro NOBITS 0000000000003d70 00000440 00000008 0 WA 0 0 8 [24] .dynamic NOBITS 0000000000003d78 00000440 00000220 16 WA 8 0 8 [25] .got NOBITS 0000000000003f98 00000440 00000068 8 WA 0 0 8 [26] .data NOBITS 0000000000004000 00000440 00000004 0 WA 0 0 1 [27] .bss NOBITS 0000000000004008 00000440 00000030 0 WA 0 0 8 [28] .comment PROGBITS 0000000000000000 00000440 0000005c 1 MS 0 0 1 [29] .gnu.build.attributes NOTE 0000000000006038 0000049c 00000238 0 L 16 0 4 [30] .debug_aranges PROGBITS 0000000000000000 000006d4 00000040 0 0 0 1 [31] .debug_info PROGBITS 0000000000000000 00000714 00000545 0 0 0 1 [32] .debug_abbrev PROGBITS 0000000000000000 00000c59 0000038d 0 0 0 1 [33] .debug_line PROGBITS 0000000000000000 00000fe6 000001e5 0 0 0 1 [34] .debug_str PROGBITS 0000000000000000 000011cb 00000111 1 MS 0 0 1 [35] .debug_line_str PROGBITS 0000000000000000 000012dc 00000130 1 MS 0 0 1 [36] .debug_loclists PROGBITS 0000000000000000 0000140c 000000e2 0 0 0 1 [37] .debug_rnglists PROGBITS 0000000000000000 000014ee 00000058 0 0 0 1 [38] .gdb_index PROGBITS 0000000000000000 00001546 00002238 0 0 0 1 [39] .gnu_debugaltlink PROGBITS 0000000000000000 0000377e 0000003f 0 0 0 1 [40] .symtab SYMTAB 0000000000000000 000037c0 00001008 24 41 148 8 [41] .strtab STRTAB 0000000000000000 000047c8 000007f5 0 0 0 1 [42] .shstrtab STRTAB 0000000000000000 00004fbd 000001d4 0 0 0 1
Downloading ELF sections with debuginfod
Until now, debuginfod has been able to provide three main types of resources for debugger tools. Executables/shared libraries, debuginfo and source code. As of elfutils version 0.188 debuginfod can now provide the raw binary contents of individual ELF/DWARF sections from executable and debuginfo files matching a given build-id. Debuginfod will attempt to retrieve a requested section from the debuginfo file corresponding to a given build-id. If the server has not indexed the debuginfo file or the section ought to be in the executable file corresponding to the given build-id, then the server will attempt to retrieve the section from the executable.
$ debuginfod-find section /usr/bin/will_segfault .gdb_index /home/amerey/.cache/debuginfod_client/619f22fe6c8e69c52b89c43b7337e1c2fb6ca145/section-.gdb_index
The contents and format of all the different ELF and DWARF sections is a rather complex topic, so I won't go into it any more in this post. But for those wishing to learn more about how a debugger uses debugging information checkout Keith Seitz's blog post on the topic. For more information about the ELF format, Wikipedia has a useful introduction.
Benefits of ELF section retrieval
This feature lets a tool download only the specific parts of debuginfo or executable files that may be of interest to it. Entire debuginfo files can get quite large. For example, the size of Firefox's libxul debuginfo is approximately 3.4 GB, and may take some time to download in its entirety. Currently GDB will attempt to read debuginfo for all shared libraries linked to an executable at the beginning of a debugging session, whether or not that debuginfo ends up actually being needed for that session. If debuginfod is enabled, GDB will attempt to download all missing debuginfo files including those that don’t end up being used.
We are working on improvements to GDB to help address this. The ELF section .gdb_index, which is typically found in debuginfo from Fedora packages, contains a concise list of all the symbols found in the corresponding debuginfo. The .gdb_index sections tend to be around just 7% of the size of an entire debuginfo file. GDB can use this index in place of the full debuginfo for shared libraries in order to successfully perform many user commands. By downloading just the index instead of the full debuginfo, GDB can potentially avoid downloading large quantities of debuginfo that otherwise wouldn’t be needed. Full debuginfo can be downloaded when an index no longer suffices for a given command and GDB requires the full debuginfo.
By using our experimental build of GDB that includes this feature, we can run some tests to see examples of the possible time and space saved by downloading indices instead of debuginfo. I tested Firefox, qemu-kvm and GDB itself by running them under our test build of GDB with .gdb_index download support. From the beginning of each of these program’s main functions, I stepped through the code until the program terminated or waited for user input. In the table below I compare the total amount of data GDB downloaded when .gdb_index support is enabled to the total amount that a standard build of GDB downloads.
Program |
.gdb_index test build |
Typical GDB build |
Reduction |
qemu-kvm |
105 MB |
360 MB |
255 MB (71%) |
GDB |
11 MB |
121 MB |
110 MB (91%) |
Firefox (without libxul) |
22 MB |
3796 MB |
3774 MB (99%) |
Firefox (with libxul) |
3743 MB |
3796 MB |
53 MB (1%) |
To step through qemu-kvm and GDB, we end up downloading significantly more data by downloading indices and using them for as long as possible. Firefox is an interesting case because the total size of all of its shared library debuginfo is dominated by the size of libxul. To step through Firefox for this experiment, GDB ends up requiring libxul. In this case the total amount of data downloaded is almost the same as a standard build of GDB. However, if we only step through Firefox up until libxul debuginfo is needed, we see huge benefits from downloading just the libxul index.
This proof of concept demonstrates the potential benefits of using debuginfo ELF section extraction to facilitate efficient downloading of debugging resources. If .gdb_index sections tend to be roughly 7% of the size of the debuginfo file it indexes, then in the best case we avoid spending time and storage downloading the vast majority of all the shared library debuginfo for a program. And in the absolute worst case scenario (download all .gdb_index then require and download full debuginfo for each index) will on average result in downloading 107% of the size of all shared library debuginfo. Except for special cases, the benefits significantly outweigh the costs. Look forward to this feature in future versions of GDB!
If you have any questions or comments about debuginfod or elfutils feel free to contact us at elfutils-devel@sourceware.org or on the Libera.Chat IRC channel #elfutils.
About the author
Aaron Merey is a Software Engineer at Red Hat, where he is a member of the Platform Tools team.
More like this
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit