How the debuginfod project evolved in 2021

Debuginfod is a web service for distributing debugging resources over HTTP. The project began in 2019 and has seen a lot of growth in 2021. 

In this post I will share some of the new additions to debuginfod in 2021. But first I will briefly introduce debuginfod. For additional information on debuginfod you can also check out our previous blog posts: 

• Introducing debuginfod, the elfutils debuginfo server

• Deploying debuginfod servers for your developers

• Debuginfod project update: New clients and metrics

What is debuginfod?

First let me briefly recap what debug info is. Debug info relates an executable or shared library’s encoded contents with human-readable information from their source files. This includes information such as the names of functions and variables and the file and line numbers they are defined and referred to. This information can enhance developer tools like debuggers, tracers and profilers. We’ll look at some examples of this below.

Debug info is useful for enabling informative debugging, but acquiring and storing debug info so that tools can easily make use of it can be inconvenient or even impossible depending on your system, permissions and debugging environment. 

Debuginfod helps solve these problems and makes it easier to automatically acquire the necessary debug info in addition to any executables or source files that may be needed without needing special permissions. 

Using debuginfod with supported tools is as easy as setting an environment variable:

  $ export DEBUGINFOD_URLS=https://debuginfod.elfutils.org
  $ gdb ./my_prog
  [...]
  (gdb) run 
  Downloading 0.02 MB separate debug info for /lib64/libdl.so.2
  Downloading 0.01 MB separate debug info for /lib64/libutil.so.1 
  Downloading 0.03 MB separate debug info for /lib64/libuuid.so.1 
  [...]
  (gdb) list
  Downloading 0.01 MB source file /usr/src/debug/readline-8.0-5.fc33.x86_64/shlib/../readline.c
  271         if (rl_done)
  272           {
  273             line = readline_internal_teardown (eof);
  274   
  275             if (rl_deprep_term_function)
  276               (*rl_deprep_term_function) (); 
  277   #if defined (HANDLE_SIGNALS)

In the above example, GDB automatically queries a debuginfod server for any missing debug info and source files. Debuginfod clients can easily be added to debugger-like tools using shared library, libdebuginfod. Alternatively debuginfod is packaged with a simple command line tool, debuginfod-find, in order to aquire debugging resources from the command line.

  $ debuginfod-find debuginfo f189c8462cf66a8a7a618f18da7c6e7ffa140b4b
  $ debuginfod-find source f189c8462cf66a8a7a618f18da7c6e7ffa140b4b \
      /usr/src/debug/coreutils-8.32-18.fc33.x86_64/separate/../src/mkdir.c

New public servers

To improve ease of access to debugging resources, debuginfod servers are now being offered by various Linux distributions including Fedora, Ubuntu and Debian. These servers provide debuginfo, executable and source files from .rpm, .deb and .ddeb packages available from recent versions of these operating systems.

See https://sourceware.org/elfutils/Debuginfod.html for the complete list of public debuginfod servers. In the newly released Fedora 35, debuginfod is enabled by default. Tools that support debuginfod will automatically query Fedora's public debuginfod servers when missing any debugging resources.

If you are using another linux distribution then to download from any of these servers, simply add the server's URL to your DEBUGINFOD_URLS environment variable. Alternatively you can just add https://debuginfod.elfutils.org to the environment variable. This server federates public debuginfod servers and will route your queries to the distro server that has the desired resource.

New clients

Earlier in the year debuginfod support was added to valgrind. Valgrind is a collection of dynamic analysis tools, including memcheck which is a particularly well-known tool for finding memory bugs in programs. Without debuginfo, memcheck is not able able to pinpoint the location of a bug in the source code

  $ valgrind -v ./bad_write
  [...]
  ==25823== Invalid write of size 1
  ==25823==    at 0x40115C: ??? (in /home/amerey/bad_write)
  ==25823==    by 0x48941E1: (below main) (in /usr/lib64/libc-2.32.so)
  ==25823==  Address 0x4a3a040 is 0 bytes inside a block of size 100 free'd
  ==25823==    at 0x483D3A0: free (vg_replace_malloc.c:872)
  ==25823==    by 0x401157: ??? (in /home/amerey/bad_write)
  ==25823==    by 0x48941E1: (below main) (in /usr/lib64/libc-2.32.so)
  ==25823==  Block was alloc'd at
  ==25823==    at 0x483A831: malloc (vg_replace_malloc.c:381)
  ==25823==    by 0x401147: ??? (in /home/amerey/bad_write)
  ==25823==    by 0x48941E1: (below main) (in /usr/lib64/libc-2.32.so)

With debuginfo, valgrind is able to produce detailed stack traces and descriptive error messages that are otherwise not possible. If valgrind is not able to locate an executable's separate debug info then it can now query debuginfod servers when DEBUGINFOD_URLS is set.

  $ set DEBUGINFOD_URLS='https://debuginfod.elfutils.org http://my.local.server'
  $ valgrind -v ./bad_write
  [...]
  ==25883== Successfully downloaded debug file for /home/amerey/bad_write
  ==25883== Successfully downloaded debug file for /usr/lib64/libc-2.32.so
  [...]
  ==25883== Invalid write of size 1
  ==25883==    at 0x40115C: main (bad_write.c:7)
  ==25883==  Address 0x4a3a040 is 0 bytes inside a block of size 100 free'd
  ==25883==    at 0x483D3A0: free (vg_replace_malloc.c:872)
  ==25883==    by 0x401157: main (bad_write.c:6)
  ==25883==  Block was alloc'd at
  ==25883==    at 0x483A831: malloc (vg_replace_malloc.c:381)
  ==25883==    by 0x401147: main (bad_write.c:5)

Dyninst also received debuginfod support earlier this year. Dyninst is a binary rewriting framework that allows for a wide variety of binary rewriting and tooling. It can benefit from the presence of debug info and therefore it was a great candidate for debuginfod integration. 

As with valgrind, debuginfod automatically attempts to retrieve any missing resources without any hassle normally associated with acquiring debug info. Other new debuginfod clients this year also include delve, perf, bpftrace and even WinDbg. Debuginfod support is also in progress for the abrt retrace server.

To get started with debuginfod, install Fedora 35 and just start debugging or tracing with any tool containing debuginfod support! For more information on debuginfod, including tools supporting it, see elfutils debuginfod services. Debuginfod is a part of the elfutils project. For more information on elfutils and installing debuginfod from source, see the elfutils home page.