Running a Debuginfod server alongside pulp

It is often extremely helpful for debugger-like tools such as gdb and valgrind to have access to additional debugging resources. These files are generally found in package repositories and are difficult to locate manually. In this article we demonstrate how to quickly and easily set up a debuginfod server with Pulp so that all of the RPM packages managed by Pulp are available to the debuginfod client. 

Pulp is a component of Red Hat Satellite 6, the tool used to deploy and manage packages to Red Hat Enterprise Linux hosts. Pulp can fetch, upload and distribute various forms of content, including RPM packages, container images and Python package indices. It can manage content from multiple remotes and track it in internal repositories. These can be published as distributions, allowing the user easy organization of and access to resources.

Debuginfod is a HTTP file server which scans directory trees and RPM packages for debugging resources and serves them to debugger-like tools. You can find more information in this Red Hat Developer blog article which introduces debuginfod. 

Setting up the container and running debuginfod

The simplest way to set up and run Pulp is within a container, although it should be noted that at the time of writing this article the container is not ready for production. It can be set up using the Pulp provided settings. The required files can be cloned from the GitHub repository and built as follows.

`$ podman build --file pulp-oci-images/pulp/Containerfile --tag pulp .

The debuginfod server is started up as a background process when the container is run, scanning for artifacts in /var/lib/pulp. Additional debuginfod arguments can be included with --build-arg DEBUGINFOD_OPTIONS="[OPTION] ... ".

Once the container is built, Pulp can be interacted with on port 8080, and the debuginfod server on port 8081. Note that on machines without SELinux in enforcing mode, the:Z can be dropped from the code below.

$ podman run \
--detach \
--publish 8080:80 \
--publish 8081:8002 \
--name pulp \
--volume "$(pwd)/settings":/etc/pulp:Z \
--volume "$(pwd)/pulp_storage":/var/lib/pulp:Z \
--volume "$(pwd)/debuginfod":/var/cache/debuginfod:Z \
--volume "$(pwd)/pgsql":/var/lib/pgsql:Z \
--volume "$(pwd)/containers":/var/lib/containers:Z \
--device /dev/fuse \
pulp

Alternatively, the image is also available under the pulp namespace on Dockerhub and can be acquired using podman pull docker.io/pulp/pulp.

Once the administrator's password has been set:

$ podman exec -it pulp bash -c 'pulpcore-manager reset-admin-password'

Pulp can be queried to verify that it is running:

$ http localhost:8080/pulp/api/v3/status/

The standard Pulp create-sync-publish workflow

An interactive shell can be run in the container so that a standard Pulp workflow can be set up. Note this could be done using the pulp-cli from outside of the container as well. The first step is to set up an admin user, using the same password used above, with Pulp listening within the container on port 80.

$ pulp config create --username admin --base-url http://localhost:80 --password <admin password>

We will first make a repository foo to locally store the packages. Then we will create a remote bar from which foo will retrieve packages. For this demonstration we will set the remote to be the repository of debugging packages for Fedora 35 for the 64 bit x86 instruction set. Lastly, we will publish foo and distribute it under the name baz. This is the user facing object from which users will get their packages.

As a final step before exiting the container we will remove all of the upstream debuginfod server paths by clearing DEBUGINFOD_URLS. We do this purely for educational purposes, to show that only the local distribution is being used for the debuginfo, although in a real workflow this step can be skipped. 

Retrieving debuginfo files

Now that the distribution baz is set up, all we need to do is tell the debuginfod client outside of the container about the new debuginfod server running in the container:

export DEBUGINFOD_URLS=http://localhost:8081/ 

And that’s it, now the debugger-like tools which use debuginfod have access to all of the files managed by Pulp.

The following is an example from the author’s machine (trimmed for clarity). We can check the buildid of grep and see that gdb is able to find the debuginfo for this exact package and download its debuginfo:

$ readelf -n /bin/grep | grep Build.ID
       Build ID: e81e4e6e322030178260ae4f6055f781cd4997e1
$ gdb /bin/grep
       ...
       Reading symbols from /bin/grep...
       
       This GDB supports auto-downloading debuginfo from the following             URLs:
       http://localhost:8081/
       Downloading separate debug info for /bin/grep...
       Reading symbols from    /home/rgoldber/.cache/debuginfod_client/e81e4e6e322030178260ae4f6055f781cd4997e1/debuginfo...

Conclusion

We hope this demo makes it clear how to run a debuginfod server adjacent to Pulp, which results in a simple way for debuginfod to access and retrieve debuginfo and source files directly from your package repository. Since Pulp is one of the components of Red Hat Satellite 6, without too much work you could similarly run debuginfod alongside it resulting in convenient, powerful debugging capabilities when using Red Hat Enterprise Linux.