diary of a Binutils Release

Red Hat Enterprise Linux (RHEL) packages are often linked to upstream projects - projects that are created by people in the open source community and then made available to anyone who wants them. Releases from these projects are brought into Fedora, often with local changes applied on top to improve their fit. The Fedora releases are then copied into CentOS Stream and from there into RHEL.

RHEL engineers are involved in all of these stages, including contributing changes to the upstream sources, making upstream releases and bringing the releases into Fedora, CentOS and RHEL. 

Creating a release for an upstream open source package can take a lot of work. Big packages have whole teams dedicated to the process and a well-publicised release schedule. However, the GNU Binutils project is smaller and so just one or two people can handle the releases. This blog is a diary of what happened behind the scenes for the 2.41 release of a single open source project - the GNU Binutils. Many of the steps are actually documented in the binutils sources themselves, in a file called README-how-to-make-a-release, however,  that does not cover all of the activities and it notes some of the steps in general terms rather than specific ones.

June 7

Time to decide exactly when the release should happen. The previous release (2.40) went out on January 16,  2023. Since we are trying for two releases per year, that means that the next release ought to go out around July 16. (Coincidentally this coincides quite nicely with Fedora's release schedule of two releases a year as well…) But I am on vacation for July 10-14 so if there are any last minute changes that need to go in, I would not be there to handle them. So pushing the release out a week seems to be a good idea. Hence July 23. 

Normally, I prefer to make releases during the weekend as I am less likely to get an emergency request for a change to go in. But this time I have an accomplice - Yara Ahmad - so pushing the release out by a day to Monday, July 24 means that she can assist me.

The next date to choose is when to make the release branch. Ideally this should be at least two weeks before the release happens. This gives interested parties enough time to get any important bug fixes into the sources, but also keeps the branch recent enough that it really reflects the state of the development sources. Based on that, creating the release branch on July 10 would normally be good, but again, this is going to clash with my holiday. (I really should have chosen better dates for that...) So moving the branch back a week to July 3 is my current choice.

Next we have to decide when to announce the upcoming branch to the upstream GNU Binutils community. People like to have some time to get new features into the sources, so announcing the branch ahead of time is appreciated. There is no hard and fast rule for when to announce the branch, but again, I like giving at least two weeks notice. So plan to send out an email some time around the middle of June. Maybe June 18. We shall see.

Next step - rope in my partner in crime, Yara, and start testing the current sources to see if they are in good enough shape to be released.

June 8

It looks like we need to do some work on the various testsuites. Important targets like x86_64-pc-linux-gnu are showing failures in the linker testsuite's CTF test section. These need to be investigated. The goal is for zero failures in the binutils testsuites for all major targets and ideally for every single supported target. Fortunately I may not have to do the work to fix these failures. We have a volunteer who specifically looks after the CTF code, so once I have checked to make sure that I am not doing anything stupid when I run the tests, I can send him an email asking for help.

June 12

I have just heard back from the CTF maintainer. He cannot reproduce the failures. After a little bit of back and forth, we discovered that the problem comes from the fact that although I am cleaning and then building the sources, I am not rerunning the configure script. In theory this should not matter, but in practice, it does. Oh well. Nearly all builders of binutils will start by running the configure script, so they are unlikely to run into this problem. Hence, whilst I will need to investigate it one day, I do not need to look at it before 2.41 goes out.

June 13

A while ago I submitted a patch upstream that adds a new feature to the linker: an option called --remap-input, which allows input files to be renamed or removed before the link happens. No one has commented on it, however, and I want to get it into the sources before the release happens, so I have checked it in today.

June 14

Time to look at some other targets. The BFIN architecture is showing 19 failures in the linker testsuite, and the HPPA architecture has 9 failures. Some other targets have one or two failures, but I tend not to worry about them. They are usually due to the fact that the tests have not been set up to handle all possible target configurations rather than due to actual bugs. 

June 1

 *sigh* My fixes for the BFIN linker testsuite problems actually made things worse. When I logged in this morning, I found that my co-chief-maintainer, Alan Modra, had had to apply a second patch to fix the errors in my patch. How embarrassing. It turns out that there are *two* BFIN linux targets, one called bfin-uclinux and the other called bfin-linux-uclibc. I had not realized this, and so I only tested and fixed one target. Of course, this broke the other target,so Alan had to fix my patch. 

June 19

Today, I sent out the announcement of the forthcoming branch and release. Fortunately the README-how-to-make-a-binutils-release document contains a template for the email and all I have to do is customize it to put in the correct version and dates.

June 21

With my Red Hat hat on, I was looking at improving the security of generated code. One issue of particular relevance is the fact that the linker can create programs that have an executable stack. This is a bad idea, as it means that buffer overrun attacks could take advantage of the stack and write code to it, which then gets executed. There is a compiler feature to tell the linker not to create an executable stack, but it only works if all of a program is compiled with the feature enabled. If at least part of a program does not have the necessary markup - maybe because it is assembled rather than compiled - then the linker will resort to the default behavior for the architecture. Some architectures default to allowing an executable stack, and so they are vulnerable to attack. The linker has a configure option called --enable-default-execstack=no which changes the default to never creating an executable stack unless explicitly requested.

I have started experimenting with enabling this configure option and then rebuilding lots of packages to see if it breaks anything. The first thing that did break was the linker's own testsuite, which has a test for the creation of executable stacks by default. Obviously this needed to be fixed and so I spent a day trying to work out how to test a configure time option from inside a linker test case. It turned out to be quite tricky... 

June 23

Yara tested the x86_64 native toolchain and found no testsuite failures for the linker, assembler, or other binary tools. This is excellent. I ran the gold linker tests and these are all good, too. So, it looks like the sources are in good shape—at least for the x86_64 Linux target. 

June 26

Time to sync the config files and the libiberty directory. The Binutils sources are unusual in a couple of ways. The first is that they share their top-level files with the GDB project. So any changes made to these files have to be agreed upon by both the GDB folks and the Binutils folks. The second weird feature is that some of the files in the Binutils sources do not actually belong to the Binutils or GDB projects. In particular, the top-level config.sub and config.guess files are actually maintained by the GNU Config project, and the whole libiberty sub-directory is actually maintained by the GNU GCC project. So when a branch approaches, one of the tasks I take on is making sure that these non-binutils-owned sources are up to date with respect to their master versions. Usually this is just a case of copying over the files and then running a few builds to make sure that everything is still OK. But occasionally bugs need to be fixed. Fortunately it looks like this time, the merge will be easy. 

July 3:

The 2.41 branch has been created. In fact it went more quickly than expected. Normally I would spend 2-3 hours creating the branch - there is a lot more involved than just using the git branch command - but this time it only took 1.5 hours. Yara was watching over my shoulder as I went through the process, so next time, she gets to do the work and I get to watch. 

July 5

Let the translation flood begin. One of the tasks when creating a branch is to let the GNU Translation Project know about the new branch and this triggers a lot of updated and new translations to be submitted. So a couple of days after the branch is created, the binutils mailing list starts to fill up with announcements of the availability of new translations. One of my jobs, therefore, is to respond to each of these announcements and download and install the translation files.

July 18

Several patches for the branch have come in. There are all minor bug fixes, however, nothing serious. 

July 21

I have just received an email request from one of the maintainers asking for the release to be delayed whilst an issue is resolved. I have to decide if the problem - an issue with the support for the MIPS architecture - is sufficient to warrant delaying the release. But I trust this maintainer, and if they say that the issue is important, then I believe them. So I agree to the delay. 

July 31

Release Day! Finally the problems have been resolved and I can make the release. In theory the process is very simple. There is a script called src-release.sh that makes the source tarballs, and all I have to do is upload them to the binutils website. In practice, of course, things are never this easy. First off there are two websites for binutils releases, one run by the FSF and one run by the Sourceware people. So, the tarballs have to be uploaded twice, using two different methods. Then the binutils web pages need to be updated to reference the new release, and all of the binutils documentation has to be uploaded as well. But, 2 hours after starting the process, we were finally finished, and we can give ourselves a pat on the back for a job well done. 

Aug 01

And so the bug reports start coming in. It turns out that a fix intended to make building the binutils sources without building the documentation breaks in the Cygwin environment. There is a fix, but the source tarballs have already been released, so all Cygwin builders will have to apply a patch locally. 

Aug 02

Oh boy - I have mucked up. A maintainer just noticed that the git tag that is intended to point to the sources as they were when the release was created (binutils-2.41) actually points to the source as they were just before the release. In particular, the release version number in the sources is wrong. The tag points to sources for version 2.40.90 and not for 2.41. sigh I may have to create a point release just to fix this problem.

Aug 04

Finally fixed the problem. After breaking the entire master branch in the Binutils repository (by checking in the release tarball over the top of the sources) and having a contributor bail me out by reverting my commit, I was finally able to find someone with enough git knowledge to know how to create a new commit in between two already existing commits. (Tip - you cannot, instead you have to create a new branch that looks like it contains a commit in between two others). 

Aug 08

Time to start work on Fedora. The rawhide binutils tracks the upstream binutils releases, but I try to make sure that each release of Fedora uses a different release of the GNU Binutils. So since the branches for Fedora 39 have just been created, it is now safe for me to update rawhide. Unfortunately, it is not simply a case of downloading the latest 2.41 release tarball and updating the binutils.spec file to use it. Fedora binutils have their own local patches applied on top of the official release, and these sometimes have to be massaged to fit or deleted if they are no longer needed. Plus these local patches often cause new failures in the binutils testsuites, so these have to be fixed. 

Aug 16

Success - Fedora rawhide binutils is now using 2.41.  

Aug 22

At this point I consider the 2.41 release to be complete. The sources have been available upstream for three weeks now, and in the rawhide buildroot for a week, and no new bug reports have come in. So, it is time to sit back, brew a big cup of tea, and definitely not jinx anything by publishing a blog saying: "all is well."

Aug 23

Of course I jinxed things. It turns out that whilst I had fixed all of the problems for the main targets supported by Fedora rawhide, I had forgotten about the RISC-V target. This is being built as part of a separate project, so I do not normally test it. It turns out that there are some problems with the LTO linker plugin for that target, so now I need to investigate.

Aug 24

Right - those problems are fixed now, so it is time to wait for the next bug report to come in…

Epilogue

The release process is not really over yet. The Fedora rawhide binutils still have to be copied over to CentOS Stream and from there into RHEL. That will not happen for a few months at least - which gives plenty of time for any problems with the new release to surface. But in the end, RHEL users will benefit from all the hard work performed by the upstream maintainers and RHEL engineers in creating the 2.41 release.