Extras Rebuild?

[Toshio]

On Sun, 2005-05-22 at 14:05 +0200, Michael Schwendt wrote:
> On Sat, 21 May 2005 18:42:40 -0700, Toshio Kuratomi wrote:

[Note -- heavy snippage/some rearranging.  If I left anything out, or changed the
 meaning, please be sure to bring it up.]

My goal is to know what packages have problems and what the level of
problem is.  A package that has been imported into the repository from
FC3 and not rebuilt could be fine, fine once rebuilt, or broken once
rebuilt.  But we don't know which of these states a package is in right
now.  The reason I asked if we were going to have an automated rebuild
is so we could determine this WRT packages whose current FE-devel
version is broken (presumably because it's last build was too far in the
past) and who have no bugzilla entry to tell us what's going on::

> [...] whatever failed to rebuild on April 7th exists in the repository as a
> copied FC3 binary.

Based on this, I should be able to write a script that tells if a
package is in an unknown state:
1) Compare date of packages.  If April 7th or older, then check if
there's a bugzilla entry for build failure.  If not, open a bugzilla
ticket that says not-built for FC4 devel.

Can you confirm that this criteria will catch not-compiled with gcc4,
not synced packages, and anything else that could cause problems when
the package is built on FC4?

Would it be most productive to target rebuilds at these pre-April 7th
packages or does filing in bugzilla for the package maintainer to take
care of them make more sense?

> > The major problem in my mind being that previous rebuilds didn't
> > result in bugzilla bugs being filed.
> 
> Right. But notice that we imported Seth's build failure report into the
> Wiki for tracking purposes, and later on I filed a bug report about every
> package on that list. The FE4Target tracker depends on all those bugs
> plus other issues, such as crashes.
> [...]
> Or let me ask a question: what other packages should be added to
> bug 157183?
> 
scorched3d is my only example ATM.  It has no bugzilla entry.  It was
reported on Bill's list of mismatched packages.  It appears to be one of
the "copied from FC3" packages.  That's two empty bugzilla searches
(check tracker, check for scorched3d targetted bugs), one email search,
and a perusal of download.fedora.redhat.com later.  And I still haven't
looked for the (possibly non-existent) build logs.

Maybe the imported from FC3 packages are the only places where bugzilla
was not filled in?

> >   Is the package available on all architectures and has been rebuilt at a
> >   sufficiently recent date that all its buildrequirements had been updated
> >   to a final enough version.
> 
> That assumes that [minor] version upgrades introduced ABI/API
> incompatibility without the package owner paying attention to it. Can you
> provide any examples?
> 
Nope.  But it has been known to happen that a bugfix causes other,
unforeseen problems.  To make sure that this hasn't happened we need to
be sure we've compiled against recent enough versions of the tools.
Note: I'm speaking less of libraries (which can usu. be replaced with
bugfixed ones later) as the compiler, linker, and other tools that
compile source into the binaries that go into the rpm.

> > > Bill Nottingham some days ago posted a list of Extras packages to
> > > maintainers-list, which need a rebuild, because of version mismatches on
> > > the three archs we build for. We should assume that package maintainers
> > > have noticed that list.
> > > 
> > But it wasn't.  Or maintainers don't all have enough time to work on them.
> 
> "Bump release, make tag, make build" is all that would be needed.
> 
This is more an empirical observation:  either the list hasn't been
noticed or there is some constraint preventing some of these packages
from being fixed.  But it's not bugzilla'd so there's no easy way to
tell.

Also, how is "bump,tag,build" superior to a targeted automated rebuild
that dropped build failures into bugzilla?  Isn't your argument against
automated rebuilds that a maintainer needs to check buildrequires
against the new FC and run tests to see if there's any change in runtime
behaviour?  In other words, do things above and beyond, bump, tag,
build?

> > We need to have these filed in bugzilla and become a dependency of the
> > FE4Tracker bug so the community can take a look at these problems.
> 
> Are these due to "problems"? I still think most of the version mismatches
> are due to missing powerpc build support earlier on.
>
> For the noarch cases, packages have been copied over. A series of
> other packages has been rebuilt for all archs meanwhile.
> 
They're symptoms of a problem: If the package has not been rebuilt
recently enough to be rebuilt for all the arches, then it may suffer
from other problems as well.  In scorched3d's case, I started a rebuild
and found that it additionally needs fixing on 64bit platforms b/c gcc4
throws errors instead of warnings on some broken code.  The lack of
rebuild masked compile-time issues.  Seeing the package in the not-
sync'd list points out that this possibility has not been checked.

> > > In addition to that, the FE4Target tracker contains a list of broken
> > > packages, which failed to rebuild for FC4 or which fail due to serious
> > > defects. These are issues a mass rebuild is unlikely to fix. As some of us
> > > have attached patches to some of the bugs, how long do we wait for a
> > > reaction from the package owner?
> > > 
> > Email the owner.  Try to get one other person to review the change.
> > Eventually merge yourself.  We need to evolve some policy here.  To be
> > useful for FC4, though, we need to come up with something sane quickly
> > as we need to merge those fixes and test them.
> 
> The thing here is, bugzilla tickets are _assigned to_ the package owner
> and _mailed to_ the package owner. So, that is a way already to address
> the package owner. Assume that package owners do see your bug report and
> your attachments (if any) and could add a comment like "go ahead with
> the fix" or set status to ASSIGNED to give you some feedback.
> 
And when the package maintainer does not respond to bugzilla mail a
direct email that explains the patch and asks for permission to apply
will often get a response.  Perhaps we need to change our culture to
promote rapid responses to bugzilla mail and slower responses to
personal mail :-)

I completely missed the main thrust of your statement, though -- that an
automated rebuild wouldn't pull these packages out of bugzilla and merge
them.  That's entirely correct.  I'm in favor of a targetted rebuild of
packages which don't have failure information in bugzilla.  I mentioned
an automated mass rebuild in my initial message because I remembered a
thread mentioning that was going to happen and people were debating
when.  I must have missed the end of the thread where the idea was
nixed.

> It [FE4Tracker] _could_ influence when to make available the current devel
> repository in an "extras/4" channel.  That would not be fair, though, since
> some contributors make sure their packages are ready, and other packagers
> perhaps depend on upstream projects catching up with API changes.
> 
I don't think making the devel repository into extras/4 is a good idea.
It'll compound the fact that we copied packages from the FE3 tree in
with things built early in the FC4 cycle not running on FE4-final.
Maintainers should request their packages be built for the FE4
repository explicitly.

> > For Extras in its present incarnation, it makes more sense to have
> > per-package trackers.  If a package has remaining issues, it doesn't
> > make it into the FEx repository.
> 
> That is the reason why I didn't like it when the public repository was
> filled with copies of FC3 binaries without a guide for the packagers when
> to rebuild their packages. We should have built Fedora Extras Development
> from scratch, since the only thing we depend on is Fedora Core (well,
> except for future programming language packages which need bootstrapping).
> 
I agree 100%.  I'd agree 200% if there were two of me.  I think ghc is
the only present package in FE that needs bootstrapping.  Everything
else is rooted in Core.  So going forward, we should keep a list of such
packages and only push those to the next version's repository::
  http://www.fedoraproject.org/wiki/Extras/PackagesNeedingBootstrap

> > Among other things, this means build failures should go in bugzilla (perhaps
> > the buildsystem can do this just as it emails out failures.  The problem is
> > a bit harder than email as we probably want to update older build failures
> > rather than create new ones and close out old ones when the build succeeds.
> > Perhaps an approach such as having a buildsystem bugzilla account be the
> > submitter or a BUILDFAIL keyword.)  The bugzilla tickets can be used to
> > track the current status of a package.
> 
> Here once more: if Fedora Extras wants to scale, every package should have
> at least one person taking care of it. Build requests are done by human
> beings. Build status reports are mailed back to human beings. It is
> assumed that the person, who requests a build, does something useful with
> a build failure log and e.g. decides whether it should be tracked in
> bugzilla, because it might take some time until it is fixed.
> 
I'm not arguing with the mainainer >= 1 concept.  I'm asking for public
and centralized information.  And I'm asking that this availability of
information be as automated as possible to avoid human fallibility
(think absent release announcements for FC updates).  The reason I'm
asking for this is to make it easier on the community members who will
end up assisting the package maintainers.  The "if you have an itch,
scratch it" people.  We should make it easy to find out a package's
current state and what direction a maintainer is taking to fix it.

Having the buildsystem open, update, and close bugs on build
failure/success means we can free up mschwendt to do more important
things than copy build-log URLs into a bunch of very similar bugzilla
tickets :-)

> > The packages that are currently out of sync,
> > imported to the devel tree but not rebuilt, etc aren't all being tracked so
> > it takes effort for a non-maintainer (or maintainer of many (too many?)
> > packages to figure out what stll needs fixing.
> 
> That's why we need a roadmap. It's the maintainer's responsibility to
> (re)build packages in preparation for FE4.
> 
> Basically, for each package in the devel repo, which is out-of-sync with
> devel CVS, somebody would need to make sure that it is really the right
> version that shall be built and released. It's the packager who would tag
> that version and request a build. I mean, we don't want to build something
> without the "okay" from the package owner and possibly create a situation
> where a released binary must be pulled from the repo because it was a
> premature version upgrade that wasn't meant to be built.
>
Okay.  Here's my proposal.  Tear it apart:

FE4 repository will open after FC4 repository is available to the build
system.  Package maintainers or their designates are responsible for
requesting package builds so the built packages can be imported into the
FE4 repository.

Package maintainers whose packages haven't been rebuilt since April 7th
or under the new buildsystem will probably want to submit their packages
to be built ASAP so there aren't too many surprises when the new
repository opens.

Packages failing builds or with runtime issues should be placed into
bugzilla and added to the FE4Tracker if you want other community members
to help you fix things.

-Toshio
-- 
toshio \ Questions for the            Would Morticia wear it?
 @tiki- \ 21st Century!               Would it look better on me?
 lounge.com=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
                                                                GA->ME 1999
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-extras-list/attachments/20050522/f10123d0/attachment.sig>