[Pulp-dev] Proposal: Basic Functional Test Requirement with each Feature or Bugfix

[Brian Bouterse]

I have two goals with this:

1. Improve the stability of pulpcore and it's plugins

2. Enable all downstreams and packagers with tests and information on how
to run those tests, so they can run the automated tests even as the
downstream is curated with a series of backports. This closely follows the
model of a downstream maintainer who backports a feature/fix from upstream
into a RPM and runs the upstream tests to make sure their patch didn't
break functionality.

# Proposal
Upstream could require a basic functional test for each feature or a test
for each bug fix. This test would be in the same commit as the feature or
fix causing it to "follow the code" so when it's cherry picked downstream
or into a package, the test goes with it. I believe we don't change the
test during cherry pick time because although the code implementation may
change what the feature or fix provides will not.

Additionally, upstream should document how to run these tests easily and
have the goal of making that easy.

This policy change would ultimately be decided by each plugin and pulpcore
separately. I don't want to make it a requirement to be part of the
github.com/pulp/ organization in any way, so if a plugin can't, or won't,
make this change that's ok, they can still host at github.com/pulp/.

As a pulpcore and pulp_file maintainer, I'm proposing this policy for both
of those repos specifically.

In terms of enforcement, I'm pitching manual enforcement by review as a
start. We can automate it later, but I hope that's a separate discussion to
focus on the policy change and keep it simple.

# Brian's take
This is a non-trivial contribution requirement so we need to really think
about it. My personal take is that at this point in the 3.y release line
it's time to trade feature/fix velocity for stability. Also backporting is
about to become a regular activity, so I think we have a practical
motivation to adopt this, even though it will slow down the future features
and bug fix velocity.

What's your perspective?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200817/55887d42/attachment.htm>