[katello-devel] Proposal - Better Continuous Integration

[Petr Chalupa]

On 20.11.12 19:33, Jeff Weiss wrote:
> Lukas Zapletal <lzap at redhat.com> writes:
>
>>>> "As a developer I would really like to have CI for each pull-request."
>>>
>>> If we can pull off running integration tests on every pull request,
>>> I think we should do it; I have my doubts this is feasible however.
>>> 30 mins per *each* pull request (and the time is only going to go
>>> up) is quite a bit. The value of these tests is somewhat limited
>>> too, as they are not truly integration tests, since each set of
>>> changes is tested independently from changes in other pull requests.
>>>
>>> I don't think it's worth building an rpm for each pull request
>>> either. For which platform? would koji people even support such a
>>> use of infrastructure?
>>>
>>
>> I also don't think it is feasible. You need to build RPMs in koji too,
>> because building it anywhere else tells you nothing. You really need to
>> build the packages in the same build root which we only have in koji.
>> Different build root gives different output, believe me.
>>
>> Thats another 10-90 minutes (depends on how many tasks koji has enqued).
>> Full installation in a snap-guest takes about 10 minutes, then cli tests
>> are 20 minutes.
>>
>> But the biggest issue is hardware resources - if you want to integrate
>> it with travis or public service, you need public hardware. The only
>> reasonable option would be run this internally (ozzak maybe).
>>
>> Plus you need to do this on all platforms we support, Fedoras and RHEL.
>> Only RHEL or only Fedora does not make sense.
>>
>> This is not continuous integration anymore. But I am not saying it does
>> not worth trying.
>>
>> I can prepare deep dive about Koji and Beaker instead. Building a RPM is
>> ONE COMMAND for you, you can start it and it will be doing its work
>> while you can focus on a different task. Then you need to install
>> Katello nighty (Beaker or snap-guest/ciauciau) and install the RPM
>> there. Running some CLI tests is one command too, or you can check
>> manually what you need to do.
>>
>> I usually have 2-3 (virtual) machines with nightly installation running
>> and I am testing my features/bugfixed there. I usually don't need clean
>> installation for each, until I break it I tend to re-use it. And if I
>> don't want to wait for Koji, I sometimes build my RPMs manually with
>> tito. (one command again) It's not perfect, but it works most of times.
>>
>> I think you need to accept we are working on software project or
>> product that is not easy to setup and install. Work cannot be considered
>> done until you really test it. It can work hundred times on your
>> development setup, our environment is pretty different.
>
>
> Hardware is cheap, our time is not.
>
> We should be testing every single commit, and we should be running every
> test we can in the time we have. We're going to the effort to write
> these tests, running them costs us basically nothing but a few cents
> worth of electricity.
>
> It doesn't matter to me if we use travis or jenkins, koji or just plain
> mock/tito.  But I don't think we should just give up and only test
> nightly.
>
> The cycle time to build and test can be more than fast enough, it just
> needs a little attention. IMO this effort is about the most worthwhile
> investment of our time that we can make. There is a limited window to
> address a bug with almost no cost. If it's caught immediately by CI, you
> can easily address it within an hour or two. It's clear which commit
> caused the bug, usually it's clear why, and it's clear who needs to fix
> it.
>
> If the bug is not caught until the next day, none of those things are
> clear. There were 12 commits by 7 different people, often no one can
> tell which one caused the problem without a git bisect. So no one fixes
> it. QA trips over it and files a bug. The bug sits waiting for triage
> for a week. etc etc. The bug stays in the system for weeks or months. It
> blocks other tests, masks other bugs, etc.
>
> We can do a CI cycle fast enough:
>
>
> 1) unit test, static analysis, and rpm build in parallel:  10-15 minutes
> 2) Install: 15 minutes
> 3) System tests (ui/cli/api) in parallel:  30 minutes
>
> Total:  1 hour max
>
> I want to stress that we *have* hardware in the boston lab.  But we
> don't necessarily have to use that, we can use ec2 or whatever.  No
> matter who we use it's way cheaper than just not testing.
>

Jeff, This is exactly what I am thinking.

I think this is not a small problem and that it takes a lot of our time. 
Lets do a little survey to find out how much time it takes. I'll send an 
email in a while.

Petr