[dm-devel] hunt for 2.6.37 dm-crypt+ext4 corruption? (was: Re: dm-crypt barrier support is effective)

[Jon Nelson]

On Fri, Dec 10, 2010 at 10:54 AM, Jon Nelson <jnelson at jamponi.net> wrote:
> On Fri, Dec 10, 2010 at 8:58 AM, Jon Nelson <jnelson at jamponi.net> wrote:
>> On Fri, Dec 10, 2010 at 12:52 AM, Jon Nelson <jnelson at jamponi.net> wrote:
>>> On Thu, Dec 9, 2010 at 8:38 PM, Ted Ts'o <tytso at mit.edu> wrote:
>>>> On Fri, Dec 10, 2010 at 02:53:30AM +0100, Matt wrote:
>>>>>
>>>>> Try a kernel before 5a87b7a5da250c9be6d757758425dfeaf8ed3179
>>>>>
>>>>> from the tests I've done that one showed the least or no corruption if
>>>>> you count the empty /etc/env.d/03opengl as an artefact
>>>>
>>>> Yes, that's a good test.  Also try commit bd2d0210cf.  The patch
>>>> series that is most likely to be at fault if there is a regression in
>>>> between 5a87b7a5d and bd2d0210cf inclusive.
>>>>
>>>> I did a lot of testing before submitting it, but that wa a tricky
>>>> rewrite.  If you can reproduce the problem reliably, it might be good
>>>> to try commit 16828088f9 (the commit before 5a87b7a5d) and commit
>>>> bd2d0210cf.  If it reliably reproduces on bd2d0210cf, but is clean on
>>>> 16828088f9, then it's my ext4 block i/o submission patches, and we'll
>>>> need to either figure out what's going on or back out that set of
>>>> changes.
>>>>
>>>> If that's the case, a bisect of those changes (it's only 6 commits, so
>>>> it shouldn't take long) would be most appreciated.
>>>
>>> I observed the behavior on bd2d0210cf in a qemu-kvm install of
>>> openSUSE 11.3 (x86_64) on *totally* different host - an AMD quad-core.
>>>
>>> I did /not/ observe the behavior on 16828088f9 (yet). I'll run the
>>> test a few more times on 1682..
>>>
>>> Additionally, I am building a bisected kernel now (
>>> cb20d5188366f04d96d2e07b1240cc92170ade40 ), but won't be able to get
>>> back at it for a while.
>>
>> cb20d5188366f04d96d2e07b1240cc92170ade40 seems OK so far. I'm going to
>> try 1de3e3df917459422cb2aecac440febc8879d410 soon.
>
> Barring false negatives, bd2d0210cf22f2bd0cef72eb97cf94fc7d31d8cc
> appears to be the culprit (according to git bisect).
> I will test bd2d0210cf22f2bd0cef72eb97cf94fc7d31d8cc again, confirm
> the behavior, and work backwards to try to reduce the possibility of
> false negatives.

A few additional notes, in no particular order:

- For me, triggering the problem is fairly easy when encryption is involved.
- I'm now 81 iterations into testing
bd2d0210cf22f2bd0cef72eb97cf94fc7d31d8cc *without* encryption.  Out of
81 iterations, I have 4 failures: #16, 40, 62, and 64.

I will now try 1de3e3df917459422cb2aecac440febc8879d410 much more extensively.

Is this useful information?

-- 
Jon