[dm-devel] data corruption with 'splt' workload to XFS on DM cache with its 3 underlying devices being on same NVMe device

[Mike Snitzer]

On Tue, Jul 24 2018 at  9:57am -0400,
Laurence Oberman <loberman at redhat.com> wrote:

> On Tue, 2018-07-24 at 15:51 +0200, Hannes Reinecke wrote:
> > 
> > _Actually_, I would've done it the other way around; after all,
> > where't the point in running dm-multipath on a partition?
> > Anything running on the other partitions would suffer from the
> > issues dm-multipath is designed to handle (temporary path loss etc), so I'm
> > not quite sure what you are trying to achieve with your testcase.
> > Can you enlighten me?
> > 
> > Cheers,
> > 
> > Hannes

I wasn't looking to deply this (multipath on partition) in production or
suggest it to others.  It was a means to experiment.  More below.

> This came about because a customer is using nvme for a dm-cache device
> and created multiple partitions so as to use the same nvme to cache
> multiple different "slower" devices. The corruption was noticed in XFS
> and I engaged Mike to assist in figuring out what was going on.

Yes, so topology for the customer's setup is:

1) MD raid1 on 2 NVMe partitions (from separate NVMe devices).
2) Then DM cache's "fast" and "metadata" devices layered on dm-linear
   mapping ontop of the MD raid1.
3) Then Ceph's rbd for DM-cache's slow device.

I was just looking to simplify the stack to try to assess why XFS
corruption was being seen without all the insanity.

One issue was corruption due to incorrect shutdown order (network was
getting shutdown out from underneath rbd, and in turn DM-cache couldn't
complete its IO migrations during cache_postsuspend()).

So I elected to try using DM multipath with queue_if_no_path to try to
replicate rbd losing network _without_ needing a full Ceph/rbd setup.

The rest is history... a rat-hole of corruption that likely is very
different than the customer's setup.

Mike