[dm-devel] dm-crypt unpleasantness during heavy write loads and memory pressure?

[Andy Lutomirski]

I've seen a number of anomalies across a few machines.  All of them
have dm-crypt in common.

The most recent example was on Linux 3.8.  I have dm-crypt on top of
hpsa hardware RAID.  (I don't have a whole lot of confidence in hpsa's
performance, but I've seen similar issues on other machines, so this
may not be hpsa's fault.)  I did a big MySQL update.  The update took
a shockingly long time (it took longer than an identical query on a
replica on a much slower machine) and, while it was running, the
machine was basically unusable.

Here's a hypothesis:

dm-crypt appears to take incoming write bios, clone them, allocate
pages for a copy of the data, encrypt the data, and then submit to the
lower layer.  The lower layer (e.g. hpsa's queue) then sorts, merges,
and does real I/O.

Let's suppose that the actual crypto is nice and fast (on this box,
I'm using XTS with AES-NI on seven logical cores at 3.5 GHz), but that
the disks are not quite so impressive (3x 7200rpm SATA drives on
RAID5).  The machine has 16 GB of RAM.  MySQL is I/O-bound.

So... what happens?  If MySQL is willing to queue up a very large
number of writes (which is probably is -- this was one giant
transaction, so MySQL wouldn't need to flush until it filled up all
available buffers), there are only three places that those bios can
wait.  They can wait in dm-crypt's queue (which is unlikely -- AES-NI
is fast), they can block in dm-crypt waiting for free memory, or they
can block on the lower layer's queue.

None of these options are good.  Requests on the lower layer's queue
can be sorted, but they're keeping an entire extra copy of the data
pinned in kernel memory.  (I have no idea how well the writeback
algorithms can handle very large amounts of memory being in use by the
kernel.)  Requests on dm-crypt's queue cannot be intelligently sorted,
causing throughput to drop (which is only going to make the problem
worse).

So... is this a real problem?  Should I blame dm-crypt for stalls
under heavy write load?


I suppose that the situation could be improved if there was a way for
dm-crypt to remap the sector range immediately but defer encryption
until the lower device's queue is nearly ready to submit the IO.

--Andy