[dm-devel] Re: dm-userspace memory consumption in remap cache

[Benjamin Gilbert]

Dan Smith wrote:
> BG> I noticed that after I wrote 1 GB of data to a dmu device with a 4
> BG> KB blocksize, the dm-userspace-remaps slab cache consumed about 39
> BG> MB of memory.  
> Ah, right.

In fact, when the dmu device is unmapped, destroy_dmu_device() moves all 
of the dmu_maps to the end of the MRU list but does not free them, so 
that memory stays around.  If a new device is created, its dmu_maps will 
still be obtained from kmem_cache_alloc even though there are unused 
dmu_maps.

> We could push statistic information back to cowd when there
> was nothing else to do.  That might be interesting, but probably not
> the best way to solve this particular issue.

Come to think of it, that would be very interesting data in its own 
right.  Hmm... you could push statistics on a block whenever its mapping 
expires, but that doesn't help the MFU blocks.  You could provide a 
query-and-reset-counters request for individual blocks still in the 
cache, and since userspace could watch the statistics pushes to see what 
blocks had been removed, it would know which blocks to query on.  That 
would allow userspace to maintain statistics on whatever level of 
time-granularity it wanted, without requiring the kernel to do periodic 
sweeps or to do large dumps to userspace.

...I'm probably missing an obvious reason that that won't work.

> BG> (As an aside, I haven't been able to figure out the semantics of
> BG> the completion notification mechanism.  Could you provide an
> BG> example of how you expect it to be used from the userspace side?)
> Recent versions of cowd use this to prevent the completion (endio)
> From firing until it has flushed its internal metadata mapping to
> disk, to prevent the data from being written and the completion event
> sent, when the data isn't really on the disk (well, it's on the disk,
> but if we crash before we write our metadata, we can't tell that it's
> really there during recovery).

Okay, I see.

> BG> 3. Map in dm-linear when there are large consecutive ranges, to
> BG> try to keep the table size down.
> I don't think this is the best approach, because if you want to
> invalidate a mapping, you'd have to split the dm-linear back up,
> suspend/resume the device, etc.

Oh, good point.

> Now that I know at least someone is paying attention, I'll try to get
> my latest dm-userspace and cowd versions out on this list.  A small
> fix has been made to dm-userspace, and several improvements and fixes
> have been made to cowd.  After I post my current code, I'll implement
> the memory limit/aggressive reuse functionality and post that as well.

Great.  Thanks!

Thanks
--Benjamin Gilbert