[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [linux-lvm] how much memory does LVM need? oom-killer comes

Stuart D. Gathman schrieb:
On Mon, 3 Sep 2007, Tomasz Chmielewski wrote:

Out of memory: kill process 2220 (getty) score 24 or a child
Killed process 2220 (getty)
Kernel panic - not syncing: Out of memory and no killable processes...

Rebooting in 20 seconds..
Again, replying to myself, as no one on this list seem to have a clue about problems with LVM snapshots... I added RAM, and was able to boot again.

I don't have any answers, but I am very impressed by this kind of testing
and grateful for the contribution.  My thoughts on evil experiments
focused on timing and component failure. Obviously, resource limits are extremely important as well.

I suppose ideal behaviour would be to refuse to create a snapshot when
memory is "low".  But defining "low" can be tricky.

Not really.

We can still have plenty memory when we want to create a snapshot - but memory can shrink later, as the snapshot fills.

As I understand, and as experimentation shown, memory usage increases as the snapshot gets filled, with 3 MB of RAM per 1 GB of snapshot filled (more or less).

As the memory used for a snapshot is not swappable, from a certain point, it is possible to realize "I'm in trouble, but I can't do anything".

Let's watch at the process once again:

1. Make a snapshot of some large volume (it's easier than making multiple small snapshots). We still have plenty of RAM, so there is no sense in refusing to create a snapshot.

2. Start filling the snapshot by either writing to the original, or to the snapshot. Memory usage increases.

3. As the snapshot fills, and you have less and less memory, from some point, you will no longer be able to remove the snapshot.

4. Now, you can only watch as the snapshot grows, your memory shrinks, the processes are getting killed, and eventually, the kernel panics. If you only have remote access, you're screwed.

Perhaps, in 1., instead of refusing to make a snapshot, make a printk, so that the admin knows what may happen? Is the memory used for a snapshot possible to calculate? Is it always ~3 MB RAM per 1 GB changes, or is it sometimes 1 MB, and sometimes 4 MB, depending on which data changes? If it remains more or less the same, add up all space reserved for snapshots, if it's more than we have, print a message.

BTW, wasn't it like that in some kernels before 2.6.22? With 2.6.18, I was getting a "cannot allocate memory" when I tried to create several snapshots. I guess it was a bug, though.

Ideally would be to make the memory needed for snapshots swappable, but I'm not sure the kernel would really like it.

Tomasz Chmielewski

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]