[dm-devel] [PATCH 4/6] dm-multipath: remove process_queued_ios()

[Hannes Reinecke]

On 02/03/2014 12:30 PM, Junichi Nomura wrote:
> On 02/03/14 17:18, Hannes Reinecke wrote:
>> Doesn't serve any real purpose anymore; dm_table_run_queue()
>> will already move things to a workqueue, so we don't need
>> to do it ourselves.
>> We only need to take care to add a small delay when calling
>> __pg_init_all_paths() to move processing off to a workqueue;
>> pg_init_done() is run from an interrupt context and needs to
>> complete as fast as possible.
> 
> I think more explanation is needed for the patch description.
> As far as I understand, the change is based on the following reasoning:
> 
>   process_queued_ios() has served 3 functions:
>     1) select pg and pgpath if none is selected
>     2) start pg_init if requested
>     3) dispatch queued IOs when pg is ready
> 
>   Basically, a call to queue_work(process_queued_ios) can be replaced by
>   dm_table_run_queue(), which runs request queue and ends up calling
>   map_io(), which does 1), 2) and 3).
> 
Yes.

>   Exception is when !pg_ready() (= either pg_init is running or requested),
>   multipath_busy() prevents map_io() being called from request_fn.
> 
>   If pg_init is running, it should be ok as far as pg_init_done() does
>   the right thing when pg_init is completed. I.e. restart pg_init if
>   !pg_ready() or call dm_table_run_queue() to kick map_io().
> 
>   If pg_init is requested, we have to make sure the request is detected
>   and pg_init will be started.
>   pg_init is requested in 3 places:
>     a) __choose_pgpath() in map_io()
>     b) __choose_pgpath() in multipath_ioctl()
>     c) pg_init retry in pg_init_done()
>   a) is ok because map_io() calls __pg_init_all_paths(), which does 2).
>   b) needs a call to __pg_init_all_paths(), which does 2).
>   c) needs a call to __pg_init_all_paths(), which does 2).
> 
> 
> By writing the above, I found possible bugs related to 1):
> 
>> @@ -1217,9 +1185,11 @@ static void pg_init_done(void *data, int errors)
>>  
>>  	if (!m->pg_init_required)
>>  		m->queue_io = 0;
>> -
>> -	m->pg_init_delay_retry = delay_retry;
>> -	queue_work(kmultipathd, &m->process_queued_ios);
>> +	else {
>> +		m->pg_init_delay_retry = delay_retry;
>> +		__pg_init_all_paths(m, 50/HZ);
>> +		goto out;
>> +	}
>>  
>>  	/*
>>  	 * Wake up any thread waiting to suspend.
> 
> It is possible that m->current_pg is NULL.
> (E.g. pg_init failed for current_pgpath, bypass_pg() was called, etc.)
> __pg_init_all_paths() will cause oops in such a case.
> 
Ah. Right.

> So how about doing this in pg_init_done():
> 
> 	if (m->pg_init_required) {
> 		m->pg_init_delay_retry = delay_retry;
> 		if (__pg_init_all_paths(m))
> 			goto out;
> 	}
> 
> 	/* pg_init successfully completed */
> 	m->queue_io = 0;
> 
> and in __pg_init_all_paths(), do something like:
> 
> 	m->pg_init_required = 0;
> 	...
> 	if (!m->current_pg)
> 		return 0;
> 	...
> 	return m->pg_init_in_progress;
> 
> 
Hmm. That still wouldn't be doing the right thing.
'fail_path' in pg_init_done() might be setting current_pg to NULL,
but this doesn't mean that the entire path group is invalid.
I just means that this particular path is invalid, and we still
might need to retry pg_init for the other paths.

>> @@ -1593,8 +1563,13 @@ static int multipath_ioctl(struct dm_target *ti, unsigned int cmd,
>>  	if (!r && ti->len != i_size_read(bdev->bd_inode) >> SECTOR_SHIFT)
>>  		r = scsi_verify_blk_ioctl(NULL, cmd);
>>  
>> -	if (r == -ENOTCONN && !fatal_signal_pending(current))
>> -		queue_work(kmultipathd, &m->process_queued_ios);
>> +	if (r == -ENOTCONN && !fatal_signal_pending(current)) {
>> +		spin_lock_irqsave(&m->lock, flags);
>> +		if (m->current_pgpath && m->pg_init_required)
>> +			__pg_init_all_paths(m, 0);
>> +		spin_unlock_irqrestore(&m->lock, flags);
>> +		dm_table_run_md_queue_async(m->ti->table);
>> +	}
>>  
>>  	return r ? : __blkdev_driver_ioctl(bdev, mode, cmd, arg);
>>  }
> 
> Similarly, m->current_pgpath can be NULL here while pg_init_required.
> Then pg_init_required is left uncleared and all IOs in the queue will
> stall until somebody calls multipath_ioctl() to redo pg selection.
> 
Ok, correct. Will be fixing it up.

Thanks for the review.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare at suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)