[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [dm-devel] [PATCH 0/3][RESEND]multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.



Thanks for the patches Vijay,

I tested the new set of patches against the git repo, and it applied
cleanly compiled and is running as expected. I will do more indepth
testing over the weekend.

thanks
fil

On 01/19/2012 02:20 PM, Chauhan, Vijay wrote:
> Resending this patch with the fix mentioned by Mike Snitzer<snitzer redhat com> and Fil<lists internyc net>. 
> Thanks to all for sending me your review comments. Fil, Thanks for trying out mpathpersist and providing your 
> feedback. Also thanking Yanling Qi<Yanling Qi netapp com> for proposing this design for pr management feature. 
> These series of patch is made on commit 864162b08067d3e55d52996846d10a046956c6d6.
> 
> 
> PERSISTENT RESERVE OUT/IN commands are currently not supported on mpath device. Any 
> command sent to mpath device is routed to only one of the physical path (selected by 
> path selector) from the active path group. PR OUT registration service action is one 
> of the such use-case which fails as it expects all the physical path for the given mpath 
> device to be registered. Due to these limitations, most of the cluster applications needs 
> to manage persistent reservation through underlying physical path.
> 
> 
> Following are the detailed description for Persistent Reservation management feature:
> ======================================================================================
> CONTENT
> =======================================================================================
> 1. Feature Introduction
> 2. Steps to configure for persistent reservation management
> 3. mpathpersist Command Usage
> 4. PR IN/OUT  service action and use-case handling
> 	4.1 Persistent Reserve Out service action
> 		4.1.1 [Register] 
> 		4.1.2 [Reserve]
> 		4.1.3 [Release]
> 		4.1.4 [CLEAR]
> 		4.1.5 [PREEMPT] & [PREEMPT AND ABORT]		
> 	4.2 Persistent Reserve In service action
> 		4.2.1 [READ KEY] 
> 		4.2.2 [READ RESERVATION]
> 		4.2.3 [READ STATUS]
> 		4.2.4 [REPORT CAPABILITIES]
> 	4.3 Usage case handled by multipath tools for PR management
> 5. mpathpersist library and header 
> 	5.1 mpathpersist  Library  APIs.
> 		5.1.1 int mpath_lib_init (void )
> 		5.1.2 int mpath_lib_exit (void )
> 		5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact,
> 				struct prin_resp *resp, int noisy, int verbose)			
> 		5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact, 
> 				int rq_scope, unsigned int rq_type, struct rout_param_descriptor *paramp, 
> 				int noisy, int verbose)
> 6. Limitation and restriction.
> 
> 
> 
> 1. Feature Introduction
> ==========================
> 
> Persistent reservation management feature allows cluster management software to manage 
> persistent reservation through mpath device. It processes management requests from callers 
> and hides the management task details. It also handles persistent reservation management of 
> data path life cycle and state changes.
> 
> The series of patch introduces new utility named 'mpathpersist' for managing pr on multipath device.
> 'mpathpersist' is similar to sg_persist utility with respect to cli usage but 
> applicable to mpath devices only.
> 
> 
> 2. Steps to configure for persistent reservation management
> =============================================================
> This feature adds new parameter 'reservation_key' in default section and multipath section of 
> multipath.conf. 
> 
> 'reservation_key'  here represents RESERVATION KEY field of PERSISTENT RESERVE OUT parameter 
> list which contains an 8-byte value provided by the application client to the device server 
> to identify the I_T nexus.
>  
> STEP 1: Set reservation_key parameter in multipath.conf for mpath device. 'reservation_key' 
> parameter needs to be set for all the mpath device applicable for persistent management.
> STEP 2: restart multipathd daemon
> 
> Note: reservation key in multipath.com must be same as service action reservation key used/to 
> be used for mpath device registration reservation.
> 
> 
> 3. mpathpersist Command Usage
> ===============================
> 
> mpathpersist CLI usage is kept close to sg_persist usage except it takes mpath device as device name.
> 
> Usage: mpathpersist [OPTIONS] [DEVICE]
>  Options:
>     --verbose|-v level         verbosity level
>                    0           Critical messages
>                    1           Error messages
>                    2           Warning messages
>                    3           Informational messages
>                    4           Informational messages with trace enabled
>     --clear|-C                 PR Out: Clear
>     --device=DEVICE|-d DEVICE  query or change DEVICE
>     --help|-h                  output this usage message
>     --hex|-H                   output response in hex
>     --in|-i                    request PR In command
>     --out|-o                   request PR Out command
>     --param-aptpl|-Z           PR Out parameter 'APTPL'
>     --read-keys|-k             PR In: Read Keys
>     --param-sark=SARK|-S SARK  PR Out parameter service action
>                                reservation key (SARK is in hex)
>     --param-rk=RK|-K RK        PR Out parameter reservation key
>                                (RK is in hex)
>     --preempt|-P               PR Out: Preempt
>     --preempt-abort|-A         PR Out: Preempt and Abort
>     --prout-type=TYPE|-T TYPE  PR Out command type
>     --read-status|-s           PR In: Read Full Status
>     --read-keys|-k             PR In: Read Keys
>     --read-reservation|-r      PR In: Read Reservation
>     --register|-G              PR Out: Register
>     --register-ignore|-I       PR Out: Register and Ignore
>     --release|-L               PR Out: Release
>     --report-capabilities|-c   PR In: Report Capabilities
>     --reserve|-R               PR Out: Reserve
>     --transport-id=TIDS|-X TIDS  TransportIDs can be mentioned
>                                in several forms
> 
> 
> 4. PR IN/OUT  service action and use-case handling
> ===================================================
> 
> 4.1 Persistent Reserve Out service action:
> ----------------------------------
> 4.1.1 [Register] 
> The caller provides multipath device name, reservation key, and service action reservation 
> key and optionally APTPL bit value. The persistent reservation management command issues PROUT 
> Register service action to all active data paths of the multipath device. If reservation key or 
> service action reservation key is not provided, fill with all 0s. For each path, a thread is 
> created to process PROUT register service action. Consolidated results of all the paths are 
> returned to caller. In case if this PROUT command fails with reservation conflict, all the 
> successful registration by other threads will be rolled back.
> Example:	mpathpersist --out --register --param-sark=123abc --prout-type=8 /dev/mapper/mpath9
> 			/*Unregister*/
> 		mpathpersist --out --register --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
> 
> 
> 4.1.2 [Reserve]
> The caller provides multipath device name, reservation type and reservation key. The reservation SCOPE 
> value is not required since persistent reservation supports only LU_SCOPE. The persistent reservation 
> management command issues PROUT Reserve service action to any active data paths of the multipath
> device. PROUT command is send to one of the active path of multipath device. Result is returned to the 
> caller of this module.
> Example:	mpathpersist --out --reserve --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
> 
> 
> 4.1.3 [Release]
> The caller provides multipath device name, reservation type and reservation key. The persistent reservation 
> management command issues PROUT Release service action to all active data paths of the multipath device. The 
> reason why issuing PROUT Release service action is because it is unclear which data path is the reservation 
> holder. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or
> reservation type is invalid. Return error status back to the caller. After PROUT Release service action command 
> is issued to all active data paths, a PRIN Read Reservation service action should be issued to an active data 
> path. If the Read Reservation parameter data indicates that the logical unit is still reserved and the registration 
> key matches the reservation key caller provided, this indicates that the reservation holder belongs to a data 
> path in failed state or removed data path of the multipath device. 
> The following operations should be taken for this error case. 
> 1. Issue a PRIN Report Full Status service action to any active data path
> 2. Remember the full status descriptors returned from the step 1.
> 3. Issue a PROUT clear reservation service action, via any active data path, to clear the reservation and registrants
>  and
> 4. Restore all registrants by issuing PROUT register service action with transport IDs from the full status 
>  descriptors saved in the step 2.
> Example:	mpathpersist --out --release --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
> 
> 
> 4.1.4 [CLEAR]
> The caller provides multipath device name and reservation key. The persistent reservation management
> command issues PROUT Preempt service action to any active data paths of the multipath device.
> If RESERVATION CONFLICT SCSI status is received, it indicates that the reservation key is invalid or the
> data path is not a registrant. Return error status back to the caller.
> Example:	mpathpersist --out --clear --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>   
> 
> 4.1.5 [PREEMPT] & [PREEMPT AND ABORT]
> The caller provides multipath device name, reservation type, reservation key and service action reservation key. 
> The persistent reservation management command issues PROUT Preempt service action to any active data paths of the 
> multipath device. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or 
> service action reservation key is invalid. Return error status back to the caller. PROUT command is send to one of 
> the active path of multipath device. Result is returned to the caller of this module.
> Example: mpathpersist --out --preempt-abort --param-rk=0 --paramsark=789def --prout-type=8 /dev/mapper/mpath9
> 
> 
> PS: This feature does not support ALL_TG_PT bit on. With a storage array having 8 or 16 target ports, the 
> total number of registrants will be equal to total-number-of-initiator-ports X total-number-of target-ports. 
> If ALL_TG_PT is on, too much storage array resources will be wasted and it also creates implementation 
> challenge of managing registration rollback. In a use case where one initiator port (such as a FC port) connects 
> to multiple target ports on the same storage array via a switch, ALL_TG_PT=1 will cause registration failed with 
> RESERVATION CONFLICT status when issuing PROUT registration to the second data path which shares the same initiator port.
> 
> PS: This RFC version does not support REGISTER AND MOVE service action.
> 
> 4.2 Persistent Reserve In service action:
> --------------------------------------------
> 4.2.1 [READ KEY] 
> The caller provides multipath device. The persistent reservation management command selects any active data path 
> and issues a PRIN Read Keys service action to the selected data path of the multipath device. Parameter data is sent 
> to the caller with appropriate formatted. The persistent reservation management command will not filter registration 
> keys returned from the storage array. In other words, multiple registration keys is reported in general.
> Example:	mpathpersist -i -k /dev/mapper/mpath9 
> 
> 
> 4.2.2 [READ RESERVATION]
> The caller provides multipath device. The persistent reservation management command selects any active data path and 
> issues a PRIN Read Reservation service action to the selected data path of the multipath device. Parameter data is 
> sent to the caller with appropriate formatted.
> Example:	mpathpersist -i -r /dev/mapper/mpath9 
> 
> 
> 4.2.3 [READ STATUS]
> The caller provides multipath device. The persistent reservation management command selects any active data path and 
> issues a PRIN Read Full Status service action to the selected data path of the multipath device. Parameter data is sent 
> to the caller with appropriate formatted. The persistent reservation management command will not filter full status 
> descriptor entries returned from the storage array. In other words, full statuses for multiple registrants is reported 
> in general.
> Example:	mpathpersist -i -s /dev/mapper/mpath9
> 
> 
> 4.2.4 [REPORT CAPABILITIES]
> The caller provides multipath device. The persistent reservation management command selects any active data path
> and issues a PRIN Report Capabilities service action to the selected data path of the multipath device. Parameter
> data is sent to the caller with appropriate formatted.
> Example:	mpathpersist -i -c /dev/mapper/mpath9
> 
> 
> 4.3 Use-case handled by multipath tools for PR management
> - For any reinstated path, multipath daemon checks for persistent reservation on the path and registers the 
> path with reservation_key (from multipath.conf) if required.
> - For any new path discovery, multipath daemon checks for persistent reservation on the path and registers the path if required.
> - In case of device all path failure condition, command is returns with failed status.
> - PR command is retried by pr management for any Unit Attention.
> - While performing persistent reservation registration on mpath device, if any of the path receives reservation conflict 
>   then successful registration on the other paths will be rolled back.
> 
> 5. mpathpersist Library and Header
> ======================================
> 
> 
> mpathpersist library provides four APIs for managing persistent reservation on device mapper multipath device. 
> 
> 5.1 mpathpersist  Library  APIs.
> ---------------------------------
> 5.1.1 int mpath_lib_init (void)
> DESCRIPTION :
> Initialize device mapper multipath configuration. This function must be invoked first before performing reservation 
> management functions.
> 
> RETURNS:
> 0->Success, 1->Failed
> Note: mpath_lib_init function must be called at the start of the usage of PR management APIs (such as 
> mpath_persistent_reserve_in or mpath_persistent_reserve_out). This function is responsible for initializing 
> config table. (Any dynamic allocation done by mpath_lib_init function will be de-allocated only by mpath_lib_exit API).
> 
> 
> 5.1.2  int mpath_lib_exit (void)
> DESCRIPTION:
> Release device mapper multipath configuration. This function must be invoked after performing reservation management functions.
> 
> RETURNS:
> 0->Success, 1->Failed.
> 
> Note: mpath_lib_exit function must be called at the end of the usage of PR management APIs (such as mpath_persistent_reserve_in 
> or mpath_persistent_reserve_out). This function is responsible for freeing up the dynamic allocation made by mpath_lib_init function.
> 
> 
> 5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp, int noisy, int verbose)
> DESCRIPTION:
> The function in the mpath_persistent_reserve_in() sends PRIN command to the DM device and gets the response.
> 
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact  -  PRIN command service action. Input argument
> resp - The response from PRIN service action. The caller should manage the memory allocation of this structure
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable
> verbose -  Set verbosity level. Input argument. value:[0-3]. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
> 
> RETURNS
> 	MPATH_PR_SUCCESS if PR command successful
> 	MPATH_PR_SYNTAX_ERROR if syntax error or invalid parameter
> 	MPATH_PR_SENSE_NOT_READY if command fails with [sk,asc,ascq: 0x2,*,*]
> 	MPATH_PR_SENSE_MEDIUM_ERROR if command fails with [sk,asc,ascq: 0x3,*,*]
> 	MPATH_PR_SENSE_HARDWARE_ERROR if command fails with [sk,asc,ascq: 0x4,*,*]
> 	MPATH_PR_SENSE_INVALID_OP if command fails with [sk,asc,ascq: 0x5,0x20,0x0]
> 	MPATH_PR_ILLEGAL_REQ if command fails with [sk,asc,ascq: 0x5,*,*]
> 	MPATH_PR_SENSE_UNIT_ATTENTION if command fails with [sk,asc,ascq: 0x6,*,*]
> 	MPATH_PR_SENSE_ABORTED_COMMAND if command fails with [sk,asc,ascq: 0xb,*,*]
> 	MPATH_PR_NO_SENSE if command fails with [sk,asc,ascq: 0x0,*,*]
>       MPATH_PR_SENSE_MALFORMED if command fails with SCSI command malformed
>       MPATH_PR_FILE_ERROR if command fails while accessing file (device node) problems(e.g. not found)
>       MPATH_PR_DMMP_ERROR if Device Mapper related error.(e.g Error in getting dm info)
>       MPATH_PR_OTHER if other error/warning has occurred(e.g transport or driver error)
> 
> 
> 5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope, unsigned int rq_type,
>                 struct rout_param_descriptor *paramp, int noisy, int verbose)
> DESCRIPTION
> The function in the mpath_persistent_reserve_out() sends PR OUT command to the DM device and gets the response.
> 
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact  - PROUT command service action. Input argument
> rq_scope - Persistent reservation scope. The value should be always LU_SCOPE (0h).
> rq_type - Persistent reservation type. The valid values of persistent reservation types are
>              5h (Write exclusive - registrants only)
>              8h (Exclusive access - registrants only)
>              7h (Write exclusive - All registrants)
>              8h (Exclusive access - All registrants).
> paramp - PROUT  command  parameter  data.  The paramp is a struct which describes PROUT parameter list. Caller should
>                  manage the memory  allocation of this structure. 
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable.
> verbose - Set verbosity level. Input argument. value: 0 to 3. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
> 
> 6. Limitation and restriction
> 	-This feature doesn't support persistent reservation types of Write-Exclusive (1h) 
> 	 and Exclusive-Access(3h).
> 	- This feature will not support ALL_TG_PT bit on. 
> 	- Current implementation is limited to FC SAS and ISCSI transport protocol. Other protocol support could be added in future.
> 
> 
> Regards,
> Vijay Chauhan
> 
> 
> 
> git status for newly added or changed files in multipath-tools:
> 
> # git diff -p --stat
>  Makefile                  |    2 +
>  Makefile.inc              |    2 +
>  libmultipath/config.c     |    2 +
>  libmultipath/config.h     |    2 +
>  libmultipath/configure.c  |    1 +
>  libmultipath/dict.c       |  114 ++++++++++++++++++++++++++++++++++++
>  libmultipath/discovery.c  |    2 +-
>  libmultipath/propsel.c    |   46 +++++++++++++++
>  libmultipath/propsel.h    |    1 +
>  libmultipath/structs.h    |    4 +
>  multipathd/Makefile       |    4 +-
>  multipathd/cli.c          |   29 ++++++----
>  multipathd/cli.h          |   16 ++++--
>  multipathd/cli_handlers.c |   72 +++++++++++++++++++++++
>  multipathd/cli_handlers.h |    4 +
>  multipathd/main.c         |  140 +++++++++++++++++++++++++++++++++++++++++++++
>  multipathd/main.h         |   14 +++++
> 
> # git status -s
>  M Makefile
>  M Makefile.inc
> A  libmpathpersist/Makefile
> A  libmpathpersist/mpath_persist.c
> A  libmpathpersist/mpath_persist.h
> A  libmpathpersist/mpath_persistent_reserve_in.3
> A  libmpathpersist/mpath_persistent_reserve_out.3
> A  libmpathpersist/mpath_pr_ioctl.c
> A  libmpathpersist/mpath_pr_ioctl.h
> A  libmpathpersist/mpath_updatepr.c
> A  libmpathpersist/mpathpr.h
>  M libmultipath/config.c
>  M libmultipath/config.h
>  M libmultipath/configure.c
>  M libmultipath/dict.c
>  M libmultipath/discovery.c
>  M libmultipath/propsel.c
>  M libmultipath/propsel.h
>  M libmultipath/structs.h
> A  mpathpersist/Makefile
> A  mpathpersist/main.c
> A  mpathpersist/main.h
> A  mpathpersist/mpathpersist.8
>  M multipathd/Makefile
>  M multipathd/cli.c
>  M multipathd/cli.h
>  M multipathd/cli_handlers.c
>  M multipathd/cli_handlers.h
>  M multipathd/main.c
>  M multipathd/main.h 
> 


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]