[dm-devel] [PATCH 0/3][RESEND]multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.
Fil
lists at internyc.net
Fri Jan 20 14:49:15 UTC 2012
Thanks for the patches Vijay,
I tested the new set of patches against the git repo, and it applied
cleanly compiled and is running as expected. I will do more indepth
testing over the weekend.
thanks
fil
On 01/19/2012 02:20 PM, Chauhan, Vijay wrote:
> Resending this patch with the fix mentioned by Mike Snitzer<snitzer at redhat.com> and Fil<lists at internyc.net>.
> Thanks to all for sending me your review comments. Fil, Thanks for trying out mpathpersist and providing your
> feedback. Also thanking Yanling Qi<Yanling.Qi at netapp.com> for proposing this design for pr management feature.
> These series of patch is made on commit 864162b08067d3e55d52996846d10a046956c6d6.
>
>
> PERSISTENT RESERVE OUT/IN commands are currently not supported on mpath device. Any
> command sent to mpath device is routed to only one of the physical path (selected by
> path selector) from the active path group. PR OUT registration service action is one
> of the such use-case which fails as it expects all the physical path for the given mpath
> device to be registered. Due to these limitations, most of the cluster applications needs
> to manage persistent reservation through underlying physical path.
>
>
> Following are the detailed description for Persistent Reservation management feature:
> ======================================================================================
> CONTENT
> =======================================================================================
> 1. Feature Introduction
> 2. Steps to configure for persistent reservation management
> 3. mpathpersist Command Usage
> 4. PR IN/OUT service action and use-case handling
> 4.1 Persistent Reserve Out service action
> 4.1.1 [Register]
> 4.1.2 [Reserve]
> 4.1.3 [Release]
> 4.1.4 [CLEAR]
> 4.1.5 [PREEMPT] & [PREEMPT AND ABORT]
> 4.2 Persistent Reserve In service action
> 4.2.1 [READ KEY]
> 4.2.2 [READ RESERVATION]
> 4.2.3 [READ STATUS]
> 4.2.4 [REPORT CAPABILITIES]
> 4.3 Usage case handled by multipath tools for PR management
> 5. mpathpersist library and header
> 5.1 mpathpersist Library APIs.
> 5.1.1 int mpath_lib_init (void )
> 5.1.2 int mpath_lib_exit (void )
> 5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact,
> struct prin_resp *resp, int noisy, int verbose)
> 5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact,
> int rq_scope, unsigned int rq_type, struct rout_param_descriptor *paramp,
> int noisy, int verbose)
> 6. Limitation and restriction.
>
>
>
> 1. Feature Introduction
> ==========================
>
> Persistent reservation management feature allows cluster management software to manage
> persistent reservation through mpath device. It processes management requests from callers
> and hides the management task details. It also handles persistent reservation management of
> data path life cycle and state changes.
>
> The series of patch introduces new utility named 'mpathpersist' for managing pr on multipath device.
> 'mpathpersist' is similar to sg_persist utility with respect to cli usage but
> applicable to mpath devices only.
>
>
> 2. Steps to configure for persistent reservation management
> =============================================================
> This feature adds new parameter 'reservation_key' in default section and multipath section of
> multipath.conf.
>
> 'reservation_key' here represents RESERVATION KEY field of PERSISTENT RESERVE OUT parameter
> list which contains an 8-byte value provided by the application client to the device server
> to identify the I_T nexus.
>
> STEP 1: Set reservation_key parameter in multipath.conf for mpath device. 'reservation_key'
> parameter needs to be set for all the mpath device applicable for persistent management.
> STEP 2: restart multipathd daemon
>
> Note: reservation key in multipath.com must be same as service action reservation key used/to
> be used for mpath device registration reservation.
>
>
> 3. mpathpersist Command Usage
> ===============================
>
> mpathpersist CLI usage is kept close to sg_persist usage except it takes mpath device as device name.
>
> Usage: mpathpersist [OPTIONS] [DEVICE]
> Options:
> --verbose|-v level verbosity level
> 0 Critical messages
> 1 Error messages
> 2 Warning messages
> 3 Informational messages
> 4 Informational messages with trace enabled
> --clear|-C PR Out: Clear
> --device=DEVICE|-d DEVICE query or change DEVICE
> --help|-h output this usage message
> --hex|-H output response in hex
> --in|-i request PR In command
> --out|-o request PR Out command
> --param-aptpl|-Z PR Out parameter 'APTPL'
> --read-keys|-k PR In: Read Keys
> --param-sark=SARK|-S SARK PR Out parameter service action
> reservation key (SARK is in hex)
> --param-rk=RK|-K RK PR Out parameter reservation key
> (RK is in hex)
> --preempt|-P PR Out: Preempt
> --preempt-abort|-A PR Out: Preempt and Abort
> --prout-type=TYPE|-T TYPE PR Out command type
> --read-status|-s PR In: Read Full Status
> --read-keys|-k PR In: Read Keys
> --read-reservation|-r PR In: Read Reservation
> --register|-G PR Out: Register
> --register-ignore|-I PR Out: Register and Ignore
> --release|-L PR Out: Release
> --report-capabilities|-c PR In: Report Capabilities
> --reserve|-R PR Out: Reserve
> --transport-id=TIDS|-X TIDS TransportIDs can be mentioned
> in several forms
>
>
> 4. PR IN/OUT service action and use-case handling
> ===================================================
>
> 4.1 Persistent Reserve Out service action:
> ----------------------------------
> 4.1.1 [Register]
> The caller provides multipath device name, reservation key, and service action reservation
> key and optionally APTPL bit value. The persistent reservation management command issues PROUT
> Register service action to all active data paths of the multipath device. If reservation key or
> service action reservation key is not provided, fill with all 0s. For each path, a thread is
> created to process PROUT register service action. Consolidated results of all the paths are
> returned to caller. In case if this PROUT command fails with reservation conflict, all the
> successful registration by other threads will be rolled back.
> Example: mpathpersist --out --register --param-sark=123abc --prout-type=8 /dev/mapper/mpath9
> /*Unregister*/
> mpathpersist --out --register --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.2 [Reserve]
> The caller provides multipath device name, reservation type and reservation key. The reservation SCOPE
> value is not required since persistent reservation supports only LU_SCOPE. The persistent reservation
> management command issues PROUT Reserve service action to any active data paths of the multipath
> device. PROUT command is send to one of the active path of multipath device. Result is returned to the
> caller of this module.
> Example: mpathpersist --out --reserve --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.3 [Release]
> The caller provides multipath device name, reservation type and reservation key. The persistent reservation
> management command issues PROUT Release service action to all active data paths of the multipath device. The
> reason why issuing PROUT Release service action is because it is unclear which data path is the reservation
> holder. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or
> reservation type is invalid. Return error status back to the caller. After PROUT Release service action command
> is issued to all active data paths, a PRIN Read Reservation service action should be issued to an active data
> path. If the Read Reservation parameter data indicates that the logical unit is still reserved and the registration
> key matches the reservation key caller provided, this indicates that the reservation holder belongs to a data
> path in failed state or removed data path of the multipath device.
> The following operations should be taken for this error case.
> 1. Issue a PRIN Report Full Status service action to any active data path
> 2. Remember the full status descriptors returned from the step 1.
> 3. Issue a PROUT clear reservation service action, via any active data path, to clear the reservation and registrants
> and
> 4. Restore all registrants by issuing PROUT register service action with transport IDs from the full status
> descriptors saved in the step 2.
> Example: mpathpersist --out --release --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.4 [CLEAR]
> The caller provides multipath device name and reservation key. The persistent reservation management
> command issues PROUT Preempt service action to any active data paths of the multipath device.
> If RESERVATION CONFLICT SCSI status is received, it indicates that the reservation key is invalid or the
> data path is not a registrant. Return error status back to the caller.
> Example: mpathpersist --out --clear --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.5 [PREEMPT] & [PREEMPT AND ABORT]
> The caller provides multipath device name, reservation type, reservation key and service action reservation key.
> The persistent reservation management command issues PROUT Preempt service action to any active data paths of the
> multipath device. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or
> service action reservation key is invalid. Return error status back to the caller. PROUT command is send to one of
> the active path of multipath device. Result is returned to the caller of this module.
> Example: mpathpersist --out --preempt-abort --param-rk=0 --paramsark=789def --prout-type=8 /dev/mapper/mpath9
>
>
> PS: This feature does not support ALL_TG_PT bit on. With a storage array having 8 or 16 target ports, the
> total number of registrants will be equal to total-number-of-initiator-ports X total-number-of target-ports.
> If ALL_TG_PT is on, too much storage array resources will be wasted and it also creates implementation
> challenge of managing registration rollback. In a use case where one initiator port (such as a FC port) connects
> to multiple target ports on the same storage array via a switch, ALL_TG_PT=1 will cause registration failed with
> RESERVATION CONFLICT status when issuing PROUT registration to the second data path which shares the same initiator port.
>
> PS: This RFC version does not support REGISTER AND MOVE service action.
>
> 4.2 Persistent Reserve In service action:
> --------------------------------------------
> 4.2.1 [READ KEY]
> The caller provides multipath device. The persistent reservation management command selects any active data path
> and issues a PRIN Read Keys service action to the selected data path of the multipath device. Parameter data is sent
> to the caller with appropriate formatted. The persistent reservation management command will not filter registration
> keys returned from the storage array. In other words, multiple registration keys is reported in general.
> Example: mpathpersist -i -k /dev/mapper/mpath9
>
>
> 4.2.2 [READ RESERVATION]
> The caller provides multipath device. The persistent reservation management command selects any active data path and
> issues a PRIN Read Reservation service action to the selected data path of the multipath device. Parameter data is
> sent to the caller with appropriate formatted.
> Example: mpathpersist -i -r /dev/mapper/mpath9
>
>
> 4.2.3 [READ STATUS]
> The caller provides multipath device. The persistent reservation management command selects any active data path and
> issues a PRIN Read Full Status service action to the selected data path of the multipath device. Parameter data is sent
> to the caller with appropriate formatted. The persistent reservation management command will not filter full status
> descriptor entries returned from the storage array. In other words, full statuses for multiple registrants is reported
> in general.
> Example: mpathpersist -i -s /dev/mapper/mpath9
>
>
> 4.2.4 [REPORT CAPABILITIES]
> The caller provides multipath device. The persistent reservation management command selects any active data path
> and issues a PRIN Report Capabilities service action to the selected data path of the multipath device. Parameter
> data is sent to the caller with appropriate formatted.
> Example: mpathpersist -i -c /dev/mapper/mpath9
>
>
> 4.3 Use-case handled by multipath tools for PR management
> - For any reinstated path, multipath daemon checks for persistent reservation on the path and registers the
> path with reservation_key (from multipath.conf) if required.
> - For any new path discovery, multipath daemon checks for persistent reservation on the path and registers the path if required.
> - In case of device all path failure condition, command is returns with failed status.
> - PR command is retried by pr management for any Unit Attention.
> - While performing persistent reservation registration on mpath device, if any of the path receives reservation conflict
> then successful registration on the other paths will be rolled back.
>
> 5. mpathpersist Library and Header
> ======================================
>
>
> mpathpersist library provides four APIs for managing persistent reservation on device mapper multipath device.
>
> 5.1 mpathpersist Library APIs.
> ---------------------------------
> 5.1.1 int mpath_lib_init (void)
> DESCRIPTION :
> Initialize device mapper multipath configuration. This function must be invoked first before performing reservation
> management functions.
>
> RETURNS:
> 0->Success, 1->Failed
> Note: mpath_lib_init function must be called at the start of the usage of PR management APIs (such as
> mpath_persistent_reserve_in or mpath_persistent_reserve_out). This function is responsible for initializing
> config table. (Any dynamic allocation done by mpath_lib_init function will be de-allocated only by mpath_lib_exit API).
>
>
> 5.1.2 int mpath_lib_exit (void)
> DESCRIPTION:
> Release device mapper multipath configuration. This function must be invoked after performing reservation management functions.
>
> RETURNS:
> 0->Success, 1->Failed.
>
> Note: mpath_lib_exit function must be called at the end of the usage of PR management APIs (such as mpath_persistent_reserve_in
> or mpath_persistent_reserve_out). This function is responsible for freeing up the dynamic allocation made by mpath_lib_init function.
>
>
> 5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp, int noisy, int verbose)
> DESCRIPTION:
> The function in the mpath_persistent_reserve_in() sends PRIN command to the DM device and gets the response.
>
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact - PRIN command service action. Input argument
> resp - The response from PRIN service action. The caller should manage the memory allocation of this structure
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable
> verbose - Set verbosity level. Input argument. value:[0-3]. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
>
> RETURNS
> MPATH_PR_SUCCESS if PR command successful
> MPATH_PR_SYNTAX_ERROR if syntax error or invalid parameter
> MPATH_PR_SENSE_NOT_READY if command fails with [sk,asc,ascq: 0x2,*,*]
> MPATH_PR_SENSE_MEDIUM_ERROR if command fails with [sk,asc,ascq: 0x3,*,*]
> MPATH_PR_SENSE_HARDWARE_ERROR if command fails with [sk,asc,ascq: 0x4,*,*]
> MPATH_PR_SENSE_INVALID_OP if command fails with [sk,asc,ascq: 0x5,0x20,0x0]
> MPATH_PR_ILLEGAL_REQ if command fails with [sk,asc,ascq: 0x5,*,*]
> MPATH_PR_SENSE_UNIT_ATTENTION if command fails with [sk,asc,ascq: 0x6,*,*]
> MPATH_PR_SENSE_ABORTED_COMMAND if command fails with [sk,asc,ascq: 0xb,*,*]
> MPATH_PR_NO_SENSE if command fails with [sk,asc,ascq: 0x0,*,*]
> MPATH_PR_SENSE_MALFORMED if command fails with SCSI command malformed
> MPATH_PR_FILE_ERROR if command fails while accessing file (device node) problems(e.g. not found)
> MPATH_PR_DMMP_ERROR if Device Mapper related error.(e.g Error in getting dm info)
> MPATH_PR_OTHER if other error/warning has occurred(e.g transport or driver error)
>
>
> 5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope, unsigned int rq_type,
> struct rout_param_descriptor *paramp, int noisy, int verbose)
> DESCRIPTION
> The function in the mpath_persistent_reserve_out() sends PR OUT command to the DM device and gets the response.
>
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact - PROUT command service action. Input argument
> rq_scope - Persistent reservation scope. The value should be always LU_SCOPE (0h).
> rq_type - Persistent reservation type. The valid values of persistent reservation types are
> 5h (Write exclusive - registrants only)
> 8h (Exclusive access - registrants only)
> 7h (Write exclusive - All registrants)
> 8h (Exclusive access - All registrants).
> paramp - PROUT command parameter data. The paramp is a struct which describes PROUT parameter list. Caller should
> manage the memory allocation of this structure.
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable.
> verbose - Set verbosity level. Input argument. value: 0 to 3. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
>
> 6. Limitation and restriction
> -This feature doesn't support persistent reservation types of Write-Exclusive (1h)
> and Exclusive-Access(3h).
> - This feature will not support ALL_TG_PT bit on.
> - Current implementation is limited to FC SAS and ISCSI transport protocol. Other protocol support could be added in future.
>
>
> Regards,
> Vijay Chauhan
>
>
>
> git status for newly added or changed files in multipath-tools:
>
> # git diff -p --stat
> Makefile | 2 +
> Makefile.inc | 2 +
> libmultipath/config.c | 2 +
> libmultipath/config.h | 2 +
> libmultipath/configure.c | 1 +
> libmultipath/dict.c | 114 ++++++++++++++++++++++++++++++++++++
> libmultipath/discovery.c | 2 +-
> libmultipath/propsel.c | 46 +++++++++++++++
> libmultipath/propsel.h | 1 +
> libmultipath/structs.h | 4 +
> multipathd/Makefile | 4 +-
> multipathd/cli.c | 29 ++++++----
> multipathd/cli.h | 16 ++++--
> multipathd/cli_handlers.c | 72 +++++++++++++++++++++++
> multipathd/cli_handlers.h | 4 +
> multipathd/main.c | 140 +++++++++++++++++++++++++++++++++++++++++++++
> multipathd/main.h | 14 +++++
>
> # git status -s
> M Makefile
> M Makefile.inc
> A libmpathpersist/Makefile
> A libmpathpersist/mpath_persist.c
> A libmpathpersist/mpath_persist.h
> A libmpathpersist/mpath_persistent_reserve_in.3
> A libmpathpersist/mpath_persistent_reserve_out.3
> A libmpathpersist/mpath_pr_ioctl.c
> A libmpathpersist/mpath_pr_ioctl.h
> A libmpathpersist/mpath_updatepr.c
> A libmpathpersist/mpathpr.h
> M libmultipath/config.c
> M libmultipath/config.h
> M libmultipath/configure.c
> M libmultipath/dict.c
> M libmultipath/discovery.c
> M libmultipath/propsel.c
> M libmultipath/propsel.h
> M libmultipath/structs.h
> A mpathpersist/Makefile
> A mpathpersist/main.c
> A mpathpersist/main.h
> A mpathpersist/mpathpersist.8
> M multipathd/Makefile
> M multipathd/cli.c
> M multipathd/cli.h
> M multipathd/cli_handlers.c
> M multipathd/cli_handlers.h
> M multipathd/main.c
> M multipathd/main.h
>
More information about the dm-devel
mailing list