[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] DRBD Primary/Primary + GNBD + multipath + GFS with GNBD server reboot == kaboom?



I configured an eight node cluster with two machines paired up with Primary/Primary DRBD served via GNBD to the other machines with roundrobin multipath + GFS on the client machines (the DRBD devices being served via both DRBD machines as GNBD devices). I successfully balanced the clients into mounting the GFS based file systems on boot but attempting to reboot one of the DRBD pair machines ('reboot') resulted in multipath losing its mind on all of the client machines.

It failed to transparently cut the failed GNBD server node out, instead failing _both_ multipathed GNDB devices (which makes no particular sense to me) and even after restoring the downed DRBD box and associated GNBD server one client box was left so 'jammed' on the GFS file system that I had to power cycle it to restore it (and all of the other boxes mounting the same GFS file system) to normal operation.

This *didn't* happen when I simply powered off one of the DRBD paired machines. Then multipath et al kept on working fine with the single GNBD server node just like it was supposed to. Recovery went fine as well.

Among all the other messages (see attached file) I got this in /var/log/messages on the 'jammed' GNBD client machine:

Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: fatal: I/O error
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   block = 26
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   function = gfs_dreread
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   file = /builddir/build/BUILD/gfs-kmod-0.1.16/_kmod_build_/src/gfs/dio.c, line = 576
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   time = 1201024470
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: about to withdraw from the cluster
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: telling LM to withdraw
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: reconnecting
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: reconnecting
Jan 22 09:54:31 pbox7 kernel: GFS: fsid=Production0:web1.3: withdrawn
Jan 22 09:54:31 pbox7 kernel:
Jan 22 09:54:31 pbox7 kernel: Call Trace:
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ffc94>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8001b9e6>] generic_make_request+0x1e8/0x1ff
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff88414dcf>] :gfs:gfs_io_error_bh_i+0x32/0x37
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883eca98>] :gfs:gfs_dreread+0xad/0xc7
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ecada>] :gfs:gfs_dread+0x28/0x43
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ed522>] :gfs:gfs_get_meta_buffer+0xc6/0x247
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883fa25b>] :gfs:gfs_copyin_dinode+0x1d/0x131
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f8c78>] :gfs:inode_go_lock+0x27/0x3f
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6a50>] :gfs:glock_wait_internal+0x222/0x2bb
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6e7e>] :gfs:gfs_glock_nq+0x395/0x3d5
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6ed4>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8840a782>] :gfs:gfs_getattr+0x41/0x67
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8000de0b>] vfs_getattr+0x2d/0xa9
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80028392>] vfs_stat_fd+0x32/0x4a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80064a9d>] do_page_fault+0x4eb/0x81d
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80039d83>] tty_ldisc_deref+0x68/0x7b
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80094baa>] recalc_sigpending_and_wake+0x9/0x1a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80028924>] do_sigaction+0x72/0x195
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff800231e6>] sys_newstat+0x19/0x31
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8005bde9>] error_exit+0x0/0x84
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8005b116>] system_call+0x7e/0x83

On a repeat experiment, here is what one of the client system looked like just before I 'reboot'ed one of the DRBD machines:

[root pbox7 tmp]# multipath -ll
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD [size=349G][features=0][hwhandler=0] \_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd2 252:2 [active][ready]
 \_ #:#:#:# gnbd5 252:5 [active][ready]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd1 252:1 [active][ready]
 \_ #:#:#:# gnbd4 252:4 [active][ready]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd0 252:0 [active][ready]
 \_ #:#:#:# gnbd3 252:3 [active][ready]

[root pbox7 tmp]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/pbox7-root0
                      48755504   6479324  39759600  15% /
/dev/sda1               194442     31032    153371  17% /boot
tmpfs                  8220020         0   8220020   0% /dev/shm
/dev/mapper/ProdWeb1-data0
                     359748608 178031212 181717396  50% /u1
/dev/mapper/ProdMisc1-data0
                     359748608       152 359748456   1% /u2
/dev/mapper/ProdLog1-data0
                     359748608       152 359748456   1% /u3

[root pbox7 tmp]# gnbd_import -l
Device name : log1a
----------------------
    Minor # : 0
 sysfs name : /block/gnbd0
     Server : pbox9.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

Device name : misc1a
----------------------
    Minor # : 1
 sysfs name : /block/gnbd1
     Server : pbox9.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

Device name : web1a
----------------------
    Minor # : 2
 sysfs name : /block/gnbd2
     Server : pbox9.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

Device name : log1b
----------------------
    Minor # : 3
 sysfs name : /block/gnbd3
     Server : pbox10.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

Device name : misc1b
----------------------
    Minor # : 4
 sysfs name : /block/gnbd4
     Server : pbox10.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

Device name : web1b
----------------------
    Minor # : 5
 sysfs name : /block/gnbd5
     Server : pbox10.internal.freerun.com
       Port : 14567
      State : Open Connected Clear
   Readonly : No
    Sectors : 732399448

After 'reboot'ing one of the DRBD machines (while the machine was still off):

[root pbox7 ~]# multipath -ll
gnbd0: checker msg is "directio checker reports path is down"
gnbd1: checker msg is "directio checker reports path is down"
gnbd2: checker msg is "directio checker reports path is down"
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ #:#:#:# gnbd2 252:2 [failed][faulty]
 \_ #:#:#:# gnbd5 252:5 [failed][faulty]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ #:#:#:# gnbd1 252:1 [failed][faulty]
 \_ #:#:#:# gnbd4 252:4 [failed][faulty]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ #:#:#:# gnbd0 252:0 [failed][faulty]
 \_ #:#:#:# gnbd3 252:3 [failed][faulty]

And a minute later (after the DRBD/GNBD Server machine finished rebooting):

[root pbox7 ~]# multipath -ll
mpath2 (GNBD-freerun-production0-drbd-web1) dm-11 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd2 252:2 [active][ready]
 \_ #:#:#:# gnbd5 252:5 [active][ready]
mpath1 (GNBD-freerun-production0-drbd-misc1) dm-10 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd1 252:1 [active][ready]
 \_ #:#:#:# gnbd4 252:4 [active][ready]
mpath0 (GNBD-freerun-production0-drbd-log1) dm-9 GNBD,GNBD
[size=349G][features=0][hwhandler=0]
\_ round-robin 0 [prio=2][active]
 \_ #:#:#:# gnbd0 252:0 [active][ready]
 \_ #:#:#:# gnbd3 252:3 [active][ready]

Any ideas or suggestions as to what is going wrong with 'reboot'ing one of the DRBD machines that causes multipath to lose *all* the paths?

--
Benjamin Franz

 I don't think it's a good idea to follow the catalog's suggestion to force
 the "Double Helix" on your biologist friends, or the "Wavelength" on your
 physicist friends. They have access to virulent pathogens and liquid
 nitrogen, and I just know it will end in tears before bedtime.
                            - Verzoeking, LiveJournal Entry, 22 Nov 2005
#!/bin/bash
#
# gnbd-srv      Start/Stop GNBD Server 
#
# chkconfig: 2345 90 09
# description: This shell script takes care of exporting and unexporting
#              GNBDs devices.
# processname: gnbd_serv
# pidfile: /var/run/gnbd/gnbd_serv.pid

# source function library
. /etc/rc.d/init.d/functions

[ -r /etc/sysconfig/gnbd ] && . /etc/sysconfig/gnbd

GNBD_SERV_TIMEOUT=${GNBD_SERV_TIMEOUT:-5}
prog="GNBD Server"

start() {
	if [ ! -f /etc/gnbdtab ]; then
		exit 0
	fi

	echo -n $"Starting $prog: "

	# start gnbd_serv
	if [ -e /var/run/gnbd/gnbd_serv.pid ] && \
	   [ -e /proc/`cat /var/run/gnbd/gnbd_serv.pid` ]; then
		echo -n $"gnbd_serv is already running"
		failure
		echo
		return 1
	fi
	gnbd_serv -q $GNBD_SERV_OPTS
	if [ $? -ne 0 ]; then
		echo -n $"cannot start gnbd_serv"
		failure
		return 1
	fi

	# export all GNBDs devices
	GNBD_EXPORT=`cat /etc/gnbdtab | egrep '^export'`
	if [ -n "$GNBD_EXPORT" ]; then
		cat /etc/gnbdtab | \
		awk '/^export/ { print "-d", $3, "-e", $2, "-u", $4 }' | \
		xargs -l gnbd_export $GNBD_EXPORT_OPTS > /dev/null
		if [ $? -ne 0 ]; then
			echo -n $"failed to export GNBDs devices"
			gnbd_serv -q -K
			failure
			return 1
		fi
	fi

	success
	echo
	return 0
}

stop() {
	echo -n $"Stopping $prog: "

	# check
	if [ ! -e /var/run/gnbd/gnbd_serv.pid ] || \
           [ ! -e /proc/`cat /var/run/gnbd/gnbd_serv.pid` ]; then
		echo -n $"gnbd_serv is not running"
		failure
		echo
		return 1
	fi

	# unexport all GNBDs devices
	gnbd_export -q -R
	if [ $? -ne 0 ]; then
		echo -n $"failed to unexport all GNBDs devices"
		failure
		return 1
	fi

	# stop gnbd_serv
	GNBD_SERV_PROCS="$(pgrep gnbd_serv)"
	while [ -n "$GNBD_SERV_PROCS" ] && \
	      [ $GNBD_SERV_TIMEOUT -gt 0 ]; do
		gnbd_serv -q -k
		sleep 1
		GNBD_SERV_TIMEOUT=$(( $GNBD_SERV_TIME -1 ))
		GNBD_SERV_PROCS="$(pgrep gnbd_serv)"
	done
	if [ -n "$GNBD_SERV_PROCS" ]; then
		gnbd_serv -q -K
	fi
	if [ $? -ne 0 ]; then
		echo -n $"failed to stop gnbd_serv"
		failure
		return 1
	fi

	success
	echo
	return 0
}


case "$1" in
	start)
		start
		;;
	stop)
		stop
		;;
	status)
		status gnbd_serv
		;;
	*)
		echo $"Usage: $0 {start|stop|status}"
		exit 1
		;;
esac

exit $?

# This is an example configuration file for device mapper multipath.
# For a complete list of the default configuration values, see
# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.defaults
# For a list of configuration options with descriptions, see
# /usr/share/doc/device-mapper-multipath-0.4.5/multipath.conf.annotated


# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices. 
#blacklist {
#        devnode "*"
#}

## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
#	device {
#		vendor	"IBM"
#		product	"S/390.*"
#	}
#}

## Use user friendly names, instead of using WWIDs as names.
defaults {
	user_friendly_names yes
}
##
## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
##
#defaults {
#	udev_dir		/dev
#	polling_interval 	10
#	selector		"round-robin 0"
#	path_grouping_policy	multibus
#	getuid_callout		"/sbin/scsi_id -g -u -s /block/%n"
#	prio_callout		/bin/true
#	path_checker		readsector0
#	rr_min_io		100
#	rr_weight		priorities
#	failback		immediate
#	no_path_retry		fail
#	user_friendly_name	yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid.  The 3 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line.  Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
blacklist {
#       wwid 26353900f02796769
	devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda|sdb|sdc|sdd)[0-9]*"
	devnode "^hd[a-z]"
}
#multipaths {
#	multipath {
#		wwid			3600508b4000156d700012000000b0000
#		alias			yellow
#		path_grouping_policy	multibus
#		path_checker		readsector0
#		path_selector		"round-robin 0"
#		failback		manual
#		rr_weight		priorities
#		no_path_retry		5
#	}
#	multipath {
#		wwid			1DEC_____321816758474
#		alias			red
#	}
#}
#devices {
#	device {
#		vendor			"COMPAQ  "
#		product			"HSV110 (C)COMPAQ"
#		path_grouping_policy	multibus
#		getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
#		path_checker		readsector0
#		path_selector		"round-robin 0"
#		hardware_handler	"0"
#		failback		15
#		rr_weight		priorities
#		no_path_retry		queue
#	}
#	device {
#		vendor			"COMPAQ  "
#		product			"MSA1000         "
#		path_grouping_policy	multibus
#	}
#}
<?xml version="1.0"?>
<cluster alias="Production0" config_version="23" name="Production0">
	<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
	<clusternodes>
		<clusternode name="pbox7.freerun.com" nodeid="1" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu1" option="off" port="6"/>
					<device name="apcpdu3" option="off" port="6"/>
					<device name="apcpdu1" option="on" port="6"/>
					<device name="apcpdu3" option="on" port="6"/>
				</method>
				<method name="1">
					<device name="apcpdu1" option="off" port="6"/>
					<device name="apcpdu3" option="off" port="6"/>
					<device name="apcpdu1" option="on" port="6"/>
					<device name="apcpdu3" option="on" port="6"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox6.freerun.com" nodeid="3" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu2" option="off" port="5"/>
					<device name="apcpdu3" option="off" port="5"/>
					<device name="apcpdu2" option="on" port="5"/>
					<device name="apcpdu3" option="on" port="5"/>
				</method>
				<method name="1">
					<device name="apcpdu2" option="off" port="5"/>
					<device name="apcpdu3" option="off" port="5"/>
					<device name="apcpdu2" option="on" port="5"/>
					<device name="apcpdu3" option="on" port="5"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox2.freerun.com" nodeid="4" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu1" port="1"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox8.freerun.com" nodeid="5" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu1" option="off" port="7"/>
					<device name="apcpdu4" option="off" port="7"/>
					<device name="apcpdu1" option="on" port="7"/>
					<device name="apcpdu4" option="on" port="7"/>
				</method>
				<method name="1">
					<device name="apcpdu1" option="off" port="7"/>
					<device name="apcpdu4" option="off" port="7"/>
					<device name="apcpdu1" option="on" port="7"/>
					<device name="apcpdu4" option="on" port="7"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox5.freerun.com" nodeid="6" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu2" option="off" port="4"/>
					<device name="apcpdu4" option="off" port="4"/>
					<device name="apcpdu2" option="on" port="4"/>
					<device name="apcpdu4" option="on" port="4"/>
				</method>
				<method name="1">
					<device name="apcpdu2" option="off" port="4"/>
					<device name="apcpdu4" option="off" port="4"/>
					<device name="apcpdu2" option="on" port="4"/>
					<device name="apcpdu4" option="on" port="4"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox1.freerun.com" nodeid="8" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu4" port="1"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox9.freerun.com" nodeid="2" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu2" option="off" port="2"/>
					<device name="apcpdu3" option="off" port="2"/>
					<device name="apcpdu4" option="off" port="2"/>
					<device name="apcpdu2" option="on" port="2"/>
					<device name="apcpdu3" option="on" port="2"/>
					<device name="apcpdu4" option="on" port="2"/>
				</method>
				<method name="1">
					<device name="apcpdu2" option="off" port="2"/>
					<device name="apcpdu3" option="off" port="2"/>
					<device name="apcpdu4" option="off" port="2"/>
					<device name="apcpdu2" option="on" port="2"/>
					<device name="apcpdu3" option="on" port="2"/>
					<device name="apcpdu4" option="on" port="2"/>
				</method>
				<method name="1">
					<device name="apcpdu2" option="off" port="2"/>
					<device name="apcpdu3" option="off" port="2"/>
					<device name="apcpdu4" option="off" port="2"/>
					<device name="apcpdu2" option="on" port="2"/>
					<device name="apcpdu3" option="on" port="2"/>
					<device name="apcpdu4" option="on" port="2"/>
				</method>
			</fence>
		</clusternode>
		<clusternode name="pbox10.freerun.com" nodeid="7" votes="1">
			<fence>
				<method name="1">
					<device name="apcpdu1" option="off" port="3"/>
					<device name="apcpdu2" option="off" port="3"/>
					<device name="apcpdu3" option="off" port="3"/>
					<device name="apcpdu1" option="on" port="3"/>
					<device name="apcpdu2" option="on" port="3"/>
					<device name="apcpdu3" option="on" port="3"/>
				</method>
				<method name="1">
					<device name="apcpdu1" option="off" port="3"/>
					<device name="apcpdu2" option="off" port="3"/>
					<device name="apcpdu3" option="off" port="3"/>
					<device name="apcpdu1" option="on" port="3"/>
					<device name="apcpdu2" option="on" port="3"/>
					<device name="apcpdu3" option="on" port="3"/>
				</method>
				<method name="1">
					<device name="apcpdu1" option="off" port="3"/>
					<device name="apcpdu2" option="off" port="3"/>
					<device name="apcpdu3" option="off" port="3"/>
					<device name="apcpdu1" option="on" port="3"/>
					<device name="apcpdu2" option="on" port="3"/>
					<device name="apcpdu3" option="on" port="3"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>
	<cman/>
	<fencedevices>
		<fencedevice agent="fence_apc" ipaddr="apcpdu1.freerun.com" login="device" name="apcpdu1" passwd="apassword"/>
		<fencedevice agent="fence_apc" ipaddr="apcpdu2.freerun.com" login="device" name="apcpdu2" passwd="apassword"/>
		<fencedevice agent="fence_apc" ipaddr="apcpdu3.freerun.com" login="device" name="apcpdu3" passwd="apassword"/>
		<fencedevice agent="fence_apc" ipaddr="apcpdu4.freerun.com" login="device" name="apcpdu4" passwd="apassword"/>
	</fencedevices>
	<rm>
		<failoverdomains>
			<failoverdomain name="LinuxVirtualServer" ordered="1" restricted="1">
				<failoverdomainnode name="pbox7.freerun.com" priority="3"/>
				<failoverdomainnode name="pbox6.freerun.com" priority="2"/>
				<failoverdomainnode name="pbox8.freerun.com" priority="4"/>
				<failoverdomainnode name="pbox5.freerun.com" priority="1"/>
				<failoverdomainnode name="pbox9.freerun.com" priority="5"/>
				<failoverdomainnode name="pbox10.freerun.com" priority="6"/>
			</failoverdomain>
			<failoverdomain name="SonicGateway" ordered="1" restricted="1">
				<failoverdomainnode name="pbox2.freerun.com" priority="2"/>
				<failoverdomainnode name="pbox1.freerun.com" priority="1"/>
			</failoverdomain>
		</failoverdomains>
		<resources/>
	</rm>
</cluster>
Jan 22 09:52:07 pbox7 clurgmgrd[5892]: <notice> Member 2 shutting down 
Jan 22 09:52:23 pbox7 kernel: gnbd0: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd0: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4974]: client lost connection with pbox9.internal.freerun.com : Broken pipe 
Jan 22 09:52:23 pbox7 gnbd_recvd[4974]: reconnecting 
Jan 22 09:52:23 pbox7 kernel: gnbd2: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd2: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4994]: client lost connection with pbox9.internal.freerun.com : Broken pipe 
Jan 22 09:52:23 pbox7 gnbd_recvd[4994]: reconnecting 
Jan 22 09:52:23 pbox7 kernel: gnbd1: Receive control failed (result -32)
Jan 22 09:52:23 pbox7 kernel: gnbd1: shutting down socket
Jan 22 09:52:23 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:52:23 pbox7 gnbd_recvd[4984]: client lost connection with pbox9.internal.freerun.com : Broken pipe 
Jan 22 09:52:23 pbox7 gnbd_recvd[4984]: reconnecting 
Jan 22 09:52:28 pbox7 gnbd_recvd[4974]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:28 pbox7 gnbd_recvd[4974]: reconnecting 
Jan 22 09:52:28 pbox7 gnbd_recvd[4994]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:28 pbox7 gnbd_recvd[4994]: reconnecting 
Jan 22 09:52:28 pbox7 gnbd_recvd[4984]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:28 pbox7 gnbd_recvd[4984]: reconnecting 
Jan 22 09:52:30 pbox7 kernel: gnbd0: Attempted send on closed socket
Jan 22 09:52:32 pbox7 kernel: device-mapper: multipath: Failing path 252:0.
Jan 22 09:52:32 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:52:32 pbox7 multipathd: checker failed path 252:0 in map mpath0 
Jan 22 09:52:32 pbox7 multipathd: mpath0: remaining active paths: 1 
Jan 22 09:52:33 pbox7 kernel: gnbd1: Attempted send on closed socket
Jan 22 09:52:33 pbox7 gnbd_recvd[4974]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:33 pbox7 gnbd_recvd[4974]: reconnecting 
Jan 22 09:52:33 pbox7 gnbd_recvd[4994]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:33 pbox7 gnbd_recvd[4994]: reconnecting 
Jan 22 09:52:33 pbox7 gnbd_recvd[4984]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : Connection refused 
Jan 22 09:52:33 pbox7 gnbd_recvd[4984]: reconnecting 
Jan 22 09:52:35 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:52:35 pbox7 kernel: device-mapper: multipath: Failing path 252:1.
Jan 22 09:52:35 pbox7 kernel: gnbd2: Attempted send on closed socket
Jan 22 09:52:35 pbox7 multipathd: checker failed path 252:1 in map mpath1 
Jan 22 09:52:35 pbox7 multipathd: mpath1: remaining active paths: 1 
Jan 22 09:52:37 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:52:37 pbox7 multipathd: checker failed path 252:2 in map mpath2 
Jan 22 09:52:37 pbox7 kernel: device-mapper: multipath: Failing path 252:2.
Jan 22 09:52:38 pbox7 openais[3472]: [TOTEM] entering GATHER state from 12. 
Jan 22 09:52:43 pbox7 multipathd: mpath2: remaining active paths: 1 
Jan 22 09:52:43 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering GATHER state from 0. 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Saving state aru eb high seq received eb 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Storing new sequence id for ring 50c 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering COMMIT state. 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering RECOVERY state. 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [0] member 64.142.17.1: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [1] member 64.142.17.2: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [2] member 64.142.17.5: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [3] member 64.142.17.6: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [4] member 64.142.17.7: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [5] member 64.142.17.8: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] position [6] member 64.142.17.10: 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] previous ring seq 1288 rep 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] aru eb high delivered eb received flag 1 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] Did not need to originate any messages in recovery. 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] CLM CONFIGURATION CHANGE 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] New Configuration: 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.1)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.2)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.5)  
Jan 22 09:52:43 pbox7 kernel: dlm: closing connection to node 2
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.6)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.7)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.8)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.10)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] Members Left: 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.9)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] Members Joined: 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] CLM CONFIGURATION CHANGE 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] New Configuration: 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.1)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.2)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.5)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.6)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.7)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.8)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] 	r(0) ip(64.142.17.10)  
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] Members Left: 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] Members Joined: 
Jan 22 09:52:43 pbox7 openais[3472]: [SYNC ] This node is within the primary component and will provide service. 
Jan 22 09:52:43 pbox7 openais[3472]: [TOTEM] entering OPERATIONAL state. 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.1 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.2 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.5 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.6 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.7 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.8 
Jan 22 09:52:43 pbox7 openais[3472]: [CLM  ] got nodejoin message 64.142.17.10 
Jan 22 09:52:46 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:52:48 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11430 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11431 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:52:48 pbox7 kernel: gnbd_monitor 11432 called gnbd_end_request with an error
Jan 22 09:52:48 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 7 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 8 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 4 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 6 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 3 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 1 
Jan 22 09:52:51 pbox7 openais[3472]: [CPG  ] got joinlist message from node 5 
Jan 22 09:52:52 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:52:53 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:52:53 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:52:57 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:57 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:52:57 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:52:58 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:58 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:52:58 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:52:58 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:52:58 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:02 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:02 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:02 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:04 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:04 pbox7 kernel: device-mapper: multipath: Failing path 252:3.
Jan 22 09:53:04 pbox7 multipathd: checker failed path 252:3 in map mpath0 
Jan 22 09:53:04 pbox7 multipathd: mpath0: remaining active paths: 0 
Jan 22 09:53:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:05 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:05 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:05 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:07 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:07 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:07 pbox7 kernel: device-mapper: multipath: Failing path 252:5.
Jan 22 09:53:07 pbox7 multipathd: checker failed path 252:5 in map mpath2 
Jan 22 09:53:07 pbox7 multipathd: mpath2: remaining active paths: 0 
Jan 22 09:53:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:11 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:11 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:13 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:14 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:14 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:14 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:16 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:16 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:20 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:20 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:22 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:23 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:23 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:23 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:25 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:25 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:29 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:29 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:31 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:32 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:32 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:32 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:34 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:34 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:38 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:38 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:38 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:40 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:40 pbox7 gnbd_recvd[11488]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:53:40 pbox7 gnbd_recvd[11488]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:53:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:53:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:41 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:41 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:41 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:43 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:43 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:43 pbox7 gnbd_recvd[11504]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:53:43 pbox7 gnbd_recvd[11504]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:53:43 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:53:46 pbox7 gnbd_recvd[11513]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:53:46 pbox7 gnbd_recvd[11513]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:53:46 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:53:47 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:47 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:47 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:49 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:50 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:50 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:50 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:53:52 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:53:52 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:53:56 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:56 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:53:56 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:53:58 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:53:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:59 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:53:59 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:53:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:53:59 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:01 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:01 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:05 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:05 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:05 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:06 pbox7 kernel: gnbd (pid 5029: gnbd_recvd) got signal
Jan 22 09:54:06 pbox7 kernel: gnbd5: Receive control failed (result -4)
Jan 22 09:54:06 pbox7 kernel: gnbd5: shutting down socket
Jan 22 09:54:06 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:54:06 pbox7 gnbd_recvd[5029]: client lost connection with pbox10.internal.freerun.com : Interrupted system call 
Jan 22 09:54:06 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:06 pbox7 kernel: gnbd (pid 5007: gnbd_recvd) got signal
Jan 22 09:54:06 pbox7 kernel: gnbd3: Receive control failed (result -4)
Jan 22 09:54:06 pbox7 kernel: gnbd3: shutting down socket
Jan 22 09:54:06 pbox7 kernel: exiting GNBD_DO_IT ioctl
Jan 22 09:54:06 pbox7 gnbd_recvd[5007]: client lost connection with pbox10.internal.freerun.com : Interrupted system call 
Jan 22 09:54:06 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:07 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:08 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:08 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:08 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:10 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:10 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:11 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:11 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:11 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:11 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:14 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:14 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:14 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:16 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:16 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:16 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:16 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:16 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:17 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:17 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:17 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:19 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:19 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:21 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:21 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:21 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:21 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:23 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:23 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:23 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:25 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:26 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:26 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:26 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:26 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:26 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:26 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:26 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:28 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:28 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: fatal: I/O error
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   block = 26
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   function = gfs_dreread
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   file = /builddir/build/BUILD/gfs-kmod-0.1.16/_kmod_build_/src/gfs/dio.c, line = 576
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3:   time = 1201024470
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: about to withdraw from the cluster
Jan 22 09:54:30 pbox7 kernel: GFS: fsid=Production0:web1.3: telling LM to withdraw
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:31 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:31 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:31 pbox7 kernel: GFS: fsid=Production0:web1.3: withdrawn
Jan 22 09:54:31 pbox7 kernel: 
Jan 22 09:54:31 pbox7 kernel: Call Trace:
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ffc94>] :gfs:gfs_lm_withdraw+0xc4/0xd3
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8001b9e6>] generic_make_request+0x1e8/0x1ff
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff88414dcf>] :gfs:gfs_io_error_bh_i+0x32/0x37
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883eca98>] :gfs:gfs_dreread+0xad/0xc7
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ecada>] :gfs:gfs_dread+0x28/0x43
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883ed522>] :gfs:gfs_get_meta_buffer+0xc6/0x247
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883fa25b>] :gfs:gfs_copyin_dinode+0x1d/0x131
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f8c78>] :gfs:inode_go_lock+0x27/0x3f
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6a50>] :gfs:glock_wait_internal+0x222/0x2bb
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6e7e>] :gfs:gfs_glock_nq+0x395/0x3d5
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff883f6ed4>] :gfs:gfs_glock_nq_init+0x16/0x2a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8840a782>] :gfs:gfs_getattr+0x41/0x67
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8000de0b>] vfs_getattr+0x2d/0xa9
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80028392>] vfs_stat_fd+0x32/0x4a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80064a9d>] do_page_fault+0x4eb/0x81d
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80039d83>] tty_ldisc_deref+0x68/0x7b
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80094baa>] recalc_sigpending_and_wake+0x9/0x1a
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff80028924>] do_sigaction+0x72/0x195
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff800231e6>] sys_newstat+0x19/0x31
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8005bde9>] error_exit+0x0/0x84
Jan 22 09:54:31 pbox7 kernel:  [<ffffffff8005b116>] system_call+0x7e/0x83
Jan 22 09:54:31 pbox7 kernel: 
Jan 22 09:54:32 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:32 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:32 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:34 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:35 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:35 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:35 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:35 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:35 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:36 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:36 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:36 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:36 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:37 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:37 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:39 pbox7 gnbd_recvd[11632]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:54:39 pbox7 gnbd_recvd[11632]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:54:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:54:41 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:41 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:41 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:42 pbox7 gnbd_recvd[11673]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:54:42 pbox7 gnbd_recvd[11673]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:54:42 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:54:43 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:44 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:44 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:44 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:44 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:44 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:45 pbox7 gnbd_recvd[11678]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:54:45 pbox7 gnbd_recvd[11678]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:54:45 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:54:46 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:46 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:50 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:50 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:50 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:54:50 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:50 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:50 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:50 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:52 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:54:53 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:53 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:54:53 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:53 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:54:53 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:54:55 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:54:55 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:54:55 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:55 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:54:55 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:54:55 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:54:59 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:54:59 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:54:59 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:55:00 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:00 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:00 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:00 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:01 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:02 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:02 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:55:02 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:02 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:04 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:55:04 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:05 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:05 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:05 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:05 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:08 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:08 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:08 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:55:10 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:10 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:10 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:10 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:10 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:11 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:11 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:55:11 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:11 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:13 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:55:13 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:15 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:15 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:15 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:15 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:17 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:17 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:17 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:55:19 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:20 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:20 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:55:20 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:20 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:20 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:20 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:20 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:20 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:22 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:55:22 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:25 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:25 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:25 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:25 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:26 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:26 pbox7 kernel: end_request: I/O error, dev gnbd0, sector 0
Jan 22 09:55:26 pbox7 multipathd: gnbd0: directio checker reports path is down 
Jan 22 09:55:28 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:29 pbox7 kernel: end_request: I/O error, dev gnbd1, sector 0
Jan 22 09:55:29 pbox7 multipathd: gnbd1: directio checker reports path is down 
Jan 22 09:55:29 pbox7 kernel: multipathd 3334 called gnbd_end_request with an error
Jan 22 09:55:29 pbox7 kernel: end_request: I/O error, dev gnbd2, sector 0
Jan 22 09:55:30 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:30 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:30 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:30 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:31 pbox7 multipathd: gnbd2: directio checker reports path is down 
Jan 22 09:55:31 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:31 pbox7 multipathd: mpath0: stop event checker thread 
Jan 22 09:55:35 pbox7 kernel: gnbd3: Attempted send on closed socket
Jan 22 09:55:35 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:35 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:35 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:35 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:37 pbox7 multipathd: gnbd3: checker msg is "directio checker reports path is down" 
Jan 22 09:55:37 pbox7 kernel: gnbd5: Attempted send on closed socket
Jan 22 09:55:38 pbox7 gnbd_recvd[11776]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:55:38 pbox7 gnbd_recvd[11776]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:55:38 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:55:39 pbox7 multipathd: gnbd5: checker msg is "directio checker reports path is down" 
Jan 22 09:55:41 pbox7 multipathd: gnbd3: checker msg is "directio checker reports path is down" 
Jan 22 09:55:41 pbox7 gnbd_recvd[11788]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:55:41 pbox7 gnbd_recvd[11788]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:55:41 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:55:43 pbox7 multipathd: gnbd5: checker msg is "directio checker reports path is down" 
Jan 22 09:55:43 pbox7 multipathd: mpath0: event checker started 
Jan 22 09:55:43 pbox7 multipathd: mpath1: event checker started 
Jan 22 09:55:43 pbox7 multipathd: mpath2: event checker started 
Jan 22 09:55:43 pbox7 multipathd: path checkers start up 
Jan 22 09:55:43 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:55:43 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:55:43 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:55:44 pbox7 gnbd_recvd[11793]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:55:44 pbox7 gnbd_recvd[11793]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:55:44 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:55:45 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:47 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:49 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:49 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:49 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:49 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:52 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:55:54 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:55:54 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:55:54 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:55:54 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:54 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:54 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:54 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:55:56 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:55:59 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:59 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:55:59 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:55:59 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:01 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:03 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:03 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:03 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:04 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:04 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:04 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:04 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:05 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:09 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:09 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:09 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:09 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:10 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:12 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:12 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:12 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:14 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:14 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:14 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:14 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:14 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:15 pbox7 kernel: gnbd3: Attempted send on closed socket
Jan 22 09:56:17 pbox7 kernel: gnbd5: Attempted send on closed socket
Jan 22 09:56:19 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:19 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:19 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:19 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:19 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:21 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:21 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:21 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:23 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:23 pbox7 multipathd: dm-10: add map (uevent) 
Jan 22 09:56:23 pbox7 multipathd: dm-10: devmap already registered 
Jan 22 09:56:24 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:24 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:24 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:24 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:28 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:29 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:29 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:29 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:29 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:30 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:30 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:30 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:32 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:34 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:34 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:34 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:34 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:37 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:37 pbox7 gnbd_recvd[11891]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:56:37 pbox7 gnbd_recvd[11891]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:56:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:56:39 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:39 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:39 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:40 pbox7 gnbd_recvd[11894]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:56:40 pbox7 gnbd_recvd[11894]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:56:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:56:41 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:43 pbox7 gnbd_recvd[11904]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:56:43 pbox7 gnbd_recvd[11904]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:56:43 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:56:46 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:48 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:48 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:48 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:48 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:48 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:48 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:48 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:50 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:56:53 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:53 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:53 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:53 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:55 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:56:57 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:56:57 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:56:57 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:56:58 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:58 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:56:58 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:56:58 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:56:59 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:03 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:03 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:03 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:03 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:04 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:06 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:06 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:06 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:08 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:08 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:08 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:08 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:08 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:13 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:13 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:13 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:13 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:13 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:15 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:15 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:15 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:17 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:18 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:18 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:18 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:18 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:22 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:23 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:23 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:23 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:23 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:24 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:24 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:24 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:26 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:28 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:28 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:28 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:28 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:31 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:33 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:33 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:33 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:33 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:33 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:33 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:33 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:35 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:36 pbox7 gnbd_recvd[11973]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:57:36 pbox7 gnbd_recvd[11973]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:57:36 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:57:39 pbox7 gnbd_recvd[11983]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:57:39 pbox7 gnbd_recvd[11983]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:57:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:57:40 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:40 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:40 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:42 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:42 pbox7 gnbd_recvd[11993]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:57:42 pbox7 gnbd_recvd[11993]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:57:42 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:57:44 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:47 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:47 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:47 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:47 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:49 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:57:51 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:57:51 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:57:51 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:57:52 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:52 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:52 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:52 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:53 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:57:57 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:57 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:57:57 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:57:57 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:57:58 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:00 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:00 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:00 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:02 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:02 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:02 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:02 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:02 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:07 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:07 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:07 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:07 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:07 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:09 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:09 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:09 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:11 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:12 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:12 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:12 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:12 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:16 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:17 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:17 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:17 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:17 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:18 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:18 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:18 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:20 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:22 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:22 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:22 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:22 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:25 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:27 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:27 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:27 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:27 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:27 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:27 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:27 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:29 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:32 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:32 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:32 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:32 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:34 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:35 pbox7 gnbd_recvd[12134]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:58:35 pbox7 gnbd_recvd[12134]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:58:35 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:58:36 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:36 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:36 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:38 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:38 pbox7 gnbd_recvd[12151]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:58:38 pbox7 gnbd_recvd[12151]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:58:38 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:58:41 pbox7 gnbd_recvd[12155]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:58:41 pbox7 gnbd_recvd[12155]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:58:41 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:58:43 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:43 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:43 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:45 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:46 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:46 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:46 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:46 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:47 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:51 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:51 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:51 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:51 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:58:52 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:58:54 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:58:54 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:58:54 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:58:56 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:58:56 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:56 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:58:56 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:58:56 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:01 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:01 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:01 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:01 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:01 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:03 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:03 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:03 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:05 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:06 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:06 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:06 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:06 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:10 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:11 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:11 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:11 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:11 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:12 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:12 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:12 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:14 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:16 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:16 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:16 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:16 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:19 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:21 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:21 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:21 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:21 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:21 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:21 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:21 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:23 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:26 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:26 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:26 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:26 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:28 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:30 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:30 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:30 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:31 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:31 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:31 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:31 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:32 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:34 pbox7 gnbd_recvd[12237]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:59:34 pbox7 gnbd_recvd[12237]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:59:34 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:59:37 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:37 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:37 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:37 pbox7 gnbd_recvd[12246]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:59:37 pbox7 gnbd_recvd[12246]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:59:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:59:39 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:40 pbox7 gnbd_recvd[12249]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 09:59:40 pbox7 gnbd_recvd[12249]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 09:59:40 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 09:59:41 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:45 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:45 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:45 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:45 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:46 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:48 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:48 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:48 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:50 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 09:59:50 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:50 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:50 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:50 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:55 pbox7 multipathd: gnbd0: unusable path 
Jan 22 09:59:55 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:55 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 09:59:55 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 09:59:55 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 09:59:57 pbox7 multipathd: gnbd1: unusable path 
Jan 22 09:59:57 pbox7 multipathd: gnbd2: unusable path 
Jan 22 09:59:57 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 09:59:59 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:00 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:00 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:00 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:00 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:04 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:05 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:05 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:05 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:05 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:06 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:06 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:06 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:08 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:10 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:10 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:10 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:10 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:13 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:15 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:15 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:15 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:15 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:15 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:15 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:15 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:17 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:20 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:20 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:20 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:20 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:22 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:24 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:24 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:24 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:25 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:25 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:25 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:25 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:26 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:30 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:30 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:30 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:30 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:31 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:33 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:33 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:33 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:33 pbox7 gnbd_recvd[12330]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:00:33 pbox7 gnbd_recvd[12330]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:00:33 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:00:35 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:36 pbox7 gnbd_recvd[12333]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:00:36 pbox7 gnbd_recvd[12333]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:00:36 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:00:39 pbox7 gnbd_recvd[12342]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:00:39 pbox7 gnbd_recvd[12342]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:00:39 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:00:40 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:40 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:40 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:42 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:44 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:44 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:44 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:44 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:44 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:49 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:49 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:49 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:49 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:49 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:51 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:00:51 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:00:51 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:00:53 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:00:54 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:54 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:54 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:54 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:00:58 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:00:59 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:59 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:00:59 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:00:59 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:00 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:00 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:00 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:02 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:04 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:04 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:04 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:04 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:07 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:09 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:09 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:09 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:09 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:09 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:09 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:09 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:11 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:14 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:14 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:14 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:14 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:16 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:18 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:18 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:18 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:19 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:19 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:19 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:19 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:20 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:24 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:24 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:24 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:24 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:24 pbox7 kernel: gnbd0: GNBD_DISCONNECT
Jan 22 10:01:25 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:27 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:27 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:27 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:29 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:29 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:29 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:29 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:29 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:31 pbox7 gnbd_recvd[12452]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:01:31 pbox7 gnbd_recvd[12452]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:01:31 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:01:34 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:34 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:34 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:34 pbox7 gnbd_recvd[12456]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:01:34 pbox7 gnbd_recvd[12456]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:01:34 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:01:36 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:37 pbox7 gnbd_recvd[12466]: ERROR [gnbd_recvd.c:213] cannot connect to server pbox9.internal.freerun.com (-1) : No route to host 
Jan 22 10:01:37 pbox7 gnbd_recvd[12466]: ERROR [gnbd_recvd.c:390] quitting 
Jan 22 10:01:37 pbox7 gnbd_monitor[4971]: ERROR [gnbd_monitor.c:671] gnbd_recvd failed (1) 
Jan 22 10:01:38 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:42 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:42 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:42 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:42 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:43 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:45 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:45 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:45 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:47 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:47 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:47 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:47 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:47 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:52 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:01:52 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:52 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:52 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:52 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:01:54 pbox7 multipathd: gnbd1: unusable path 
Jan 22 10:01:54 pbox7 multipathd: gnbd2: unusable path 
Jan 22 10:01:54 pbox7 multipathd: gnbd3: directio checker reports path is down 
Jan 22 10:01:56 pbox7 multipathd: gnbd5: directio checker reports path is down 
Jan 22 10:01:57 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:57 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:01:57 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:01:57 pbox7 gnbd_recvd[5007]: reconnecting 
Jan 22 10:02:01 pbox7 multipathd: gnbd0: unusable path 
Jan 22 10:02:02 pbox7 gnbd_recvd[5029]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 
Jan 22 10:02:02 pbox7 gnbd_recvd[5029]: reconnecting 
Jan 22 10:02:02 pbox7 gnbd_recvd[5007]: ERROR [gnbd_recvd.c:233] reading kill server reply from pbox10.internal.freerun.com failed : Unknown error 254 

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]