[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] CS5 & qdisk : details about problem "Node is undead" (contd.)



Hi,

I have a more precise sequence in the syslog on the node entering the infernal loop "Node is undead",
I think it could give some more indication about the problem, if someone could help me ...
Thanks a lot
Regards
Alain
[root node3 ~]# tail -f /var/log/syslog /var/log/daemons/*&
[1] 12017
[root node3 ~]# ==> /var/log/syslog <==
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initial score 1/1
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initialization complete
Mar  2 14:07:29 s_sys node3 openais[11209]: [CMAN ] quorum device registered
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <notice> Score sufficient for master operation (1/1; required=1); upgrading
Mar  2 14:07:38 s_sys node3 qdiskd[11100]: <info> Node 1 is the master
	===> Here is the poweroff -nf on  node2
Mar  2 14:07:59 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (2/16)
Mar  2 14:08:00 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (3/16)
Mar  2 14:08:01 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (4/16)
Mar  2 14:08:02 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (5/16)
Mar  2 14:08:03 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (6/16)

==> /var/log/daemons/errors <==
Mar  2 13:18:24 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:24 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:25 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:25 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:46:31 s_sys node3 mountd[16139]: Caught signal 15, un-registering and exiting.
Mar  2 13:46:44 s_sys node3 dlm_controld[15234]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 gfs_controld[15240]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 fenced[15228]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 qdiskd[15156]: <err> cman_dispatch: Host is down
Mar  2 13:46:44 s_sys node3 qdiskd[15156]: <err> Halting qdisk operations

==> /var/log/daemons/info <==
Mar  2 14:06:58 s_sys node3 openais[11209]: [MAIN ] Using default multicast address of 239.192.0.0
Mar  2 14:06:58 s_sys node3 openais[11209]: [CMAN ] CMAN 2.0.98 (built Jan 27 2009 10:53:06) started
Mar  2 14:06:59 s_sys node3 openais[11209]: [CMAN ] quorum regained, resuming activity
Mar  2 14:07:12 s_sys node3 qdiskd[11100]: <info> Quorum Partition: /dev/disk/by-id/scsi-3600601607b801c00343ab689eb03de11 Label: QDISK_0_0
Mar  2 14:07:12 s_sys node3 qdiskd[11100]: <info> Quorum Daemon Initializing
Mar  2 14:07:13 s_sys node3 qdiskd[11100]: <info> Heuristic: 'ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W3 -c1 -t3 12.11.2.1; RES=$?; fi; fi; echo $RES | grep -w 0 ' UP
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initial score 1/1
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initialization complete
Mar  2 14:07:29 s_sys node3 openais[11209]: [CMAN ] quorum device registered
Mar  2 14:07:38 s_sys node3 qdiskd[11100]: <info> Node 1 is the master

==> /var/log/daemons/warnings <==
Mar  2 10:25:54 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:25:54 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:26:14 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:26:14 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:26:24 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:26:24 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:30:31 s_sys node3 avahi-daemon[25488]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 11:11:15 s_sys node3 avahi-daemon[25474]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 11:28:24 s_sys node3 avahi-daemon[25466]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 13:56:47 s_sys node3 avahi-daemon[25408]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

[root node3 ~]#
==> /var/log/syslog <==
Mar  2 14:08:04 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (7/16)
Mar  2 14:08:05 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (8/16)
Mar  2 14:08:06 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (9/16)
Mar  2 14:08:07 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (10/16)
Mar  2 14:08:08 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (11/16)
Mar  2 14:08:09 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (12/16)
Mar  2 14:08:10 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (13/16)
Mar  2 14:08:11 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (14/16)
Mar  2 14:08:12 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (15/16)
Mar  2 14:08:13 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (16/16)
Mar  2 14:08:14 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (17/16)
Mar  2 14:08:14 s_sys node3 qdiskd[11100]: <debug> Node 1 DOWN
Mar  2 14:08:14 s_sys node3 qdiskd[11100]: <debug> Making bid for master
Mar  2 14:08:15 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (18/16)
Mar  2 14:08:16 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (19/16)
Mar  2 14:08:17 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (20/16)
Mar  2 14:08:18 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (21/16)
Mar  2 14:08:19 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (22/16)
Mar  2 14:08:20 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (23/16)
Mar  2 14:08:21 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (24/16)
Mar  2 14:08:21 s_sys node3 qdiskd[11100]: <info> Assuming master role

==> /var/log/daemons/info <==
Mar  2 14:08:21 s_sys node3 qdiskd[11100]: <info> Assuming master role

==> /var/log/syslog <==
Mar  2 14:08:22 s_sys node3 qdiskd[11100]: <debug> Node 1 missed an update (25/16)
Mar  2 14:08:22 s_sys node3 qdiskd[11100]: <notice> Writing eviction notice for node 1
Mar  2 14:08:22 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:23 s_sys node3 qdiskd[11100]: <notice> Node 1 evicted
Mar  2 14:08:24 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:24 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:24 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:25 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:25 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:25 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:26 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:26 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:26 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:27 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:27 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:27 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:28 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:28 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:28 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:29 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:29 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:29 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:29 s_sys node3 openais[11209]: [TOTEM] The token was lost in the OPERATIONAL state.
Mar  2 14:08:29 s_sys node3 openais[11209]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Mar  2 14:08:29 s_sys node3 openais[11209]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Mar  2 14:08:29 s_sys node3 openais[11209]: [TOTEM] entering GATHER state from 2.
Mar  2 14:08:30 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:30 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:30 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:31 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:31 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:31 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:32 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:32 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:32 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:33 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:33 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:33 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] entering GATHER state from 0.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] Creating commit token because I am the rep.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] Saving state aru 49 high seq received 49
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] Storing new sequence id for ring 27c
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] entering COMMIT state.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] entering RECOVERY state.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] position [0] member 12.11.2.4:
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] previous ring seq 632 rep 12.11.2.3
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] aru 49 high delivered 49 received flag 1
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] Did not need to originate any messages in recovery.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] Sending initial ORF token
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] CLM CONFIGURATION CHANGE
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] New Configuration:
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ]     r(0) ip(12.11.2.4)
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] Members Left:
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ]     r(0) ip(12.11.2.3)
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] Members Joined:
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0641] confchg, low nodeid=2, us = 2
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0651] confchg, build downlist: 1 nodes
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] CLM CONFIGURATION CHANGE
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] New Configuration:
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ]     r(0) ip(12.11.2.4)
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] Members Left:
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] Members Joined:
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0662] confchg, sent downlist
Mar  2 14:08:34 s_sys node3 openais[11209]: [SYNC ] This node is within the primary component and will provide service.
Mar  2 14:08:34 s_sys node3 openais[11209]: [TOTEM] entering OPERATIONAL state.
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0785] downlist left_list: 1
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar  2 14:08:34 s_sys node3 openais[11209]: [CLM  ] got nodejoin message 12.11.2.4
Mar  2 14:08:34 s_kernel node3 kernel: dlm: closing connection to node 1
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:0959] sending joinlist to cluster
Mar  2 14:08:34 s_sys node3 openais[11209]: [CPG  ] got joinlist message from node 2
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar  2 14:08:34 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:34 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:34 s_sys node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar  2 14:08:34 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:34 s_sys node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar  2 14:08:34 s_sys node3 fenced[11229]: fencing node "node2"

==> /var/log/daemons/info <==
Mar  2 14:08:34 s_sys node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar  2 14:08:34 s_sys node3 fenced[11229]: fencing node "node2"

==> /var/log/syslog <==
Mar  2 14:08:35 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:35 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:35 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:36 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:36 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:36 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:37 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:37 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:37 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:38 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:38 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:38 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node

[root node3 ~]# Mar  2 14:08:39 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:39 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:39 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:39 s_sys node3 fenced[11229]: fence "node2" success
Mar  2 14:08:39 s_sys node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar  2 14:08:39 s_sys node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0

==> /var/log/daemons/info <==
Mar  2 14:08:39 s_sys node3 fenced[11229]: fence "node2" success
fg
==> /var/log/syslog <==
Mar  2 14:08:40 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:40 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:40 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node

tail -f /var/log/syslog /var/log/daemons/*

[root node3 ~]# tail -f /var/log/syslog /var/log/daemons/*&
[1] 12044
[root node3 ~]# ==> /var/log/syslog <==
Mar  2 14:08:41 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:42 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:42 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:42 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:43 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:43 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:43 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:44 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:44 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:44 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node

==> /var/log/daemons/errors <==
Mar  2 13:18:24 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:24 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:25 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:18:25 s_sys node3 ccsd[15202]: Error while processing connect: Connection refused
Mar  2 13:46:31 s_sys node3 mountd[16139]: Caught signal 15, un-registering and exiting.
Mar  2 13:46:44 s_sys node3 dlm_controld[15234]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 gfs_controld[15240]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 fenced[15228]: cluster is down, exiting
Mar  2 13:46:44 s_sys node3 qdiskd[15156]: <err> cman_dispatch: Host is down
Mar  2 13:46:44 s_sys node3 qdiskd[15156]: <err> Halting qdisk operations

==> /var/log/daemons/info <==
Mar  2 14:07:12 s_sys node3 qdiskd[11100]: <info> Quorum Daemon Initializing
Mar  2 14:07:13 s_sys node3 qdiskd[11100]: <info> Heuristic: 'ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W3 -c1 -t3 12.11.2.1; RES=$?; fi; fi; echo $RES | grep -w 0 ' UP
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initial score 1/1
Mar  2 14:07:29 s_sys node3 qdiskd[11100]: <info> Initialization complete
Mar  2 14:07:29 s_sys node3 openais[11209]: [CMAN ] quorum device registered
Mar  2 14:07:38 s_sys node3 qdiskd[11100]: <info> Node 1 is the master
Mar  2 14:08:21 s_sys node3 qdiskd[11100]: <info> Assuming master role
Mar  2 14:08:34 s_sys node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar  2 14:08:34 s_sys node3 fenced[11229]: fencing node "node2"
Mar  2 14:08:39 s_sys node3 fenced[11229]: fence "node2" success

==> /var/log/daemons/warnings <==
Mar  2 10:25:54 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:25:54 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:26:14 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:26:14 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:26:24 s_sys node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar  2 10:26:24 s_sys node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar  2 10:30:31 s_sys node3 avahi-daemon[25488]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 11:11:15 s_sys node3 avahi-daemon[25474]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 11:28:24 s_sys node3 avahi-daemon[25466]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar  2 13:56:47 s_sys node3 avahi-daemon[25408]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!

[root node3 ~]#
[root node3 ~]#
[root node3 ~]#
[root node3 ~]#
==> /var/log/syslog <==
Mar  2 14:08:45 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:45 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:45 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
tail -f /var/log/syslog /var/log/daemons/Mar  2 14:08:46 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:46 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:46 s_sys node3 qdiskd[11100]: <debug> Telling CMAN Mar  2 14:08:47 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:47 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:47 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:48 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:48 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:48 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:49 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:49 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:49 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:50 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:50 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:50 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:51 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:51 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:51 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:52 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:52 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:52 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:53 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:53 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:53 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:54 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:54 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:54 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:55 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:55 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:55 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:56 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:56 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:56 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:57 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:57 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:57 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:58 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:58 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:58 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:08:59 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:08:59 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:08:59 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:09:00 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:09:00 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:09:00 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:09:01 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:09:01 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:09:01 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:09:02 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:09:02 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar  2 14:09:02 s_sys node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar  2 14:09:03 s_sys node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar  2 14:09:03 s_sys node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
M


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]