[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Linux-cluster] service stuck in "recovering", no attempt to restart



On a 3 node cluster running:
  cman-2.0.115-34.el5_5.3
  rgmanager-2.0.52-6.el5.centos.8
  openais-0.80.6-16.el5_5.9

We have a custom resource, "dn", for which I wrote the resource agent.
Service has three resources: a virtual IP (using ip.sh), and two dn children.

Normally, when one of the dn instances fails its status check,
rgmanager stops the service (stops dn_a and dn_b, then stops the IP),
then relocates to another node and starts the service there.

Several hours ago, one of the dn instances failed its status check,
rgmanager stopped it, marked the service "recovering", but then did
not seem to try to start it on any node.  It just stayed down for
hours until logged in to look at it.

Until 17:22 today, service was running on node1.  Here's what it logged:

Oct  4 17:22:12 clustnode1 clurgmgrd: [517]: <err> Monitoring Service dn:dn_b > Service Is Not Running
Oct  4 17:22:12 clustnode1 clurgmgrd[517]: <notice> status on dn "dn_b" returned 1 (generic error)
Oct  4 17:22:12 clustnode1 clurgmgrd[517]: <notice> Stopping service service:dn
Oct  4 17:22:12 clustnode1 clurgmgrd: [517]: <info> Stopping Service dn:dn_b
Oct  4 17:22:12 clustnode1 clurgmgrd: [517]: <notice> Checking if stopped: check_pid_file /dn/dn_b/dn_b.pid
Oct  4 17:22:14 clustnode1 clurgmgrd: [517]: <info> Stopping Service dn:dn_b > Succeed
Oct  4 17:22:14 clustnode1 clurgmgrd: [517]: <info> Stopping Service dn:dn_a
Oct  4 17:22:15 clustnode1 clurgmgrd: [517]: <notice> Checking if stopped: check_pid_file /dn/dn_a/dn_a.pid
Oct  4 17:22:17 clustnode1 clurgmgrd: [517]: <info> Stopping Service dn:dn_a > Succeed
Oct  4 17:22:17 clustnode1 clurgmgrd: [517]: <info> Removing IPv4 address 10.6.9.136/23 from eth0
Oct  4 17:22:27 clustnode1 clurgmgrd[517]: <notice> Service service:dn is recovering

At around that time, node2 also logged this:

Oct  4 17:21:19 clustnode2 ccsd[5584]: Unable to read complete comm_header_t.
Oct  4 17:21:29 clustnode2 ccsd[5584]: Unable to read complete comm_header_t.

[Cluster name and node names anonymized with simple search and replace]

There are no other log entries in /var/log/messages on any node around
that time, that relate to cluster suite.

Currently, the service is still "recovering", with cluster status
otherwise apparently fine.  clustat -x output on all three nodes is
identical except for which node has local="1".  It looks like this:

<?xml version="1.0"?>
<clustat version="4.1.1">
  <cluster name="clustnode" id="23048" generation="12"/>
  <quorum quorate="1" groupmember="1"/>
  <nodes>
    <node name="clustnode1" state="1" local="0" estranged="0" rgmanager="1" rgmanager_master="0" qdisk="0" nodeid="0x00000001"/>
    <node name="clustnode2" state="1" local="1" estranged="0" rgmanager="1" rgmanager_master="0" qdisk="0" nodeid="0x00000002"/>
    <node name="clustnode3" state="1" local="0" estranged="0" rgmanager="1" rgmanager_master="0" qdisk="0" nodeid="0x00000003"/>
  </nodes>
  <groups>
    <group name="service:dn" state="118" state_str="recovering" flags="0" flags_str="" owner="none" last_owner="clustnode1" restarts="0" last_transition="1317763347" last_transition_str="Tue Oct  4 17:22:27 2011"/>
  </groups>
</clustat>

And cman_tool status shows all three nodes voting and in the quorum:

Version: 6.2.0
Config Version: 2
Cluster Name: clustnode
Cluster Id: 23048
Cluster Member: Yes
Cluster Generation: 12
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Quorum: 2  
Active subsystems: 8
Flags: Dirty 
Ports Bound: 0 177  
Node name: clustnode2
Node ID: 2
Multicast addresses: 239.245.0.84 
Node addresses: 10.6.8.208 

Again, this looks the same on all three nodes.

Here's the resource section of cluster.conf (with the values of some
of the arguments to my custom resource modified so as not to expose
actual username, path, or port number):

<rm log_level="6">
  <service autostart="1" name="dn" recovery="relocate">
    <ip address="10.6.9.136" monitor_link="1">
      <dn user="username" dninstall="/dn/path" name="dn_a" monitoringport="portnum"/>
      <dn user="username" dninstall="/dn/path" name="dn_b" monitoringport="portnum"/>
    </ip>
  </service>
</rm>

Any ideas why it might be in this state, where everything is
apparently fine except that the service is "recovering" and rgmanager
isn't trying to do anything about it and isn't logging any complaints?

Attached: strace -fp output of clurgmrgd processes on node1 and node2
  -- Cos
Process 517 attached with 4 threads - interrupt to quit
[pid  9842] clock_gettime(CLOCK_REALTIME,  <unfinished ...>
[pid  1001] clock_gettime(CLOCK_REALTIME,  <unfinished ...>
[pid  1000] select(6, [3 5], NULL, NULL, {0, 935000} <unfinished ...>
[pid   517] select(12, [10 11], NULL, NULL, {8, 177000} <unfinished ...>
[pid  9842] <... clock_gettime resumed> {1317781205, 661864000}) = 0
[pid  1001] <... clock_gettime resumed> {1317781205, 661864000}) = 0
[pid  9842] futex(0x432a5cbc, FUTEX_WAIT_PRIVATE, 3573, {7, 357519000} <unfinished ...>
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81853, {0, 867658000}) = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781206, 530851000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81855, {2, 999711000} <unfinished ...>
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2}) = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  1001] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781209, 532508000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81857, {3, 0} <unfinished ...>
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  1001] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781212, 534580000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81859, {3, 0} <unfinished ...>
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  9842] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  9842] futex(0x432a5c90, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  9842] clock_gettime(CLOCK_REALTIME, {1317781213, 21497000}) = 0
[pid  9842] futex(0x432a5cbc, FUTEX_WAIT_PRIVATE, 3575, {10, 0} <unfinished ...>
[pid   517] <... select resumed> )      = 0 (Timeout)
[pid   517] socket(PF_FILE, SOCK_STREAM, 0) = 13
[pid   517] connect(13, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid   517] write(13, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid   517] read(13, "\1\0\0\0\0\0\0\0\350D9\3\0\0\0\0\0\0\0\0", 20) = 20
[pid   517] close(13)                   = 0
[pid   517] socket(PF_FILE, SOCK_STREAM, 0) = 13
[pid   517] connect(13, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid   517] write(13, "\3\0\0\0\0\0\0\0\350D9\3\0\0\0\0\31\0\0\0/cluster/@co"..., 45) = 45
[pid   517] read(13, "\3\0\0\0\0\0\0\0\350D9\3\0\0\0\0\2\0\0\0", 20) = 20
[pid   517] read(13, "2\0", 2)          = 2
[pid   517] close(13)                   = 0
[pid   517] socket(PF_FILE, SOCK_STREAM, 0) = 13
[pid   517] connect(13, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid   517] write(13, "\2\0\0\0\0\0\0\0\350D9\3\0\0\0\0\0\0\0\0", 20) = 20
[pid   517] read(13, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
[pid   517] close(13)                   = 0
[pid   517] clone(Process 18772 attached
child_stack=0x40ebc240, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x40ebc9c0, tls=0x40ebc930, child_tidptr=0x40ebc9c0) = 18772
[pid   517] select(12, [10 11], NULL, NULL, {10, 0} <unfinished ...>
[pid 18772] set_robust_list(0x40ebc9d0, 0x18) = 0
[pid 18772] rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT USR1 USR2 TERM], NULL, 8) = 0
[pid 18772] _exit(0)                    = ?
Process 18772 detached
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  1001] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781215, 536718000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81861, {3, 0} <unfinished ...>
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  1001] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781218, 538706000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81863, {3, 0} <unfinished ...>
[pid  1000] <... select resumed> )      = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2}) = 0 (Timeout)
[pid  1000] read(5, 0x428a4f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid  1000] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid  1001] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid  1001] futex(0x12fac8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  1001] clock_gettime(CLOCK_REALTIME, {1317781221, 540821000}) = 0
[pid  1001] futex(0x12fac8ec, FUTEX_WAIT_PRIVATE, 81865, {3, 0}
Process 28445 attached with 4 threads - interrupt to quit
[pid 28962] clock_gettime(CLOCK_REALTIME,  <unfinished ...>
[pid 28931] clock_gettime(CLOCK_REALTIME,  <unfinished ...>
[pid 28930] select(6, [3 5], NULL, NULL, {1, 725000} <unfinished ...>
[pid 28445] select(12, [10 11], NULL, NULL, {6, 894000} <unfinished ...>
[pid 28962] <... clock_gettime resumed> {1317781260, 477926000}) = 0
[pid 28931] <... clock_gettime resumed> {1317781260, 477926000}) = 0
[pid 28962] futex(0x429dacbc, FUTEX_WAIT_PRIVATE, 24531, {4, 991782000} <unfinished ...>
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81869, {2, 613666000} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28931] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28931] futex(0x1bdea8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28931] clock_gettime(CLOCK_REALTIME, {1317781263, 93684000}) = 0
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81871, {3, 0} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28962] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28962] futex(0x429dac90, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28962] clock_gettime(CLOCK_REALTIME, {1317781265, 471616000}) = 0
[pid 28962] futex(0x429dacbc, FUTEX_WAIT_PRIVATE, 24533, {10, 0} <unfinished ...>
[pid 28931] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28931] futex(0x1bdea8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28931] clock_gettime(CLOCK_REALTIME, {1317781266, 95446000}) = 0
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81873, {3, 0} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28445] <... select resumed> )      = 0 (Timeout)
[pid 28445] socket(PF_FILE, SOCK_STREAM, 0) = 14
[pid 28445] connect(14, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid 28445] write(14, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 20) = 20
[pid 28445] read(14, "\1\0\0\0\0\0\0\0\"\rc\4\0\0\0\0\0\0\0\0", 20) = 20
[pid 28445] close(14)                   = 0
[pid 28445] socket(PF_FILE, SOCK_STREAM, 0) = 14
[pid 28445] connect(14, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid 28445] write(14, "\3\0\0\0\0\0\0\0\"\rc\4\0\0\0\0\31\0\0\0/cluster/@co"..., 45) = 45
[pid 28445] read(14, "\3\0\0\0\0\0\0\0\"\rc\4\0\0\0\0\2\0\0\0", 20) = 20
[pid 28445] read(14, "2\0", 2)          = 2
[pid 28445] close(14)                   = 0
[pid 28445] socket(PF_FILE, SOCK_STREAM, 0) = 14
[pid 28445] connect(14, {sa_family=AF_FILE, path="/var/run/cluster/ccsd.sock"...}, 110) = 0
[pid 28445] write(14, "\2\0\0\0\0\0\0\0\"\rc\4\0\0\0\0\0\0\0\0", 20) = 20
[pid 28445] read(14, "\2\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\0\0\0\0", 20) = 20
[pid 28445] close(14)                   = 0
[pid 28445] clone(Process 29968 attached
child_stack=0x40705240, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x407059c0, tls=0x40705930, child_tidptr=0x407059c0) = 29968
[pid 28445] select(12, [10 11], NULL, NULL, {10, 0} <unfinished ...>
[pid 29968] set_robust_list(0x407059d0, 0x18) = 0
[pid 29968] rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT USR1 USR2 TERM], NULL, 8) = 0
[pid 29968] _exit(0)                    = ?
Process 29968 detached
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28931] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28931] futex(0x1bdea8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28931] clock_gettime(CLOCK_REALTIME, {1317781269, 97451000}) = 0
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81875, {3, 0} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28445] <... select resumed> )      = 1 (in [10], left {6, 643000})
[pid 28445] accept(10, 0, NULL)         = 14
[pid 28445] fcntl(14, F_GETFD)          = 0
[pid 28445] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 28445] select(15, [14], NULL, [14], {1, 0}) = 1 (in [14], left {1, 0})
[pid 28445] read(14, "\30\0\0\0\4\0\0\0", 8) = 8
[pid 28445] select(15, [14], NULL, [14], {1, 0}) = 1 (in [14], left {1, 0})
[pid 28445] read(14, "\22:\274\0\0\0\0\30\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
[pid 28445] clone(Process 29977 attached
child_stack=0x40705240, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x407059c0, tls=0x40705930, child_tidptr=0x407059c0) = 29977
[pid 28445] select(12, [10 11], NULL, NULL, {6, 643000} <unfinished ...>
[pid 29977] set_robust_list(0x407059d0, 0x18) = 0
[pid 29977] rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT USR1 USR2 TERM], NULL, 8) = 0
[pid 29977] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 29977] write(14, "x\0\0\0\4\0\0\0\22:\274\0\0\0\0x\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128
[pid 29977] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 29977] write(14, "\30\0\0\0\4\0\0\0\22:\274\0\0\0\0\30\0\0\0\0\0\0\0\0\0\0\0\0N\213y\23", 32) = 32
[pid 29977] select(15, [14], NULL, [14], {10, 0}) = 1 (in [14], left {10, 0})
[pid 29977] read(14, "\30\0\0\0\4\0\0\0", 8) = 8
[pid 29977] select(15, [14], NULL, [14], {10, 0}) = 1 (in [14], left {10, 0})
[pid 29977] read(14, "\22:\274\0\0\0\0\30\0\0\0\0\0\0\0\0\0\0\0\0\30\0\0\0", 24) = 24
[pid 29977] close(14)                   = 0
[pid 29977] _exit(0)                    = ?
Process 29977 detached
[pid 28445] <... select resumed> )      = 1 (in [10], left {6, 642000})
[pid 28445] accept(10, 0, NULL)         = 14
[pid 28445] fcntl(14, F_GETFD)          = 0
[pid 28445] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 28445] select(15, [14], NULL, [14], {1, 0}) = 1 (in [14], left {1, 0})
[pid 28445] read(14, "\30\0\0\0\4\0\0\0", 8) = 8
[pid 28445] select(15, [14], NULL, [14], {1, 0}) = 1 (in [14], left {1, 0})
[pid 28445] read(14, "\22:\274\0\0\0\0\30\0\0\0\f\0\0\0\0\0\0\0\0\0\0\0\0", 24) = 24
[pid 28445] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 28445] write(14, "\30\0\0\0\4\0\0\0\22:\274\0\0\0\0\30\0\0\0\f\0\0\0\1\0\0\0\0\377\377\377\377", 32) = 32
[pid 28445] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 28445] write(14, "\30\0\0\0\4\0\0\0\22:\274\0\0\0\0\30\0\0\0\f\0\0\0\2\0\0\0\0\377\377\377\377", 32) = 32
[pid 28445] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 28445] write(14, "\30\0\0\0\4\0\0\0\22:\274\0\0\0\0\30\0\0\0\f\0\0\0\3\0\0\0\0\377\377\377\377", 32) = 32
[pid 28445] select(15, NULL, [14], [14], NULL) = 1 (out [14])
[pid 28445] write(14, "\30\0\0\0\4\0\0\0\22:\274\0\0\0\0\30\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377", 32) = 32
[pid 28445] select(15, [14], NULL, [14], {10, 0}) = 1 (in [14], left {10, 0})
[pid 28445] read(14, "\30\0\0\0\4\0\0\0", 8) = 8
[pid 28445] select(15, [14], NULL, [14], {10, 0}) = 1 (in [14], left {10, 0})
[pid 28445] read(14, "\22:\274\0\0\0\0\30\0\0\0\0\0\0\0\0\0\0\0\0\30\0\0\0", 24) = 24
[pid 28445] close(14)                   = 0
[pid 28445] select(12, [10 11], NULL, NULL, {6, 642000} <unfinished ...>
[pid 28931] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28931] futex(0x1bdea8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28931] clock_gettime(CLOCK_REALTIME, {1317781272, 99389000}) = 0
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81877, {3, 0} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2}) = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2} <unfinished ...>
[pid 28931] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28931] futex(0x1bdea8c0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28931] clock_gettime(CLOCK_REALTIME, {1317781275, 101463000}) = 0
[pid 28931] futex(0x1bdea8ec, FUTEX_WAIT_PRIVATE, 81879, {3, 0} <unfinished ...>
[pid 28962] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28962] futex(0x429dac90, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 28962] clock_gettime(CLOCK_REALTIME, {1317781275, 474705000}) = 0
[pid 28962] futex(0x429dacbc, FUTEX_WAIT_PRIVATE, 24535, {9, 999904000} <unfinished ...>
[pid 28930] <... select resumed> )      = 0 (Timeout)
[pid 28930] read(5, 0x41587f6b, 1)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 28930] select(6, [3 5], NULL, NULL, {2, 2}

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]