[Fedora-directory-users] MMR broken, can't get it started again

[Chris St. Pierre]

I noticed today that replication between a few of my four FDS machines
(all MMR agreements going every which way) had failed with errors like
this:

[30/Aug/2007:00:02:04 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental
protocol: event update_window_opened should not occur in state
start_backoff
[31/Aug/2007:00:03:59 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental
protocol: event update_window_opened should not occur in state
start_backoff
[31/Aug/2007:07:35:59 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Incremental
update failed and requires administrator action

I tried using mmr.pl to reinitialize the agreements, but that failed
miserably and wiped all of the data from the database I was
re-initializing.  I got the following errors on the supplier:

[31/Aug/2007:10:24:56 -0500] NSMMReplicationPlugin - Beginning total
update of replica "agmt="cn="Replication to chico (o=isp)""
(chico:389)".
[31/Aug/2007:10:25:36 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Failed to send
extended operation: LDAP error 81 (Can't contact LDAP server)
[31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error
89: NULL for total update operation
[31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error
89: NULL for total update operation
[31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error
89: NULL for total update operation
[31/Aug/2007:10:25:38 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error
89: NULL for total update operation
[31/Aug/2007:10:25:39 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Received error
89: NULL for total update operation
[31/Aug/2007:10:25:39 -0500] NSMMReplicationPlugin -
agmt="cn="Replication to chico (o=isp)"" (chico:389): Warning: unable
to send endReplication extended operation (Bad parameter to an ldap
routine)

And on the consumer:

[31/Aug/2007:10:24:54 -0500] NSMMReplicationPlugin -
multimaster_be_state_change: replica o=isp is going offline; disabling
replication
[31/Aug/2007:10:24:56 -0500] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to
access the database
[31/Aug/2007:10:25:16 -0500] - import userRoot: Processed 2036 entries
-- average rate 101.8/sec, recent rate 101.8/sec, hit ratio 0%
[31/Aug/2007:10:25:24 -0500] - ERROR bulk import abandoned
[31/Aug/2007:10:25:24 -0500] - import userRoot: Aborting all import
threads...
[31/Aug/2007:10:25:31 -0500] - import userRoot: Import threads
aborted.
[31/Aug/2007:10:25:31 -0500] - import userRoot: Closing files...
[31/Aug/2007:10:25:35 -0500] - libdb: userRoot/owner.db4: unable to
flush: No such file or directory
[...lots of lines like that...]
[31/Aug/2007:10:25:36 -0500] - libdb: userRoot/id2entry.db4: unable to
flush: No such file or directory
[31/Aug/2007:10:25:36 -0500] - import userRoot: Import failed.
[31/Aug/2007:10:25:36 -0500] - process_bulk_import_op: NULL backend

At that point, the supplier crashes and we get the "Can't contact LDAP
server" error in the consumer.

This looked really similar to an error I'd had before, in this thread:
http://www.mail-archive.com/fedora-directory-users@redhat.com/msg04969.html

I had eventually resolved that by dismantling all replication
information, using db2ldif to import the database from the supplier to
the consumer while the consumer was down, bringing the consumer back
up, and reinitializing the replication agreements.  This has not
worked this time; I get the exact same errors.

Anyone have any suggestions?  Thanks!

Chris St. Pierre
Unix Systems Administrator
Nebraska Wesleyan University