[Fedora-directory-users] Can't locate CSN in Multi-Master replica

[Rich Megginson]

Dael Maselli wrote:
> Dear Richard,
>
> The problem came back, this time in one node.
>
> We have 4-way replica with the nodes: ds-m1, ds-2, ds-m3, ds-m4.
>
> Yesterday all RW replica works fine, this morning one node (ds-m3) 
> crashed
What was the cause of the crash?
> and restarted with this log:
>
> [14/Nov/2007:09:16:37 +0100] - Fedora-Directory/1.0.4 B2006.312.1621 
> starting up
> [14/Nov/2007:09:16:37 +0100] - Detected Disorderly Shutdown last time 
> Directory Server was running, recovering database.
> [14/Nov/2007:09:16:38 +0100] NSMMReplicationPlugin - 
> replica_check_for_data_reload: Warning: data for replica dc=infn,dc=it 
> was reloaded and it no longer matches the data in the changelog 
> (replica data > changelog). Recreating the changelog file. This could 
> affect replication with replica's consumers in which case the 
> consumers should be reinitialized.
If you see this again, try this:
Shutdown m3, then start it with the replica log level:
cd /opt/fedora-ds/slapd-m3
./stop-slapd
./start-slapd -d 8192
Then shut it down as soon as you see the replica_check_for_data_reload: 
error message.  Then paste the error log to pastebin.com or 
rafb.net/paste and paste the link here.  Be sure to obscure any 
sensitive information first.

>
> Then I tried to reinitialize ds-m3 from ds-m1 and in ds-m3 log it wrote:
> [14/Nov/2007:15:21:36 +0100] NSMMReplicationPlugin - 
> multimaster_be_state_change: replica dc=infn,dc=it is going offline; 
> disabling replication
> [14/Nov/2007:15:21:36 +0100] - WARNING: Import is running with 
> nsslapd-db-private-import-mem on; No other process is allowed to 
> access the database
> [14/Nov/2007:15:21:38 +0100] - import userRoot: Workers finished; 
> cleaning up...
> [14/Nov/2007:15:21:39 +0100] - import userRoot: Workers cleaned up.
> [14/Nov/2007:15:21:39 +0100] - import userRoot: Indexing complete.  
> Post-processing...
> [14/Nov/2007:15:21:39 +0100] - import userRoot: Flushing caches...
> [14/Nov/2007:15:21:39 +0100] - import userRoot: Closing files...
> [14/Nov/2007:15:21:39 +0100] - import userRoot: Import complete.  
> Processed 10 entries in 2 seconds. (5.00 entries/sec)
> [14/Nov/2007:15:21:39 +0100] NSMMReplicationPlugin - 
> multimaster_be_state_change: replica dc=infn,dc=it is coming online; 
> enabling replication
> [14/Nov/2007:15:21:39 +0100] NSMMReplicationPlugin - 
> replica_reload_ruv: Warning: new data for replica dc=infn,dc=it does 
> not match the data in the chang elog.
>  Recreating the changelog file. This could affect replication with 
> replica's  consumers in which case the consumers should be reinitialized.
>
> So I tried to make changes on directory from node ds-m1,2 or 4 and it 
> propagates to
> all 4 node (including ds-m3). BUT when I try to make changes from 
> ds-m3 it will not
> propagates and in the ds-m3 log there is angain:
>
> [14/Nov/2007:15:42:22 +0100] agmt="cn=m3-m2" (ds-m2:636) - Can't 
> locate CSN 4739d5a5000000030000 in the changelog (DB rc=-30990). The 
> consumer may need to be reinitialized.
> [14/Nov/2007:15:42:22 +0100] agmt="cn=m3-m4" (ds-m4:636) - Can't 
> locate CSN 4739d5a5000000030000 in the changelog (DB rc=-30990). The 
> consumer may need to be reinitialized.
> [14/Nov/2007:15:42:22 +0100] agmt="cn=m3-m1" (ds-m1:636) - Can't 
> locate CSN 4739d5a5000000030000 in the changelog (DB rc=-30990). The 
> consumer may need to be reinitialized.
>
> So, please help me! What can I do now, we can't reinstall from scratch 
> anytime one
> server goes down.

To get up and running again, try this, assuming you have no pending 
changes in the m3 database that you care about:
shutdown m3
remove all of the files in the changelog directory (e.g. 
/opt/fedora-ds/slapd-instance/cldb)
restart m3
do a replica reinit of m3 from one of the other masters
>
> Thank you. Best regards.
>
> Dael Maselli.
>
> Dael Maselli wrote:
>> Well. I restarted from scratch. Now all works fine.
>>
>> Now I have 4-way RW replicas with agreements from all to all.
>>
>> Thank you for assistance.
>>
>> Regards.
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> -- 
>> Fedora-directory-users mailing list
>> Fedora-directory-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
> ------------------------------------------------------------------------
>
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3245 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://listman.redhat.com/archives/fedora-directory-users/attachments/20071115/e5d249cb/attachment.bin>