[Fedora-directory-users] Re: Migration from i-planet 52

Tue Dec 19 15:10:32 UTC 2006

>>Any errors?
No as far as I can tell the import goes smooth. Our upstream applications
seem to have no problem with the data it seems to be all imported.

I am really troubleshooting and index and performance issues, I figure out
of order keys could be slowing searches down. This a fairly large db 160000
entires) some objects have upwards of 20 attributes.

db_verify and db_2index both run fairly quickly. It just seems like they
cant both agree on what clean data looks like.

FWI I upgraded to the latest 1.0.4 before all this testing.

[18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import job...
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering enabled
with bucket size 15
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file
"/opt/idsk/downloads/idsk.services.com.ldif"
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning file
"/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries)
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished;
cleaning up...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up.
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up producer
thread...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing complete.
Post-processing...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete.
Processed 2037 entries in 1 seconds. (2037.00 entries/sec)
[18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline...
[18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access the
database
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job...
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering enabled
with bucket size 15
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file
"/opt/idsk/downloads/idsk.com.ldif"
[18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288 entries --
average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0%
[18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956 entries --
average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100%
[18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning file
"/opt/idsk/downloads/idsk.com.ldif" (163684 entries)
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished; cleaning
up...
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up.
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer
thread...
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete.
Post-processing...
[18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches...
[18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files...
[18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete.  Processed
163684 entries in 116 seconds. (1411.07 entries/sec)

Edward

On 12/18/06, Noriko Hosoi <nhosoi at redhat.com> wrote:
>
> Eddie C wrote:
> > All,
> >
> > I tested this from scratch.
> > I used the ldif_2db function which did work much faster! 112
> > seconds...rather then about 20 minutes.
> > However I think the verify_db and db2_index functions are not in
> > agreement.
> >
> > Create indexes.
> > After my initial import.
> > [root at ldap3 slapd-ldap3]# more out.after.final.txt
> > *****************************************************************
> > verify-db: This tool should only be run if recovery start fails
> > and the server is down.  If you run this tool while the server is
> > running, you may get false reports of corrupted files or other
> > false errors.
> > *****************************************************************
> > Verify log files in db ... Good
> > Verify db/o_idsk_com/id2entry.db4 ... Good
> > Verify db/userRoot/ancestorid.db4 ... Good
> > Verify db/userRoot/entrydn.db4 ... Good
> > Verify db/userRoot/cn.db4 ... Good
> > Verify db/userRoot/numsubordinates.db4 ... Good
> > Verify db/userRoot/aci.db4 ... Good
> > Verify db/userRoot/parentid.db4 ... Good
> > Verify db/userRoot/objectclass.db4 ... Good
> > Verify db/userRoot/id2entry.db4 ... Good
> > Verify db/userRoot/nsUniqueId.db4 ... Good
> > Verify db/idsk_services/ancestorid.db4 ...
> > DB ERROR: db_verify: Page 4: out-of-order key at entry 252
> > DB ERROR: db_verify: Page 7: out-of-order key at entry 194
> > DB ERROR: db_verify: Page 7: out-of-order key at entry 450
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 69
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 325
> > DB ERROR: db_verify: Page 11: out-of-order key at entry 581
> > DB ERROR: db_verify: Page 12: out-of-order key at entry 22
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 249
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 498
> > DB ERROR: db_verify: Page 16: out-of-order key at entry 754
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 195
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 451
> > DB ERROR: db_verify: Page 17: out-of-order key at entry 707
> > DB ERROR: db_verify: Page 18: out-of-order key at entry 148
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 254
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 510
> > DB ERROR: db_verify: Page 21: out-of-order key at entry 766
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 207
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 463
> > DB ERROR: db_verify: Page 22: out-of-order key at entry 719
> > DB ERROR: db_verify: Page 23: out-of-order key at entry 160
> > DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4:
> > DB_VERIFY_BAD: Database verification failed
> > Secondary index file ancestorid.db4 in db/idsk_services is corrupted.
> > Please run db2index(.pl) for reindexing.
> >
> Is this after you imported your ldif file to the backend 'idsk_services'?
>
> When you imported the ldif file, did you get any errors from ldif2db?
> This is an sample output when we import Example.ldif.  Did you see any
> messages other than these?
> [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
> pages: 1009265, procpages: 23476
> [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
> [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
> import_pages: 51200, pagesize: 4096
> [18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to access
> the database
> [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
> pages: 1009265, procpages: 23476
> [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
> [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
> import_pages: 51200, pagesize: 4096
> [18/Dec/2006:04:00:49 -0800] - import example: Beginning import job...
> [18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled
> with bucket size 100
> [18/Dec/2006:04:00:49 -0800] - import example: Processing file
> "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
> [18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file
> "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" (160
> entries)
> [18/Dec/2006:04:00:50 -0800] - import example: Workers finished;
> cleaning up...
> [18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up.
> [18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer
> thread...
> [18/Dec/2006:04:00:50 -0800] - import example: Indexing complete.
> Post-processing...
> [18/Dec/2006:04:00:50 -0800] - import example: Flushing caches...
> [18/Dec/2006:04:00:50 -0800] - import example: Closing files...
> [18/Dec/2006:04:00:51 -0800] - All database threads now stopped
> [18/Dec/2006:04:00:51 -0800] - import example: Import complete.
> Processed 160 entries in 2 seconds. (80.00 entries/sec)
>
> Thanks,
> --noriko
> > So then i reran db2index....verify_db again...same result.
> >
> > Edward
> >
> >
> > On 12/15/06, *Eddie C* <edlinuxguru at gmail.com
> > <mailto:edlinuxguru at gmail.com>> wrote:
> >
> >     I recently did an ldif backup of our iplanet 52 database. Its
> >     about an 88 MB ldif file.
> >     I took this to a new FDS server Dell 850 3 ghz duel core 2 sata
> >     hard disks.
> >     I ran an ldapadd  the data imported perfectly.
> >     Then I tried to cutover some systems and give the database some
> load.
> >
> >     System went 200% processor
> >
> >     Eventually I realized I was missing indexes so I added them
> >     through the graphical tool.
> >
> >     The log seemed to do something like this
> >     generating index 1%
> >     generating index 2%
> >     ....
> >     generating index 49%
> >     Done
> >     Seemed weird that they would jump from 49% to Done
> >     At this point the new system was running at 100% processor
> >     But the queries are running faster on our old 440 MHZ sparc t1
> >     server52 database
> >
> >     I ran
> >     DB ERROR: db_verify: Page 30: out-of-order key at entry 498
> >     DB ERROR: db_verify: DB->verify: db/o_com/channelcontentowner.db4:
> >     DB_VERIFY_BAD: Database verification failed
> >
> >     then I tried db2_index. The program seemed to be in a tight loop
> >     complaining about 1 missing entry.
> >
> >     I do not realize how the data can be so corrupted right after an
> >     import.
> >
> >     These are someone generic symptoms. Any ideas? Thanks
> >
> >
> > ------------------------------------------------------------------------
> >
> > --
> > Fedora-directory-users mailing list
> > Fedora-directory-users at redhat.com
> > https://www.redhat.com/mailman/listinfo/fedora-directory-users
> >
>
>
>
> --
> Fedora-directory-users mailing list
> Fedora-directory-users at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/fedora-directory-users/attachments/20061219/76d486c4/attachment.htm>