[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Fedora-directory-users] Re: Migration from i-planet 52



Eddie C wrote:
>>Any errors?
No as far as I can tell the import goes smooth. Our upstream applications seem to have no problem with the data it seems to be all imported. I am really troubleshooting and index and performance issues, I figure out of order keys could be slowing searches down. This a fairly large db 160000 entires) some objects have upwards of 20 attributes. db_verify and db_2index both run fairly quickly. It just seems like they cant both agree on what clean data looks like. FWI I upgraded to the latest 1.0.4 before all this testing.
Just to confirm, was the server running when you ran db_verify?
[18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database [18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import job... [18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering enabled with bucket size 15 [18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file "/opt/idsk/downloads/idsk.services.com.ldif" [18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning file "/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries) [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished; cleaning up...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up.
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up producer thread... [18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing complete. Post-processing...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete. Processed 2037 entries in 1 seconds. ( 2037.00 entries/sec)
[18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline...
[18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job...
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering enabled with bucket size 15 [18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file "/opt/idsk/downloads/idsk.com.ldif" [18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288 entries -- average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0% [18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956 entries -- average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100% [18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning file "/opt/idsk/downloads/idsk.com.ldif" (163684 entries) [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished; cleaning up...
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up.
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer thread... [18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete. Post-processing...
[18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches...
[18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files...
[18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete. Processed 163684 entries in 116 seconds. ( 1411.07 entries/sec)

Edward
On 12/18/06, *Noriko Hosoi* <nhosoi redhat com <mailto:nhosoi redhat com>> wrote:

    Eddie C wrote:
    > All,
    >
    > I tested this from scratch.
    > I used the ldif_2db function which did work much faster! 112
    > seconds...rather then about 20 minutes.
    > However I think the verify_db and db2_index functions are not in
    > agreement.
    >
    > Create indexes.
    > After my initial import.
    > [root ldap3 slapd-ldap3]# more out.after.final.txt
    > *****************************************************************
    > verify-db: This tool should only be run if recovery start fails
    > and the server is down.  If you run this tool while the server is
    > running, you may get false reports of corrupted files or other
    > false errors.
    > *****************************************************************
    > Verify log files in db ... Good
    > Verify db/o_idsk_com/id2entry.db4 ... Good
    > Verify db/userRoot/ancestorid.db4 ... Good
    > Verify db/userRoot/entrydn.db4 ... Good
    > Verify db/userRoot/cn.db4 ... Good
    > Verify db/userRoot/numsubordinates.db4 ... Good
    > Verify db/userRoot/aci.db4 ... Good
    > Verify db/userRoot/parentid.db4 ... Good
    > Verify db/userRoot/objectclass.db4 ... Good
    > Verify db/userRoot/id2entry.db4 ... Good
    > Verify db/userRoot/nsUniqueId.db4 ... Good
    > Verify db/idsk_services/ancestorid.db4 ...
    > DB ERROR: db_verify: Page 4: out-of-order key at entry 252
    > DB ERROR: db_verify: Page 7: out-of-order key at entry 194
    > DB ERROR: db_verify: Page 7: out-of-order key at entry 450
    > DB ERROR: db_verify: Page 11: out-of-order key at entry 69
    > DB ERROR: db_verify: Page 11: out-of-order key at entry 325
    > DB ERROR: db_verify: Page 11: out-of-order key at entry 581
    > DB ERROR: db_verify: Page 12: out-of-order key at entry 22
    > DB ERROR: db_verify: Page 16: out-of-order key at entry 249
    > DB ERROR: db_verify: Page 16: out-of-order key at entry 498
    > DB ERROR: db_verify: Page 16: out-of-order key at entry 754
    > DB ERROR: db_verify: Page 17: out-of-order key at entry 195
    > DB ERROR: db_verify: Page 17: out-of-order key at entry 451
    > DB ERROR: db_verify: Page 17: out-of-order key at entry 707
    > DB ERROR: db_verify: Page 18: out-of-order key at entry 148
    > DB ERROR: db_verify: Page 21: out-of-order key at entry 254
    > DB ERROR: db_verify: Page 21: out-of-order key at entry 510
    > DB ERROR: db_verify: Page 21: out-of-order key at entry 766
    > DB ERROR: db_verify: Page 22: out-of-order key at entry 207
    > DB ERROR: db_verify: Page 22: out-of-order key at entry 463
    > DB ERROR: db_verify: Page 22: out-of-order key at entry 719
    > DB ERROR: db_verify: Page 23: out-of-order key at entry 160
    > DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4:
    > DB_VERIFY_BAD: Database verification failed
    > Secondary index file ancestorid.db4 in db/idsk_services is
    corrupted.
    > Please run db2index(.pl) for reindexing.
    >
    Is this after you imported your ldif file to the backend
    'idsk_services'?

    When you imported the ldif file, did you get any errors from ldif2db?
    This is an sample output when we import Example.ldif.  Did you see any
    messages other than these?
    [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
    pages: 1009265, procpages: 23476
    [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache:
    204800k
    [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
    import_pages: 51200, pagesize: 4096
    [18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with
    nsslapd-db-private-import-mem on; No other process is allowed to
    access
    the database
    [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
    pages: 1009265, procpages: 23476
    [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
    [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
    import_pages: 51200, pagesize: 4096
    [18/Dec/2006:04:00:49 -0800] - import example: Beginning import job...
    [18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled
    with bucket size 100
    [18/Dec/2006:04:00:49 -0800] - import example: Processing file
    "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
    [18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file
    "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
    (160
    entries)
    [18/Dec/2006:04:00:50 -0800] - import example: Workers finished;
    cleaning up...
    [18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up.
    [18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer
    thread...
    [18/Dec/2006:04:00:50 -0800] - import example: Indexing complete.
    Post-processing...
    [18/Dec/2006:04:00:50 -0800] - import example: Flushing caches...
    [18/Dec/2006:04:00:50 -0800] - import example: Closing files...
    [18/Dec/2006:04:00:51 -0800] - All database threads now stopped
    [18/Dec/2006:04:00:51 -0800] - import example: Import complete.
    Processed 160 entries in 2 seconds. (80.00 entries/sec)

    Thanks,
    --noriko
    > So then i reran db2index....verify_db again...same result.
    >
    > Edward
    >
    >
    > On 12/15/06, *Eddie C* <edlinuxguru gmail com
    <mailto:edlinuxguru gmail com>
    > <mailto: edlinuxguru gmail com <mailto:edlinuxguru gmail com>>>
    wrote:
    >
    >     I recently did an ldif backup of our iplanet 52 database. Its
    >     about an 88 MB ldif file.
    >     I took this to a new FDS server Dell 850 3 ghz duel core 2 sata
    >     hard disks.
    >     I ran an ldapadd  the data imported perfectly.
    >     Then I tried to cutover some systems and give the database
    some load.
    >
    >     System went 200% processor
    >
    >     Eventually I realized I was missing indexes so I added them
    >     through the graphical tool.
    >
    >     The log seemed to do something like this
    >     generating index 1%
    >     generating index 2%
    >     ....
    >     generating index 49%
    >     Done
    >     Seemed weird that they would jump from 49% to Done
    >     At this point the new system was running at 100% processor
    >     But the queries are running faster on our old 440 MHZ sparc t1
    >     server52 database
    >
    >     I ran
    >     DB ERROR: db_verify: Page 30: out-of-order key at entry 498
    >     DB ERROR: db_verify: DB->verify:
    db/o_com/channelcontentowner.db4:
    >     DB_VERIFY_BAD: Database verification failed
    >
    >     then I tried db2_index. The program seemed to be in a tight loop
    >     complaining about 1 missing entry.
    >
    >     I do not realize how the data can be so corrupted right after an
    >     import.
    >
    >     These are someone generic symptoms. Any ideas? Thanks
    >
    >
    >
    ------------------------------------------------------------------------
    >
    > --
    > Fedora-directory-users mailing list
    > Fedora-directory-users redhat com
    <mailto:Fedora-directory-users redhat com>
    > https://www.redhat.com/mailman/listinfo/fedora-directory-users
    <https://www.redhat.com/mailman/listinfo/fedora-directory-users>
    >



    --
    Fedora-directory-users mailing list
    Fedora-directory-users redhat com
    <mailto:Fedora-directory-users redhat com>
    https://www.redhat.com/mailman/listinfo/fedora-directory-users




------------------------------------------------------------------------

--
Fedora-directory-users mailing list
Fedora-directory-users redhat com
https://www.redhat.com/mailman/listinfo/fedora-directory-users

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]