[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Fedora-directory-users] Re: Migration from i-planet 52



>>Any errors?
No as far as I can tell the import goes smooth. Our upstream applications seem to have no problem with the data it seems to be all imported. 
 
I am really troubleshooting and index and performance issues, I figure out of order keys could be slowing searches down. This a fairly large db 160000 entires) some objects have upwards of 20 attributes.
 
db_verify and db_2index both run fairly quickly. It just seems like they cant both agree on what clean data looks like.
 
FWI I upgraded to the latest 1.0.4 before all this testing.
 
[18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import job...
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering enabled with bucket size 15
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file "/opt/idsk/downloads/idsk.services.com.ldif"
[18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning file "/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries)
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished; cleaning up...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up.
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up producer thread...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing complete.  Post-processing...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files...
[18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete.  Processed 2037 entries in 1 seconds. ( 2037.00 entries/sec)
[18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline...
[18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job...
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering enabled with bucket size 15
[18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file "/opt/idsk/downloads/idsk.com.ldif"
[18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288 entries -- average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0%
[18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956 entries -- average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100%
[18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning file "/opt/idsk/downloads/idsk.com.ldif" (163684 entries)
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished; cleaning up...
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up.
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer thread...
[18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete.  Post-processing...
[18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches...
[18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files...
[18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete.  Processed 163684 entries in 116 seconds. ( 1411.07 entries/sec)

Edward
 
On 12/18/06, Noriko Hosoi <nhosoi redhat com> wrote:
Eddie C wrote:
> All,
>
> I tested this from scratch.
> I used the ldif_2db function which did work much faster! 112
> seconds...rather then about 20 minutes.
> However I think the verify_db and db2_index functions are not in
> agreement.
>
> Create indexes.
> After my initial import.
> [root ldap3 slapd-ldap3]# more out.after.final.txt
> *****************************************************************
> verify-db: This tool should only be run if recovery start fails
> and the server is down.  If you run this tool while the server is
> running, you may get false reports of corrupted files or other
> false errors.
> *****************************************************************
> Verify log files in db ... Good
> Verify db/o_idsk_com/id2entry.db4 ... Good
> Verify db/userRoot/ancestorid.db4 ... Good
> Verify db/userRoot/entrydn.db4 ... Good
> Verify db/userRoot/cn.db4 ... Good
> Verify db/userRoot/numsubordinates.db4 ... Good
> Verify db/userRoot/aci.db4 ... Good
> Verify db/userRoot/parentid.db4 ... Good
> Verify db/userRoot/objectclass.db4 ... Good
> Verify db/userRoot/id2entry.db4 ... Good
> Verify db/userRoot/nsUniqueId.db4 ... Good
> Verify db/idsk_services/ancestorid.db4 ...
> DB ERROR: db_verify: Page 4: out-of-order key at entry 252
> DB ERROR: db_verify: Page 7: out-of-order key at entry 194
> DB ERROR: db_verify: Page 7: out-of-order key at entry 450
> DB ERROR: db_verify: Page 11: out-of-order key at entry 69
> DB ERROR: db_verify: Page 11: out-of-order key at entry 325
> DB ERROR: db_verify: Page 11: out-of-order key at entry 581
> DB ERROR: db_verify: Page 12: out-of-order key at entry 22
> DB ERROR: db_verify: Page 16: out-of-order key at entry 249
> DB ERROR: db_verify: Page 16: out-of-order key at entry 498
> DB ERROR: db_verify: Page 16: out-of-order key at entry 754
> DB ERROR: db_verify: Page 17: out-of-order key at entry 195
> DB ERROR: db_verify: Page 17: out-of-order key at entry 451
> DB ERROR: db_verify: Page 17: out-of-order key at entry 707
> DB ERROR: db_verify: Page 18: out-of-order key at entry 148
> DB ERROR: db_verify: Page 21: out-of-order key at entry 254
> DB ERROR: db_verify: Page 21: out-of-order key at entry 510
> DB ERROR: db_verify: Page 21: out-of-order key at entry 766
> DB ERROR: db_verify: Page 22: out-of-order key at entry 207
> DB ERROR: db_verify: Page 22: out-of-order key at entry 463
> DB ERROR: db_verify: Page 22: out-of-order key at entry 719
> DB ERROR: db_verify: Page 23: out-of-order key at entry 160
> DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4:
> DB_VERIFY_BAD: Database verification failed
> Secondary index file ancestorid.db4 in db/idsk_services is corrupted.
> Please run db2index(.pl) for reindexing.
>
Is this after you imported your ldif file to the backend 'idsk_services'?

When you imported the ldif file, did you get any errors from ldif2db?
This is an sample output when we import Example.ldif.  Did you see any
messages other than these?
[18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
pages: 1009265, procpages: 23476
[18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
[18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
import_pages: 51200, pagesize: 4096
[18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with
nsslapd-db-private-import-mem on; No other process is allowed to access
the database
[18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096,
pages: 1009265, procpages: 23476
[18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k
[18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50,
import_pages: 51200, pagesize: 4096
[18/Dec/2006:04:00:49 -0800] - import example: Beginning import job...
[18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled
with bucket size 100
[18/Dec/2006:04:00:49 -0800] - import example: Processing file
"/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif"
[18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file
"/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" (160
entries)
[18/Dec/2006:04:00:50 -0800] - import example: Workers finished;
cleaning up...
[18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up.
[18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer
thread...
[18/Dec/2006:04:00:50 -0800] - import example: Indexing complete.
Post-processing...
[18/Dec/2006:04:00:50 -0800] - import example: Flushing caches...
[18/Dec/2006:04:00:50 -0800] - import example: Closing files...
[18/Dec/2006:04:00:51 -0800] - All database threads now stopped
[18/Dec/2006:04:00:51 -0800] - import example: Import complete.
Processed 160 entries in 2 seconds. (80.00 entries/sec)

Thanks,
--noriko
> So then i reran db2index....verify_db again...same result.
>
> Edward
>
>
> On 12/15/06, *Eddie C* <edlinuxguru gmail com
> <mailto: edlinuxguru gmail com>> wrote:
>
>     I recently did an ldif backup of our iplanet 52 database. Its
>     about an 88 MB ldif file.
>     I took this to a new FDS server Dell 850 3 ghz duel core 2 sata
>     hard disks.
>     I ran an ldapadd  the data imported perfectly.
>     Then I tried to cutover some systems and give the database some load.
>
>     System went 200% processor
>
>     Eventually I realized I was missing indexes so I added them
>     through the graphical tool.
>
>     The log seemed to do something like this
>     generating index 1%
>     generating index 2%
>     ....
>     generating index 49%
>     Done
>     Seemed weird that they would jump from 49% to Done
>     At this point the new system was running at 100% processor
>     But the queries are running faster on our old 440 MHZ sparc t1
>     server52 database
>
>     I ran
>     DB ERROR: db_verify: Page 30: out-of-order key at entry 498
>     DB ERROR: db_verify: DB->verify: db/o_com/channelcontentowner.db4:
>     DB_VERIFY_BAD: Database verification failed
>
>     then I tried db2_index. The program seemed to be in a tight loop
>     complaining about 1 missing entry.
>
>     I do not realize how the data can be so corrupted right after an
>     import.
>
>     These are someone generic symptoms. Any ideas? Thanks
>
>
> ------------------------------------------------------------------------
>
> --
> Fedora-directory-users mailing list
> Fedora-directory-users redhat com
> https://www.redhat.com/mailman/listinfo/fedora-directory-users
>



--
Fedora-directory-users mailing list
Fedora-directory-users redhat com
https://www.redhat.com/mailman/listinfo/fedora-directory-users





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]