raid1 + lilo + boot

[Roger <grunky rockriver net>]

I have several systems with identical hardware that all have software 
raid1 running on RH9 on hot-swappable serial ata drives.

One of our machines has developed a problem recently. Running 'lilo -v' 
used to write out boot sectors info for both drivers - now it only does 
one drive.

lilo -v
LILO version 21.4-4, Copyright (C) 1992-1998 Werner Almesberger
'lba32' extensions Copyright (C) 1999,2000 John Coffman

boot = /dev/hda, map = /boot/map.0301
Reading boot sector from /dev/hda
Merging with /boot/boot.b
Mapping message file /boot/message
Boot image: /boot/vmlinuz-2.4.20-20.9smp
Mapping RAM disk /boot/initrd-2.4.20-20.9smp.img
Added 2.4.20-20.9smp *
Boot image: /boot/vmlinuz-2.4.20-20.9
Mapping RAM disk /boot/initrd-2.4.20-20.9.img
Added 2.4.20-20.9
/boot/boot.0300 exists - no backup copy made.
Writing boot sector.

On other machines(exact same hardware+partitions) 'lilo -v' writes info 
to BOTH disks.

LILO version 21.4-4, Copyright (C) 1992-1998 Werner Almesberger
'lba32' extensions Copyright (C) 1999,2000 John Coffman

boot = /dev/hda, map = /boot/map.0301
Reading boot sector from /dev/hda
Merging with /boot/boot.b
Mapping message file /boot/message
Boot image: /boot/vmlinuz-2.4.20-20.9smp
Mapping RAM disk /boot/initrd-2.4.20-20.9smp.img
Added 2.4.20-20.9smp *
Boot image: /boot/vmlinuz-2.4.20-20.9
Mapping RAM disk /boot/initrd-2.4.20-20.9.img
Added 2.4.20-20.9
/boot/boot.0300 exists - no backup copy made.
Writing boot sector.
boot = /dev/hdb, map = /boot/map.0341
Reading boot sector from /dev/hdb
Merging with /boot/boot.b
Mapping message file /boot/message
Boot image: /boot/vmlinuz-2.4.20-20.9smp
Mapping RAM disk /boot/initrd-2.4.20-20.9smp.img
Added 2.4.20-20.9smp *
Boot image: /boot/vmlinuz-2.4.20-20.9
Mapping RAM disk /boot/initrd-2.4.20-20.9.img
Added 2.4.20-20.9
/boot/boot.0340 exists - no backup copy made.
Writing boot sector.

Since a number of machines have identical hardware+setup I took the 
lilo.conf file from another system and put it in.

Still lilo just write partition info to hda and not hdb.

Below I have the output of several setup files all take from other 
systems. Keep in mind the systems, partitions, /proc/md5stat, 
/etc/raidtab, are EXACTLY the same. I've double and tripled checked - 
used diff to..

Basically lilo isn't writing its info to /dev/hdb anymore - for what 
reason is beyond me.

I've used raidhotremove and raidsetfaulty then re-added /dev/hda1 and 
same to /dev/hdb1 thinking that a 're-sync' would jar something loose.

Someone suggested a coping of the boot record..

dd if=/dev/hda1 of=/dev/hdb1 count=1 bs=512

didn't help either.

Fdisk on BOTH /dev/hda and /dev/hdb looks as following. Partitions are 
the same size, hdX1 is bootable, with the same types of partitions

Disk /dev/hda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 2677 21502971 fd Linux raid autodetect
/dev/hda2 2678 12355 77738535 fd Linux raid autodetect
/dev/hda3 12356 14331 15872220 fd Linux raid autodetect
/dev/hda4 14332 14593 2104515 f Win95 Ext'd (LBA)
/dev/hda5 14332 14592 2096451 fd Linux raid autodetect


/proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md5 : active raid1 hda5[0] hdb5[1]
2096384 blocks [2/2] [UU]

md3 : active raid1 hda3[0] hdb3[1]
15872128 blocks [2/2] [UU]

md2 : active raid1 hda2[0] hdb2[1]
77738432 blocks [2/2] [UU]

md1 : active raid1 hda1[0] hdb1[1]
21502848 blocks [2/2] [UU]

unused devices: <none>

/etc/lilo.conf
prompt
timeout=50
default=2.4.20-20.9smp
boot=/dev/md1
map=/boot/map
install=/boot/boot.b
message=/boot/message
#linear
lba32

image=/boot/vmlinuz-2.4.20-20.9smp
label=2.4.20-20.9smp
initrd=/boot/initrd-2.4.20-20.9smp.img
read-only
root=/dev/md1

image=/boot/vmlinuz-2.4.20-20.9
label=2.4.20-20.9
initrd=/boot/initrd-2.4.20-20.9.img
read-only
root=/dev/md1

/etc/raidtab
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda1
raid-disk 0
device /dev/hdb1
raid-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda2
raid-disk 0
device /dev/hdb2
raid-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda3
raid-disk 0
device /dev/hdb3
raid-disk 1
raiddev /dev/md5
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda5
raid-disk 0
device /dev/hdb5
raid-disk 1