[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Confirmed premature hard disk failure



Now one occurrence doesn't prove it was the Load_Cycle_Count problem
that killed the drive, but this drive is definitely prematurely failed.
This is going to be spammy, I think it is important to document this so
we can all learn from it...

The drive I posted about a while back:

https://www.redhat.com/archives/fedora-devel-list/2008-July/msg00426.html

Is now definitely failing as of about 3-4 weeks ago. It's at the stage
where the drive still seems to be mostly readable, but IO bandwidth has
dropped like a rock. Booting Fedora takes like 20 minutes. Same thing
two previous failed drives did.

The drive has a manufacture date of May 2007. I have no reason to
believe the drive has been mistreated, no more than typical laptop wear
and tear. It is a "desktop replacement" laptop and is rarely
transported.

Fortunately /home seems to be readable. Media errors show up in the
syslog, but no IO errors are turning up in user space. The WinXP
partition seems to be trashed, however. So much for my wife's Half Life
2 and Spore savegames...

ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: BMDMA stat 0x25
ata1.00: cmd 25/00:00:00:58:08/00:04:00:00:00/e0 tag 0 dma 524288 in
         res 51/40:0f:f1:59:08/40:02:00:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
ata1.00: configured for UDMA/100
sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
        00 08 59 f1 
sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
end_request: I/O error, dev sda, sector 547313
ata1: EH complete
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Full smartctl -a output, after salvaging some data:

smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Scorpio family
Device Model:     WDC WD600VE-11KWT0
Serial Number:    WD-WXEY06009078
Firmware Version: 01.03K01
User Capacity:    60,011,642,880 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Dec 30 17:09:36 2008 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline 
data collection: 		 (3180) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					No General Purpose Logging support.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  44) minutes.
Conveyance self-test routine
recommended polling time: 	 (   6) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   198   198   051    Pre-fail  Always       -       5472
  3 Spin_Up_Time            0x0003   150   119   021    Pre-fail  Always       -       1466
  4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3359
  5 Reallocated_Sector_Ct   0x0033   197   197   140    Pre-fail  Always       -       19
  7 Seek_Error_Rate         0x000f   200   200   051    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       6564
 10 Spin_Retry_Count        0x0012   092   085   051    Old_age   Always       -       22
 11 Calibration_Retry_Count 0x0012   100   100   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       359
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       350
193 Load_Cycle_Count        0x0032   061   061   000    Old_age   Always       -       419536
194 Temperature_Celsius     0x0022   087   071   000    Old_age   Always       -       56
196 Reallocated_Event_Count 0x0032   196   196   000    Old_age   Always       -       4
197 Current_Pending_Sector  0x0012   192   190   000    Old_age   Always       -       86
198 Offline_Uncorrectable   0x0010   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0009   200   200   051    Pre-fail  Offline      -       3

SMART Error Log Version: 1
ATA Error Count: 2803 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2803 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 f1 41 e8 e2  Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 40 c9 41 e8 02 08      01:07:07.212  READ DMA
  27 00 00 00 00 00 00 08      01:07:07.211  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 0a      01:07:07.208  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 0a      01:07:07.206  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 08      01:07:07.205  READ NATIVE MAX ADDRESS EXT

Error 2802 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 f1 41 e8 e2  Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 40 c9 41 e8 02 08      01:07:04.430  READ DMA
  27 00 00 00 00 00 00 08      01:07:04.429  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 0a      01:07:04.427  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 0a      01:07:04.427  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 08      01:07:04.426  READ NATIVE MAX ADDRESS EXT

Error 2801 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 f1 41 e8 e2  Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 40 c9 41 e8 02 08      01:07:01.254  READ DMA
  27 00 00 00 00 00 00 08      01:07:01.254  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 0a      01:07:01.251  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 0a      01:07:01.248  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 08      01:07:01.248  READ NATIVE MAX ADDRESS EXT

Error 2800 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 f1 41 e8 e2  Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 40 c9 41 e8 02 08      01:06:58.277  READ DMA
  27 00 00 00 00 00 00 08      01:06:58.277  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 0a      01:06:58.274  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 0a      01:06:58.271  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 08      01:06:58.271  READ NATIVE MAX ADDRESS EXT

Error 2799 occurred at disk power-on lifetime: 6563 hours (273 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 18 f1 41 e8 e2  Error: UNC 24 sectors at LBA = 0x02e841f1 = 48775665

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 40 c9 41 e8 02 08      01:06:55.684  READ DMA
  27 00 00 00 00 00 00 08      01:06:55.684  READ NATIVE MAX ADDRESS EXT
  ec 00 00 00 00 00 00 0a      01:06:55.681  IDENTIFY DEVICE
  ef 03 45 00 00 00 00 0a      01:06:55.678  SET FEATURES [Set transfer mode]
  27 00 00 00 00 00 00 08      01:06:55.678  READ NATIVE MAX ADDRESS EXT

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%      6564         789370
# 2  Conveyance offline  Completed: read failure       90%      6564         789370
# 3  Extended offline    Completed: read failure       90%      6562         789370
# 4  Extended offline    Completed: read failure       90%      6562         789369
# 5  Extended offline    Completed: read failure       90%      6560         85755114
# 6  Extended offline    Completed without error       00%      6455         -
# 7  Extended offline    Interrupted (host reset)      60%      6455         -
# 8  Extended offline    Completed without error       00%      4402         -
# 9  Extended offline    Completed without error       00%         0         -
#10  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Self tests 1-4 were done just today. Test 5 was done when the drive
first started failing 3-4 weeks ago, after which the machine was kept
off until now. 6 and 7 are from when I first noticed
Current_Pending_Sector was nonzero, maybe a month or two ago. 8 is from
my post in July. 9 and 10 are from when the drive was new.

Things to note:

1) The SMART health status is PASSED. Just like every other failed drive
I've had...

2) None of the attributes are failed.

3) By the time the self tests start failing, it is too late, you are
already losing data.

We can not trust the drive to assess itself properly. If we ever get a
SMART desktop alert thing going, we MUST look directly at the
attributes, and come to our own conclusion.

Any questions before I RMA the drive?

Attachment: signature.asc
Description: This is a digitally signed message part


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]