출처 : http://web.bii.a-star.edu.sg/~james/100TB/14TB/faults/2007-03-05.html
Fault Log : 5th Mar 2007
Description : Disk warning on storage node 1, port 1
One of the drives on storage node 1 developed a hardware fault which was detected by automated scripts. A notification email was sent from the management node at 19:38 today. The following message was observed in storage node 1's /var/log/messages :
- Mar 5 19:28:21 linux kernel: [12972720.367977] 3w-9xxx: scsi0: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=1.
The following command shows the drive error :
# ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 OK* - - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK WARNING - p1 - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744 # smartctl -a -d 3ware,1 /dev/twa0 smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: HDS725050KLA360 Serial Number: KRVN03ZAHRS7BD Firmware Version: K2AOAB0A User Capacity: 500,107,862,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 T13 1532D revision 1 Local Time is: Thu Mar 8 13:04:40 2007 SGT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (10419) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 174) minutes. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0 3 Spin_Up_Time 0x0007 106 106 024 Pre-fail Always - 655 (Average 655) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 41 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 2 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 100 100 020 Pre-fail Offline - 0 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 3826 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 41 192 Power-Off_Retract_Count 0x0032 100 100 050 Old_age Always - 195 193 Load_Cycle_Count 0x0012 100 100 050 Old_age Always - 195 194 Temperature_Celsius 0x0002 171 171 000 Old_age Always - 32 (Lifetime Min/Max 18/36) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 2 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] Warning! SMART Selective Self-Test Log Structure error: invalid SMART checksum. SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
It appears that the 3ware card has not marked the disk as "failed", since the RAID5 array was not considered degraded. I would like to force the controller to rebuild the disk, by "removing" and "reinserting" the disk. However, since a hotspare was already assigned, I would have to delete the hotspare drive first.
# ./tw_cli maint deleteunit c0 u1 Deleting unit c0/u1 ...Done. # ./tw_cli maint remove c0 p1 Exporting port /c0/p1 ... Done. # ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 DEGRADED - - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK DEGRADED - - - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744 # ./tw_cli maint rescan c0 Rescanning controller /c0 for units and drives ...Done. Found the following unit(s): [none]. Found the following drive(s): [/c0/p1]. # ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 REBUILDING 0 - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK DEGRADED - p15 - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744
It was then that I realized that despite removing the hotspare drive, the 3ware card had decided to use the disk on p15 for rebuilding the RAID5 volume. Thus, I had to perform another "removal" of the drive on p15, to force the 3ware in to using p1 for rebuilding. Note that there was a time delay of a few minutes before the 3ware decided to commence rebuilding on p1 (ie, DEGRADED -> REBUILD-PAUSED -> REBUILDING).
# ./tw_cli maint remove c0 p15 Exporting port /c0/p15 ... Done. # ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 DEGRADED - - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK DEGRADED - - - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744 # ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 REBUILD-PAUSED 0 - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK DEGRADED - p1 - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744 # ./tw_cli /c0/u0 show Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ----------------------------------------------------------------------- u0 RAID-5 REBUILDING 0 - 64K 6519.12 13671583744 u0-0 DISK OK - p0 - 465.651 976541696 u0-1 DISK DEGRADED - p1 - 465.651 976541696 u0-2 DISK OK - p2 - 465.651 976541696 u0-3 DISK OK - p3 - 465.651 976541696 u0-4 DISK OK - p4 - 465.651 976541696 u0-5 DISK OK - p5 - 465.651 976541696 u0-6 DISK OK - p6 - 465.651 976541696 u0-7 DISK OK - p7 - 465.651 976541696 u0-8 DISK OK - p8 - 465.651 976541696 u0-9 DISK OK - p9 - 465.651 976541696 u0-10 DISK OK - p10 - 465.651 976541696 u0-11 DISK OK - p11 - 465.651 976541696 u0-12 DISK OK - p12 - 465.651 976541696 u0-13 DISK OK - p13 - 465.651 976541696 u0-14 DISK OK - p14 - 465.651 976541696 u0/v0 Volume - - - - 6519.12 13671583744
Finally, I redesignate p15 as the hotspare.
# ./tw_cli maint rescan c0 Rescanning controller /c0 for units and drives ...Done. Found the following unit(s): [none]. Found the following drive(s): [/c0/p15]. # ./tw_cli maint createunit c0 rspare p15 Creating new unit on controller /c0 ... Done. The new unit is /c0/u1.
The following entries were found in /var/log/messages. They correspond to the actions mentioned above.
Mar 8 13:15:47 linux kernel: [13209543.278994] 3w-9xxx: scsi0: AEN: ERROR (0x04:0x0002): Degraded unit:unit=0, port=1. Mar 8 13:17:05 linux kernel: [13209621.293550] 3w-9xxx: scsi0: AEN: INFO (0x04:0x001A): Drive inserted:port=1. Mar 8 13:17:09 linux kernel: [13209625.704754] 3w-9xxx: scsi0: AEN: INFO (0x04:0x000B): Rebuild started:unit=0. Mar 8 13:22:51 linux kernel: [13209968.144051] 3w-9xxx: scsi0: AEN: ERROR (0x04:0x0002): Degraded unit:unit=0, port=15. Mar 8 13:25:39 linux kernel: [13210135.641076] 3w-9xxx: scsi0: AEN: INFO (0x04:0x000B): Rebuild started:unit=0. Mar 8 13:29:22 linux kernel: [13210358.314647] 3w-9xxx: scsi0: AEN: INFO (0x04:0x001A): Drive inserted:port=15. Mar 8 18:02:41 linux kernel: [13226756.373219] 3w-9xxx: scsi0: AEN: INFO (0x04:0x0005): Rebuild completed:unit=0.
'Linux > Tip&Tech' 카테고리의 다른 글
| apache 특정 ip log 남기지 않기 (0) | 2008.10.09 |
|---|---|
| mysql 접속 test with PHP (0) | 2008.10.09 |
| SUlinux 1.5 SSH 접근 BUG (0) | 2008.10.09 |
| Switch Cisco password recover (0) | 2008.10.09 |
| ubuntu mysql 설치시 termcap 에러 발생 해결 (0) | 2008.10.09 |