Adventures with Advanced Format HDD's and third party firmware

Recently we bought a couple of cheap servers with 24 x 2.5″ SAS disks; and a large batch of 900GB 2.5″ 10K disks seperately.

These are for a (very cheap) development platform that doesn’t require vendor support or replacement components within a specific time. Hence buying off eBay.

Unfortunately; while the cheap as chips servers were great, the cheap disk batch was slightly less so.

They were sold as “Seagate Savvio 10K.5 900GB 2.5″ SAS Hard Drive, Model Number ST9900805SS” which are genuinely good enterprise drives that should work on any 6Gbps SAS backplane/controller.

However… while the labels on them confirmed they’re supposed to be that… the didn’t behave as such.

The first problem was that fdisk/cfdisk wouldn’t read them. It’d just error and bail. What was the issue? 520 byte sectors, aka. Advanced Format sectors.

520 or 528 byte sectors can be used on many enterprise HDD’s; though they’re very rarely used directly by O/S’es; rather they’re used as part of vendor specific and proprietary disk formats which include firmware/software level error detection. e.g. the extra 8 or 16 bytes are used for checksums or some other form of error detection mechanism.

The Linux kernel doesn’t support anything other than 512 byte sectors at this time. So this is the kind of kernel message you’ll get if you’ve got a disk that’s formatted with something else,

[48276.003988] sd 1:0:12:0: [sdm] Unsupported sector size 520.
[48276.003994] sd 1:0:12:0: [sdm] 0 512-byte logical blocks: (0 B/0 B)
[48276.003996] sd 1:0:12:0: [sdm] 520-byte physical blocks
[48276.005987] sdm: detected capacity change from 900185481216 to 0

This is a not uncommon problem. Even many enterprisey-style RAID controllers/cards for will use 520/528 byte sectors if the disks support them. You move them disks and expose them directly to the O/S and you’ll be unable to read or write to the disks.

This is easily fixed . On Linux you can use sg_format or st (from SeaGate Enterprise Tools download) to re-format the disk to 512-bytes. e.g.

sg_format --format --size=512 --verbose /dev/sdm

I did this for all of the disks without issue.

All of the disks also passed SeaGate Enterprise Tools initiated Drive Self Test (DST) routines without issue. In short… at a basic level the disks appeared to be fine.

But, while fdisk/cfdisk could interact with the disks after the 512-byte format, any attempt to write to the disk failed. They all appeared to be locked in some kind of read-only mode for some bizarre reason.

Long story short: it turned out that the firmware was not SeaGate firmware.

Using smartctl and the SeaGate Enterprise tools; this is what one of the disks would report when first inspected.

[root@san02 ~]# smartctl --all /dev/sdm
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-327.36.3.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              DKS5D-J900SS
Revision:             5F0B
User Capacity:        900,185,481,216 bytes [900 GB]
Logical block size:   512 bytes
Rotation Rate:        10000 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c50042c6e8af
Serial number:        6XS1DCPJ0000M207L70N
Device type:          disk
Transport protocol:   SAS
Local Time is:        Fri Nov 18 20:30:12 2016 AEST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     28 C
Drive Trip Temperature:        68 C

Manufactured in week 35 of year 2011
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  168
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  168
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 2020612814
  Blocks received from initiator = 1145904566
  Blocks read from cache and sent to initiator = 2128
  Number of read and write commands whose size  segment size = 0

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 35656.68
  number of minutes until next internal SMART test = 23

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   4039431666        0         0  4039431666          0       1050.719           0
write:         0        0         0         0          0       2829.753           0
verify: 26330877        0         0  26330877          0          0.000           0

Non-medium error count:       86


[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No self-tests have been logged

[root@san02 ~]# st -i /dev/sg12
	/dev/sg12
	Vendor = SEAGATE
	Product = DKS5D-J900SS
	Version = 5F0B
	Serial Number = 6XS1DCPJ
	Copyright = Copyright (c) 2012 Seagate All rights reserved
	SCSI Firmware = 04195F0B
	Servo RAM Release = 2011B106
	Servo ROM Release = 00000000
	Servo RAM Date = B106
	Servo ROM Date = 2011

	Blocksize = 512, Highblock = 1758174767, Capacity = 879087 MB
	-this is a Seagate drive
	-this drive supports DST
		-short DST time = 120 seconds
		-long DST time = 5 seconds
	-Mode Page Settings [current value (default)]:
		-WCE bit = 0 (1)
		-RCD bit = 0 (0)
		-AWRE bit = 0 (1)
		-ARRE bit = 0 (1)
		-DExcpt bit = 0 (0)
		-Number of cache segments = 3 (32)
		-PM  bit = 0 (0)
		-DLM bit = 0 (0)
		-DTOLI bit = 0 (0)
		-JIT bit 0 = 1 (1)
		-JIT bit 1 = 1 (1)
		-JIT bit 2 = 1 (1)
		-JIT bit 3 = 1 (1)

[root@san02 ~]#

Note the Product = DKS5D-J900SS which is NOT the ST9900805SS model that was expected.

A quick Google will lead to the conclusion that these disks are actually manufactured by Hitachi. What gives?

Firmware. They’re SeaGate disks with Hitachi firmware. Because they were pulled from a Hitachi storage system.

Many storage system providers reinvent the wheel and do their own things with disks; this includes writing customised firmware that does everything differently for no good reason.

After tracking down the latest official SeaGate firmware I did some research on doing HDD firmware updates from within Linux, which boil down to the following:

  1. Use official vendor tools if available, e.g. “st” from SeaGate Enterprise Tools
  2. Use ‘hdparm –fwdownload’
  3. Use ‘sg_write_buffer’ command

If you’ve got firmware from another vendor on you disk; you may or may not be able to use their tools.

e.g. because the disks are reporting a different product ID, they may fail.

[root@san02 firmware]# st -d CP-SAS-0004.LOD /dev/sg12
GetSGVersions - open error: No such file or directory
Downloading file CP-SAS-0004.LOD to drive /dev/sg12
	Drive PRODUCT string does not match download file - Firmware Download aborted.
[root@san02 firmware]#

So you try hdparm… which may not work because the firmware on the disk doesn’t behave as hdparm expects.

[root@san02 firmware]# sg_map | grep sg12
/dev/sg12  /dev/sdm
[root@san02 firmware]# hdparm --fwdownload CP-SAS-0004.LOD --yes-i-know-what-i-am-doing --please-destroy-my-drive /dev/sdm

/dev/sdm:
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 13 00 00 00 00 20 00 01 cf 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
DOWNLOAD_MICROCODE: not supported by device
[root@san02 firmware]#

So you resort to sg_write_buffer… which doesn’t do much more than blindly dump code to the disk… not safe… but if you’re certain the firmware file is legit, appropriate for the drive, and isn’t corrupted it should be OK.

[root@san02 firmware]# st -l | grep sg12
/dev/sg12 SEAGATE  DKS5D-J900SS     5F0B 1758174767 blocks
[root@san02 firmware]# sg_write_buffer -v -m 5 -I CP-SAS-0004.LOD /dev/sg12
tried to read 8388608 bytes from CP-SAS-0004.LOD, got 1238016 bytes
will write 1238016 bytes
sending single write buffer, mode=0x5, mpsec=0, id=0, offset=0, len=1238016
    Write buffer cmd: 3b 05 00 00 00 00 12 e4 00 00
[root@san02 firmware]# st -l | grep sg12
/dev/sg12 SEAGATE  ST9900805SS      0004 1758174767 blocks
[root@san02 firmware]#

Now the disks report like this,

[root@san02 firmware]# smartctl --all /dev/sdm
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-327.36.3.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SEAGATE
Product:              ST9900805SS
Revision:             0004
User Capacity:        900,185,481,216 bytes [900 GB]
Logical block size:   512 bytes
Rotation Rate:        10000 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c50042c6e8af
Serial number:        6XS1DCPJ0000M207L70N
Device type:          disk
Transport protocol:   SAS
Local Time is:        Fri Nov 18 20:57:36 2016 AEST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     30 C
Drive Trip Temperature:        68 C

Manufactured in week 35 of year 2011
Specified cycle count over device lifetime:  10000
Accumulated start-stop cycles:  1
Specified load-unload count over device lifetime:  300000
Accumulated load-unload cycles:  1
Elements in grown defect list: 0

Vendor (Seagate) cache information
  Blocks sent to initiator = 0
  Blocks received from initiator = 392901
  Blocks read from cache and sent to initiator = 0
  Number of read and write commands whose size  segment size = 0

Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 35657.13
  number of minutes until next internal SMART test = 48

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0          0.000           0
write:         0        0         0         0          0          0.201           0

Non-medium error count:        1


[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No self-tests have been logged

[root@san02 firmware]# st -i /dev/sg12
	/dev/sg12
	Vendor = SEAGATE
	Product = ST9900805SS
	Version = 0004
	Serial Number = 6XS1DCPJ
	Copyright = Copyright (c) 2012 Seagate All rights reserved
	SCSI Firmware = 03160004
	Servo RAM Release = 2011B107
	Servo ROM Release = 00000000
	Servo RAM Date = B107
	Servo ROM Date = 2011

	Blocksize = 512, Highblock = 1758174767, Capacity = 879087 MB
	-this is a Seagate drive
	-this drive supports DST
		-short DST time = 120 seconds
		-long DST time = 7560 seconds
	-Mode Page Settings [current value (default)]:
		-WCE bit = 0 (1)
		-RCD bit = 0 (0)
		-AWRE bit = 0 (1)
		-ARRE bit = 0 (1)
		-DExcpt bit = 0 (0)
		-Number of cache segments = 32 (32)
		-PM  bit = 0 (0)
		-DLM bit = 0 (0)
		-DTOLI bit = 0 (0)
		-JIT bit 0 = 1 (1)
		-JIT bit 1 = 1 (1)
		-JIT bit 2 = 1 (1)
		-JIT bit 3 = 1 (1)

[root@san02 firmware]#

Writes by fdisk/cfdisk work; and the partitions can be formatted using mkfs.* tools.

Best of luck…

UPDATE 2018/03/15 - For anyone looking for the firmware file CP-SAS-0004.LOD you can grab the original Seagate .zip I pulled that from my Google Drive here

Author image
About Colin Stubbs
Brisbane, Queensland, Australia
Space monkey meat popsicle with technology and noise addictions.