HDD Testing on GNU/Linux & macOS

HDD Testing on Linux and macOS.md

HDD Testing on GNU/Linux & macOS

Author: Rohan Barar
Date: 10/09/2024

Surface Testing

Warning

Performing surface tests on SSDs is a bad idea given the risk of unnecessary wear. SSDs have a limited number of write cycles. Surface tests, which repeatedly write and read data to every block, can significantly reduce the lifespan of a SSD.

badblocks is a command-line program within e2fsprogs, a set of utilities for maintaining ext2, ext3 and ext4 file systems. badblocks can be used to perform surface tests on HDDs to check the entire disk surface for bad sectors.

A port of e2fsprogs is available for macOS, and can be installed via Homebrew.

1. Identify Target Disk

GNU/Linux

lsblk

Example GNU/Linux Output:

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0   2.7T  0 disk
├─sda1        8:1    0   2.7T  0 part /media/HDD
└─sda2        8:2    0    32M  0 part
sdb           8:16   0 931.5G  0 disk
├─sdb1        8:17   0   200M  0 part
└─sdb2        8:18   0 931.2G  0 part
zram0       252:0    0   7.5G  0 disk [SWAP]
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0   256M  0 part /boot/efi
├─nvme0n1p2 259:2    0   512M  0 part /boot
└─nvme0n1p3 259:3    0 476.2G  0 part /home
                                      /

macOS

diskutil list

Example macOS Output

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk0
   1:             Apple_APFS_ISC Container disk1         524.3 MB   disk0s1
   2:                 Apple_APFS Container disk3         994.7 GB   disk0s2
   3:        Apple_APFS_Recovery Container disk2         5.4 GB     disk0s3

/dev/disk3 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +994.7 GB   disk3
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD            9.6 GB     disk3s1
   2:              APFS Snapshot com.apple.os.update-... 9.6 GB     disk3s1s1
   3:                APFS Volume Preboot                 9.7 GB     disk3s2
   4:                APFS Volume Recovery                1.6 GB     disk3s3
   5:                APFS Volume Data                    777.0 GB   disk3s5
   6:                APFS Volume VM                      4.3 GB     disk3s6

/dev/disk4 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                                                   *1.0 TB     disk4

In this case, the desired storage device is /dev/sdb on GNU/Linux and /dev/disk4 on macOS. All subsequent example command snippets will use this device.

2. Determine Device Block Size

badblocks assumes a default value of 1024 bytes, but according to this StackExchange answer, this should be specified to match what the device reports. This is because if the device block size assumed by badblocks is larger than the true device block size, the integrity of badblocks results can be compromised. On the other hand, assuming a device block size smaller than the true device block size will make running the surface test unnecessarily slow.

GNU/Linux

sudo blockdev --getbsz /dev/sdb

Example GNU/Linux Output

512

macOS

diskutil info /dev/disk4

Example macOS Output

Device Identifier:         disk4
Device Node:               /dev/disk4
Whole:                     Yes
Part of Whole:             disk4
Device / Media Name:       Portable

Volume Name:               Not applicable (no file system)
Mounted:                   Not applicable (no file system)
File System:               None

Content (IOContent):       None
OS Can Be Installed:       No
Media Type:                Generic
Protocol:                  USB
SMART Status:              Not Supported

Disk Size:                 1.0 TB (1000204886016 Bytes) (exactly 1953525168 512-Byte-Units)
Device Block Size:         512 Bytes

Media OS Use Only:         No
Media Read-Only:           No
Volume Read-Only:          Not applicable (no file system)

Device Location:           External
Removable Media:           Fixed

Solid State:               Info not available
Virtual:                   No

In this case, the device block size is 512 bytes.

3. Determine Number of Blocks to Test Simultaneously

According to this StackExchange answer, this option does not affect the integrity of your results, but it does affect the speed at which badblocks runs. If this value is too small, badblocks will take an unnecessarily long time to run (since queueing and processing a separate IO request incurs overhead). On the other hand, setting a value that is too high may cause badblocks to run out of memory. The default value is 64, but can be increased significantly. Values used by others online include 32768 and 65536.

4. Run badblocks

In this example, we will:

• -w: Run a destructive write-mode test. This involves scanning for bad blocks by writing some patterns on every block of the device, reading every block and comparing the contents. Destructive write-mode tests WILL OVERWRITE ALL DATA ON THE DISK!!!
• -s: Show the progress of the scan by writing out a rough percentage completion of the current badblocks pass over the disk to the console.
• -v: Run in verbose mode.
• -b: Specify the correct device block size of 512 bytes.
• -c: Request 65536 blocks be tested at a time.
• -t: (Optional) Use a single custom test pattern (e.g. 0xDEADBEEF).

By default, badblocks writes four test patterns (0xaa, 0x55, 0xff, and 0x00) to each block when used with the -w flag, verifying the functionality of each block. This behavior can be customized with a specific test pattern, but it is recommended to use at least two different patterns. As noted in this StackExchange answer, multiple patterns help detect certain errors, such as a block stuck storing a value like 0xff, which may not be flagged as bad if only the 0xff pattern is tested.

macOS

sudo badblocks -wsv -b 512 -c 65536 -t 0xDEADBEEF /dev/rdisk4 # Use raw (unbuffered) device node

GNU/Linux

sudo badblocks -wsv -b 512 -c 65536 -t 0xDEADBEEF /dev/sdb

Other badblocks parameters and options can be found here.

SMART Monitoring

Although badblocks may not find any bad sectors, this does not necessarily mean that the HDD is healthy or without damage. This is because HDDs use a technique called "bad sector management" to detect and remap bad sectors. This process involves identifying problematic sectors and remapping their logical block addresses (LBAs) to spare sectors from a reserved pool. Since this process is handled internally by the HDD firmware, remapping of bad sectors is entirely transparent to the Operating System.

Although it is not possible to obtain details regarding which exact sectors have been remapped, it is possible to use SMART (Self-Monitoring, Analysis and Reporting Technology) to obtain a count of how many bad sectors a HDD has remapped (Reallocated Sectors Count), alongside several other diagnostic and monitoring values. More information about SMART can be found here.

An increasing number of remapped sectors is indicative of impending HDD failure, and so running a SMART self-test can be useful when evaluating the health of a HDD.

macOS-Specific Configuration Steps

Unfortunately, SMART support for external HDDs is not included by default in macOS, and requires the installation of a third-party kernel extension. On Apple Silicon, this requires modifying the system security policy.

1. Boot into Recovery Mode.
2. Select Startup Security Utility from Utilities.
3. Select Reduced Security and enable Allow user management of kernel extensions from identified developers.
4. Reboot.

Once this is done, the SAT SMART Driver can be installed. This is an open-source driver that provides access to SMART information for external USB and FireWire drives. More information about the driver can be found on Github. To simplify the driver installation process, a signed version of the SAT SMART Driver is available from BinaryFruit. The signed driver installer can be found here.

Even with SAT SMART Driver installed, SMART information cannot be accessed via Disk Utility. Instead, either DriveDx (available here) or smartmontools (available via Homebrew) need to be used. Since DriveDx is not free software, it is best to install the FOSS smartmontools. To avoid having to learn CLI commands to operate smartmontools, an open-source multi-platform graphical user interface called GSmartControl can also be installed via Homebrew.

Note that GSmartControl will only be able to use smartmontools if Homebrew is added to PATH via a file at /etc/paths.d/homebrew instead of via .bash_profile.

More specifically, instead of having the following inside .bash_profile:

export PATH="~/opt/homebrew/bin:$PATH"
export PATH="~/opt/homebrew/sbin:$PATH"

A file named homebrew should be created at /etc/paths.d containing the following:

/Users/RohanBarar/opt/homebrew/bin
/Users/RohanBarar/opt/homebrew/sbin

Once this is done, the relevant lines in .bash_profile can be commented out or removed entirely.

Note: Your Homebrew installation path will differ, so modify the above two code blocks as necessary.

Finally, GSmartControl can be run by typing gsmartcontrol in Terminal.

Extended SMART Test Bash Script

The following bash script utilises smartmontools to perform an extended SMART test on a selected HDD or SSD. This script should run on both GNU/Linux and macOS, but has only been tested on GNU/Linux.

#!/usr/bin/env bash

# Define constants.
readonly INTERVAL=1 # Status update interval in seconds.
readonly RUNNING_STRING_1="Self-test routine in progress..."
readonly RUNNING_STRING_2="Extended self-test in progress"
readonly BRIGHT_CYAN="\033[96m"
readonly BOLD_BRIGHT_CYAN="\033[1;96m"
readonly BOLD_RED="\033[1;31m"
readonly BOLD_GREEN="\033[1;32m"
readonly GREEN="\033[38;2;0;255;0m"
readonly BOLD_YELLOW="\033[1;33m"
readonly RESET="\033[0m"

# Define global variables.
TEST_RUNNING=0
DEVICE_NUM=0
DEVICE=""
INIT_OUTPUT=""
OFFLINE_MODE=0
DURAT_MINS=0
DURAT_SECS=0
END_TIME=0
PCENT=0
TSTAMP=""
INFO=""

# TRAPS
# SIGINT (Ctrl+C)
trap clean_up SIGINT

# FUNCTIONS
# Cleanup on SIGINT (Ctrl + C).
function clean_up() {
    if [ "$TEST_RUNNING" -eq 1 ]; then
        # Move to the next line after completion (prevent overwriting progress).
        echo ""

        # Print feedback.
        echo -en "${BOLD_YELLOW}WARNING:${RESET} "
        echo -e "TERMINATING EXTENDED SMART TEST ON ${DEVICE}!"

        # Terminate test.
        sudo smartctl -X "$DEVICE" &>/dev/null
    fi

    # Terminate script.
    exit
}

# Get the current timestamp.
function get_timestamp() {
    date "+%H:%M:%S"
}

# List available devices.
function list_devices() {
    local DEVICES=""
    local SMART_DEVICES=""

    echo -e "${BOLD_BRIGHT_CYAN}AVAILABLE DEVICES${RESET}"
    
    case "$(uname)" in
        Linux) DEVICES=$(lsblk) ;;
        Darwin) DEVICES=$(diskutil list) ;;
    esac

    echo -e "${BRIGHT_CYAN}\
================================================================
${DEVICES}
================================================================\
${RESET}\n"

    SMART_DEVICES=$(smartctl --scan | awk '{print NR ": " $1}')
    echo -e "${GREEN}\
================================================================
${SMART_DEVICES}
================================================================\
${RESET}"
}

# Check if the test is still running.
function check_test_status() {
    local OUTPUT=""
    # Note: 'sudo smartctl -l selftest' does not always list running tests.
    OUTPUT=$(sudo smartctl -a "$DEVICE" 2>/dev/null)
    echo "$OUTPUT" | grep -q -E "$RUNNING_STRING_1|$RUNNING_STRING_2"
}

# Print progress bar.
function progress_bar() {
    local progress=$1
    local total=$2
    local width=50
    local percent
    local time_remaining
    local hrs
    local mins
    local secs
    local filled
    local empty
    local timestamp

    # Restrict the maximum percentage to 100%.
    if [ "$progress" -ge "$total" ]; then
        percent=100
        progress=$total
    else
        percent=$((progress * 100 / total))
    fi
    
    # Calculate the remaining time in seconds.
    time_remaining=$((total - progress))

    # Format remaining time as hours, minutes and seconds.
    hrs=$((time_remaining / 3600))
    mins=$(((time_remaining % 3600) / 60))
    secs=$((time_remaining % 60))

    # Determine the length of the filled and empty sections of the progress bar.
    filled=$((width * percent / 100))
    empty=$((width - filled))

    # Print the timestamp.
    timestamp=$(get_timestamp)
    echo -ne "${BOLD_GREEN}[${timestamp}]${RESET} "

    # Print the progress bar.
    printf "["
    printf "%${filled}s" | tr ' ' '#'
    printf "%${empty}s" | tr ' ' '-'
    printf "] "
    printf "%d%% (Est. %02d:%02d:%02d remaining)\r" \
    "$percent" "$hrs" "$mins" "$secs"
}

# Welcome user.
echo -e "${BOLD_YELLOW}\
################################################################################
#                            EXTENDED SMART TESTING                            #
################################################################################

                                                            Author:  Rohan Barar
                                                            Revision: 10/09/2024
${RESET}"

# Check if platform is supported.
if [[ "$(uname)" != "Linux" && "$(uname)" != "Darwin" ]]; then
    echo -e "${BOLD_RED}ERROR:${RESET} UNSUPPORTED PLATFORM."
    exit 1
fi

# Check if 'smartctl' is available.
if ! command -v smartctl &>/dev/null; then
    echo -e "${BOLD_RED}ERROR:${RESET} INSTALL 'smartmontools'."
    exit 2
fi

# Display available devices and prompt user for selection.
list_devices
echo ""
read -rp "ENTER NUMBER OF DEVICE TO TEST: " DEVICE_NUM

# Get the device path based on user input.
DEVICE=$(smartctl --scan | awk -v num="$DEVICE_NUM" 'NR == num {print $1}')

# Exit the script if an invalid device was specified.
if [ -z "$DEVICE" ]; then
    echo -e "${BOLD_RED}ERROR:${RESET} INVALID DEVICE SELECTION."
    exit 3
fi

# Verify if the device is accessible.
if ! sudo smartctl -i "$DEVICE" &>/dev/null; then
    echo -en "${BOLD_RED}ERROR:${RESET} "
    echo -e "${BOLD_YELLOW}${DEVICE}${RESET} INACCESSIBLE."
    exit 4
fi

# Enable flag.
TEST_RUNNING=1

# Start the extended SMART test.
TSTAMP=$(get_timestamp)
echo ""
echo -ne "${BOLD_GREEN}[${TSTAMP}]${RESET} "
echo -e "STARTING EXTENDED SMART TEST ON ${BOLD_YELLOW}${DEVICE}${RESET}."

INIT_OUTPUT=$(sudo smartctl -t long "$DEVICE")

# Extract the expected duration and estimated completion time in seconds.
if grep -qF "OFFLINE" <<< "$INIT_OUTPUT"; then
    OFFLINE_MODE=1
    DURAT_MINS=$(echo "$INIT_OUTPUT" | grep -oP "\b\d+(?= minutes)" | head -1)
    DURAT_SECS=$((DURAT_MINS * 60))
    END_TIME=$(( $(date +%s) + DURAT_SECS ))
else
    OFFLINE_MODE=0
fi

# Wait for the test to complete.
while check_test_status; do
    if [ "$OFFLINE_MODE" -eq 1 ]; then
        progress_bar $(( DURAT_SECS - (END_TIME - $(date +%s)) )) $DURAT_SECS
    else
        INIT_OUTPUT=$(sudo smartctl -l selftest "$DEVICE")
        PCENT=$(echo "$INIT_OUTPUT" | \
        grep -oP '(?<=Extended self-test in progress \().*?(?=% completed)' | \
        sed 's/%//')
        TSTAMP=$(get_timestamp)
        echo -ne "${BOLD_GREEN}[${TSTAMP}]${RESET} "
        echo -ne "${PCENT}% COMPLETE...\r"
    fi
    sleep "$INTERVAL"
done

# Disable flag.
TEST_RUNNING=0

# Move to the next line after completion (prevent overwriting progress).
echo ""

# Get the timestamp for completion.
TSTAMP=$(get_timestamp)

# Notify the user regarding test completion.
echo -en "${BOLD_GREEN}[${TSTAMP}]${RESET} "
echo -e "COMPLETED EXTENDED SMART TEST ON ${BOLD_YELLOW}${DEVICE}${RESET}!\n"

# Print the extended SMART information.
echo -e "FETCHING EXTENDED SMART DATA FOR ${BOLD_YELLOW}${DEVICE}${RESET}..."
INFO=$(sudo smartctl -x "$DEVICE")
echo -e "${BRIGHT_CYAN}\
================================================================
${INFO}
================================================================\
${RESET}"

Save and run the above script.

chmod +x Extended_SMART_Testing.sh
./Extended_SMART_Testing.sh