Part 5: Linux Storage Depth: Inodes, Disks, RAID & LVM

This part covers filesystem internals (inodes vs data blocks), filesystem comparisons (ext4 vs XFS), software RAID arrays, Logical Volume Management (LVM), and diagnosing disk I/O performance bottlenecks.

01: Inodes vs. Data Blocks

A Linux filesystem splits storage into two distinct components:

1. Data Blocks: Store the actual raw contents of files.

2. Inodes (Index Nodes): Store metadata about the file (size, owner, group, permissions, creation/modification timestamps, and pointers to the allocated data blocks).

Key Rule: An inode does not store the file name. The mapping between file names and inode numbers is stored inside the parent directory's data blocks.

The "No Space Left on Device" Outage

It is possible to receive a No space left on device error even when df -h shows gigabytes of free capacity. This occurs when you run out of available inodes (common on systems generating millions of tiny files).

Bash

# Inspect available and used INODES per mounted filesystem
df -i

# Count files per directory to identify the inode-exhausting culprit
for d in /var/*/; do echo "$(find "$d" -xdev -type f | wc -l) $d"; done | sort -n

Flag Breakdown for the Inode Command

• find "$d": Searches inside directory $d.

• -xdev: Crucial flag that prevents find from traversing into other mounted filesystems (like /proc or external mounts).

• -type f: Restricts search results strictly to regular files.

• wc -l: Counts total line output (total files found).

02: Filesystem Architecture Comparison

03: Software RAID (Redundant Array of Independent Disks)

RAID combines multiple physical disks into a single logical array managed by the kernel's mdadm tool to provide redundancy, performance, or both.

Building and Managing a RAID 5 Array

Bash

# Create a RAID 5 array named /dev/md0 from 3 raw block devices
sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd

# Monitor live rebuild/sync status
cat /proc/mdstat

# Inspect detailed status of an array
sudo mdadm --detail /dev/md0

# Mark a degraded disk as failed manually
sudo mdadm /dev/md0 --fail /dev/sdc

# Remove the failed disk from the array
sudo mdadm /dev/md0 --remove /dev/sdc

# Hot-swap and add a new replacement disk (rebuild begins automatically)
sudo mdadm /dev/md0 --add /dev/sde

04: Logical Volume Management (LVM) Architecture

LVM abstracts physical storage devices away from the operating system, allowing volumes to be resized dynamically across physical drives.

The LVM hierarchy consists of three layers:

1. Physical Volumes (PV): Raw block devices or partitions initialized for LVM (/dev/sdb, /dev/sdc).

2. Volume Groups (VG): Pools of storage combining one or more Physical Volumes into a single storage pool.

3. Logical Volumes (LV): Virtual partitions carved out of a Volume Group and formatted with a filesystem.

Plaintext

[ Physical Disks (/dev/sdb, /dev/sdc) ] 
                   │
                   ▼ (pvcreate)
      [ Physical Volumes (PV) ]
                   │
                   ▼ (vgcreate)
       [ Volume Group (VG) ]
                   │
                   ▼ (lvcreate)
      [ Logical Volumes (LV) ] ───► [ Formatted Filesystem (ext4/xfs) ]

Complete Step-by-Step LVM Workflow

Bash

# Step 1: Initialize physical block devices as Physical Volumes
sudo pvcreate /dev/sdb /dev/sdc

# Step 2: Combine physical volumes into a Volume Group named 'vg_data'
sudo vgcreate vg_data /dev/sdb /dev/sdc

# Step 3: Create a 50GB Logical Volume named 'lv_app' inside 'vg_data'
sudo lvcreate -L 50G -n lv_app vg_data

# Step 4: Format the Logical Volume with an ext4 filesystem
sudo mkfs.ext4 /dev/vg_data/lv_app

# Step 5: Create a mount point and mount the filesystem
sudo mkdir -p /mnt/appdata
sudo mount /dev/vg_data/lv_app /mnt/appdata

Dynamic Online Volume Expansion

LVM allows you to extend logical volumes and their underlying filesystems live without unmounting or taking applications offline:

Bash

# Extend the Logical Volume by 20GB AND resize the filesystem in one pass (-r)
sudo lvextend -r -L +20G /dev/vg_data/lv_app

• -r (Resize Filesystem): Automatically invokes the correct filesystem resize utility (resize2fs for ext4 or xfs_growfs for XFS) to expand into the newly allocated block space.

05: Diagnosing Disk I/O Bottlenecks

When system load spikes, determine whether the bottleneck is CPU, Memory, or Storage I/O.

Bash

# Monitor disk I/O metrics refreshed every 1 second
iostat -x 1

Key iostat Performance Indicators

• %util: The percentage of elapsed time during which I/O requests were issued to the device. Values consistently near 100% indicate the storage device is saturated.

• await: The average time (in milliseconds) for I/O requests to be served (includes queue time plus hardware service time).

  • Healthy spinning disks: $< 10\text{ms}$.

  • Healthy SSDs: $< 1\text{ms}$.

  • High await with high %util confirms a severe storage bottleneck.

Bash

# Identify top processes consuming disk read/write bandwidth in real-time
sudo iotop -o

• -o (Only): Restricts display strictly to processes or threads actively performing I/O instead of showing all idle processes.