Part 7: System Observability, Metrics & Linux Networking Core

Now, we focus on two foundational skills for production engineering: System Observability (knowing why a system is struggling) and Core Linux Networking (routing, DNS, and packet filtering).

01: System Observability: Metrics, Logs & Traces

Observability is built on three distinct telemetry pillars:

• Metrics: Numeric time-series data. Cheap to store and ideal for real-time alerting, showing that a system is failing.

• Logs: Discrete, timestamped text events describing what happened at a specific moment.

• Traces: End-to-end request journeys showing where latency or errors were introduced across service hops.

Brendan Gregg’s USE Method

When investigating performance issues, systematically evaluate three metrics for every resource (CPU, Memory, Disks, Network):

1. Utilization: The percentage of time the resource was busy performing work.

2. Saturation: Extra work queued up that could not be serviced immediately.

3. Errors: The count of explicit error events.

Bash

# Check CPU Utilization and Run-Queue Saturation (r column > CPU core count = Saturation)
vmstat 1 5

# Check Memory Utilization and Active Swapping (si/so columns > 0 = Memory Saturation)
free -m

Deep-Dive Kernel Tracing (strace)

When a process is "stuck" or frozen, attach strace to intercept system calls live:

Bash

# Attach strace to a running PID and display execution time per system call
sudo strace -p 4821 -T

• -p 4821: Attaches directly to a running Process ID 4821.

• -T: Displays the time spent inside each system call, pinpointing exact blocking functions.

02: Practical 4-Layer Linux Networking Model

When troubleshooting network connectivity, always diagnose bottom-up:

Plaintext

[ 4. Application ] ──► (curl, dig, openssl)
[ 3. Transport ]   ──► (ss, nc, firewalld)
[ 2. Network ]     ──► (ip route, ping, mtr)
[ 1. Link ]        ──► (ip link, ethtool)

Golden Rule: A failure at a lower layer (e.g., missing IP route) makes every higher-layer application (e.g., curl) fail. Diagnose bottom-up to avoid chasing ghost bugs.

03: Routing & Interface Management

The Linux kernel routes traffic based on its local routing table.

Bash

# Display the active kernel IPv4 routing table
ip route show

# Check which interface and gateway the kernel will select for a specific destination IP
ip route get 8.8.8.8

Interface Configuration via nmcli (NetworkManager)

NetworkManager manages interface profiles on RHEL/Ubuntu systems:

Bash

# View active physical network interface status
nmcli dev status

# Add a persistent static IP configuration to interface eth0
sudo nmcli con add type ethernet con-name static-eth0 ifname eth0 ip4 192.168.1.50/24 gw4 192.168.1.1

# Set primary and secondary DNS servers for the profile
sudo nmcli con mod static-eth0 ipv4.dns "8.8.8.8 8.8.4.4"

# Activate the updated profile
sudo nmcli con up static-eth0

04: DNS Resolution Order & Diagnostics

When a hostname fails to resolve, follow the OS resolution chain:

1. /etc/nsswitch.conf: Defines lookup priority (typically hosts: files dns).

2. /etc/hosts: Local static hostname-to-IP overrides (checked first).

3. /etc/resolv.conf: Nameservers configured for public/internal queries.

Bash

# Query DNS records directly using dig
dig example.com A

# Query a specific remote DNS server directly, bypassing local caches
dig @8.8.8.8 example.com

# Trace the delegation path from root DNS servers down to authoritative servers
dig +trace example.com

05: Firewalls: firewalld & nftables

Modern Linux distributions use nftables inside the kernel for packet filtering. firewalld acts as a high-level management daemon on top of nftables.

Bash

# View active firewalld zones and enabled rules
sudo firewall-cmd --list-all

# Open TCP port 8443 in the public zone temporarily (runtime only)
sudo firewall-cmd --zone=public --add-port=8443/tcp

# Persist the rule permanently across system reboots
sudo firewall-cmd --zone=public --add-port=8443/tcp --permanent

# Reload firewalld to apply permanent rules to the live engine
sudo firewall-cmd --reload