Part 3: Production Automation: Process Control & Robust Bash Scripting

Welcome to Part 3 of the Linux: Master Series Guide. In Part 2, we navigated the Filesystem Hierarchy Standard and learned how Linux manages multi-user security through permissions and ACLs.

Now, we move into operational automation: managing running workloads and writing robust shell scripts. A runbook step like "clear old logs when disk usage exceeds 90%" is only documentation until it is converted into a script. This part covers process lifecycles, signal handling, and building production-grade, fail-safe Bash scripts.

01: Process Lifecycles and Signal Management

A process is a running instance of a program, identified by a unique Process ID (PID) and tied to a parent process (PPID).

Process States and Monitoring

Inter-Process Signals

Linux uses signals to communicate with running processes. Signals can be sent using the kill or pkill commands:

Bash

# Gracefully request process termination by PID
kill 1234

# Forcefully terminate a process by name
pkill -9 nginx

Job Control: Foreground and Background Execution

• Append & to start a command in the background: python3 long_task.py &

• List background jobs running in the current terminal session: jobs

• Bring a background job to the foreground: fg %1

• Resume a suspended foreground job in the background: bg %1
  

02: Defensive Bash Scripting Standards

A script only earns trust in production when it is idempotent—safe to run once or ten times without duplicating work or erroring on existing state.

The Defensive Header: set -euo pipefail

Place set -euo pipefail at the very top of every production script:

Bash

#!/usr/bin/env bash
set -euo pipefail

• set -e: Exit immediately if any command exits with a non-zero (failure) status.

• set -u: Treat unset variables as errors and exit immediately.

• set -o pipefail: Ensures a pipeline fails if any command in the chain fails, rather than only checking the last command.

Parameter Expansions and Guard Clauses

Avoid bare variable expansions; double-quote all variables to prevent word splitting:

Bash

# Default parameter if variable is unset or empty
APP_ENV="${APP_ENV:-production}"

# Abort with an explicit error message if a variable is missing
LOG_DIR="${LOG_DIR:?LOG_DIR environment variable must be set}"

# Strip standard file extensions
filename="backup-2026.tar.gz"
echo "${filename%.tar.gz}"  # Outputs: "backup-2026"

Resource Locking and Cleanup Traps

Use trap to release locks or delete temporary directories when a script exits or receives an error signal:

Bash

#!/usr/bin/env bash
set -euo pipefail

# Create a temporary working directory
TMP_DIR=$(mktemp -d)

# Guarantee cleanup executes regardless of how the script terminates
cleanup() {
    rm -rf "${TMP_DIR}"
}
trap cleanup EXIT ERR

# Prevent duplicate concurrent executions using flock
exec 200>/var/run/myapp-archive.lock
flock -n 200 || { echo "Another instance is already running" >&2; exit 1; }

03: Text Processing Mastery

Ops work heavily involves filtering logs, API outputs, and configuration files. Three key utilities form the core text-processing pipeline:

1. grep (Pattern Matching)

• grep -E "^(ERROR|WARN)" app.log: Extended regular expressions.

• grep -v "DEBUG" app.log: Invert match (exclude lines matching DEBUG).

• grep -r "DATABASE_URL" /etc/: Search recursively through directories.

• zgrep "FATAL" /var/log/syslog*.gz: Search directly inside rotated, compressed .gz logs.

2. sed (Stream Editor)

• sed -i 's/max_connections=100/max_connections=200/' config.conf: In-place string substitution.

• sed -i '/^$/d' config.conf: Delete blank lines in place.

3. awk (Field Processing & Summaries)

• awk -F: '$3 >= 1000 {print $1}' /etc/passwd: Print usernames with UID >= 1000

• awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 10: Generate a top-10 list of IP addresses hitting a web server.

04: Parsing Script Arguments (getopts)

getopts is the POSIX standard way to handle CLI flags safely inside scripts:

Bash

#!/usr/bin/env bash
set -euo pipefail

usage() { echo "Usage: $0 -d <directory> -n <days>" >&2; exit 2; }

target_dir=""
days=30

while getopts ":d:n:" opt; do
  case "${opt}" in
    d) target_dir="${OPTARG}" ;;
    n) days="${OPTARG}" ;;
    \?) echo "Invalid option: -${OPTARG}" >&2; usage ;;
    :) echo "Option -${OPTARG} requires an argument." >&2; usage ;;
  esac
done
shift $((OPTIND - 1))

if [[ -z "${target_dir}" ]]; then
    usage
fi

echo "Cleaning logs older than ${days} days in ${target_dir}..."

05: Task Scheduling (Cron & At)

Automation requires running tasks on a schedule or delaying execution.

Scheduling Recurring Tasks with Cron

Edit user cron jobs using crontab -e. The syntax consists of five time fields followed by the command:

Plaintext

# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12)
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0)
# *  *  *  *  *  command to be executed

Bash

# Run log archiving every night at 02:30 AM
30 2 * * * /usr/local/bin/archive-logs.sh -d /var/log/myapp >> /var/log/cron.log 2>&1

One-Off Deferred Tasks with at

For one-off scheduled actions where you do not want to create a permanent cron entry:

Bash

# Schedule a system maintenance check in 15 minutes
echo "/usr/local/bin/health-check.sh" | at now + 15 minutes

# View pending deferred jobs
atq

# Cancel a queued job by ID
atrm 3

Mastering process lifecycles and writing defensive, idempotent scripts lays the groundwork for system administration. In Part 4, we will take service supervision to the next level by exploring systemd, custom units, timers, and journald.