Part 4: Managing Everything with systemd & Service Supervision

Welcome to Part 4 of the Linux: Master Series Guide. In Part 3, we covered process lifecycles, signal management, and writing defensive Bash scripts.

Now, we take service management to an enterprise level. On modern Linux distributions, PID 1—the ancestor of every process—is managed by systemd. This part covers systemd unit file anatomy, creating custom background services, replacing traditional cron with precise systemd timers, and querying binary logs journalctl.

01: Architecture: Units and Dependencies

systemd manages system resources through units. The most common unit types include:

• .service: Background daemons and applications.

• .timer: Monotonic or calendar-based task schedulers (replacing cron).

• .socket: Inter-process communication or network sockets for socket-activated daemons.

• .mount: Filesystem mount points managed by systemd.

• .target: Grouping units to define boot goals (replacing legacy SysV runlevels).

  
  

Unit Precedence Order

When loading unit definitions, systemd checks directories in this strict order:

1. /etc/systemd/system/: Local system administrator overrides (Highest precedence).

2. /run/systemd/system/: Runtime units created dynamically (Volatile, lost on reboot).

3. /usr/lib/systemd/system/: Units installed by system package managers (apt, dnf) (Lowest precedence).

Rule: Never modify files directly under /usr/lib/systemd/system/ as package updates will overwrite them. Always put custom or modified units under /etc/systemd/system/.

02: Unit File Anatomy & Creating Custom Services

A .service unit is broken into three distinct sections: [Unit], [Service], and [Install].

Here is an example unit file created at /etc/systemd/system/healthcheck.service:

Ini, TOML

[Unit]
Description=Application Health Check Daemon
After=network-online.target
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
ExecStart=/usr/local/bin/healthcheck.sh
ExecStop=/usr/local/bin/healthcheck-stop.sh
Restart=on-failure
RestartSec=5
User=nobody
Group=nobody

[Install]
WantedBy=multi-user.target

Detailed Unit File Option Breakdown

[Unit] Section (Metadata & Ordering)

• Description=: A human-readable title describing the unit's purpose.

• After=network-online.target: Ordering dependency. Ensures this unit starts after network connectivity is established. It does not force the network to start; it only controls timing if both are queued.

• Wants=network-online.target: Soft requirement dependency. Tells systemd to attempt to activate network-online.target alongside this service, but if it fails, this service continues starting anyway.

• Requires=postgresql.service: Hard requirement dependency. If postgresql.service fails or stops, this service immediately fails or stops as well.

[Service] Section (Execution & Supervision)

• Type=simple: (Default) Assumes the process started via ExecStart is the main service process.

• ExecStart=: Absolute path to the command or binary executed when starting the service.

• ExecStop=: Absolute path to the command executed when explicitly stopping the service.

• Restart=on-failure: Automatically restarts the service if it exits with a non-zero exit code or is terminated by a signal.

• RestartSec=5: Configures a 5-second delay before systemd attempts to restart a failed service.

• User=nobody / Group=nobody: Runs the process under a low-privilege service account instead of root to enforce security isolation.

[Install] Section (Target Association)

• WantedBy=multi-user.target: Specifies which target links this service when enabled. multi-user.target corresponds to standard multi-user, non-graphical server boot-up.

Essential Service Management Commands

Bash

# Reload systemd configuration after adding or editing a unit file
sudo systemctl daemon-reload

# Start a service immediately
sudo systemctl start healthcheck.service

# Stop a running service
sudo systemctl stop healthcheck.service

# Enable a service to auto-start on system boot
sudo systemctl enable healthcheck.service

# Combine enable and start into a single command
sudo systemctl enable --now healthcheck.service

# Check live operational status, PID, and recent logs
systemctl status healthcheck.service

03: Precision Automation with systemd Timers

systemd timers replace traditional cron jobs. They offer tighter logging integration, dependency tracking, and monotonic scheduling (e.g., run 15 minutes after boot).

A timer requires two files with matching names:

1. /etc/systemd/system/backup.service (A Type=oneshot execution unit).

2. /etc/systemd/system/backup.timer (The scheduling unit).

Execution Unit (backup.service):

Ini, TOML

[Unit]
Description=Nightly Database Backup Job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-db.sh

Timer Unit (backup.timer):

Ini, TOML

[Unit]
Description=Trigger backup.service nightly at 02:30 AM

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Detailed Breakdown of Timer Options

• OnCalendar=*-*-* 02:30:00: Uses systemd calendar syntax (Year-Month-Day Hour:Minute:Second). *-*-* means every day.

• Persistent=true: If the system was powered off when the timer was scheduled to trigger, systemd executes the missed job immediately upon the next boot.

• WantedBy=timers.target: Ensures the timer activates automatically when systemd initializes timer targets.

Bash

# Validate calendar expressions before applying
systemd-analyze calendar "Mon..Fri 09:00"

# List all active system timers, showing last run and next scheduled run
systemctl list-timers --all

04: Querying Logs with journald

systemd-journald collects logs from the kernel, system services, and process standard output (stdout/stderr) into a structured binary format queried via journalctl.

Bash

# Follow live logs for a specific service (like tail -f)
journalctl -u healthcheck.service -f

# Filter service logs by severity level (error or higher)
journalctl -u healthcheck.service -p err

# Query logs within a specific time window
journalctl --since "1 hour ago" --until "10 minutes ago"

# View logs generated during the current boot cycle only
journalctl -b

# View logs generated during the previous boot cycle
journalctl -b -1

# Show disk space consumed by binary journal logs
journalctl --disk-usage

# Reclaim space immediately by deleting logs older than 2 weeks
sudo journalctl --vacuum-time=2weeks