Diagnosing Linux Systems

Troubleshooting a slow Linux server involves a systematic approach to isolate and resolve the issue.

Here are things that you can initially check. I categorized them and provided basic tools that are in most distros. There are other programs, but they might not be included in your distro.

Hardware CPU Memory

Identify system load and processes. High CPU usage by processes can cause slowness. Look for processes consuming excessive CPU and whether it’s system %sy or user %us time.

CPU

# Check CPU usage
uptime

# Found in most distros
top
htop

# Many options and grep the output
mpstat -P ALL

Memory

A lack of RAM causes the server to use swap space, which is much slower. Watch for high swap usage and low available memory.

free -m

top

# Virtual memory and statistics
vmstat

Disc I/O

High I/O wait %iowait indicates processes are waiting on disk access. Also check if any filesystem is full.

iostat -xz 1

# Like top, but for io
iotop

# File system info
df -h

# Summarize disc usage
du -sh

Processes

Identifies which processes are consuming the most resources. A runaway process can slow the entire system.

Also look for zombie processes. Zombie processes may indicate that child processes have not been properly handled and can signal deeper issues in process management.

# top again
top

# Process info
ps aux --sort=%cpu
ps aux --sort=%mem
ps aux |grep defunct

Network

A busy or misconfigured network interface can affect server performance, especially for web or database servers.

iftop

nload

# Many options check manpages
netstat -plant

# Many options check manpages
ss -tulnp

Logs

Logs can reveal hardware errors, kernel panics, service crashes, or misbehaving daemons.

Looking through logs can be tedious, so use grep to filter what you are looking for.

# Look through the Journal
journalctl -xe

# And of course var logs
/var/log/syslog
/var/log/*

Cron Jobs

Poorly timed or intensive cron jobs can spike resource usage at regular intervals.

crontab -l

ls /etc/cron.*

Compromised or Rootkits

A compromised system may be running unauthorized processes, mining cryptocurrency, or participating in a botnet, leading to slowness.

Check out:

chkrootkit

rkhunter

clamav

Applications

If the system seems okay, look into the application logs (Nginx/Apache, PostgreSQL/MySQL).

SMART Hardware

Failing disks, bad RAM, or overheating CPUs can cause intermittent or chronic slowness.

# Use smartctl on your device, check the many options
smartctl -a /dev/sda

Last Resort Reboot

If the system has become unresponsive or other checks fail, a reboot may temporarily resolve the issue, though it’s better to find and fix the root cause.

Summary

Many of these tools have different options (flags) to get the output you are looking for. Also, it is common to grep the output.

The ideas I listed above are common programs found in most distros. You may need to install and configure what you need. Especially if you are running Arch. Check the manpages for your distro.

Once you configured the tools you like, you could automate them using a Bash script.