Set Up Server Alerts

Metrics on a graph are useless at 3 a.m. — nobody's looking. What wakes someone up is an alert, and to set up server alerts that fire at the right moment, on the right condition, and actually reach a human is trickier than it sounds. Alert too eagerly and people mute you; too late and the outage found your customers first. This server alerting guide covers the two ways SNMP delivers alerts, why duration matters more than any single reading, the real OIDs to trigger on, and the reason the most important alert — "the server is gone" — can only come from outside the server.

The problem with naive alerting

The instinct when you first set up server alerts is to draw a line and fire the instant a value crosses it. CPU over 90%? Page. Disk over 80%? Page. It feels responsible. In practice it's a machine that cries wolf.

Real servers spike constantly and harmlessly, and it's worth naming the usual culprits:

• A nightly backup pins the CPU for ninety seconds.
• A deploy briefly saturates a disk.
• A traffic burst maxes an interface for a single poll, then settles.
• A cron batch job hammers memory once an hour and lets go.

Alert on the raw crossing and every one of those becomes a page — so people start ignoring the channel, and then they miss the one alert that mattered. That's alert fatigue, and it's how monitoring quietly stops working while still technically running.

The opposite failure is just as bad. Set the threshold so loose that only a catastrophe trips it, and you find out about problems from users instead of the dashboard. The whole craft of alerting is threading that needle: catch genuine, sustained trouble early, stay silent through the noise. Get that balance right and people trust the alerts. Get it wrong in either direction and the system is worse than none.

Two ways SNMP delivers an alert

SNMP gives you two mechanisms, and they solve different problems.

The first is polling with thresholds. A monitor reads an OID on a schedule — say every minute — compares the value against your limits, and raises an alert when the condition holds. You own the logic, so you decide exactly what "too high, for too long" means. This is the workhorse for resource metrics like CPU and disk, where you care about a trend, not an instant.

The second is SNMP traps. Here the agent pushes a message the moment a defined event happens, sending it unsolicited to a trap receiver on UDP port 162 (polling uses UDP 161). Traps are event-driven and immediate — good for discrete happenings like a link going down or a reboot — but they're fire-and-forget over UDP, so a lost trap is simply gone, and there's no built-in confirmation it arrived.

Most solid setups use both — polling for the slow-burn resource problems, traps for the sudden ones. Neither is a replacement for the other.

Step-by-step: set up server alerts

1. Confirm the metrics respond. Before any alerting logic, prove the OIDs you'll trigger on return values. Walk CPU at 1.3.6.1.2.1.25.3.3.1.2 and read uptime at 1.3.6.1.2.1.25.1.1.0. If these are empty, fix the agent first — the snmpd.conf setup covers it.
2. Pick what deserves an alert. Not every metric needs to page someone. Start with the ones that mean real trouble: sustained high CPU, disk approaching full (low disk space alerts go deep on this), an unexpected uptime reset, and abnormal interface traffic.
3. Set two levels. Define a warning threshold (worth a look) and a critical threshold (act now). Warnings can go to a quiet channel; criticals should page.
4. Add a duration condition. This is the step that separates a usable server alerting guide from a noisy one. Require the value to stay across the line for several minutes before firing — five is a common choice. A backup or deploy blip never lasts that long, so it never pages.
5. Baseline per server. A database node at 45% CPU all day is normal; a web node at 45% might not be. Set thresholds against each host's own quiet, not one global number.
6. Route the notification. Decide where each severity goes — email, SMS, chat — and to whom. An alert nobody receives is not an alert.
7. Add an off-host check. Put an independent monitor outside the server so its own failure can't silence the alert. More on that below.
8. Test each path. Deliberately trip a threshold and confirm the message lands. Do this before you rely on it, not during an incident.

Do these in order. There's no point tuning thresholds (step 4) before the OIDs even respond (step 1), and there's no point trusting any of it until you've tested delivery (step 8).

Threshold and trap configuration in practice

A workable alerting policy is mostly a small table of conditions plus the duration rule. Written out, a starting point looks like this:

# Alert policy — poll every 60s, require the condition to hold before firing
CPU  (1.3.6.1.2.1.25.3.3.1.2)  : warn > 80% for 5m, crit > 95% for 5m
Disk (1.3.6.1.2.1.25.2.3.1.6  : warn > 85% used,  crit > 95% used
      vs .5 hrStorageSize)
Uptime (1.3.6.1.2.1.25.1.1.0) : crit if value resets  (unexpected reboot)
Traffic (1.3.6.1.2.1.2.2.1)   : warn on sustained abnormal throughput

The two habits baked in here matter more than the exact numbers. First, every condition carries a duration or a comparison, never a bare instantaneous crossing — that's what stops the nightly-backup false alarm. Second, disk is judged as a percentage: divide hrStorageUsed (1.3.6.1.2.1.25.2.3.1.6) by hrStorageSize (1.3.6.1.2.1.25.2.3.1.5) rather than trusting a raw byte figure, since a "large" number means nothing without the capacity behind it.

For traps, point the agent's notification target at a receiver on UDP 162 and treat what arrives as a hint to investigate, not gospel — because UDP can drop a trap and never tell you. That fragility is exactly why polling stays your backbone and traps ride alongside as an early nudge for discrete events.

Alerting from outside the server

Here's the failure mode that no on-host alert can escape. If the tool that watches the server, evaluates the threshold, and sends the message all live on that server, then the server going down takes the alarm down with it. A kernel panic doesn't email you. A CPU pinned to 100% may never schedule the notifier. A full disk can jam the very process that would have warned you the disk was full. The single most important alert — "this machine is gone" — is the one an on-host system structurally cannot send.

That's why the last step points outward. This is double durability: an independent watcher, off the box and outside your network, keeps checking and alerting even when the host is dark. ostr.io Monitoring delivers exactly that — real-time email and SMS alerts on downtime, slow responses, resource spikes, and even SSH logins, sent from outside your infrastructure. It's better to get notified from two independent sources than to trust the one that shares the server's fate. Pair external alerting with your on-host thresholds and neither blind spot is left uncovered.