SNMP Process Monitoring

SNMP process monitoring lets you watch what's actually running on a server — how many processes exist and what state each is in — without shelling in to run ps. It's how you catch a fork bomb before it exhausts the process table, notice that a critical daemon has quietly died, or spot a zombie pile-up that signals a misbehaving parent. The data comes straight from the Host Resources MIB, so any standard agent already publishes it. This guide covers what the process table represents, the exact OID that reports run status, how to count processes over SNMP, sensible thresholds, and why an outside checker is the one that still answers when the box is drowning.

What is Process monitoring and why monitor it

A process is a running instance of a program — the kernel tracks every one of them, each with a state: running, runnable, waiting, or not-runnable. On a healthy server that count sits in a fairly stable band, drifting up and down with load. When it does something unusual, that's your early warning.

Two failure modes make process monitoring worth the effort. The first is too many: a runaway loop that keeps forking, a leaked connection pool spawning workers, or an outright fork bomb. Left alone, the process count climbs until the system hits its process-table limit, at which point nothing new can start — you can't even log in to fix it, because your login shell is itself a new process that can't be created. The second is too few: a service you depend on has crashed, and the process that should be there simply isn't. Both are invisible from a CPU or memory graph. Only counting and inspecting processes surfaces them.

There's also the zombie problem. A process that has exited but whose parent never reaped it lingers in the table as a defunct entry. A handful is normal; a steadily growing pile points to a bug in a parent process that's failing to clean up after its children. Watching process state over SNMP catches that drift. The symptoms operators feel — "can't spawn new connections", "service is down but the box is up", sluggishness with no obvious resource hog — all map back to the process table.

The OID: 1.3.6.1.2.1.25.4.2.1.7

The core OID for process monitoring is 1.3.6.1.2.1.25.4.2.1.7, named hrSWRunStatus in the Host Resources MIB (RFC 2790). It's a table column, not a scalar: the agent returns one row per running process, each keyed by an index, and each value is the run state of that process as an enumerated integer.

The enumerated values for hrSWRunStatus are: 1 = running, 2 = runnable (waiting for a resource), 3 = notRunnable (loaded but not runnable), 4 = invalid (not a valid entry). Because it's a table, walking 1.3.6.1.2.1.25.4.2.1.7 gives you the state of every process at once — you can count the rows to get a process total, or filter by state to find, say, everything stuck in notRunnable.

For a straight process count there's a shortcut. hrSystemProcesses at 1.3.6.1.2.1.25.1.6.0 is a scalar that reports the total number of processes in one read, no walking required. So the process count OID and the per-process status column are complementary: use hrSystemProcesses for the headline number, and hrSWRunStatus when you need to know what state those processes are in.

How to query it

Any Net-SNMP client handles this. These examples assume an SNMPv2c agent with a read-only community; if that's not configured, see the snmpd.conf guide.

1. Get the total process count in a single call:

   snmpget -v2c -c public 10.0.0.5 1.3.6.1.2.1.25.1.6.0
   

   HOST-RESOURCES-MIB::hrSystemProcesses.0 = Gauge32: 214
   

2. Walk the run-status column to see every process's state:

   snmpwalk -v2c -c public 10.0.0.5 1.3.6.1.2.1.25.4.2.1.7
   

   HOST-RESOURCES-MIB::hrSWRunStatus.1 = INTEGER: running(1)
   HOST-RESOURCES-MIB::hrSWRunStatus.2 = INTEGER: runnable(2)
   HOST-RESOURCES-MIB::hrSWRunStatus.418 = INTEGER: running(1)
   

3. Force numeric output if the MIB isn't installed locally:

   snmpwalk -v2c -c public -On 10.0.0.5 1.3.6.1.2.1.25.4.2.1.7
   

Count the rows from the walk for a status-aware total, or just read hrSystemProcesses for the quick number. To monitor a specific service you'd pair this with the process-name column in the same table, matching by name and then checking its status.

Normal vs alarming values & thresholds

Process counts are host-specific — a busy app server runs far more than a minimal appliance — so baseline each machine against its own normal. General guidance for what to monitor:

• Stable count within the host's usual band — healthy. Learn what "normal" looks like per server and alert on departure from it.
• A sharp, sustained climb — investigate for a fork loop or leaked workers before the process table fills.
• Approaching the system's process limit — critical. New processes will start failing, including your remote login; page immediately.
• A sudden drop — a service may have crashed. If a count you expect around 210 falls to 150, something exited that shouldn't have.
• Growing count of non-runnable or defunct entries — a reaping bug or resource starvation; worth a warning-level trigger.

The most reliable approach is relative, not absolute: alert on deviation from each host's baseline rather than a single global number, and require the condition to persist for a few minutes so a brief burst of short-lived processes doesn't page anyone.

Alerting on this metric externally

Polling hrSWRunStatus or hrSystemProcesses tells you the process table looks wrong right now. What prevents the incident is being told the moment it goes wrong — and a server whose process table is full is the textbook case of a machine that can't help you. When no new process can be forked, a local alerting agent that needs to spawn a helper, open a shell, or send a mail can't. The very condition you're trying to detect is the one that gags the on-host notifier.

Reading it from outside dodges that trap completely. That's the principle of double durability: an independent watcher polls 1.3.6.1.2.1.25.4.2.1.7 and 1.3.6.1.2.1.25.1.6.0 on its own hardware, so a host with an exhausted process table — or one that's crashed entirely — still trips an alert. SNMP is pull-based; the external side just asks, and it doesn't need the sick server to fork anything to get an answer.

You can build that yourself or hand it off. ostr.io Monitoring polls process and health OIDs from outside your network and fires real-time email and SMS alerts, so a server buried under runaway processes still reaches you even when it can't start a single new one.