Demo
Demo

Knowledge base

PMTA knowledge base›Mechanism§ 2.3

The watchdog that restarts the engine

pmtawatch
A node can crash and come back, and you will not find out

The manual mentions the watchdog in passing — while explaining why the postmaster address must be set. But the consequence of that sentence is much bigger than the sentence itself.

A node can crash and come back up on its own. The watchdog notices the crash and starts the engine again; sending resumes without human intervention.

And the only documented notification of this is an email. The crash report goes to the postmaster address set by the postmaster directive. The event is not reflected anywhere else: in the status summary the node looks like it is working, because it really is.

This is important for a fleet: a node that crashes once a day and comes back up in seconds is indistinguishable from a healthy one from the outside. It will be visible only indirectly — through sending gaps as long as a restart takes.

The main trap is that nobody reads the email. The postmaster address is either not set at all or points to a mailbox nobody opens. Then crash reports go into the void, and the crashes simply do not exist for those watching the fleet. Moreover, the address must be outside this node: the crash report is sent by whatever crashed, and a local mailbox may be unavailable exactly when it matters.
What the Hub does. 🔴 It does not set the postmaster address — there is no postmaster directive in the configuration.

So the watchdog’s reports have nowhere to go: a node that crashed and came back up looks no different in the cabinet from one that worked continuously. Telemetry might at most see a gap in polling, if the crash fell between two measurements.

This is not the only thing lost: the manual names the postmaster address as the channel for the warning about the license expiring soon as well. Neither message reaches anyone right now.

The Hub accepts mail to role addresses (postmaster@, abuse@, dmarc@) and discards it — but that is about inbound messages from other systems, not about the watchdog’s outgoing reports.

Recommended values

What the mechanism needs in order to work:

What What it must be
postmaster address set by the directive and outside this node
the mailbox behind it one that is read, not a role address into the void
the path to it not through this same node — it may have crashed

An address outside the node is not nitpicking. The crash report is sent by a machine that has just crashed; if the email goes through it, it may not go out at all.

For a fleet, email is not the best channel. It would be better for the Hub to learn about a restart; but the manual describes no such way, and email remains the only one.

The analysis of the address outside the node and the “email is not the best channel for a fleet” assessment are ours. From the manual here come the existence of the watchdog itself, the automatic restart and sending the report to the postmaster address.
Example configuration
# the only thing the watchdog needs is where to write
postmaster admin@example.net
The highlighted line must point to a mailbox outside this node. The report is sent by whatever crashed — the path to the recipient must not go through it.
yellow — — directive names, blue — blocks. They are highlighted the same way in the text of the card.
▲

Common mistakes and problems

what people get wrong and how it ends3 mistakes▶

Leaving the postmaster address unset

§ 2.3

What it costsCrash reports have nowhere to go. A node that crashes regularly and comes back up in seconds looks healthy from the outside — and stays that way for everyone watching the fleet.
How to spot itSending shows periodic short gaps with no visible cause, while the status summary is fine.
How to do it rightSet the postmaster address — to a mailbox outside this node. Check the uptime: it will reveal recent restarts.

Specifying an address on the same node

§ 2.3

What it costsThe report is sent by a machine that has just crashed. If the email’s path goes through it, it may not go out at all — and exactly when it is needed most.
How to spot itThere were crashes, and not a single email arrived.
How to do it rightUse an external mailbox whose delivery does not depend on this node.

Assuming a working node has not crashed

§ 2.3

What it costsThe watchdog brings the engine back up by itself, and in the status summary the node looks like it is working — because it is. The crash is visible only through uptime and a gap in sending.
How to spot itReports show gaps in sending, while the node is “fine.”
How to do it rightLook at uptime, not only status. And set up a notification channel that someone reads.
Compatibility
PowerMTA 6.0
described in the manual of this version · platform: any
The vendor does not state the first version for every directive — there is no note here. The name also appears in the 5.0 edition of the manual.
Inheritance
The watchdog has no inheritance — it either runs on the node or it does not. What is inherited is the postmaster address: the directive is set at the node level and has no scope.
⇄

Related entries

2 relations

Does not work without

Initial configurationThat is where the postmaster address is set — without it the watchdog’s reports have nowhere to go. § 2.3

Works together with

What the node tells about itself and through which channelsThe watchdog’s email is the only surface where a crash with an automatic restart is visible. inference

Referenced from

postmasterGlobalThe watchdog sends crash reports here — it is its only way to report them. § 3.3.1.4backoff-notify<domain>A second channel for learning that something is wrong: the watchdog watches the engine itself, the notification watches queue behavior. § 2.3Starting and stopping the engineA manual stop and start is the same action the watchdog performs by itself after a crash. § 4.1.3What to do after a crashThe mechanism this step relies on. § 4.2.5

How to check on the node
Check whether the channel is set and whether the node has crashed:
grep -n '^postmaster' /etc/pmta/config
pmta show status | grep -i 'uptime\|started'
Empty output from the first means the watchdog’s reports have nowhere to go. The second shows how long the node has been running without interruption: a shorter time than expected is exactly the trace of a recent crash with an automatic restart.
◇

What else to read

3 articles

◇What the node tells about itself and through which channelsSeven observation surfaces: the web monitor, the console in four formats, the log, accounting, accounting search, metrics and export to external analytics.◇What makes a virtual machine riskyRunning on a virtual machine is possible, but the manual does not recommend it without sufficient hardware and properly tuned hypervisor settings: under load the node may stop responding.◇How the policy for a receiver is assembledSettings are assembled from the most general pattern to the most precise; the order of blocks in the file means nothing.

In the Hub
Cost of a mistake
safe: a wrong value breaks nothing
Scenarios
Troubleshoot deliverability · Untangle a stuck queue