Demo
Demo

Knowledge base

PMTA knowledge base›Procedure§ 1.8

Node backup and restore

Backup and recovery
Five directories — and a stop for the duration of the restore

The manual lists five directories that make up the node’s recoverable state: configuration, internal state, logs, spool and crash dumps. The paths are given as defaults and may differ on a particular installation.

Restoring is simple and crude: stop the engine, put the directories back, start it. The manual offers no way to restore on the fly.

What stands behind each directory. The configuration is settings and secrets; losing it is reversible if the file is assembled again. The spool is messages not yet sent: losing it means losing messages. Internal state is what the engine remembers between starts. Logs and dumps are needed for investigation, not for operation.

The order of directories in the list is not the order of importance. The configuration can be assembled again, the spool cannot.

The main trap is copying the spool on the fly. This is the directory the engine is writing to right now: a copy taken from a running node will almost certainly catch a message in the middle of being written. On restore such a copy will produce either corrupted messages or the engine refusing them. A spool copy makes sense only from a stopped node or through an instant volume snapshot.
What the Hub does. 🔴 It does not back up nodes at all — there is no backup module; it is planned.

What exists: the Hub assembles the configuration itself, keeps its versions and can roll back to the previous one. That is, the configuration is recoverable — from the Hub, not from a node copy.

But the spool cannot be recovered by anything. Messages queued on a node are lost together with the machine: the Hub does not have them; it only knows they were handed to the node.

Until the module appears, this means a simple rule: no destructive actions on a production node without testing on a staging environment. There will be nowhere to restore from.

Recommended values

What is in which directory and what losing it means:

Directory What is inside If lost
configuration settings, passwords, signing keys recoverable from the Hub; keys — if saved
internal state what the engine remembers between starts recoverable, but the node will “forget” what it accumulated
spool messages not yet sent messages lost irreversibly
logs traces for investigation investigating the past becomes impossible
crash dumps material for the vendor nothing needed for operation

A copy from a running node can be taken only via a volume snapshot. Ordinary copying of the spool on the fly gives an unusable copy — the engine is writing to it at that very moment.

Signing keys are worth storing separately from the node. Losing them means changing the selector and the DNS record, not restoring; this is the most expensive part of the configuration.

The “what losing it means” analysis and the volume snapshot rule are ours. From the manual here come the list of five directories with paths for both platforms, the restore order and the caveat that the paths are given as defaults.
Example configuration
# taking a copy: configuration and state can be copied on the fly
tar czf /backup/pmta-etc.tgz /etc/pmta/
tar czf /backup/pmta-state.tgz /var/lib/pmta/
# the spool — only from a volume snapshot or a stopped node# tar czf /backup/pmta-spool.tgz /var/spool/pmta
The highlighted line is commented out on purpose. A copy of the spool taken from a running node will catch a message in the middle of being written, and on restore the engine will trip over it.
yellow — — directive names, blue — blocks. They are highlighted the same way in the text of the card.
▲

Common mistakes and problems

what people get wrong and how it ends3 mistakes▶

Copying the spool on a running node

§ 1.8

What it costsThe engine writes to the spool at the very moment of copying. The copy will almost certainly catch a message in the middle of being written, and on restore will produce either corrupted messages or a refusal.
How to spot itThe restored node starts, but some messages cannot be parsed, and the log shows spool read errors.
How to do it rightTake the spool from a stopped node or with an instant volume snapshot. The configuration and state can be copied on the fly.

Taking paths from the manual without checking them against the installation

§ 1.8

What it costsThe paths are given as defaults, and the manual says so explicitly. A copy made using the wrong paths will collect empty directories, and this will be discovered exactly when a restore is needed.
How to spot itThe backup archive is suspiciously small, and restoring from it yields nothing.
How to do it rightCheck the paths against the actual installation when setting up copying, and check archive sizes at least once.

Assuming the configuration in the Hub replaces a node copy

§ 1.8

What it costsIt replaces only the configuration. The spool — messages not yet sent — is not stored in the Hub: it only knows they were handed to the node. Losing the machine means losing those messages.
How to spot itAfter a node was replaced, part of a campaign did not go out, and there are no traces of those messages either on the node or in the Hub.
How to do it rightRemember the boundary: the configuration is recoverable, queued messages are not. Until the backup module appears, do nothing destructive on a production node without testing on a staging environment.
Compatibility
PowerMTA 6.0
described in the manual of this version · platform: any
The vendor does not state the first version for every directive — there is no note here. The name also appears in the 5.0 edition of the manual.
⇄

Related entries

3 relations

Depends on

spoolGlobalThe directive that sets the very directory whose copy is the most valuable part of a backup. § 1.8

Affects

Accept, route, deliverThe spool is where a message lives between receipt and sending; that is why copying it means preserving the messages. § 1.8

Applies to

The node's main configuration fileThe configuration directory is the first thing listed for restoring a node. § 1.8

Referenced from

Running without root privilegesA related procedure: they share some of the steps. § 1.8Removing the engineA related procedure: they share some of the steps. § 1.8spool-delete-corruptedGlobalA corrupted file is what restoring from a backup will not save: the message in it can no longer be read. inferencelog-fileGlobalThe log is usually not included in a backup: it recovers by itself and takes up a lot of space. § 1.8deliver-only<spool>The standard way to bring up a failed node’s queue: attach its spool and drain it. § 3.3.2.2Switching over to a standby nodeA related procedure: they share some of the steps. inference

How to check on the node
After a restore, check not that the files exist but that the node came back to life:
pmta test config
pmta show status
The first checks that the configuration parsed. The second checks that the engine is running: the Queues line shows whether recipients came back from the spool, and Spool how many files are in use.
◇

What else to read

3 articles

◇The activation key is a prerequisite for installation, not a step after itWithout an activation key from SparkPost support the engine neither installs nor runs, and the license tier sets the outbound connection ceiling: on Standard everything above 200 is ignored.◇Cloud restrictionsClouds restrict outbound mail on port 25 until you ask them to lift the limit, and mailbox providers treat their address ranges with suspicion.◇How to turn a sending plan into hardwareTwo constants for sizing any node: about 2 KB of memory per queued recipient and spool space proportional to the total size of the messages.

In the Hub
Cost of a mistake
dangerous: effects outlast a rollback
Scenarios
Emergency stop · Prepare a node for sending