Knowledge base

Untangle a stuck queue

entries in this scenario: 51
log-fileGlobalset by the HubDirectiveThe path to the log file; the line is mandatory, and it also sets the location of the monitor and SNMP logs.max-memory-usage-percentGlobalDirectiveWhen the share of physical memory is exceeded, the node responds 452 to MAIL FROM until usage drops back below it.spool-max-recipientsGlobalDirectiveLimits how many recipients the node holds in queues; above the ceiling, submission is refused with a temporary error. The default is to calculate from the machine’s resources.log-commands<source>set by the HubDirectiveWrites the full exchange of commands and responses to the log; the vendor calls this the first step when investigating connectivity.log-rotateGlobalDirectiveThe number of files, including the current one; the default is eight, and zero turns rotation off entirely.allow-priority-interruption<domain>DirectiveWith the value no the queue becomes untouchable: its connections are not taken away, however high the neighbor’s priority.allow-priority-interruption-during-transfer<domain>DirectiveWith no the interruption is postponed until the connection frees up; with yes it happens immediately, in the middle of a transfer. Only for SMTP queues.log-connections<source>set by the HubDirectiveWrites a log line when a connection from this source opens and closes; off by default.log-data<source>set by the HubDirectiveRecords the exchange byte by byte, in hexadecimal and as text at the same time; the most detailed and heaviest level.log-verbose<signals>DirectiveAdds messages about the export of events to SparkPost Signals to the log; off by default.max-message-hops<source>DirectiveLoop protection: the node does not accept a message that has passed through too many mail systems.spool-delete-corruptedGlobalDirectiveMessage files with encoding errors — traces of data corruption on disk — are deleted instead of lying around idle.log-hires-timestampGlobalDirectiveAdds milliseconds to log timestamps; off by default.A file, a program or an endpoint — and a file under each of the threecriticalMechanismThree ways to hand off accounting records; an endpoint still has a file underneath, which is deleted only after a successful send, and a non-2xx response triggers a retry.What the node tells about itself and through which channelscriticalMechanismSeven observation surfaces: the web monitor, the console in four formats, the log, accounting, accounting search, metrics and export to external analytics.A queue is a receiving server together with a sending addresscriticalMechanismThe engine groups messages by the “recipient domain plus sending address” pair: one domain from two addresses is two independent queues with their own counters, connections and mode.Recipients deferred after a temporary refusalMechanismA recipient to whom the receiver responded with 4xx moves into a separate waiting area with a per-receiver interval; two commands can release it early.A retry ladder of your ownMechanismretry-after and backoff-retry-after accept a list of intervals — up to 30 steps, and the last one applies until the bounce.The receiver declares an encryption requirement in its DNS zoneMechanismThe node looks at the recipient domain’s DANE records and, if STARTTLS is required, sends only over an encrypted channel with a certificate verified by fingerprint.Directives with the same nameMechanismA directive name defines a setting only together with its block: 48 of 417 names are documented in more than one place, and one name in different blocks is not always one setting.The watchdog that restarts the engineMechanismKeeps an eye on the engine and brings it back up after a crash by itself, sending a crash report to the postmaster address.The node needs a resolver that does all the work itselfMechanismThe engine requires recursive name resolution; a resolver that answers with a referral to another server forces the query to be repeated and triggers a log warning.What makes a virtual machine riskyMechanismRunning on a virtual machine is possible, but the manual does not recommend it without sufficient hardware and properly tuned hypervisor settings: under load the node may stop responding.Sending windows set in the message itselfMechanismA header carries one or more intervals during which the message may be sent; the schedule cancels the message’s lifetime, and the message bounces when the windows run out.The log and accounting — two files and two different questionsMechanismThe engine keeps a separate log and a separate accounting file; the separation is made for the sake of format, performance and the ability to hand over the log without disclosing data.Keeping selected domains resolved at all timesMechanismA file with a list of domains whose DNS records are refreshed on their own schedule, independently of the queue and regardless of their own time to live.Figure out why a queue to one receiver is growingProcedureFirst check the route and reachability with one command, then make the node attempt delivery with the session logged, and only for an intermittent problem enable per-receiver logging.What to do after a crashProcedureThe procedure after a crash: make sure the watchdog brought the engine back up, keep the report email, find the crash dump, pass it to the vendor.show queuescriticalCommandShows the selected queues and, unless told otherwise, the last fifty deliveries, bounces and delays; selection is by connection, mode and pause state.clear dnscacheCommandRemoves all records of a domain from the engine’s DNS cache; with a star, the whole cache.listCommandLists recipients in queues by selection; with –pause it interrupts a recipient’s delivery to show it.resolveCommandQueries a domain’s MX records and addresses and with –connect connects to the first available mail host.scheduleCommandImmediately schedules delivery attempts for queues without waiting for the retry interval; with an option it also releases deferred recipients.traceCommandOpens a connection to a domain and tries to deliver one message from the queue, writing the SMTP conversation to the log.rotate logCommandCloses the current log and starts a new one; the previous one stays in place.show precacheCommandPrints the domains whose records the node refreshes regardless of the queue contents.Return a node filecriticalAPI endpointReturns a log or accounting file by name from the /logs list.Messages in the queueAPI endpointLists messages in the selected queues, filtered by job, envelope, recipient and sender.List of logs and accounting filesAPI endpointLists the log and accounting files the monitor can serve; the contents themselves come through /getFile.Temporary queue failureAccounting recordThe attempt failed at the queue level: the connection was refused, was not established or dropped before the conversation about messages.The receiver refused the connectionError codeThe connection was not established: the remote host rejected it — either the mail server is down, or a filter is rejecting this particular sender.The receiver went silent mid-conversationError codeThe connection is established, but the receiver stopped responding; the engine waits ten minutes, and this time is not configurable.The name lookup did not complete — and no conclusion is drawnError codeThe name service query failed, usually on timeout; the engine deliberately does not bounce the message on such a failure.ConnectionsMetricThe number of simultaneous connections: now, the maximum and the allowed limit, in both directions.Name cacheMetricThe number of names in the cache and the number of queries waiting for a response; separately, active queries.Queue depthMetricRecipients, recipient domains and kilobytes in the queue.Three short histories for each queuecriticalMonitor screenThe page of a single queue carries the last fifty recipient events, the last ten queue errors and the last ten mode changes.LogscriticalMonitor screenA list of log and accounting files with download links.QueuesMonitor screenThe top queues — “receiver and sending address” pairs — and a filter form: by connections, mode, list length and order.Node statusMonitor screenNode state on one page: traffic, peaks, connections with the ceiling, the DNS cache, the spool and queues by delivery method.The node's own logFileThe engine’s own log: startup and shutdown, errors, connection traces and warnings; rotation is automatic, the number of kept files is configurable.