Knowledge base

PMTA knowledge base›Mechanism§ 12.1.2

CSV or JSON — chosen by the file extension, not by a directive

record serialisation

The format is set by the file name

There are two formats, and the choice between them is not configured with a directive. It is made by the extension in the block’s path: .json gives JSON, anything else gives CSV. The pipe form always writes CSV.

CSV — per RFC 4180, in UTF-8, with a header line. The first column is always the record type — that is what makes it possible to keep several types in one file. The price of such a mix: the header carries all columns of all selected types, and each line fills only its own.

JSON — one object per line, no header, the NDJSON format. Each record is self-describing: all configured fields are present in it, even empty ones. The shared header problem does not exist at all.

🔴 The extension also decides the webhook’s content type header: .json gives application/x-ndjson, everything else text/csv. So a path changed for a completely different reason silently changes the format of the data sent, and the receiver will break on data that looks correct.

The timestamp format is configured separately — iso-times, and it is shared by both formats.

The main trap is mixing record types in one CSV file. The header starts carrying all columns of all types and grows with every added type, while each line fills only its own part. The parsing program does not break right away but on the first added type.

What the Hub does. It uses both — in different ways.

Telemetry goes to .json, that is, NDJSON: the receiver parses the batch line by line, and each record is self-describing. The extension here is part of the protocol, not decoration — the configurator code marks this with a comment, because with any other extension CSV would arrive and there would be nothing to parse the batch with.

The safety-net file is .csv, with the default record types (d,b), and a mix of two types in it is tolerable.

Recommended values

The mechanism has no values of its own: the path chooses the format.

The path ends with	Format	Webhook content type
.json	NDJSON, no header	application/x-ndjson
anything else	CSV with a header	text/csv
a pipe	CSV	—

The content type can be overridden only with the http-webhook-extra-headers file.

Both formats, the extension-based selection rule, the CSV header line and the presence of empty fields in JSON come from the manual (§ 12.1.2). The remark about the silent format change when editing the path is ours.

Example configuration

# the extension is part of the protocol, not decoration
<acct-file /var/log/pmta/mtahub-webhook.json>    records d, b, rb, t, feedback-loop
    http-webhook-url https://hub.example.com/api/telemetry/pmta/7
</acct-file>

Replace .json with .log — and the receiver will get CSV with the text/csv type instead of NDJSON. The configuration will remain syntactically correct.

yellow — — directive names, blue — blocks. They are highlighted the same way in the text of the card.

▲

Common mistakes and problems

what people get wrong and how it ends2 mistakes▶

Changing the file path and silently changing the format

our experience

What it costsThe extension decides both the format and the content type header. Editing the path for a completely different reason breaks parsing on the receiver side, while the configuration stays correct.

How to spot itThe receiver responds with a parse error right after the path is edited.

How to do it rightTreat the extension as part of the protocol and mark this with a comment in the configuration itself.

Keeping many record types in one CSV file

§ 12.1.2.2

What it costsThe header carries all columns of all selected types and grows with each new one added. Lines fill only their part, and the parsing program breaks on the very first addition.

How to spot itThe header has dozens of columns, while a third of each line is filled.

How to do it rightSplit the types into files or switch to the JSON format, which does not have this problem.

Compatibility

PowerMTA 6.0

described in the manual of this version · platform: any

The vendor does not state the first version for every directive — there is no note here. The name also appears in the 5.0 edition of the manual.

Inheritance

The mechanism has no inheritance: the format is a property of the path in each <acct-file> block separately. The same stream of events can be written in two blocks in different formats.

⇄

Related entries

2 relations

Affects

Accounting sent by the node instead of accounting collected from the nodeThe path extension decides not only the record format but also the content type header in the request sent. § 12.1.1

Applies to

How an accounting record is structuredThe same set of records written in two ways: that entry is about their composition, this one about how they land in the file. § 12.1.2

Referenced from

http-webhook-url<acct-file>The extension in the path decides both the record format and the content type header in the request. § 12.1.1 Search through accounting filesThe mechanism the command works with. § 12.5

How to check on the node

The format is visible in the first line of the file:

head -1 "$(ls -t /var/log/pmta/acct-*.csv | head -1)"

A line like type,timeLogged,rcpt,… is a CSV header. A line starting with a curly brace is NDJSON. For a webhook the content type is checked on the receiver side, in its web server log.

◇

What else to read

3 articles

◇What the node tells about itself and through which channelsSeven observation surfaces: the web monitor, the console in four formats, the log, accounting, accounting search, metrics and export to external analytics.◇Two inbound report processors and what to do with themBounces by message and complaints are parsed by two identically structured blocks; each has its own address list and its own pair of decisions about the message’s fate.◇Five ways for a queue to hand off a messageA queue does not have to use SMTP: it can write to a file, run a program, call over HTTP or discard.

In the Hub

Cost of a mistake: reversible: harms only while it is set
Scenarios: Collect telemetry