fluentbit postfix stream bounced emails to bigquery

Installing fluent bit described in docs

wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
echo 'deb https://packages.fluentbit.io/ubuntu/focal focal main' | sudo tee /etc/apt/sources.list.d/fluentbit.list
sudo apt update
sudo apt-get install td-agent-bit
sudo service td-agent-bit start
sudo systemctl status td-agent-bit

As described in docs we can run fluentbit without any configs by passing all configurations as command line arguments like:

/opt/td-agent-bit/bin/td-agent-bit -i systemd \
    -p systemd_filter=_SYSTEMD_UNIT=[email protected] \
    -p tag='host.*' -o stdout

Also we can grab some journals from /var/log/journal/xxxxxxxx/*.journal and pass Path parameter to fluentbit

parsers.conf

[PARSER]
    name postfix_smtp_bounced
    format regex
    regex (?<id>[^:]+): to=<(?<to>[^>]+)>, relay=(?<relay>(none|[^\[]+))(\[(?<ip>[^\]]+)\]:(?<port>\d+))?, delay=(?<delay>[^,]+), delays=(?<receive>[^\/]+)\/(?<queue>[^\/]+)\/(?<conn>[^\/]+)\/(?<send>[^,]+), dsn=(?<dsn>[^,]+), status=(?<status>[^ ]+) \((?<msg>[^\)]+)\)

There are many more interesting stuff in postfix logs, here are few other parsers

And here is a sample configuration which was used to parse dumped journal

td-agent-bit.conf

[SERVICE]
    Flush        1
    daemon       Off
    Log_Level    warn
    Parsers_File bqp.conf

[INPUT]
    name             systemd
    Systemd_Filter   [email protected]
    Systemd_Filter   _COMM=smtp
    Systemd_Filter_Type And
    Read_From_Tail   Off
    Path             /data/
    Strip_Underscores On
    alias smtp
    tag smtp
    Mem_Buf_Limit 128MB

[FILTER]
    Name record_modifier
    Match smtp
    Whitelist_key MESSAGE

[FILTER]
    name grep
    match smtp
    regex MESSAGE status=bounced

[FILTER]
    name parser
    match smtp
    key_name MESSAGE
    parser postfix_smtp_bounced

[FILTER]
    name grep
    match smtp
    # regex MESSAGE .+
    exclude MESSAGE .+

[FILTER]
    Name record_modifier
    Match smtp
    # Whitelist_key id
    Whitelist_key to

[OUTPUT]
    name   stdout
    format json_lines
    match smtp

[OUTPUT]
    name   bigquery
    match  smtp
    google_service_credentials /data/creds.json
    project_id demo
    dataset_id dev
    table_id   postfix
    skip_invalid_rows On
    ignore_unknown_values On
    Retry_Limit 5

With this setup we have our bounced emails in bigquery table

Notes:

While testing bigquery output can be commented out
To test whether all messages were parsed replace exclude MESSAGE .+ with regex MESSAGE .+ in filter
We can have multiple input -> filter -> output chains (e.g. if we want to have successfull emails side by side with bounced)

Additional headers

Same way as HTTP requests, emails have their headers which also can be logged

For this to work, somewhere in main.cf add

header_checks = regexp:/etc/postfix/header_checks

with contents like:

/^subject:/      WARN
/^X-FOO:/        INFO

after that in logs there will be records like:

Jan 23 16:29:08 demo postfix/smtpd[448165]: 4JflJ44pRZzVg5M: client=unknown[173.110.42.141]
Jan 23 16:29:08 demo postfix/cleanup[448186]: 4JflJ44pRZzVg5M: info: header X-Foo: hello from unknown[173.110.42.141]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<bsg>
Jan 23 16:29:08 demo postfix/cleanup[448186]: 4JflJ44pRZzVg5M: info: header Subject: hello from unknown[173.110.42.141]; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<bsg>
Jan 23 16:29:08 demo postfix/cleanup[448186]: 4JflJ44pRZzVg5M: message-id=<>
Jan 23 16:29:08 demo postfix/qmgr[448161]: 4JflJ44pRZzVg5M: from=<[email protected]>, size=438, nrcpt=1 (queue active)
Jan 23 16:29:08 demo postfix/smtpd[448165]: disconnect from unknown[173.110.42.141] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
Jan 23 16:29:09 demo postfix/smtp[448187]: 4JflJ44pRZzVg5M: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[108.177.14.26]:25, delay=0.96, delays=0.13/0.01/0.48/0.35, dsn=2.0.0, status=sent (250 2.0.0 OK  1642688949 g25si2880938lfh.217 - gsmtp)
Jan 23 16:29:09 demo postfix/qmgr[448161]: 4JflJ44pRZzVg5M: removed