Logstash decode mail message subject

We did tried to parse and send logs from MTA to BigQuery

But inside logs mail message subject was mime encoded and not readable aka something like =?UTF-8?B?0K7RgNC40YHRgiAo0JrQuNC10LIpICjQvdC+0LI=?=

Thankfully there is still a way to perform some Ruby code in pipeline

Ended up with following

input {
    stdin {}
}
filter {
    # extract json from incomming "message"
    json {
        source => "message"
    }
    # remove logstash fields
    mutate {
        remove_field => ["@version","@timestamp","path","host","type","message","event"]
    }
    # just for demo remove all fields except "timeLogged", "rcpt", "totalSecondsQueued" and "header_Subject"
    mutate {
        remove_field => ["type", "timeQueued", "orig", "orcpt", "dsnAction", "dsnStatus", "dsnDiag", "dsnMta", "bounceCat", "srcType", "srcMta", "dlvType", "dlvSourceIp", "dlvDestinationIp", "dlvEsmtpAvailable", "dlvSize", "vmta", "jobId", "envId", "queue", "vmtaPool", "timeFirstAttempt", "dlvTlsProtocol", "dlvTlsCipher", "rcvSmtpUser"]
    }
    # decode header
    ruby {
        init => "require 'mail'"
        code => "event.set('[subject]', Mail::Encodings.value_decode(event.get('[header_Subject]')))"
    }
    # remove encoded header
    mutate {
        remove_field => ["header_Subject"]
    }
}
output {
    stdout {}
}

Which gives us desired decoded output and can be further processed and send to bigquery

Demos were tested with container

docker run --rm -it -v ${PWD}/bigquery.conf:/usr/share/logstash/pipeline/pipeline.conf docker.elastic.co/logstash/logstash:8.5.0

by pasting to stdin sample log row