DKIM signing and verification filters for OpenSMTPD

README.md

Important

Current version of this code has moved into a proper GitHub repository: https://github.com/palant/opensmtpd-filters

The OpenSMTPD documentation currently suggests using either opensmtpd-filter-dkimsign or opensmtpd-filter-rspamd for DKIM support. The former lacks functionality and requires you to compile code from some Austrian web server yourself. The latter is overdimensioned for my needs. So I’ve written my own fairly simple filters in Python.

Prerequisites

These filters require Python 3 with dkimpy module installed. You can optionally install pyspf module as well, if you want dkimverify.py to perform SPF verification as well.

Setting up

Your smtpd.conf file should contain directives like the following:

filter dkimverify proc-exec "/usr/local/bin/dkimverify.py example.com"
filter dkimsign proc-exec "/usr/local/bin/dkimsign.py example.com:mydkim:/etc/mail/dkim/mydkim.key"

listen on eth0 tls filter dkimverify
listen on eth0 port 587 tls-require auth filter dkimsign

This sets up dkimverify filter for port 25 (incoming mail) and dkimsign filter for port 587 (outgoing mail).

dkimverify.py takes a single command line parameter: the host name to appear in the Authentication-Results email header. It will add a header like Authentication-Results: example.com; dkim=pass; spf=fail (sender is example.com/1.2.3.4) [email protected] to emails, this header can then be considered in further processing.

dkimsign.py takes one or multiple parameters of the form domain:selector:keyfile on the command line. Instead of configuring all domains on the command line, you can also pass this script -c /etc/mail/dkim/dkim.conf parameter, with the file /etc/mail/dkim/dkim.conf containing domain configurations in the same format, one per line.

opensmtpd.py module

The opensmtpd.py module here allows implementing OpenSMTPD filters easily. It is used like following:

from opensmtpd import FilterServer

server = FilterServer()
server.register_handler('report', 'link-auth', handle_auth)
server.register_handler('filter', 'connect', handle_connect)
server.serve_forever()


def handle_auth(session, username, result):
    if result == 'pass':
        print('Session {} authenticated'.format(session), file=sys.stderr)


def handle_connect(session, rdns, fcrdns, src, dest):
    if fcrdns == 'pass':
        return 'proceed'
    else:
        return 'junk'

See smtpd-filters man page for the description of the existing report events and filter requests and their parameters. The FilterServer class also exposes a convenience method register_message_filter() that allows filtering complete email messages:

  server.register_message_filter(handle_message)


  def handle_message(context, lines):
      return map(lambda line: line.replace('xyz', 'abc'), lines)

There is also method track_context(). If called during registration phase, the server will create a context object for each session and pass it to the handlers instead of the session ID.

dkimsign.py

#!/usr/bin/env python3
import argparse
import email
import re
import sys

from dkim import dkim_sign

from opensmtpd import FilterServer

def start():
    parser = argparse.ArgumentParser(description='DKIM signing filter for OpenSMTPD.')
    parser.add_argument('--config', '-c', metavar='config_path', help='Config file listing domain configurations (one per line)')
    parser.add_argument('domains', nargs='*', metavar='domain:selector:key_path', help='Domain configuration')
    args = parser.parse_args()

    if args.config:
        with open(args.config, 'r') as input:
            for line in input:
                line = line.strip()
                if line:
                    args.domains.append(line)
    if not args.domains:
        parser.print_help()
        sys.exit(1)

    config = {}
    for entry in args.domains:
        domain, selector, key = entry.split(':', 2)
        config[domain] = {'selector': selector, 'key': key}

    server = FilterServer()
    server.register_message_filter(lambda context, lines: sign(config, lines))
    server.serve_forever()


def sign(config, lines):
    parsed = email.message_from_string('\n'.join(lines))
    sender = parsed.get('from', '')
    match = re.search(r'<([^<>]+)>', sender)
    if match:
        sender = match.group(1)
    domain = re.sub(r'.*@', '', sender)
    if domain in config:
        with open(config[domain]['key'], 'rb') as input:
            key = input.read()
        signature = dkim_sign(
            '\n'.join(lines).encode('latin-1'),
            config[domain]['selector'].encode('latin-1'),
            domain.encode('latin-1'),
            key
        ).decode('latin-1')
        header, value = re.split(r':\s*', signature, 1)
        parsed[header] = value
        lines = parsed.as_string().splitlines(False)
    return lines


if __name__ == '__main__':
    start()

dkimverify.py

#!/usr/bin/env python3
import argparse
import email
import re
import traceback

from dkim import dkim_verify

from opensmtpd import FilterServer

try:
    import spf
except:
    spf = None


def start():
    parser = argparse.ArgumentParser(description='DKIM verifying filter for OpenSMTPD.')
    parser.add_argument('hostname', nargs='?', default='localhost')
    args = parser.parse_args()

    server = FilterServer()
    server.register_handler('report', 'link-identify', save_identity)
    server.register_handler('report', 'tx-mail', save_sender)
    server.register_message_filter(lambda context, lines: verify(server, args.hostname, context, lines))
    server.serve_forever()


def save_identity(context, method, identity):
    context['identity'] = identity


def save_sender(context, message_id, result, sender):
    context['sender'] = sender


def verify(server, hostname, context, lines):
    message = '\n'.join(lines)
    parsed = email.message_from_string(message)
    if 'authentication-results' in parsed:
        del parsed['authentication-results']

    dkim_result = 'unknown'
    if 'dkim-signature' in parsed:
        if dkim_verify(message.encode('latin-1')):
            dkim_result = 'pass'
        else:
            dkim_result = 'fail'

    if spf:
        try:
            ip = re.sub(r':\d+$', '', context['src'])
            ip = re.sub(r'^\[(.*)\]$', r'\1', ip)
            spf_result, code, message = spf.check(i=ip, s=context['sender'], h=context['identity'])
            clean = lambda value: re.sub(r'\s', '', value)
            spf_result = '{} (sender is {}/{}) smtp.mailfrom={}'.format(
                spf_result,
                clean(context['identity']),
                clean(ip),
                clean(context['sender'])
            )
        except:
            server.log_exception()
            spf_result = 'unknown'
        spf_result = '; spf={}'.format(spf_result)
    else:
        spf_result = ''

    parsed['Authentication-Results'] = '{}; dkim={}{}'.format(hostname, dkim_result, spf_result)
    return parsed.as_string().splitlines(False)


if __name__ == '__main__':
    start()

opensmtpd.py

#!/usr/bin/env python3
import os
import re
import sys
import traceback

class FilterServer:
    """Filter server implementation, communicates with OpenSMTPD via stdin and stdout."""

    def __init__(self):
        """Handles the initial communication with OpenSMTPD."""

        self._stdin = os.fdopen(sys.stdin.fileno(), 'r', encoding='latin-1', buffering=1)
        self._stdout = os.fdopen(sys.stdout.fileno(), 'w', encoding='latin-1', buffering=1)
        self._stderr = os.fdopen(sys.stderr.fileno(), 'w', encoding='latin-1', buffering=1)
        self._handlers = {}
        self._contexts = None
        while self.recv() != 'config|ready':
            pass


    def recv(self):
        """Low-level functionality. Receives one line from OpenSMTPD."""

        return self._stdin.readline().rstrip('\r\n')


    def send(self, line):
        """Low-level functionality. Sends one line to OpenSMTPD."""

        print(line, file=self._stdout)


    def log_exception(self):
        """Prints the current exception message to stderr."""
        traceback.print_exc(file=self._stderr)


    def register_handler(self, event, phase, handler):
        """Registers an event processor, has to be called before serve_forever().
        Supported event types are 'report' and 'filter'. Handlers for report
        events will receive session ID and phase-specific parameters as
        arguments, no response expected. Handlers for most filter events will
        receive the same parameters but have to return a result like 'proceed'
        or 'reject|550 Spam'. Handler for 'data-line' filter will receive an
        additional send_dataline handler as last parameter which can be called
        any number of times to send lines to OpenSMTPD. Return value is ignored
        for this filter.
        """

        key = '{}|{}'.format(event, phase)
        if key in self._handlers:
            raise Exception('Handler for {} is already registered'.format(key))
        self._handlers[key] = handler
        self.send('register|{}|smtp-in|{}'.format(event, phase))


    def _call_handlers(self, result_handler, event, phase, session, *args):
        key = '{}|{}'.format(event, phase)
        if key in self._handlers:
            try:
                if self._contexts is not None and key != 'report|link-connect':
                    session = self._contexts[session]
                result_handler(self._handlers[key](session, *args))
            except:
                self.log_exception()


    def _filter_response(self, version, kind, session, token, payload):
        if re.search(r'^0\.[1-4]$', version):
            self.send('|'.join([kind, token, session, payload]))
        else:
            self.send('|'.join([kind, session, token, payload]))


    def serve_forever(self):
        """Ends initialization phase and processes any requests coming from
        OpenSMTPD. This function never returns.
        """

        def noop(result):
            pass

        def send_filter_response(result):
            self._filter_response(version, 'filter-result', session, token, result)

        def send_dataline(line):
            self._filter_response(version, 'filter-dataline', session, token, line)

        self.send('register|ready')
        while True:
            line = self.recv()
            count = line.count('|')
            if count < 5:
                continue
            elif count == 5:
                event, version, timestamp, subsystem, phase, session = line.split('|', 5)
                payload = None
            else:
                event, version, timestamp, subsystem, phase, session, payload = line.split('|', 6)

            if event == 'report':
                args = []
                if payload is not None:
                    args = payload.split('|')
                if phase == 'tx-mail' and re.search(r'^0\.[1-4]$', version) and len(args) == 3:
                    # Older protocol versions had result and sender reversed
                    args[1], args[2] = (args[2], args[1])
                self._call_handlers(noop, event, phase, session, *args)
            elif event == 'filter':
                if payload is not None and '|' in payload:
                    token, payload = payload.split('|', 1)
                else:
                    token = payload
                    payload = None
                if phase == 'data-line':
                    self._call_handlers(noop, event, phase, session, payload, send_dataline)
                else:
                    args = []
                    if payload is not None:
                        args = payload.split('|')
                    self._call_handlers(send_filter_response, event, phase, session, *args)


    def track_context(self):
        """Calling this method before serve_forever() ensures that the first
        parameter passed to all handlers will be the session context rather
        than merely a session ID. A session context is a dict object containing
        the following keys by default: 'session', 'rdns', 'fcrdns', 'src',
        'dest'. These match the parameters of the link-connect report event.
        Handlers can add and modify context object at will.
        """
        if self._contexts is not None:
            return

        self._contexts = {}

        def handle_link_connect(session, rdns, fcrdns, src, dest):
            self._contexts[session] = dict(session=session, rdns=rdns, fcrdns=fcrdns, src=src, dest=dest)


        def handle_link_disconnect(context):
            del self._contexts[context['session']]


        self.register_handler('report', 'link-connect', handle_link_connect)
        self.register_handler('report', 'link-disconnect', handle_link_disconnect)


    def register_message_filter(self, handler):
        """Convenience method allowing to filter message bodies without
        registering multiple event handlers. The handler will be called with
        the session context and a list of message lines. It has to return a
        filtered list of lines. Note: this will call track_context().
        """

        def escape_line(line):
            if line.startswith('.'):
                return '.' + line
            else:
                return line

        def unescape_line(line):
            if line.startswith('..'):
                return line[1:]
            else:
                return line

        def handle_dataline(context, line, send_dataline):
            try:
                if line != '.':
                    context.setdefault('message_lines', []).append(unescape_line(line))
                else:
                    lines = handler(context, context.get('message_lines', []))
                    for l in lines:
                        send_dataline(escape_line(l))
                    send_dataline('.')
                    context.pop('message_lines', None)
            except:
                send_dataline('.')
                context['message_error'] = True
                context.pop('message_lines', None)
                raise


        def handle_commit(context, *args):
            if 'message_error' in context:
                del context['message_error']
                return 'reject|451 Internal server error'
            else:
                return 'proceed'


        self.track_context()
        self.register_handler('filter', 'data-line', handle_dataline)
        self.register_handler('filter', 'commit', handle_commit)

@skrause87 what do you mean by broken DKIM signature? Do you mean that it is not calculated correctly or something else?

@jarmo broken means, that DKIM verifiers like rspamd mark the mail as R_DKIM_REJECT. After some debugging at rspamd i found mismatching hash values inside the signature.

2023-09-14 13:29:34 #28619(rspamd_proxy) <63f5b4>; dkim; rspamd_dkim_check: bh value mismatch: 73794e7dc4b38643febef62cb793c44a6810b94713d6aee82dfef9b59de04a35 versus fd67b50a88067747f497666b704a925fdc84a4b1869e755dc6d5957cc8526742, try add LF; try adding CRLF 2023-09-14 13:29:34 #28619(rspamd_proxy) <63f5b4>; dkim; rspamd_dkim_check: bh value mismatch after added CRLF: 73794e7dc4b38643febef62cb793c44a6810b94713d6aee82dfef9b59de04a35 versus 97a5fd6a2b11f37512a8e2deda4add1f950bd7959cef37bee35ce 39e15f06787, try add LF 2023-09-14 13:29:34 #28619(rspamd_proxy) <63f5b4>; dkim; rspamd_dkim_check: bh value mismatch after added LF: 73794e7dc4b38643febef62cb793c44a6810b94713d6aee82dfef9b59de04a35 versus fc7051de34e1074b3bad12ed2e64cfd8c4f904b1e4eb45bae4c0085 8c81a6546

@palant @jarmo a leading dot in any body line can cause the problem. opened a bug report.
https://bugs.launchpad.net/dkimpy/+bug/2036189

@skrause87 I can confirm - this one e-mail I mentioned above with failing DKIM signature also had a dot at the beginning of a line and after I removed that dot then a valid signature was created by dkimpy. Good catch!

After some further research the problem is not related to dkimpy!

http://www.opendkim.org/libopendkim/dkim_body.html
"Dot stuffing and the terminating dot in the message body are expected to be removed by the caller." see the notes

openSMTPD masks a leading dot with an additional dot, which is a correct behaviour according to RFC.
https://www.rfc-editor.org/rfc/rfc5321#section-4.5.2

The problem was caused by the used OpenSMTPD -> dkimpy integration by the published dkimsign.py script, which does not comply with "Dot stuffing".

The solution was to manipulate the lines array before signing. So the script pays attention to "Dot stuffing".

for linekey, line in enumerate(lines):
if line.startswith('..'):
  lines[linekey] = line[1:]
  
signature = dkim_sign(
 '\n'.join(lines).encode('latin-1'),
 config[domain]['selector'].encode('latin-1'),
 domain.encode('latin-1'),
 key

@skrause87 but what if there are dots part of the message? For example, what if body is something like this:

previous line
... and so on

It's pretty common (https://editorsmanual.com/articles/ellipsis/) and I think (have not double checked) it's allowed by RFC too.

Maybe should check that there's only two dots on the entire line and not just starting with two dots?

Hej @jarmo,

only the first character of a message line is important. So if it's a dot, it will be masked with another dot.

Your example

previous line
... and so on

becomes

previous line
.... and so on

by OpenSMTPD.

For DKIM signing this has to be normalized back to your example. Signing with OpenSMTPD lines leads to a body hash that does not match the original message, because the recipient also generates a body hash from the "normalized" message.

We tested our solution in several ways and could not find any message with a wrong signature again.

@skrause87 got it. Thanks for the explanation and fix.

PS! Did you also experience the problem I described above with multi-part messages or have you not seen it yet?

@jarmo Could not confirm the multipart failure. Messages headers look nearly the same, but message will always pass dkim verification.

@skrause87 I tried your solution and managed to create e-mail body, which does not create a valid signature:

$ sendmail -f [email protected] -t [email protected]
dkim test!...
...
wohoo!
.

Notice the line starting with .... Now, I assumed that your code changed lines array in-place thus also returning it from the sign function.

Here's a solution which works for me even with the e-mail body above. Including the whole sign function for clarity:

def sign(config, lines):
    parsed = email.message_from_string('\n'.join(lines))
    sender = parsed.get('from', '')
    match = re.search(r'<([^<>]+)>', sender)
    if match:
        sender = match.group(1)
    domain = re.sub(r'.*@', '', sender)
    if domain in config:
        with open(config[domain]['key'], 'rb') as input:
            key = input.read()
        signature = dkim_sign(
            # handle dot stuffing http://www.opendkim.org/libopendkim/dkim_body.html and https://www.rfc-editor.org/rfc/rfc5321#section-4.5.2
            '\n'.join([line[1:] if line.startswith("..") else line for line in lines]).encode('latin-1'),
            config[domain]['selector'].encode('latin-1'),
            domain.encode('latin-1'),
            key
        ).decode('latin-1')
        header, value = re.split(r':\s*', signature, 1)
        lines = signature.splitlines(False) + lines
    return lines

Difference between my and your code is that I remove additional dot only for the input for signing by dkim_sign function, but do not change the original lines array so it will have all the dots there.

However, it seems that there is still a bug somewhere in opensmtpd itself because even after not changing original lines array, signature will be valid, but the body, which is sent to the recipient will only have two dots where in the original body there were three - this means that opensmtpd is removing a dot from the original message body, which it should not be doing and that's why signature will be valid. So I guess filters should get raw body lines without any dot-stuffing magic going on so that DKIM signing should not change contents of any lines, but calculating signature AS IS and dot stuffing should happen in the end before sending out e-mail by opensmtpd.

Anyway, the code provided by me seems to be working with all e-mails as long as opensmtpd does remove dots from the original message body (e.g. keeping two dots instead of three, which seems to be a bug).

PS! I also simplified adding signature to the beginning of e-mail regarding the multi-part body problem I discovered since To: header might not even exist.

Ahaa! It seems that this ... becoming into .. is somehow sendmail related or I should have entered these dots with already stuffed - when I used Thunderbird to create similar body then all three dots were preserved.

I can confirm that this works as expected:

$sendmail -f [email protected] -t [email protected]
dkim test!...
....
wohoo!
.

There will be three dots in the end-result and signature will be valid.

@jarmo @skrause87 Thank you for looking into this issue and in particular for figuring out how escaping in OpenSMTPd works (not sure about now but this definitely wasn’t documented back when I wrote this script). I updated the register_message_filter() function in opensmtpd.py to handle this correctly. You can see the patch in the dmarc2html repository which is using the same module: palant/opensmtpd-filters@e097734

@palant thanks for your comment! I agree that your change seems more appropriate related to dot-stuffing than the changes above. Tested and it works. However it looks like you missed the problem I also brought out about signature ending inside multi-part body sometimes.

However it looks like you missed the problem I also brought out about signature ending inside multi-part body sometimes.

I didn’t miss it, I merely cannot reproduce an issue here. And there doesn’t seem to be anything wrong with your example either. Yes, the signature is placed after the Content-Type header here – but that’s perfectly fine, the ordering of the headers is irrelevant. The multipart body only starts at the ------=_Part… line, and that one comes after the signature.

@palant thanks for putting your time into it - now, after reading your reply I took a second look on my comment here and agree that my example should not have caused any problems at the time. I restored the sign function into the same one in this gist and now it works (e.g. signature passes).

I suppose it was somehow related to the "dot-stuffing" at time too, but I don't have any idea how changing signature header order made signature PASS at the time...