I was having issues with fail2ban not catching invalid dovecot login attempts because for whatever reason invalid login attempts we're going into the mail.log. So, here is what I changed to get it to work.
In /var/log/auth.log the failed dovecot login attempts look like
Jan 2 19:41:14 shittie auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=derp rhost=10.0.0.1
Awesome, failed attempt. However, the dovecot filter in fail2ban (v0.8.6) for dovecot wasn't catching failed attempts even when I switched it to look at /var/log/auth.log instead of /var/log/mail.log.
so, all I did was open up /etc/fail2ban/filter.d/dovecot.conf and replace the failregex line with
failregex = .*(?:dovecot:auth).*(?:authentication failure).*(rhost=)(?P<host>\S+)
Next all I had to do was modify my jail.local down around line 266
[dovecot]
enabled = true
port = smtp,ssmtp,imap2,imap3,imaps,pop3,pop3s
filter = dovecot
logpath = /var/log/auth.log
Reload fail2ban and that was it.