I was having issues with fail2ban not catching invalid dovecot login attempts because for whatever reason invalid login attempts we're going into the mail.log. So, here is what I changed to get it to work.
In /var/log/auth.log the failed dovecot login attempts look like
Jan 2 19:41:14 shittie auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=derp rhost=10.0.0.1
Awesome, failed attempt. However, the dovecot filter in fail2ban (v0.8.6) for dovecot wasn't catching failed attempts even when I switched it to look at /var/log/auth.log instead of /var/log/mail.log.
so, all I did was open up /etc/fail2ban/filter.d/dovecot.conf and replace the failregex
line with