MTA-STS Cloudflare worker

mta-sts.js

// This worker is designed to be able to neatly handle MTA-STS policies for multiple domains.

// Make a new worker with this script and add your domains to the stsPolicies dict like the example.
// Add a DNS AAAA record for mta-sts.yourdomain.com pointing to 100:: and set to proxied,
// then add a workers route for mta-sts.yourdomain.com/* pointing to this worker.

// You should probably also create a Cloudflare configuration rule disabling Browser Integrity Check for the mta-sts subdomain
// to ensure MTAs aren't blocked from retrieving your policy.

// You'll still need to manually add the appropriate _mta-sts.yourdomain.com TXT record to enable the policy, 
// and the _smtp._tls.yourdomain.com TXT record for reporting.

const stsPolicies = {
  "yourdomain.com":
`version: STSv1
mode: enforce
mx: mail.yourdomain.com
max_age: 86400`
}


const respHeaders = {
  "Content-Type": "text/plain;charset=UTF-8",
  "X-Clacks-Overhead": "GNU Terry Pratchett, Jon Postel, Alan Turing, Dan Kaminsky"
}

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const reqUrl = new URL(request.url)

  if (!reqUrl.hostname.startsWith("mta-sts.")) {
    return new Response(`Incorrect worker route. mta-sts policies must be served on the mta-sts subdomain\n`, {status: 500, headers: respHeaders})
  }

  const policyHost = reqUrl.hostname.slice(8)

  if (!stsPolicies.hasOwnProperty(policyHost)) {
    return new Response(`${policyHost} is not defined in the mta-sts worker\n`, {status: 500, headers: respHeaders})
  }

  if (reqUrl.protocol !== "https:" || reqUrl.pathname !== "/.well-known/mta-sts.txt") {
    reqUrl.protocol = "https:"
    reqUrl.pathname = "/.well-known/mta-sts.txt"
    return Response.redirect(reqUrl, 301)
  }

  return new Response(stsPolicies[policyHost] + "\n", {status: 200, headers: respHeaders})
}

I would imagine it's most likely Cloudflare's Browser Integrity Check, which from memory is enabled by default and blocks requests from some User-Agents. The main one I run into is Python urllib's default UA, but it's possible CF has some MTA UAs in their naughty list. I'll add a note to the gist

With all due respect, Browser Integrity Check is not solely responsible for issuing a challenge. BIC is a mechanism in which Cloudflare fingerprints a browser to determine whether it's a legitimate browser or a headless device.

BIC is loosely based Google's Picasso - not to be confused with Picasa - Google's photo editor.

BIC is indeed taken into consideration when it comes to determining if a device is suspect - but there's a lot more that needs to happen before Cloudflare would take any action (automated). I can all but guarantee this is manually invoked by a WAF Custom Rule...especially if this is a Free/Pro/Biz customer. I would suspect something like Bot Management could be a culprit - but BM is a very expensive product and anyone employing it would know to create exclusions for automated checks (i.e. APIs - or automated checks like quering mta-sta.mydomain.com for a mail policy).

If you call out anything in your Gist - I would suggest adding something like this:

Ensure your Cloudflare security policy does not invoke a challenge for any requests to the URL associated with the Worker binding.

But it might be good to get a response back from @Ry3nlNaToR first to see if they can provide more insight into what actually happened.

The Cloudflare docs are quite clear that this is exactly the behaviour of BIC:

Cloudflare’s Browser Integrity Check (BIC) looks for common HTTP headers abused most commonly by spammers and denies access to your page.

It also challenges visitors without a user agent or with a non-standard user agent such as commonly used by abusive bots, crawlers, or visitors.

I can confirm that on one of my domains where I use this worker, on which I don't have any non-default rules configured, simply setting the User-Agent header to Python-urllib/3.11 is enough to get BIC to block the request.

Okie dokie - I will step away from the conversation... It's your post, anyway. Thanks so much for sharing the information in the first place! It was IMMENSELY helpful!

@Tugzrida @W4JEW Its not my CF account I host my policy on AWS using CloudFront and S3, anyway back the subject I asked my friend for more details wasn't a managed challenge it was a block as suspected by @Tugzrida it was due to BIC Service: Browser integrity check Action taken:Block the user agent was libwww-perl/6.68 disabling Browser Integrity Check would fix it or just do what you did WAF Custom Rule and skip all of the security features.