With the recent chaos at Twitter and the future of the platform looking unclear, many are looking for alternatives. One popular alternative that has seen tremendous growth over the last several weeks has been Mastodon. Mastodon builds on a federated set of individual communities, each backed by their own running server instance. It's too early to tell if Mastodon will be the "replacement" for Twitter, but it does appear to be facilitating a different kind of discussion format.
If you are looking to join Mastodon, it is best to jump into one of the existing communities and join the conversations ongoing there. However, you can also start your own community and host your own Mastodon instance. You can always follow people in different communities as part of the fediverse.
This document briefly describes how to get Mastodon running on Fly.io. That's actually not 100% accurate because it actually uses a combination of services. For a small server with few users, you can operate a Mastodon instance with minor spend by leveraging the free tiers of many services. This is somewhat of a cobbled mess, so if you want a quick solution the Digital Ocean pre-built instances are likely easier to get started with.
Caveat, this is for a small setup at the moment. It is unclear how this will scale or which components will be bottlenecks in the future.
UPDATE: Nov 24, 2022: This was updated to use Cloudflare R2 in place of AWS S3. This required setting an additional environment variable. See the revisions of this doc for instructions on how to use AWS S3.
UPDATE: Feb, 2023: Upstash redis on Fly.io now supports HyperLogLog, so this article was updated to use the free Fly.io Redis instead.
For this setup I'm using the following active services. I don't have any skin in the game on any of these, so it should be relatively unbiased.
- Google Domains: domain registrar
- Fly.io: VMs, Redis, and Postgres
AWS: S3- Cloudflare: R2, DNS, and cached hosting of Mastodon Assets and user-uploaded content
- Sendgrid: email delivery
As to costs, the current usage of Fly.io comes in under their minimum monthly charge of $5, so the costs are waived.
This document walks through the approximate order I would use to set up these services. However, I built this with a lot of trial and error, so this was not the order of operations I followed initially.
You will need a custom domain for your community. Chose your favorite domain registrar, it shouldn't matter too much which you chose. I used Google Domains because I had bought domains from there before.
For the remainder of this doc I'll assume a domain of mastiff.party,
a Mastodon dedicated to the love of Mastiffs. (I have no idea if this
is an active community or not)
You will need a service that can reliably deliver emails for sign up email verification, notifications and other transactional updates. You should be able to use any service that offers SMTP delivery. I chose Sendgrid, they have a free plan that allows up to 100 emails/day. I have few users on my instance, so that felt like plenty.
Sign up for an account and enter your domain name. You will need to
authenticate your domain and verify a sender address. Follow the
instructions for domain authentication, it will require adding some
records to your DNS settings from the registrar. Then you'll want to
setup a single sender that emails will be sent from. Following the
example here, I used [email protected] as my sender
address.
Ensure that link branding is turned off, even if the domain verification says that it should work. SSL problems mean that the links won't work. Use the direct Sendgrid links for now (see Hiccup below).
Lastly, setup an API key in your settings and remember the key value. You'll use this when setting up the configuration for your site.
My setup uses Cloudflare for a few aspects:
- R2 object storage, similar to AWS S3
- Exposing the R2 bucket as public under the domain name
files.mastiff.party - Caching of the Mastodon static assets, as
assets.mastiff.party - General DNS
Create a new account with Cloudflare or start a new site. During the setup you will be instructed on how to move your DNS to Cloudflare. This is not an optional step.
Once setup, first create a R2 bucket in Cloudflare, for this guide I'll name it
files-mastiff-party. Once created go to the "Manage R2 API Tokens"
to grab the key and secret key. These will be set as the AWS_*
variables below.
Under the new R2 bucket, grab the endpoint URL under the bucket name
at the top of the page. Second, go to Bucket Settings and create a
domain under the section "domain access". Enter the name
files.mastiff.party, it will then create a DNS entry for files
under your site that points to the bucket. This will expose the files
in the bucket as public even though they are by default private.
In the environment variables, shown later in the fly.toml, you will
need to set S3_PERMISSION=private. This prevents Mastodon from
trying to use a public-read object ACL that is not supported on
Cloudflare. Instead, the custom domain exposes the objects as public.
In your DNS settings you will want to setup the following records:
- A record:
mastiff.party(@root) -> Fly.io IPv4 address (configured next)- Do not configure Proxying
- AAAA record:
mastiff.party-> Fly.io IPv6 address configured next)- Do not configure Proxying
- CNAME record:
assets->mastiff.party- Configure Proxying (caching)
Under the SSL/TLS settings for your domain, enable Full (strict) encryption mode. Otherwise Cloudflare will attempt an HTTP connection to your origin server on Fly and get an HTTPS redirect. This will end up causing a redirect loop.
Second, under Rules -> Transform Rules, select the Header Modification box on the right. You will need to create a HTTP Response Header Modification rule. These are the settings I used:
- rule name: access-control-allow-origin
- Under matches:
- Field: Request method
- Operator: is in
- Value: GET, POST, HEAD, OPTIONS
- Then,
- Enable Set static
- Header name:
Access-Control-Allow-Origin - Value
*
This CORS header rule will permit Javascript resources loaded from
assets.mastiff.party to access mastiff.party.
The remainder of the components will be setup on Fly.io. Mastodon provides pre-built Docker images on Dockerhub so it is easy to simply use those directly. I've broken the steps out by the component pieces.
An example fly.toml is attached to this gist with the relevant
values that should match the rest of the config in this doc. I'm going
to use the Fly app name of mastiff for this example.
The app is split between three VMs: web, sidekiq and streaming. This is my current scaling sizes for the VMs. You will definitely need more than the default 256MB for web and sidekiq or you'll see OOM's even at small scale. Streaming doesn't appear to need as much so can be kept at 256MB.
- web
- Size: shared-cpu-1x
- Memory: 1GB
- sidekiq
- Size: shared-cpu-1x
- Memory: 1GB
- streaming
- Size: shared-cpu-1x
- Memory: 256MB
To get started, the shared-cpu-1x Postgres VM with 256MB of memory
should be fine. Provision the DB with flyctl and once it is running
attach it to your Mastodon application. This will set the
DATABASE_URL in the app allowing Rails to connect.
Mastodon relies on Redis as part of its architecture. The Redis provided on Fly.io is based on the Upstash distribution. For a small site, the free Redis tier of 100MB was fine to get started with.
Just start with the following:
$ flyctl redis createFollow the steps to provision a 100MB instance in the same region as your VMs.
Record the password exported by fly redis status <instance> in the "Private URL"
and set that as the secret REDIS_PASSWORD. You'll notice the REDIS_HOST is part
of the fly.toml and should match the Private URL.
You'll want to follow the steps
here to setup
your Fly application with your custom mastiff.party domain. This
will be where you'll plug the IPv4 and IPv6 addresses into your
Cloudflare DNS.
When you add your custom domain Fly will auto-provision an SSL cert
for your domain mastiff.party using Let's Encrypt. However,
Cloudflare will need to proxy requests for assets.mastiff.party to
your site and hence will need a wildcard cert as well. Use the
flyctl command to create a wildcard
cert. (Unfortunately
you can not change the name of the origin Cloudflare uses to proxy to
in their free plans).
Along with the configuration in the provided fly.toml, you will need
to set the following secrets with flyctl secrets set. I've tried to
limit this to only values that are actually secret. All other values
are specified in the fly.toml.
AWS_SECRET_ACCESS_KEY The secret user key for the R2 bucket
DATABASE_URL Set automatically when the Postgres DB is attached
OTP_SECRET Generated below
REDIS_PASSWORD Redis password, see Redis section above.
SECRET_KEY_BASE Generated below
SMTP_PASSWORD From Sendgrid's API auth creds
VAPID_PRIVATE_KEY Generated below
VAPID_PUBLIC_KEY Generated below
For these commands you should deploy the Mastodon container and then
use the following to access the VM: flyctl ssh console. Once you
have a terminal in the app, cd to /mastodon. This is where the app
code exists in the VM.
- Setup the DB:
RAILS_ENV=production bundle exec rake db:create db:schema:load
- Generate the secrets, this will provide
SECRET_KEY_BASEandOTP_SECRET
bundle exec rake secret
- Generate the web push secrets, will provide
VAPID_*_KEYvalues
bundle exec rake mastodon:webpush:generate_vapid_key
See the provided fly.toml example for the remaining settings. A few
mentions:
- ALTERNATE_DOMAINS: any potential name that may be used in a request should be listed here or else Rails will 403 the request
- CDN_HOST: Domain that will be used to cache and offload the static
assets, make sure to include the scheme (
https://) here or else you'll have problems - S3_ALIAS_HOST: This is the Cloudflare proxy rule that will expose the R2 assets under your custom domain name in case you want to move them in the future
- STREAMING_API_BASE_URL: This allows websocket streaming to work by requesting the streaming on a different port. In theory you would have nginx in front of your app to redirect those requests to the node.js app, but since we're running that in a separate VM this was a work around. Would love to know if there's a better way to handle this on Fly.
This should be all you need to get the site loading. Deploy and check
out https://mastiff.party. The monitoring logs on Fly are also
critical to debugging any potential issues.
Once you have the server running and have created a user, you can give
yourself admin privileges by logging into the rails console (see
above) and running the following (replace myusername):
RAILS_ENV=production ./bin/tootctl accounts modify myusername --role admin
I believe that is the full setup, but again I arrived here somewhat through trial and error. Let me know if anything needs clarification.
During my earlier trial and error I messed up the DB enough that I
figured I'd just delete it and start over. Unfortunately, deleting the
DB doesn't appear to automatically detach it from the running
apps. This isn't terrible until you attempt to reattach a new DB
because it will complain that DATABASE_URL is already defined.
I had to set the experimental auto_rollback = false option in
fly.toml because I couldn't delete the secret
DATABASE_URL. Removing it clearly failed the app so it would
continually rollback to the previous version. Once I deleted this I
could attach a new DB back again.
During my early trail and error I found that account images were not working. This was when I was trying to find an S3 solution before settling on the original. I found this command that appeared to fix things and redownload the right media:
Log into the VM with flyctl ssh console
cd /mastodon
./bin/tootctl accounts refresh --all --verbose
Sendgrid allows link branding by having you setup a custom subdomain
like <foo>.mastiff.party that will point back to Sendgrid. Email
links will appear under your domain instead of
Sendgrid's. These links don't use SSL, which normally would
be fine. However, Fly sets the response header
strict-transport-security: max-age=63072000; includeSubDomains when
accessing your Mastodon site at mastiff.party. Browsers will respect
that and try to access the email links using https, but Sendgrid won't
have an accurate SSL cert and you'll see the big danger page.
I haven't explored this much since I was OK with the Sendgrid branded email links. I'm guessing I could use Cloudflare to host these links behind https if needed.
This is very cool! By the way, Upstash Redis now supports
Hyperloglog. https://docs.upstash.com/redis/overall/rediscompatibility