ashfurrow

May 12 2018 Intermittent Outage Retrospective

On May 12, for about five hours in the afternoon EDT, mastodon.technology experienced intermittent outages, lasting from twenty seconds to five minutes. The outages were caused in the course of an investigation into a failure to backup the database. The database is fine, and turned out that the backup method itself wasn't working properly.

Timeline

March 10, 2018: I upgrade the instance to Mastodon 2.3.0, from 2.1.x. After the upgrade, I notice Sidekiq keeps segfaulting and dropping jobs. I investigate and open an issue, which is later resolved. However, in the process of investigation, I upgrade my host's Docker version.

May 12, 2018, Around Noon EDT: I notice that local backups are not nearly as large as they should be. The offsite backups on S3 indicate the problem started on the March 11. I reproduce the failure to run pg_dump, which hangs every time. It gets stuck around the same table

ashfurrow

ashfurrow

#!/bin/sh
df -H | grep '/mnt/volume-nyc1-01' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 80 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)"
    curl -s --user "api:$MAILGUN_API_SECRET" \
      https://api.mailgun.net/v3/mastodon.technology/messages \

ashfurrow

The Eidolon Build Process

Since we originally built Eidolon – an auction bidding kiosk app – the project has largely remained in maintenance mode. Eidolon was one of the first projects that we used automated deploys for, and the deploy process has remained largely unchanged. I believe this stability of the deploy process is a testament to how well the automated deploys have gone.

This post is going to detail the mechanics of automated deploys for an enterprise-distributed iOS application, discuss lessons we learned and applied to other projects' deploy processes, and describe some of the changes we'd like to make. Our project is entirely open source, so you can check out any part of the code on your own or open an issue with questions.

Deploying Eidolon

It's one command on the terminal to deploy Eidolon:

ashfurrow

Apps to install from macOS App Store:

• Pastebot
• GIF Brewery
• Slack
• Keynote/Pages/Numbers
• 1Password
• OmniFocus 3
• Airmail 3
• iA Writer

ashfurrow

Pod::Spec.new do |s|
  s.name = 'ReactiveCocoa'
  s.version = '4.2.1'
  s.summary = 'A framework for composing and transforming streams of values.'
  s.description = <<-EOS
    ReactiveCocoa (RAC) is an Objective-C framework for Functional Reactive Programming.
    It provides APIs for composing and transforming streams of values.
  EOS
  s.homepage = 'https://github.com/ReactiveCocoa/ReactiveCocoa'
  s.license = { :type => 'MIT', :file => 'LICENSE.md' }

ashfurrow

Over the weekend, I lamented the death of Dropbox. Well, they're not dead, just dead to me. I discussed setting up BItTorrent Sync and some of the security problems with common setup tutorials, and got some great feedback. A few people pointed to Sync (referral link), which after investigating, I'm pleased to say is awesome and has become my Dropbox replacement.

My goal online is to minimize the surface area I expose to internet mass surveillance. An absolutist approach would be to only use open source software, with keys that I generate, and on a server that I physically control. That's just... way too much work. The reason that Dropbox is so popular is that it's so convenient, and I was hoping to get something like Dropbox in terms of convenience, but with a reduced risk of being surveilled.

And this is why I like Sync: it's premise is really simple. It's a folder, and

ashfurrow

Over the weekend, I lamented the death of Dropbox. Well, they're not dead, just dead to me. I discussed setting up BItTorrent Sync and some of the security problems with common setup tutorials, and got some great feedback. A few people pointed to Sync (referral link), which after investigating, I'm pleased to say is awesome and has become my Dropbox replacement.

My goal online is to minimize the surface area I expose to internet mass surveillance. An absolutist approach would be to only use open source software, with keys that I generate, and on a server that I physically control. That's just... way too much work. The reason that Dropbox is so popular is that it's so convenient, and I was hoping to get something like Dropbox in terms of convenience, but with a reduced risk of being surveilled.

And this is why I like Sync: it's premise is really simple. It's a folder, and

ashfurrow

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>DVTConsoleDebuggerInputTextColor</key>
	<string>1 0.721569 0.819608 1</string>
	<key>DVTConsoleDebuggerInputTextFont</key>
	<string>Monaco - 14.0</string>
	<key>DVTConsoleDebuggerOutputTextColor</key>
	<string>0.760784 1 0.87451 1</string>

ashfurrow

Title

Building Open Source Communities Without Burnout

Duration

25 minutes.

Abstract