Paddy Byers paddybyers

Summary of incident affecting the Ably production cluster 3 January 2019 - preliminary investigation and conclusions

Overview

See the incident on our status site at https://status.ably.io/incidents/574

There was an incident affecting the Ably production cluster on 3 January that caused a significant level of disruption for multiple accounts, primarily in the us-east-1 region.

The incident was most severe for a 2-hour period between 1520 and 1720 (all times in UTC) in which there were elevated error rates across all regions, with the greatest impact in us-east-1. During that time, channel attach error rates averaged nearly 50%, and message publish error rates averaged around 42%. For a further period of 2½ hours, as the system recovered, error rates were also escalated; taking the entire period of the incident, channel attach error rates averaged around 33% and message publish error rates averaged 22%.

Summary of incident affecting the Ably production cluster 3 December 2018 - preliminary investigation and conclusions

Overview

See incident on our status site at https://status.ably.io/incidents/572

There was an incident affecting the Ably production cluster on 3 December that caused a significant level of disruption for multiple accounts, primarily in the us-east-1 region, but with some adverse impact on other regions.

The incident lasted for a period of 65 minutes between 1520 and 1625 (all times in UTC) in which there were elevated error rates across all regions, with the greatest impact in us-east-1. During that time, channel creation error rates averaged 23%, and message publish rates averaged around 46%. The us-east-1 region was taken out of service, and traffic diverted to other regions, during that time. The us-east region was reinstated at 1839, without any further disruption.

	package io.ably.example.androidpushexample;

	import android.content.Intent;
	import com.google.firebase.messaging.FirebaseMessagingService;
	import com.google.firebase.messaging.RemoteMessage;
	import io.ably.lib.util.Log;

	import java.util.HashMap;
	import java.util.Map;

	using System;
	using System.Collections.Generic;
	using System.Linq;

	namespace IO.Ably.ConsoleTest
	{
	/// <summary>ILoggerSink implementation that outputs colored messages to console.</summary>
	class MyLogger : ILoggerSink
	{
	static readonly Dictionary<LogLevel, ConsoleColor> s_colors = new Dictionary<LogLevel, ConsoleColor>()

	"use strict";
	const fs = require('fs');
	const util = require('util');
	const slice = [].slice;
	const format = util.format;

	const unloggedIterationCount = 10000000;
	const loggedIterationCount = 10000;

	const fileName = './test.log';

	Java:

	String encrypt(String inputString) {
	IvParameterSpec ivSpec = new IvParameterSpec("c1234567-zipl-03".getBytes());
	SecretKeySpec keySpec = new SecretKeySpec("GeN8IfCrdxKxMxFA".getBytes(), "AES");

	Cipher cipher;
	try {
	cipher = Cipher.getInstance("AES/CBC/NoPadding");
	cipher.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);

	IvParameterSpec ivSpec = new IvParameterSpec("c1234567-zipl-03".getBytes());
	SecretKeySpec keySpec = new SecretKeySpec("GeN8IfCrdxKxMxFA".getBytes(), "AES");

	Cipher cipher = Cipher.getInstance("AES/CBC/NoPadding");
	cipher.init(Cipher.ENCRYPT_MODE, keySpec, ivSpec);
	return "1" + bytesToHex(cipher.doFinal("Test".getBytes()));

	console.log(process.argv[0]);
	console.log(process.argv[1]);
	console.log(process.argv[2]);
	var port = process.argv.length > 2 ? Number(process.argv[2]) : 1337;
	var http = require('http');
	var svr = http.createServer(function (req, res) {
	if(req.url.indexOf('exit') != -1) {
	res.write('exiting');
	res.end();
	svr.close();