dannycoates

Friends, Links, Reactions

I just read a great article that was really cool and I want to share it with my friends. How should I do it? Twitter, Facebook, email? I guess it depends. Twitter's pretty good but its easy to miss things in the stream. I don't know what Facebook will do with it or who will see it. Email is only for things I think are important to specific people.

All I really want is two things:

• to tell my friends "this is cool" or "this made me [laugh|angry|sad|happy]"
• to see what my friends are sharing and their reactions

Lets make that super easy!

dannycoates

Sync Metrics

Goal

Metrics help us understand how people are using our products which in turn helps us improve them in ways that increase the value folks get out of them. We aren't interested in using metrics to turn data about our users into a product that we can sell, we just want to provide the best possible user experience.

Artifacts

Engagement and retention are our bread and butter metrics. By improving these measurements we can be confident that we're improving the user experience.

dannycoates

-- this produces a table of daily retention of new users over a period of 10 days

CREATE TABLE cohorts10 AS
SELECT DAY,
       period,
       users,
       retained
FROM
  ( SELECT r.DAY,
          (f.DAY - r.DAY) AS period,

dannycoates

Experimenting with AWS Lambda for ETL

A lot of us are interested in doing more analysis with our service logs so I thought I'd share an experiment I'm doing with Sync. The main idea is to transform the raw logs into something that'll be nice to query and generate reports with in Redshift.

The Pipeline

Logs make their way into an S3 bucket (lets call it the 'raw' bucket) where we've got a lambda listening for new data. This lambda reads the raw heka protobuf gzipped data, does some transformation and writes a new file to a different S3 bucket (the 'processed' bucket) in a format that is redshift friendly (like json or csv). There's another lambda listening on the processed bucket that loads this data into Redshift.

dannycoates

var Zlib = require('zlib')
var AWS = require('aws-sdk')
var HekaDecodeStream = require('heka-decode-stream')
var through = require('through2')

var s3 = new AWS.S3()

exports.handler = function (event, context) {
	var i = 0
	var record = event.Records[0].s3

dannycoates

Data

To produce the statistics that we're currently interested in, here are the core data points I've identified for each category.

MAU / Device Usage / Retention

• account_id (uid)
• account_created_at
• device_id
• device_created_at

dannycoates

Device info

Here are the fields stored on the server for each device a user adds.

{
  // The user this device belongs to
  "uid": "4c352927cd4f4a4aa03d7d1893d950b8",
  // The active session for this device
  "sessionToken": "27cd4f4a4aa03d7d186a2ec81cbf19d5c8a604713362df9ee15c4f4a4aa03d7d",

dannycoates

Best Berry Cobbler

• Topping

  • 1 1/2 cups all-purpose flour
  • 1/2 cup firmly packed light brown sugar
  • 1 1/2 teaspoons double-acting baking powder
  • 3/4 teaspoon salt
  • 1 tablespoon cinnamon

• 1 tsp cardamom

dannycoates

var redis = require('redis')

function match(event, filter) {
  var filterNames = Object.keys(filter)
  for (var i = 0; i < filterNames.length; i++) {
    var name = filterNames[i]
    if (event[name] && event[name] === filter[name]) {
      continue
    }
    else {

dannycoates

server.register(
  {
    register: require('hapi-able'),
    options: {
      git: 'git://github.com/dannycoates/able-demo.git',
      addRoutes: true
    }
  },
  function (err) {
    if (err) {