Missing ending fragments, part 2

Bug 1185123 (part 2).ipynb

{"nbformat_minor": 0, "cells": [{"execution_count": 3, "cell_type": "code", "source": "import binascii\nimport pandas as pd\n\nfrom operator import attrgetter, itemgetter\nfrom moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client, get_clients_history\nfrom collections import defaultdict\nfrom __future__ import division\n\n%pylab inline", "outputs": [{"output_type": "stream", "name": "stdout", "text": "Populating the interactive namespace from numpy and matplotlib\n"}], "metadata": {"scrolled": true, "collapsed": false, "trusted": true}}, {"execution_count": 4, "cell_type": "code", "source": "sc.defaultParallelism", "outputs": [{"execution_count": 4, "output_type": "execute_result", "data": {"text/plain": "80"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Get all main pings for a set of recent build-ids:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 5, "cell_type": "code", "source": "build_ids = (\"20150710000000\", \"20150717999999\")\n\npings = get_pings(sc,\n                  app=\"Firefox\",\n                  channel=\"nightly\",\n                  build_id=build_ids,\n                  doc_type=\"main\",\n                  schema=\"v4\")\n\ncrashes = get_pings(sc,\n                    app=\"Firefox\",\n                    channel=\"nightly\",\n                    build_id=build_ids,\n                    doc_type=\"crash\",\n                    schema=\"v4\")", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Take a subset of nightly clients:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 6, "cell_type": "code", "source": "def sample(ping):\n    client_id = ping.get(\"clientId\", None)\n    return client_id and binascii.crc32(ping[\"clientId\"]) % 100 < 10\n\nsampled_pings = pings.filter(sample)\nsampled_crashes = crashes.filter(sample)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 7, "cell_type": "code", "source": "crashes_by_client = sampled_crashes.map(lambda c: (c[\"clientId\"], c[\"meta\"])).groupByKey().collectAsMap()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Get a subset of fields:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 8, "cell_type": "code", "source": "subset = get_pings_properties(sampled_pings, [\"clientId\",\n                                              \"meta/documentId\",\n                                              \"meta/submissionDate\",\n                                              \"meta/creationTimestamp\",\n                                              \"environment/system/os/name\",\n                                              \"payload/info/reason\",\n                                              \"payload/info/sessionId\",\n                                              \"payload/info/subsessionId\",\n                                              \"payload/info/previousSessionId\",\n                                              \"payload/info/previousSubsessionId\",\n                                              \"payload/info/subsessionCounter\",\n                                              \"payload/info/profileSubsessionCounter\",\n                                              \"payload/simpleMeasurements/firstPaint\",\n                                              \"payload/simpleMeasurements/savedPings\",\n                                              \"payload/simpleMeasurements/uptime\",\n                                              \"payload/histograms/STARTUP_CRASH_DETECTED\"])", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Group fragments by client and dedupe by documentId:", "cell_type": "markdown", "metadata": {}}, {"execution_count": 9, "cell_type": "code", "source": "def dedupe_and_sort(group):\n    key, history = group\n    \n    seen = set()\n    result = []\n    \n    for fragment in history:\n        id = fragment[\"meta/documentId\"]\n        if id in seen:\n            continue\n            \n        seen.add(id)\n        result.append(fragment)\n    \n    result.sort(key=itemgetter(\"payload/info/profileSubsessionCounter\"))\n    return result\n\ngrouped = subset.groupBy(lambda x: x[\"clientId\"]).map(dedupe_and_sort).collect()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "**< Digression>** What's the percentage of clients that have at least one pair of fragments with different documentIds but the same profileSubsessionCounter?", "cell_type": "markdown", "metadata": {}}, {"execution_count": 10, "cell_type": "code", "source": "def duplicate_pssc(grouped):\n    dupes = 0\n    dupe_clients = set()\n\n    for history in grouped:\n        counts = defaultdict(int)\n\n        for fragment in history:\n            key = fragment[\"payload/info/profileSubsessionCounter\"]\n            counts[key] += 1\n\n        for _, v in counts.iteritems():\n            if v > 1:\n                dupes += 1\n                dupe_clients.add(history[0][\"clientId\"])\n                break\n\n    print 100.0*dupes/len(grouped)\n    return dupe_clients\n    \ndupe_clients = duplicate_pssc(grouped)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "3.08981408746\n"}], "metadata": {"scrolled": true, "collapsed": false, "trusted": true}}, {"source": "**< /Digression\\>** Let's remove those clients to be safe.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 11, "cell_type": "code", "source": "dd_grouped = filter(lambda h: h[0][\"clientId\"] not in dupe_clients, grouped)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "Given the set of chain breaks, how many of them are due to missing starting/ending fragments?", "cell_type": "markdown", "metadata": {}}, {"execution_count": 40, "cell_type": "code", "source": "class AdjacentBreaks:\n    def __init__(self):\n        self.missing_total = 0\n        self.missing_start = 0\n        self.missing_end = 0\n        self.missing_both = 0\n        self.crashed_prev = 0\n        self.reason = defaultdict(int)\n    \n    \n    def process(self, prev, curr):        \n        if prev[\"payload/info/sessionId\"] == curr[\"payload/info/previousSessionId\"]:\n            # Ignore fake missing fragments? See IncrementError class\n            if prev[\"payload/info/reason\"] in (\"aborted-session\", \"shutdown\") and \\\n               curr[\"payload/info/subsessionCounter\"] == 1:\n                return\n            \n            self.missing_total += 1\n            self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n                    \n            # Are there missing starting fragments?\n            missing_start = curr[\"payload/info/subsessionCounter\"] != 1\n                    \n            # Are there missing ending fragments?\n            missing_end = prev[\"payload/info/reason\"] not in (\"aborted-session\", \"shutdown\")\n                    \n            if missing_start and missing_end:\n                self.missing_both += 1\n            elif missing_start:\n                self.missing_start += 1\n            elif missing_end:\n                self.missing_end += 1\n                \n            self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n    \n                \n    def stats(self, total):\n        print \"ADJACENT SESSIONS STATS\"\n        print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n        print \"{:5.2f}% of edges are missing one or more starting fragments\".format(100*self.missing_start/total)\n        print \"{:5.2f}% of edges are missing one or more ending fragments\".format(100*self.missing_end/total)\n        print \"{:5.2f}% of edges are missing both starting and ending fragments\".format(100*self.missing_both/total)\n        print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n        \n        print \"\"\n        print \"Reason distribution:\"\n        print dict(self.reason)\n        print \"\"\n    \n\nclass WithinBreaks:\n    def __init__(self):\n        self.missing_total = 0\n        self.crashed_prev = 0\n        self.reason = defaultdict(int)\n        \n        \n    def process(self, prev, curr):\n        if prev[\"payload/info/sessionId\"] == curr[\"payload/info/sessionId\"]:\n            self.missing_total += 1\n            self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n            self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n            \n            \n    def stats(self, total):\n        print \"WITHIN SESSIONS STATS\"\n        print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n        print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n        \n        print \"\"\n        print \"Reason distribution:\"\n        print dict(self.reason)\n        print \"\"\n\n        \nclass NonAdjacentBreaks:\n    def __init__(self):\n        self.missing_total = 0\n        self.reason = defaultdict(int)\n        self.difference = defaultdict(int)\n        self.crashed_prev = 0\n        \n        \n    def process(self, prev, curr):\n        if prev[\"payload/info/sessionId\"] != curr[\"payload/info/sessionId\"] and \\\n           prev[\"payload/info/sessionId\"] != curr[\"payload/info/previousSessionId\"]:\n            self.missing_total += 1\n            self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n            self.difference[curr[\"payload/info/profileSubsessionCounter\"] - prev[\"payload/info/profileSubsessionCounter\"]] += 1\n            self.crashed_prev += curr[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev, curr)\n            \n            \n    def stats(self, total):\n        print \"NON-ADJACENT SESSIONS STATS\"\n        print \"{:5.2f}% of edges have fragments missing\".format(100*self.missing_total/total)\n        print \"{:5.2f}% of edges have a crash in-between\".format(100*self.crashed_prev/self.missing_total)\n        print \"\"\n        \n        print \"Reason distribution:\"\n        print dict(self.reason)\n        print \"\"\n        \n        print \"Difference distribution:\"\n        dist = pd.Series(self.difference)\n        dist.sort_index()\n        print dist\n        \n        print \"\"\n        \n        \nclass IncrementError:\n    def __init__(self):\n        self.errors_total = 0\n        self.reason = defaultdict(int)\n        \n    def process(self, prev, curr):\n        if prev[\"payload/info/sessionId\"] == curr[\"payload/info/previousSessionId\"] and \\\n           prev[\"payload/info/reason\"] in (\"aborted-session\", \"shutdown\") and \\\n           curr[\"payload/info/subsessionCounter\"] == 1:\n            self.errors_total += 1\n            self.reason[\"{} -> {}\".format(prev[\"payload/info/reason\"], curr[\"payload/info/reason\"])] += 1\n            \n    def stats(self, total):\n        print \"PROFILESUBSESSIONCOUNTER INCREMENT ERRORS\"\n        print \"{:5.2f}% of edges have a mismatching profileSubsessionCounter\".format(100*self.errors_total/total)\n        print \"\"\n        \n        print \"Reason distribution:\"\n        print dict(self.reason)        \n        print \"\"\n\n\ndef has_crash_ping(prev, curr):\n    client_id = prev[\"clientId\"]\n    client_crashes = crashes_by_client.get(client_id, None)\n    \n    if client_crashes:\n        for crash in list(client_crashes):\n            if crash[\"creationTimestamp\"] >= prev[\"meta/creationTimestamp\"] and \\\n               crash[\"creationTimestamp\"] <= curr[\"meta/creationTimestamp\"]:\n                return True\n            \n    return False\n        \n\ndef missing(grouped):\n    broken_clients = set()\n    correct_clients = set()\n    num_broken_chains = 0\n    num_crashed = 0\n    total_edges = 0\n   \n    adjacent_breaks = AdjacentBreaks()\n    within_breaks = WithinBreaks()\n    non_adjacent_breaks = NonAdjacentBreaks()\n    increment_errors = IncrementError()\n    \n    for history in grouped:\n        correct_clients.add(history[0][\"clientId\"])\n\n        for i in range(1, len(history)):\n            prev_fragment = history[i - 1]\n            prev_pss_counter = prev_fragment[\"payload/info/profileSubsessionCounter\"]\n            \n            curr_fragment = history[i]\n            current_pss_counter = curr_fragment[\"payload/info/profileSubsessionCounter\"]\n\n            num_crashed += curr_fragment[\"payload/histograms/STARTUP_CRASH_DETECTED\"] or has_crash_ping(prev_fragment, curr_fragment)\n            total_edges += 1\n\n            # Is a fragment missing?\n            if prev_pss_counter + 1 != current_pss_counter:\n                broken_clients.add(curr_fragment[\"clientId\"])\n                num_broken_chains += 1\n                \n                adjacent_breaks.process(prev_fragment, curr_fragment)\n                within_breaks.process(prev_fragment, curr_fragment)\n                non_adjacent_breaks.process(prev_fragment, curr_fragment)\n                increment_errors.process(prev_fragment, curr_fragment)\n           \n    correct_clients = correct_clients.difference(broken_clients)\n    \n    print \"GENERAL STATS\"\n    print \"{:5.2f}% clients have a broken session chain\".format(100*len(broken_clients)/len(grouped))        \n    print \"{:5.2f}% of clients with a missing fragment experienced at least one crash\".format(100*len(broken_clients.intersection(crashes_by_client.keys()))/len(broken_clients))\n    print \"{:5.2f}% of clients without a missing fragment experienced at least one crash\".format(100*len(correct_clients.intersection(crashes_by_client.keys()))/len(correct_clients))\n    print \"{:5.2f}% of edges have a crash in-between\\n\".format(100*num_crashed/total_edges)\n    \n    increment_errors.stats(num_broken_chains)\n    adjacent_breaks.stats(num_broken_chains)\n    within_breaks.stats(num_broken_chains)\n    non_adjacent_breaks.stats(num_broken_chains)\n    \nmissing(dd_grouped)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "GENERAL STATS\n 4.51% clients have a broken session chain\n23.65% of clients with a missing fragment experienced at least one crash\n10.00% of clients without a missing fragment experienced at least one crash\n 2.44% of edges have a crash in-between\n\nPROFILESUBSESSIONCOUNTER INCREMENT ERRORS\n 3.72% of edges have a mismatching profileSubsessionCounter\n\nReason distribution:\n{'aborted-session -> environment-change': 4, 'aborted-session -> shutdown': 10, 'aborted-session -> aborted-session': 6, 'aborted-session -> daily': 1}\n\nADJACENT SESSIONS STATS\n19.82% of edges have fragments missing\n15.93% of edges are missing one or more starting fragments\n 3.89% of edges are missing one or more ending fragments\n 0.00% of edges are missing both starting and ending fragments\n 3.57% of edges have a crash in-between\n\nReason distribution:\n{'shutdown -> shutdown': 31, 'environment-change -> shutdown': 5, 'daily -> shutdown': 10, 'environment-change -> environment-change': 1, 'daily -> daily': 5, 'environment-change -> daily': 1, 'shutdown -> environment-change': 2, 'aborted-session -> shutdown': 2, 'aborted-session -> aborted-session': 13, 'shutdown -> aborted-session': 42}\n\nWITHIN SESSIONS STATS\n 1.95% of edges have fragments missing\n 0.00% of edges have a crash in-between\n\nReason distribution:\n{'daily -> daily': 1, 'daily -> aborted-session': 7, 'environment-change -> aborted-session': 3}\n\nNON-ADJACENT SESSIONS STATS\n74.51% of edges have fragments missing\n10.21% of edges have a crash in-between\n\nReason distribution:\n{'shutdown -> shutdown': 334, 'aborted-session -> environment-change': 1, 'environment-change -> shutdown': 1, 'daily -> shutdown': 1, 'environment-change -> environment-change': 2, 'shutdown -> daily': 22, 'environment-change -> daily': 1, 'aborted-session -> daily': 2, 'shutdown -> environment-change': 12, 'aborted-session -> shutdown': 20, 'aborted-session -> aborted-session': 5, 'shutdown -> aborted-session': 20}\n\nDifference distribution:\n2     210\n3      55\n4      28\n5      16\n6       8\n7       5\n8       8\n9       7\n10      4\n11      3\n12      1\n13      1\n14      4\n15      5\n16      2\n...\n146    1\n170    1\n171    1\n178    1\n192    1\n200    1\n240    1\n259    1\n268    1\n276    1\n286    1\n342    1\n410    1\n424    1\n428    1\nLength: 69, dtype: int64\n\n"}], "metadata": {"scrolled": false, "collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.9", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}