Compare search counts in paired v2 and v4 FHR pings.

search_pairs.ipynb

{"nbformat_minor": 0, "cells": [{"execution_count": 1, "cell_type": "code", "source": "from moztelemetry import get_pings, get_pings_properties\nimport ujson as json", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "Load v2-v4 paired pings", "cell_type": "markdown", "metadata": {}}, {"execution_count": 2, "cell_type": "code", "source": "pairs_dir_path = 's3n://net-mozaws-prod-us-west-2-pipeline-analysis/bcolloran/mergedDataPerClient/nightly/2015-07-09/8937clients/part-*'", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 3, "cell_type": "code", "source": "f = sc.sequenceFile(pairs_dir_path)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 4, "cell_type": "code", "source": "all_pairs = f.mapValues(json.loads)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "A function to find the overlapping window of active dates\nbetween the paired v2 and v4 pings, and perform some\noperation on them. With unit tests at:\n  https://gist.gitthub.com/SamPenrose/8316485d0b7d7ba881fc", "cell_type": "markdown", "metadata": {}}, {"execution_count": 5, "cell_type": "code", "source": "from collections import defaultdict\ndef get_overlap(pair, v2_extractor=None, v4_extractor=None):\n    v2_blobs = pair['v2'].get('data', {}).get('days', {}) # {'YYYY-MM-DD': dict}\n    v4_blobs = pair['v4'] # [{'creationDate': 'YYYY-MM-DD:...', 'k': val, ...}, ...]\n    # One blob per date in v2, multiple per date in v4\n    results = {'v2': {}, 'v4': defaultdict(list)}\n    if not (v2_blobs and v4_blobs):\n        return results\n\n    v2_dates = v2_blobs.keys()\n    v2_dates.sort()\n    v4_blobs.sort(key=lambda d: d['creationDate']) # probably redundant\n    v2_start, v2_end = v2_dates[0], v2_dates[-1] # possibly same\n    start = end = None\n\n    # Find overlap and walk v4 at same time.\n    for v4 in v4_blobs:\n        v4_date = v4['creationDate'][:10]\n        # Walk start up as far as we must.\n        if v4_date < v2_start:\n            # If v2 is entirely after v4, we never get past here.\n            continue\n        elif not start:\n            start = v4_date\n        # We have at least one overlapping date.\n        if v4_date <= v2_end:\n            # Walk end up as far as we can.\n            end = v4_date\n        else:\n            break\n        value = v4_extractor(v4) if v4_extractor else v4\n        results['v4'][v4_date].append(value)\n    if end is None: # We never reached last line of the loop.\n        return results\n\n    for v2_date in v2_dates:\n        if v2_date < start:\n            continue\n        if v2_date > end:\n            break\n        value = v2_extractor(v2_blobs[v2_date]) if v2_extractor else v2_blobs[v2_date]\n        results['v2'][v2_date] = value\n    return results", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 53, "cell_type": "code", "source": "import numbers\ndef v2_search_extractor(d):\n    total = 0\n    search_dict = d.get('org.mozilla.searches.counts', {})\n    counts = [v for k, v in search_dict.items() if k != '_v'\n              and isinstance(v, numbers.Number)]\n    total += sum(counts)\n    return int(total)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 73, "cell_type": "code", "source": "def v4_search_extractor(d):\n    '''\n    Count searches in v4_pings.\n    '''\n    total = 0\n    hists = d.get('payload/keyedHistograms/SEARCH_COUNTS')\n    if hists is None:\n        try:\n            hists = d.get('payload', {}).get('keyedHistograms', {}).get('SEARCH_COUNTS', {})\n        except AttributeError: # float sneaking in in Nightly somewhere\n            return 0\n    if hists == 'MISSING':\n        return 0\n    for blob in hists.values():\n        total += blob.get('sum', 0)\n    return total", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "v4_search_extractor() run manually on twenty pings.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 134, "cell_type": "code", "source": "twenty = [p[1] for p in all_pairs.take(20)]\ntovs = [get_overlap(p) for p in twenty]", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 74, "cell_type": "code", "source": "for ov in tovs:\n    for d in ov['v4'].values():\n        for blob in d:\n            print v4_search_extractor(blob),", "outputs": [{"output_type": "stream", "name": "stdout", "text": "0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 1 0 0 0 0 10 0 0 0 0 1 0 5 1 29 0 24 1 0 0 1 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 3 1 1 0 0 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 1 2 0 0 1 1 0 2 4 2 1 4 0 53 59 21 1 10 0 2 0 31 21 1 3 0 11 0 0 0 0 0 0 11 0 13 0 32 45 6 26 1 13 0 0 5 0 40 7 0 3 0 4 0 4 45 22 2 13 0 2 8 1 4 7 4 1 0 0 0 12 11 2 12 3 34 0 0 1 1 0 0 0 1 1 1 1 1 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 1 3 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 3 2 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 92, "cell_type": "code", "source": "sample = all_pairs.sample(False, 0.1, 37).map(lambda v: v[1])\nsovs = sample.map(lambda d: get_overlap(d, v2_search_extractor, v4_search_extractor))", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 96, "cell_type": "code", "source": "# Note that this 20 taken from sample is different from twenty\n# taken from all_pairs.\nsovs20 = sovs.take(20)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 103, "cell_type": "code", "source": "t2 = t4 = 0\nfor ov in sovs20:\n    for date in ov['v2'].keys():\n        t2 += ov['v2'][date]\n        t4 += sum(ov['v4'][date])\n        template = \" *%s-%s*  |\" if (ov['v2'][date] - sum(ov['v4'][date])) else \"%s-%s |\"\n        print template % (ov['v2'][date], sum(ov['v4'][date])),\nprint '\\n', t2, t4", "outputs": [{"output_type": "stream", "name": "stdout", "text": "0-0 |  *2-1*  | 1-1 | 5-5 | 0-0 | 2-2 |  *0-7*  | 0-0 | 0-0 | 2-2 | 0-0 | 0-0 |  *2-4*  | 0-0 | 2-2 | 3-3 |  *5-1*  | 3-3 | 5-5 | 5-5 | 6-6 | 8-8 | 4-4 |  *29-25*  | 3-3 |  *0-6*  | 5-5 | 6-6 | 5-5 | 11-11 | 7-7 | 3-3 |  *13-9*  |  *6-0*  |  *6-10*  | 16-16 | 3-3 | 3-3 |  *12-5*  | 6-6 | 2-2 | 4-4 | 3-3 | 5-5 | 3-3 | 6-6 | 0-0 | 4-4 | 3-3 | 1-1 | 18-18 |  *0-10*  | 7-7 | 1-1 | 4-4 | 14-14 | 1-1 | 4-4 | 7-7 | 4-4 | 16-16 | 1-1 | 3-3 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 |  *2-1*  | 4-4 |  *1-0*  | 9-9 | 14-14 | 0-0 | 0-0 | 0-0 |  *1-0*  | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 1-1 | 0-0 | 0-0 |  *0-2*  |  *0-1*  | 0-0 | 2-2 | 0-0 | 1-1 | 0-0 | 0-0 | 2-2 | 0-0 | 0-0 | 1-1 | 0-0 | 0-0 | 3-3 | 0-0 | 0-0 | 1-1 | 0-0 | 0-0 | 0-0 | 2-2 | 0-0 | 1-1 | 0-0 | 1-1 | 0-0 | 2-2 | 2-2 | 2-2 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 1-1 | 0-0 | 1-1 | 0-0 | 1-1 | 2-2 | 0-0 |  *2-1*  | 1-1 | 1-1 | 1-1 | 4-4 | 2-2 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 |  *1-0*  | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 1-1 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | 0-0 | \n370 371\n"}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 110, "cell_type": "code", "source": "def search_combiner(d):\n    '''\n    v2 has one blob per date; v4 has multiple. sum() the latter.\n    '''\n    v2_total = 0\n    v4_total = 0\n    for date in d['v2']:\n        v2_total += d['v2'][date]\n        v4_total += sum(d['v4'][date])\n    return (v2_total, v4_total)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 113, "cell_type": "code", "source": "combined = sovs.map(search_combiner)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 114, "cell_type": "code", "source": "c20 = combined.take(20)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 116, "cell_type": "code", "source": "for t in c20: print t,", "outputs": [{"output_type": "stream", "name": "stdout", "text": "(0, 0) (10, 9) (0, 7) (4, 6) (175, 160) (111, 121) (0, 0) (0, 0) (30, 28) (1, 0) (1, 3) (0, 1) (2, 2) (0, 0) (14, 14) (20, 19) (1, 0) (0, 0) (0, 0) (1, 1)\n"}], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 118, "cell_type": "code", "source": "import plotly.plotly as PL\nimport plotly.graph_objs as GO", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": 119, "cell_type": "code", "source": "v2c = combined.map(lambda tup: tup[0])\nv4c = combined.map(lambda tup: tup[1])", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 123, "cell_type": "code", "source": "v2r = v2c.collect()", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 124, "cell_type": "code", "source": "v4r = v4c.collect()", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 125, "cell_type": "code", "source": "trace = GO.Scatter(x=v2r, y=v4r)", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": 126, "cell_type": "code", "source": "data = GO.Data([trace])", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "If a point is on x == y, then the paired v2 and v4 pings had the same count.\nThis graphic does not show how many ping pairs share (x, y) values.\nOverall, it shows many pairs with very close agreement, and a tendency\nfor v4 to fall short of v2 when both are less than 400.", "cell_type": "markdown", "metadata": {}}, {"execution_count": 130, "cell_type": "code", "source": "PL.iplot(data, filename='searches-X-v2-Y-v4')", "outputs": [{"execution_count": 130, "output_type": "execute_result", "data": {"text/plain": "<plotly.tools.PlotlyDisplay object>", "text/html": "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\"seamless=\"seamless\" src=\"https://plot.ly/~mozilla/162.embed\" height=\"525\" width=\"100%\"></iframe>"}, "metadata": {}}], "metadata": {"collapsed": false, "trusted": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.9", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}