Anthony Miyaguchi acmiyaguchi

Keybase proof

I hereby claim:

I am acmiyaguchi on github.
I am acmiyaguchi (https://keybase.io/acmiyaguchi) on keybase.
I have a public key whose fingerprint is A4EB A869 A93C 0409 2CBD 9BB8 C3BD 35E4 6DD4 5DA5

To claim this, I am signing this object:

Nightly Fennec on TaskCluster

These are some notes for the development of nightly fennec on TaskCluster.

Launching tasks

This is a script that I use to emulate the workflow of launching a task from a hook registered in the Hook Manager on tools.taskcluster.net. By default, this will launch a task definition that will start a fennec nightly.

A hook will start a start a task using the current head of a repository. This is a painful process to work locally since you have to edit the task definition to get the correct revision for every launch.

Running

This gist lets you keep IPython notebooks in git repositories. It tells git to ignore prompt numbers and program outputs when checking that a file has changed.

To use the script, follow the instructions given in the script's docstring.

For further details, read this blogpost.

The procedure outlined here is inspired by this answer on Stack Overflow.

Registering hyperloglog functions.

import org.apache.spark.sql.SparkSession
import com.mozilla.spark.sql.hyperloglog.functions._
import com.mozilla.spark.sql.hyperloglog.aggregates._

object UDFs{
  val HllCreate = "hll_create"
 val HllCardinality = "hll_cardinality"

Databricks Airflow Workflow

Procedure

Modified repositories:

	amiyaguchi> mshal: Thanks for the link to the bug, it was very informative. Are there examples of clients that look at artifacts stored at specific routes or that would break if builds were moved to a new route?
	10:32 AM <mshal> one sec, in a meeting
	11:42 AM <mshal> amiyaguchi: sorry about that! I know of two at least - mozregression and the artifact downloads
	11:42 AM <mshal> mozregression uses gecko.v2: https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/artifacts.py#610
	11:42 AM <mshal> whoops, that's the link for artifacts
	11:42 AM <mshal> mozregression is here: https://github.com/mozilla/mozregression/blob/96391566492f553becd0942630bf4ea99f954e75/mozregression/fetch_configs.py#L382
	11:43 AM <mshal> but artifact builds will hopefully be moved to gecko.v2 as well (bug 1250700), and then we can remove buildbot.X and gecko.v1.X (bug 1279221, bug 1277881)
	11:43 AM <firebot> https://bugzil.la/1250700 — NEW — Artifact builds should use the gecko.v2 index instead of buildbot.revisions.X
	11:4

	1:56 PM <aki> https://github.com/escapewindow/signingscript
	1:56 PM <aki> https://github.com/escapewindow/scriptworker
	1:57 PM <aki> http://scriptworker.readthedocs.io/en/latest/
	1:58 PM <aki> https://tools.taskcluster.net/auth/clients/

	virtualenv signingscript scriptworker

	https://tools.taskcluster.net/auth/clients/

	These are the

	[hadoop@ip-172-31-11-113 ~]$ aws s3 ls s3://net-mozaws-prod-us-west-2-pipeline-analysis/mreid/crash/v4
	PRE v4/
	2016-11-02 00:46:10 0 v4_$folder$
	[hadoop@ip-172-31-11-113 ~]$ aws s3 ls s3://net-mozaws-prod-us-west-2-pipeline-analysis/mreid/crash/v4/
	PRE crash_date=20161101/
	2016-11-02 00:46:10 0 crash_date=20161101_$folder$
	[hadoop@ip-172-31-11-113 ~]$ aws s3 ls s3://net-mozaws-prod-us-west-2-pipeline-analysis/mreid/crash/v4/crash_date=20161101/
	2016-11-02 00:50:48 0 _SUCCESS
	2016-11-02 00:50:49 9338 _common_metadata
	2016-11-02 00:50:49 126496 _metadata