Dustin's proposal for a new in-tree task-graph generation system

00-TODO.md

Main Goals

• Schedule all branches using new taskgraph

• Implement optimization

• Factor legacy away

  • tests
  • builds, etc (notes)

• use the new big-graph scheduler (garndt)

  • make all tasks without other dependencies depend on the decision task
  • refactor mozilla-taskcluster to create the decision task with the appropriate schedulerid, taskgroupid
  • mozilla-taskcluster should use, and grant, the repo's role to decision tasks
  • use the taskGroupId from mozilla-taskcluster in mach taskgraph decision

• mop-up

  • fix platforms (pmoore?)
  • combine test kinds (wcosta)
  • better serialization/unserialization
    • remove root option to taskgraph.from_json
  • ensure routes match

Distractions

Tools / Utilities

• dot output
• filtering of output (mach taskgraph tasks --label build-opt-linux64)

Nits

• move docker images out of testing/docker into taskcluster/docker
• fix docker worker to not require cache scopes to be listed explicitly
• allow --parameters to take a URL (or a taskid) and fetch from there
• remove TOOLTOOL_REPO - we should always be using an in-image version of tooltool anyway
• install fixed flake8, tox versions in lint image
• rename the "cross-opt" config variant to "cross-opt-st-an"
• update (or remove) caches document

Enhancements

• factor out directory hashing (and use requests, and check artifact expiration for about-to-expire artifacts)
• combine all invocations of mozharness into a single way of doing it (so, combine test-linux and build-linux.sh)
• optimization:
  • use an Azure table, etc. for optimization
  • allow concurrent requests
  • refactor so each task has a list of optimizations?
  • log a reason something was optimized
• error checking
  • check things against docs where possible (e.g., attributes, parameters, kinds, caches) -- just parse the docs with a regex
  • check that (TH platform / symbol) is unique
  • check that index paths are unique

01-builds.md

TODO

• re-implement builds
• refactor into a few kinds, based on feedback
• re-implement jobs (TODO in graphdiff.py)
• re-implement post-build (TODO in graphdiff.py)
• implement job optimization based on 'files'
• unify compare.py and graphdiff.py to actually compare everything
• verify docker images
• fix try, mapping jobs and builds properly
• test and fix branch-specific stuff (caches, etc.)
• is image opt not working anymore?
• check that everything has from future
• jobs, job run-using, documented
• use build transforms to set attributes based on job name (??)
• secrets: true everywhere, instead of [*].
• docstrings for all common job methods; same args; name docker_worker_..
• use add_gecko_vcs_env_vars everywher
• run tests on each commit
• all kind.ymls have comments, transforms at the top
• all new modules have docstrings
• all attributes documented
• all transforms have docstrings
• test something with android partner builds (try)
• check that JSON/un-JSON still work
• optimization.only-if-files-changed -> when.files-changed
• update faq/how-tos to outline kind approaches
• another pass for TODO
• tests again
• lint!
• taskdiff commits
• remove index routes where signed off, including unnecessary product options
• update kinds.rst
• fix -j support (job attribute? remove from attrs.rst?)
• one more lint and test run-through

Review fixes;

• test more interesting try pushes
• rebase
• Callek - add diff in https://reviewboard.mozilla.org/r/77436/#review75970
• remove 'all_builds_and_tests' from target_tasks.py in final taskdiff commit
• why are image builds not optimizing? why are hashes different?
• callek - add pine to b2d-device run-on-projects
• callek - more branch-specific tests for target_tasks.py:all_builds_and_tests
• ahal - move mozlint into mozlint.yml
• ahal - use mach run-using for some lint tasks if possible
• ahal - indent list in python-lint.yml
• ahal - just "..not require a build" in kind.yml
• maja - remove newline in a taskdiff commit
• sfink - taskdiff commit to set mulet run-on-projects: [] and add delete it from shell-haz
• sfink - sm-package/opt descr to "Spidermonkey source package and test"
• sfink - use defaults for files-changed public, src, remove from sm-package -
• sfink - taskdiff commit to remove buildbot indexes from spidermonkey
• sfink - move using: spidermonkey to job-defaults
• sfink - remove explicit tier for sm-msan/opt
• sfink - add a note to docs that rnu-on-projects [] means "nowhere by default"
• mshal - fix mozreview ID problem re "drop gecko.v1 routes"
• ehsan - reword kind.rst to indicate that st-an builds are built, just not uploaded
• bhearsum - make PR for redo
• Ms2ger - extra already exists in "# only set up treeherder information if the task contained any to begin with" in legacy.py
• pmoore - whitespace in how-tos.rst, transforms.rst
• pmoore - write a new task implementation => write a new kind implementation how-tos.rst -
• mshal - fixed moved/removed routes.json, make sure it's not in temp kinds
• mshal, new bug - use underscores in route templates
• mshal, new bug - For the first, the plan is to modify routes.json such that both tools can use it so, same variable names -, and pull that data into both tools. I'll probably change to something other than JSON so I can include comments, and those comments will include a note that once BB is gone, the file can be removed and its functionality folded into task.py.
• mshal, new bug - For the second, the plan is to include a whitelist of v2 job names in task.py, and if a job comes through that's not on the whitelist, throw an exception sugesting double-checking that the index routes haven't changed from what buildbot generates, and assuming that checks out to add the new job-name to the whitelist. Again, the whitelist can be deleted once we're completely migrated and will have comments to that effect.
• gps - invert if index: logic in taskcluster/taskgraph/transforms/task.py, rename vars to something else, use .get('treeherder', {}.get('tier', 3) instead of try/except with []
• gps, newbug - rewrite == try to level == 1
• gps - use backticks in transforms.rst
• gps - raise exception in job/__init__, fix docstring on run_job_using
• gps - make run_task only optionally do checkout (in rebase?)
• gps - mock requests in test_files_changed
• gps - use key rather than tuple arg files_changed
• gps - log matching file at debug level in files_changed
• gps - include response.json() in retry operation
• gps - add a comment about _cache
• gps - alphabetize build kind
• gps - early return in common.py
• gps - remove "Ideally" from mozharness.py, fix triple-space, fix mh_command slashes, only use r when necessary
• gps - factor common code in mulet.py
• jlund - partner description should be "Android armv7 API 15+ partner sample 1"
• jlund - drop @run_job_using("docker-engine", "mozharness-via-build.sh")
• jlund - mulet jobs use builder image; fix comment on line 37, check dependencies
• gps - kinds.rst -
• gerard-majax - add pine to b2g-device run-on-project
• prettify "triggers ridealong" logging

Followup Bugs

• Merge test kinds & base on jobs
• update try-by-default bug to dup of 1301762
• update l10n tasks to modern standards (Callek)
• tc(Mn-h) marionette-harness/opt -- needs to be refactored to invoke like other things (Maja)
• android-stuff kind (Nick)

10-transform-docs-notes.txt

Docs:
 - Kinds
   - list of kinds
   - Implementing Kinds
     - Tests in general; specific types point here
       - yaml files
       - platform splitting
       - link to transforms
 - Transforms
   - Transforms in general
     - schema validation
     - documentation
     - get_keyed_by
   - Task generation transforms
     - `config` contents -- point to source
   - Test-specific transforms
     - `config` contents -- point to source
 - Tutorial
   - add a test
   - modify how a test works
   - make sweeping changes or introduce a special case
     - enumerate options, encourage best choice

Refactor:
 - support methods for transforms in base
 - validation optional -> mandatory
 - test.py -- some way to link between test.py and make_task_description.py?
 - consistent way to represent deps for tests (just dict? docker-image-{} after all)
 --- later
 - get rid of class methods (h/t gps)
 - use a different dir than taskgraph/kinds (i.e., tasks)

Fix:
 - remove extra transform stuff from TestTask
 - use kwargs in get_keyed_by tests

Tests:
 - check docstrings for transforms
 - check that every kind is documented
 - treeherder
 - TestTask

20-optimization.md

We want to remove as many tasks from the graph as possible, as efficiently as possible.

Observation: a task cannot be optimized if one or more of its dependencies cannot be optimized. So "no" flows downstream in the graph. This is the most effective way to minimize the amount of redundant checking for optimizability of tasks.

Even so, we may consider tasks which are depended on by tasks that will eventually be optimized away -- wasted effort. Ideally this can be countered by a kind of lazy evaluation where we do the minimum amount of work. For example, in some cases it may be possible to determine that a task can be optimized, without determining the task it should be replaced with.

So, optimization proceeds recursively from the "leaves" (nodes on which nothing depends, like tests) to the roots. First, each node is queried for whether it can be optimized. Each node begins this determination by asking the same of its dependencies; once a dependency says "no", the answer for this node is "no".

The graph is then traversed again, with any optimizable leaf nodes trimmed entirely -- there is no point in replacing them. Non-leaf nodes that cannot be optimized are passed over. Non-leaf nodes that can be optimized are asked for their replacement taskId, and traversal stops there.

---

Parallelizing calls to the index API and such can wait until we port this to Python3 and asyncio.

29-try-syntax.txt

overall
-------

old way: flags say what to parse
new way: parse it all, filter by flags

transition:
 * parse '-b do -p all -u all', tag with filename (and build filename) as attribute, filter on that
 * add sufficient attributes via yml files to allow filtering
   - some kind of testing to ensure things work OK

NOTE: talos not supported
NOTE: additional-parameters not used

proposed attributes:
 ** do we want k/v attributes or just string matches?
 - kind={legacy,build,test,..}
 - build-type={dbg, opt} (-b, expanded)
 - platform={linux,linux64,..} (-p, post aliases)
 - unittest-suite={mochitests,..} (-u, post aliases)
 - talos-suite={..} (-t)
 - chunk=N
 - job=.. (-j)

what to do about post-build jobs like symbol uploads?
 - select them with the corresponding build jobs
 - still need to be able to enable/disable for different branches

load_tasks
----------

load branch-appropriate flags/builds file
call parse_commit with it
collect set of changed files from vcs
filter out results with non-matching "when" (by implication all downstream tasks too)
for each result:
  load build template with lots of parameters
  format URLs for build products
  lots of magic transformations
  for each post-build:
    make task with configure_dependent_task (set requires, copy over TH config
  for each test:
    for each chunk:
      make task with configure_dependent_task
      for range(trigger_tests):
        add task

parse_commit
------------

jobs arg = from testing/taskcluster/tasks/branches/try/job_flags.yml; see below

args.jobs = -j, comma-separated
build_types = ['opt', 'debug']
platforms = list of build names (-p)

parse_test_opts parses -u (-t is totally ignored!); parses testname[platform]
into [{'test': 'name', 'platforms': ['..']}]; platforms omitted if not specified
tests = [{test: .., platforms: [..], only_chunks: [..]}, ..]

loop over platforms (-p)
  platform builds is jobs['builds'][platform] (skip if missing)
  loop over build types (-b)
  platform_build = platform_builds['types'][build_type], so {'task': <path>, (optional) additional_parameters: {}}
  adds post-build based on flags.post-build, where alloewd_build_tasks matches (or doesn't exist) (anthony)
  adds "dependents" w/ extrac_tests_from_platform:
    for each test in tests (from options):
      if build job path doesn't match alloewd_build_tasks, ignore
      if build_platform['platforms'] doesn't match test['platforms'], ignore
      add test_task {'allowed_build_tasks': ..}
      copy only_chunks into test_task if given
add all tasks aka jobs that are not excluded with -j (including via tags, but there are no tags)

result is
 task: <path>
 post-build:
   - allowed_build_tasks: ..
     - <build path>
     - <build path>
     - ..
     task: <path>
 dependents:
   - allowed_build_tasks:
     <build path>:
       task: <test path>
 additional-parameters: {..}
 build_name: <plat name>
 build_type: <debug or opt>
 interactive: t/f
 when:
   file_patterns: ..


job_flags
---------
flags:
  aliases:
    fb: foobar

  builds:
    - <build names>

  tests:
    - <test names>

  post-build:
    - upload-symbols

builds:
  <build name>
    platforms:
      - b2g, Android, Linux, etc.
    types:
      opt:
        task: <path>
      debug:
        task: <path>

post-build:
  upload-symbols:
    allowed_build_tasks:
      - <paths>
    task: <path>

tests:
  <test name>:
    allowed_build_tasks:
      <build path>:
        task: <test path>

tasks:  # -j flag (because why give it one name when you can give it two instead)
  <task>:
    task: <path>
    root: t/f -- if false, don't run
    (tags: ..) -- basically aliases, apparently unused
    when:
      file_patterns:
        ..

99-done.md

(1258497) Create taskgraph mach commands and in-tree implementation

Minimum viable product, still pulling from the existing YML files, and with few or no changes to try behavior.

• create taskcluster/mach_commands with mach subcommands

• create a LegacyKind that can parse the existing YAML files

• handle parameters as a data structure

• make it clear that kinds can keep local state

• refactor TGG as a generator

• pick better data types

• write a decision task that stops short of creating tasks

• automate comparison of old and new decision tasks

• write a new try parser

• support only target_task_method; one of those methods can be "read from parameters['target_set']; decision task should separately dump its target set

• dump parameters in yaml, not json

• parse try syntax, applying aliases etc., into a set of attribute patterns

• add platform, build_type, test_suite, etc. attributes to all tasks

• switch to filtering try tasks based on attributes, so always generate the full task grpah

• move target_tasks_xxx methods out of the decision task and use them for mach taskgraph target too

• support named dependencies, fixup tasks to refer to them

• update docs

• unit testing for taskgraph framework (not for legacy support)

• hook tests up to a task that runs in try (mach python-test)

• actually create tasks in decision task

• remove sandbox stuff and just use yml

• set up scopes correctly to create tasks

• fix decision task artifact paths

• fix task linking so that task IDs in URLs are correct

• successful try push

• review/needinfo comments

  • longer variable names for (l, r, n)
  • document root arg to TaskGraphGenerator
  • graph.py: note that edges link nodes
  • graph.py: s/anme/name/
  • unittest_suite and unittest_flavor are unrelated to treeherder
  • make all decision task args required, and -p for taskgraph commands
  • better handling for missing keys in parameters (use a dict with better KeyError?)
  • try to move more decision logic out of mach_commands.py
  • replace RuntimeError with Exception in Properties
  • use with open in load_properties and yaml.load, etc.
  • don't use KeyError for duplicate tasks
  • LegacyKind docstring
  • split mach_util refactor into a separate commit
  • update copyright headers
  • simplify test aliases
  • fold refactor cset in
  • wrap task set method stuff into TaskGraphGenerator so there's none of this set_target_tasks madness
  • folded armen's renaming stuff into "add partial tests for legacy kind" - put it in the right place
  • remove unused imports in taskcluster/taskgraph/test/test_target_tasks.py
  • remove __slots__
  • no [] when feeding a generator to a set (set literal??) taskcluster/taskgraph/try_option_syntax.py
  • fix negated task platforms
  • update sm-* jobs
  • grammar nits for old docs
  • don't mention diffing in describing task labels

• review round II

  • license header for mach_util.py
  • re-order patches 3 and 4
  • make sure docs build (mach doc)
  • examples of unittest_suite
  • check doc refs
  • line wrap docs
  • check docs numbering
  • use a DL in the docs
  • use keyword arguments with @SubCommand
  • move longer command descriptions to docstrings, with one-line descriptions
  • use mozunit runner
  • drop second argument to show_taskgraph
  • remove optimization
  • try yet another set of variable names in graph.py
  • fix Graph docstrig (3-tuple)
  • sort nodes into deque in visit_postorder
  • test if nodes have been seen before adding to deque, then remove redundant check
  • use defaultdict(set) without a lambda
  • use explicit list in static sets in tests
  • test visit_postorder with dijoint sets
  • move make_empty into constructor for TryOptionSyntaxParser
  • fix spaces in braces
  • more docstrings for test specs
  • use a request.Session to create tasks
  • docstring for taskgraph_decision
  • import yaml at module level
  • fix ref to PARAMETERS.md
  • use relative imports
  • use a dict comprehension to get params from options
  • move imports to top level in taskgraph modules
  • attributes = attributes or {}
  • raise TypeError if attrs aren't strings

• Make kind a class of Task

• call types/kinds in post-order so they can generate dependencies, or iterate to a fixed point

Related

• support -t in try parser
• Try commands with "-p win32" spawn jobs on other platforms
• "Pulling docker image has failed" anytime the image task runs
• retriggering doesn't work with big-graph scheduler
• optimized task graph has no dependencies
• --rebuild doesn't work (armen)

(1286075) builds and tests

• build task cleanup
• test and lint the new code

Tools / Utilities

• JSON output
  • with jq compatibility (does it accept JSON streams?)
• label-only output
• enable comparing task graphs

Nits

• fix yml dumping to not prefix everything with !!python/unicode, duh
• add a task to run mach taskgraph python-tests
• remove explicit hgfingerprint secret scope
• better logging: hook python logging up to self.log in the mach commands
• create tasks in parallel
• document how to navigate between kinds, kind impls, and transforms
• remove extra.build_* from all the build tasks -- it was only used by legacy.py
• look harder at how marionette-harness works and simplify
• use a common load_yaml method
• all_builds_and_tests needs something more reliable than filtering by kinds
• redo's logging is oo-glay