Sync Azure DevOps Boards GitHub external connections to include all repositories per org (with dry-run, verification, and browser-context fallback)

sync_ado_github_connections.py

#!/usr/bin/env python3
"""
ADO GitHub connection sync runbook (single-file memory)
=======================================================

What this script does
---------------------
- Synchronizes Azure DevOps Boards external GitHub connections so the selected
  repositories match all repos discoverable for each target GitHub organization.
- Uses ADO Contribution providers:
  - connection inventory: ms.vss-work-web.azure-boards-external-connection-data-provider
  - repo discovery:      ms.vss-work-web.github-repository-search-data-provider
  - apply selection:     ms.vss-work-web.github-unified-installation-experience-data-provider
- Protects against cross-connection conflicts by skipping repos already owned by
  another connection in the same project.

Important behavior
------------------
- Selection updates are replace-like in ADO UI/backend. Always submit the full
  desired set (this script does that) to avoid dropping previously selected repos.
- For some recreated connections, repo discovery may require browser headers
  context (cookie, x-tfs-session, serviceHost).

Quick usage
-----------
1) Dry-run + list visible connections:
   python3 sync_ado_github_connections.py \
     --token '<ADO_BEARER_TOKEN>' \
     --org your-github-org-1 \
     --org your-github-org-2 \
     --dry-run \
     --print-connections

2) Apply sync:
   python3 sync_ado_github_connections.py \
     --token '<ADO_BEARER_TOKEN>' \
     --org your-github-org-1 \
     --org your-github-org-2 \
     --print-connections

3) If ADO discovery returns zero/unexpected repos, run with browser context:
   python3 sync_ado_github_connections.py \
     --token '<ADO_BEARER_TOKEN>' \
     --cookie '<FULL_COOKIE_HEADER_VALUE>' \
     --tfs-session '<X_TFS_SESSION_FROM_BROWSER>' \
     --service-host '<SERVICE_HOST_FROM_BROWSER_ROUTE_VALUES>' \
     --org your-github-org \
     --print-connections

When --service-host is needed
-----------------------------
- Usually optional.
- Use it when repo discovery returns 0, TF400864, or permission/context errors
  even though the same action works in the ADO web UI.
- This happens when ADO requires the exact ContributedPage routing context used
  by your browser session.

How to find serviceHost
-----------------------
1) Open Azure DevOps in browser:
   Project Settings -> Boards -> External connections.
2) Open DevTools (Network) and capture the request to:
   /_apis/Contribution/HierarchyQuery
3) In request payload (dataProviderContext.properties.sourcePage.routeValues),
   copy the `serviceHost` value exactly.
4) Pass that value to `--service-host`.

Example source path in payload:
- sourcePage.routeValues.serviceHost
- e.g. "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (your-host-name)"

How to find auth/context values (token, cookie, x-tfs-session)
--------------------------------------------------------------
1) Open Azure DevOps in browser:
   Project Settings -> Boards -> External connections.
2) Open DevTools -> Network, then trigger a repo list action (search/filter/paginate)
   so a request to `/_apis/Contribution/HierarchyQuery` appears.
3) Open that request and copy:
   - bearer token:
     Request Headers -> `authorization: Bearer <...>` (copy only the token part).
   - cookie:
     Request Headers -> `cookie: ...` (entire value).
   - tfs session:
     Request Headers -> `x-tfs-session: ...`.
   - service host:
     Request Payload -> `dataProviderContext.properties.sourcePage.routeValues.serviceHost`.
4) Pass values to script:
   - `--token '<token>'`
   - `--cookie '<cookie>'`
   - `--tfs-session '<x-tfs-session>'`
   - `--service-host '<serviceHost>'` (only when needed)

Security notes
--------------
- Treat token/cookie/session as secrets.
- Do not commit them into source code, gists, or shell history files.
- Prefer temporary shell variables and rotate/re-sign-in if they are exposed.

4) If multiple connections exist for the same org, pin the connection ID:
   python3 sync_ado_github_connections.py \
     --token '<ADO_BEARER_TOKEN>' \
     --org your-github-org \
     --connection your-github-org=<CONNECTION_ID>

Reading output
--------------
- available: repos discovered for the org
- current: currently selected in the target connection
- blocked_elsewhere: repos linked to a different connection (excluded)
- desired_submit: full set sent to ADO for this connection
- verification_ok selected=N: post-apply state validated
"""

import argparse
import json
import sys
import time
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Set, Tuple

import requests
from requests import RequestException

SEARCH_PROVIDER = "ms.vss-work-web.github-repository-search-data-provider"
ADD_PROVIDER = "ms.vss-work-web.github-unified-installation-experience-data-provider"
CONNECTIONS_PROVIDER = "ms.vss-work-web.azure-boards-external-connection-data-provider"


@dataclass
class Connection:
    id: str
    org: str
    is_valid: bool
    repos: Dict[str, str]  # id -> full_name


class AdoClient:
    def __init__(
        self,
        base_url: str,
        project: str,
        bearer_token: str,
        timeout: int = 120,
        cookie: Optional[str] = None,
        tfs_session: Optional[str] = None,
        service_host: Optional[str] = None,
    ):
        self.base_url = base_url.rstrip("/")
        self.project = project
        self.timeout = timeout
        self.service_host = service_host
        self.endpoint = f"{self.base_url}/_apis/Contribution/HierarchyQuery"
        self.session = requests.Session()
        headers = {
            "accept": "application/json;api-version=5.0-preview.1;excludeUrls=true;enumsAsNumbers=true;msDateFormat=true;noArrayWrap=true",
            "accept-language": "en-US,en;q=0.9",
            "content-type": "application/json",
            "authorization": f"Bearer {bearer_token}",
            "origin": self.base_url,
            "referer": f"{self.base_url}/{self.project}/_settings/boards-external-integration",
            "x-requested-with": "Vss-Fetch",
            "x-tfs-fedauthredirect": "Suppress",
            "x-vss-clientauthprovider": "MsalTokenProvider",
            "user-agent": "Mozilla/5.0",
        }
        if cookie:
            headers["cookie"] = cookie
        if tfs_session:
            headers["x-tfs-session"] = tfs_session
        self.session.headers.update(headers)

    def source_page(self) -> Dict[str, Any]:
        route_values: Dict[str, Any] = {
            "project": self.project,
            "adminPivot": "boards-external-integration",
            "controller": "ContributedPage",
            "action": "Execute",
        }
        if self.service_host:
            route_values["serviceHost"] = self.service_host
        return {
            "url": f"{self.base_url}/{self.project}/_settings/boards-external-integration",
            "routeId": "ms.vss-admin-web.project-admin-hub-route",
            "routeValues": route_values,
        }

    def query(self, provider: str, properties: Dict[str, Any]) -> Dict[str, Any]:
        payload = {"contributionIds": [provider], "dataProviderContext": {"properties": properties}}
        last_err: Optional[Exception] = None
        for attempt in range(1, 5):
            try:
                resp = self.session.post(self.endpoint, json=payload, timeout=self.timeout)
                if resp.status_code >= 400:
                    raise RuntimeError(f"{resp.status_code}: {resp.text[:800]}")
                return resp.json()
            except (RequestException, RuntimeError) as err:
                last_err = err
                if attempt == 4:
                    break
                time.sleep(attempt * 2)
        raise RuntimeError(f"provider={provider} failed after retries: {last_err}")

    def list_connections(self) -> List[Connection]:
        props = {"includeInvalidConnections": True, "sourcePage": self.source_page()}
        data = self.query(CONNECTIONS_PROVIDER, props)
        provider = data.get("dataProviders", {}).get(CONNECTIONS_PROVIDER, {})
        conns = provider.get("externalConnections", []) if isinstance(provider, dict) else []
        out: List[Connection] = []
        for c in conns:
            if not isinstance(c, dict):
                continue
            cid = c.get("id")
            svc = c.get("serviceEndpoint") or {}
            org = svc.get("gitHubHandle")
            is_valid = bool(svc.get("isEndpointValid"))
            repos: Dict[str, str] = {}
            for r in c.get("externalGitRepos", []) or []:
                rid = r.get("id")
                name = r.get("name")
                if isinstance(rid, str) and isinstance(name, str):
                    repos[rid] = name
            if isinstance(cid, str) and isinstance(org, str):
                out.append(Connection(id=cid, org=org, is_valid=is_valid, repos=repos))
        return out

    def list_org_repositories(self, org: str) -> Dict[str, str]:
        repos: Dict[str, str] = {}
        token: Optional[str] = None
        seen_tokens: Set[str] = set()
        for _ in range(50):
            props: Dict[str, Any] = {
                "filter": "",
                "orgOrUserName": org,
                "isUserType": False,
                "strongBoxKey": "useWellKnownStrongBoxLocation",
                "sourcePage": self.source_page(),
            }
            if token:
                props["continuationToken"] = token
            data = self.query(SEARCH_PROVIDER, props)
            provider = data.get("dataProviders", {}).get(SEARCH_PROVIDER, {})
            rows = provider.get("data", []) if isinstance(provider, dict) else []
            for r in rows:
                if not isinstance(r, dict):
                    continue
                rid = r.get("id")
                add = r.get("additionalProperties") or {}
                full = add.get("repoNameWithOwner")
                name = r.get("name")
                if isinstance(rid, str):
                    if isinstance(full, str) and full.startswith(org + "/"):
                        repos[rid] = full
                    elif isinstance(name, str):
                        repos[rid] = f"{org}/{name}"
            token = provider.get("continuationToken") if isinstance(provider, dict) else None
            if not isinstance(token, str) or not token or token in seen_tokens:
                break
            seen_tokens.add(token)
        return repos

    def apply_connection_selection(self, org: str, connection_id: str, selected_ids: List[str]) -> Dict[str, Any]:
        props = {
            "orgName": org,
            "externalRepositoryExternalIds": selected_ids,
            "existingConnectionId": connection_id,
            "selectedRepoNodeIds": selected_ids,
            "sourcePage": self.source_page(),
        }
        return self.query(ADD_PROVIDER, props)


def choose_target_connection(org: str, connections: List[Connection], override_id: Optional[str]) -> Connection:
    scoped = [c for c in connections if c.org == org]
    if override_id:
        match = next((c for c in scoped if c.id == override_id), None)
        if not match:
            raise RuntimeError(f"connection override id not found for org {org}: {override_id}")
        return match
    if not scoped:
        visible_orgs = sorted({c.org for c in connections})
        raise RuntimeError(
            f"no connection found for org {org}. visible_orgs={visible_orgs}"
        )
    scoped.sort(key=lambda c: (not c.is_valid, -len(c.repos), c.id))
    return scoped[0]


def parse_org_mapping(values: List[str]) -> Dict[str, str]:
    out: Dict[str, str] = {}
    for v in values:
        if "=" not in v:
            raise RuntimeError(f"invalid --connection format: {v}")
        org, cid = v.split("=", 1)
        out[org.strip()] = cid.strip()
    return out


def main() -> int:
    ap = argparse.ArgumentParser(description="Guarantee org repos are selected in ADO GitHub connections.")
    ap.add_argument("--base-url", default="https://dev.azure.com/your-organization")
    ap.add_argument("--project", default="your-project")
    ap.add_argument("--token", required=True)
    ap.add_argument("--cookie", default=None, help="Optional Cookie header value from browser request.")
    ap.add_argument("--tfs-session", default=None, help="Optional x-tfs-session header value.")
    ap.add_argument(
        "--service-host",
        default=None,
        help="Optional sourcePage routeValues.serviceHost value.",
    )
    ap.add_argument("--org", action="append", required=True, help="GitHub organization. Repeat for multiple orgs.")
    ap.add_argument("--dry-run", action="store_true", help="Do not apply changes; only report what would be submitted.")
    ap.add_argument(
        "--print-connections",
        action="store_true",
        help="Print visible ADO external connections before syncing.",
    )
    ap.add_argument(
        "--connection",
        action="append",
        default=[],
        help="Optional org-to-connection-id mapping: org=connectionId (repeatable).",
    )
    args = ap.parse_args()

    client = AdoClient(
        args.base_url,
        args.project,
        args.token,
        cookie=args.cookie,
        tfs_session=args.tfs_session,
        service_host=args.service_host,
    )
    overrides = parse_org_mapping(args.connection)
    all_connections = client.list_connections()
    if args.print_connections:
        print("[connections]")
        for c in sorted(all_connections, key=lambda x: (x.org, x.id)):
            print(f"  - org={c.org} id={c.id} valid={c.is_valid} repos={len(c.repos)}")

    global_owner: Dict[str, Tuple[str, str]] = {}
    for c in all_connections:
        for rid in c.repos.keys():
            global_owner[rid] = (c.id, c.org)

    overall_ok = True
    for org in args.org:
        target = choose_target_connection(org, all_connections, overrides.get(org))
        available = client.list_org_repositories(org)
        current_ids = set(target.repos.keys())
        available_ids = set(available.keys())

        # Repos already connected to another connection block this one.
        blocked: Set[str] = set()
        for rid in available_ids:
            owner = global_owner.get(rid)
            if owner and owner[0] != target.id:
                blocked.add(rid)

        desired_ids = sorted((available_ids - blocked) | current_ids)
        print(
            f"[{org}] connection={target.id} available={len(available_ids)} current={len(current_ids)} blocked_elsewhere={len(blocked)} desired_submit={len(desired_ids)}"
        )

        if args.dry_run:
            print(f"[{org}] dry_run_apply_skipped")
        else:
            response = client.apply_connection_selection(org, target.id, desired_ids)
            provider = response.get("dataProviders", {}).get(ADD_PROVIDER, {})
            if isinstance(provider, dict) and provider.get("errorMessage"):
                print(f"[{org}] apply_error={provider.get('errorMessage')}", file=sys.stderr)
                overall_ok = False

        # Refresh and verify.
        if args.dry_run:
            print(f"[{org}] dry_run_verification_skipped expected_selection={len(desired_ids)}")
        else:
            refreshed = client.list_connections()
            new_target = choose_target_connection(org, refreshed, target.id)
            new_ids = set(new_target.repos.keys())
            expected = set(desired_ids)
            missing = sorted(expected - new_ids)

            if missing:
                overall_ok = False
                print(f"[{org}] verification_missing={len(missing)}", file=sys.stderr)
                for rid in missing[:20]:
                    print(f"  - missing {available.get(rid, rid)} ({rid})", file=sys.stderr)
                if len(missing) > 20:
                    print(f"  - ... and {len(missing)-20} more", file=sys.stderr)
            else:
                print(f"[{org}] verification_ok selected={len(new_ids)}")

        if blocked:
            print(f"[{org}] blocked_repositories={len(blocked)}")
            shown = 0
            for rid in sorted(blocked):
                owner = global_owner.get(rid)
                name = available.get(rid, rid)
                print(f"  - {name} blocked_by_connection={owner[0]} org={owner[1]}")
                shown += 1
                if shown >= 20:
                    if len(blocked) > 20:
                        print(f"  - ... and {len(blocked)-20} more")
                    break

    return 0 if overall_ok else 1


if __name__ == "__main__":
    raise SystemExit(main())