Resource Groups Tagging API Support Probe

probe-analysis.md

RGT tag-resources support probe

Testing cloud-custodian/cloud-custodian#11021 (aws/taggable).

Question: can we determine, ahead of time, which resource types the Resource Group Tagging API is unable to tag — so aws.taggable stops dispatching batches that are guaranteed to fail?

Method: synthesize one unique ARN for each of the 2260 ARN types in tests/data/arn-types.json (380 services), pointed at a non-existent account (111111111111), and call tag-resources on them in batches of 20. Nothing real can be tagged because no emitted ARN names a resource that exists; the script hard-fails if any generated ARN contains the caller's own account id.

Probe script: rgt_probe.py Run: python rgt_probe.py tests/data/arn-types.json us-east-2 from a cloud-custodian checkout. Roughly 340 API calls (113 batches plus binary-split isolation and throttle retries), about 4 minutes, mutates nothing.

The signal

RGT emits a service-specific error only when it has successfully routed the ARN to that service's tagging API:

arn:aws:ec2:us-east-2:111111111111:instance/c7nprobe0689    InvalidID          The ID 'c7nprobe0689' is not valid
arn:aws:s3:::c7nprobe1853                                   NoSuchBucket       The specified bucket does not exist
arn:aws:sqs:us-east-2:111111111111:c7nprobe2041             InvalidAddress     The address ... is not valid
arn:aws:cassandra:us-east-2:111111111111:/keyspace/fake/    InvalidParameterException  Unrecognized service or resource type for tagging

So InvalidParameterException: Unrecognized service or resource type for tagging is the only response that means "not taggable". Every other per-resource error is evidence of support.

Results (us-east-2, 2026-08-21)

verdict	count	meaning
ROUTED	1688	RGT routed to the service — type is taggable
NOT_TAGGABLE	348	Unrecognized service or resource type for tagging
CALL_REJECTED	223	whole call 400'd, no per-resource detail (see below)
REPORTED_SUCCESS	1	no failure reported for a resource that cannot exist

Top ROUTED error codes: AccessDeniedException 647, InvalidParameterException 261, InvalidID 109, BadRequestException 98, ValidationException 85, InvalidRequestException 80.

1. 348 untaggable types; 95 services with no taggable type at all

a4b, aidevops, aiops, airflow-serverless, application-autoscaling, application-signals-mcp,
appmesh-preview, appstudio, aws-external-anthropic, backup-search, bedrock-mantle, bugbust,
cassandra, chime, cleanrooms-ml, cloudsearch, cloudshell, codeguru-security,
codestar-notifications, codewhisperer, cognito-sync, cur, dax, drs, elastictranscoder,
elemental-appliances-software, elemental-inference, evidently, evs, finspace, freertos,
gameliftstreams, health-agent, honeycode, identity-sync, inspector2, interconnect,
iotfleethub, iotmanagedintegrations, launchwizard, license-manager-linux-subscriptions,
license-manager-user-subscriptions, lookoutequipment, lookoutmetrics, lookoutvision,
mediastore, mgn, migrationhub-orchestrator, monitron, mpa, neptune-db, neptune-graph,
networkflowmonitor, nimble, nova-act, observabilityadmin, odb, one, opensearch,
pca-connector-scep, pcs, pi, polly, private-networks, q, qdeveloper, rds-db, repostspace,
rolesanywhere, rtbfabric, s3-object-lambda, s3express, s3files, scn, sdb, security-ir,
securityagent, securitylake, serverlessrepo, simspaceweaver, snow-device-management,
social-messaging, sqlworkbench, sso-oauth, thinclient, timestream, transform,
transform-custom, translate, trustedadvisor, voiceid, wickr, workdocs, workmail,
workmailmessageflow

A further 16 services are partially unsupported, which matters more than the fully-dead ones because a policy will happily discover the taggable siblings:

service	untaggable / total	untaggable types
personalize	15/18
s3	7/13	accessgrant, accessgrantsinstance, accessgrantslocation, accesspoint, accesspointobject, objectlambdaaccesspoint, storagelensgroup
iam	6/14	access-report, assumed-role, delegation-request, federated-user, group, sms-mfa
aws-marketplace	6/16
sso	5/6
mgh	4/5
controltower	3/4	EnabledBaseline, EnabledControl, LandingZone
es	2/5	application, datasource
artifact, codecatalyst, qapps, signin	2 each
execute-api, payment-cryptography, sts, workspaces-instances	1 each

Note iam:group and the entire s3 access-point family — both plausible discovery results.

2. The probe's best find: global services fail the whole call outside us-east-1

54 of the CALL_REJECTED rows returned Invocation of TagResources for this resource is not supported in this region, all in global services:

service	types
cloudfront	19
waf	12
networkmanager	9
route53	8
globalaccelerator	4
cloudfront-keyvaluestore	1
savingsplans	1

Confirmed directly:

us-east-2: WHOLE CALL FAILED InvalidParameterException: ...not supported in this region
us-east-1: call OK -> FailedResourcesMap={...InvalidClientTokenId...}
mixed batch of 3 valid + 1 global arn: WHOLE CALL FAILED -> InvalidParameterException

Three consequences for aws.taggable, ascending in severity:

1. Taggable.resource_type does not set global_resource, so UniversalTag.get_client() (c7n/tags.py:1012) builds its client in the policy region. Global resources discovered outside us-east-1 are untaggable by construction.
2. One such ARN fails its entire batch of 20. The rejection is a whole-call 400, not a FailedResourcesMap entry, so the 19 innocent ARNs are lost with it.
3. It raises a raw ClientError, not a ResourceGroupTagError, so TagActionDispatch.process (c7n/resources/taggable.py:295-307) does not catch it — it kills the policy. Identical abort mechanism to the empty-ResourceARNList crash in the smoke test analysis.

The 2026-08-21 smoke test never hit this: us-east-2 discovery returned no global resources (grep -c "not supported in this region" custodian-run.log → 0). A us-east-1 run, or any account with CloudFront/WAF/Route53, will.

3. RGT reports success for IAM ARNs that cannot exist

One row, es:opensearchservice_role → arn:aws:iam::111111111111:role/aws-service-role/opensearchservice.amazonaws.com/AWSServiceRoleForAmazonOpenSearchService, came back absent from FailedResourcesMap entirely — i.e. reported as tagged, in an account that does not exist. For IAM ARNs, "no failure reported" is not evidence the tag landed.

Limitations — read before trusting a row

• 169 of the 223 CALL_REJECTED rows are probe artifacts, not findings. arn-types.json describes IAM policy resource patterns, not real ARNs: 19 contain a literal *, 100 have an empty region field, the remainder empty account fields (iq, partnercentral, geo-*). These fail as ... is not a valid AmazonResourceName (ARN). Check the message before believing a CALL_REJECTED verdict. The file is not a usable source of real ARNs.
• Runtime, state, and permission failures are invisible. cloudformation, kms, events, cloudwatch and ec2 all came back cleanly ROUTED. The probe cannot see No updates are to be performed, key-policy AccessDenied, ManagedRuleException, or throttling — which is to say, most of what actually failed in the smoke test.
• Dead end, recorded so nobody repeats it: bucketing ROUTED types by whether they rejected the synthetic id on format grounds (the ssm:managed-instance mi-[0-9a-f]{17} class of failure) flags 209 types — nearly every type in every service, because a generic filler never matches any real id format. It cannot predict which real ARNs a service will reject. Removed from the script.
• 2 rows (bcm-data-exports:export, bcm-data-exports:table) stayed throttled through 6 backoff attempts; verdicts for those are unknown.
• Single region, single partition (aws), single point in time. Results drift as AWS adds services.

Validation against the smoke test

service	smoke-test failure	probe verdict	predicted?
application-autoscaling	Unrecognized service or resource type	NOT_TAGGABLE 1/1	yes
cassandra	Unrecognized service or resource type	NOT_TAGGABLE 3/3	yes
sqlworkbench	(in the never-attempted tail)	NOT_TAGGABLE	yes
athena	InvalidRequestException resourceARN regex	ROUTED	no
ssm	ValidationException resourceId must be mi-*	ROUTED	no
cloudformation	No updates are to be performed	ROUTED	no
events	ManagedRuleException	ROUTED	no
kms	AccessDeniedException (key policy)	ROUTED	no
cloudwatch / ec2	Throttling / RequestLimitExceeded	ROUTED	no

Against the smoke test's 3014 discovered resources, the skip list would have pre-filtered 7 (4 application-autoscaling + 2 cassandra + 1 sqlworkbench). The skip list is real but small; the global-region bug is the valuable catch.

Recommendations

Keep the probe as a test-data generator, not a live test — it needs real credentials, so it cannot run in CI.

• New P0 — global-service region routing. Partition ARNs by service in TagActionDispatch.process and tag the global set (cloudfront, waf, route53, globalaccelerator, networkmanager, savingsplans, cloudfront-keyvaluestore) through us-east-1. More urgent than the skip list: it is a policy-killer on any account holding those resources. The already-planned P0 (catch broadly per service group and continue) contains the blast radius for this and the empty-ARN crash — that one fix does double duty.
• P1 — static skip list. Check the 348 NOT_TAGGABLE service/type pairs into tests/data/, filter against it before dispatch. Zero runtime cost, refreshed manually by re-running the probe when AWS adds services. Include the 16 partially-unsupported services; per-type granularity matters there.
• Do not try to predict id-format rejections from a probe. That class of failure (athena:named-query, ssm:managed-instance) needs handling at dispatch time — treat InvalidRequestException/ValidationException on a well-formed ARN as a skip-and-log, not a batch failure.

rgt_probe.py

#!/usr/bin/env python3
"""Probe which AWS resource types the Resource Group Tagging API can actually tag.

Synthesizes one unique ARN for every ARN type in cloud-custodian's
tests/data/arn-types.json, pointed at a deliberately non-existent account, and
calls tag-resources on them in batches. Nothing real can be tagged, because no
ARN we emit names a resource that exists.

The signal: RGT only emits a *service-specific* error (InvalidID from ec2,
NoSuchBucket from s3, InvalidAddress from sqs) if it successfully routed the ARN
to that service's tagging API. So any service-specific error is evidence of
support, and only "InvalidParameterException: Unrecognized service or resource
type for tagging" means the type is genuinely untaggable.

Usage:
    python rgt_probe.py [path/to/arn-types.json] [region]

Needs credentials for any account (the call must authenticate; the ARNs it
carries are fake). Guards against tagging anything real by refusing to run if a
generated ARN names the caller's own account.

Buckets:
    NOT_TAGGABLE      RGT does not recognize the service/resource type
    ROUTED            RGT routed to the service; the type is taggable
    CALL_REJECTED     the whole call 400'd, no per-resource detail. Two causes:
                      a global-only service invoked outside us-east-1, and
                      malformed templates (see caveat below)
    REPORTED_SUCCESS  no failure reported for a resource that cannot exist

Caveat: arn-types.json describes IAM *policy resource patterns*, not real ARNs.
Some entries carry literal wildcards or empty account/region fields and can
never be valid ARNs; those land in CALL_REJECTED as probe artifacts rather than
findings. Check the message before believing a CALL_REJECTED row.

Dead end, recorded so nobody repeats it: bucketing ROUTED types by whether they
rejected the synthetic id on *format* grounds looks promising but flags nearly
every type in every service, since a generic filler never matches any real id
format. It cannot predict which real ARNs a service will reject.
"""
import json
import os
import re
import sys
import time
from collections import Counter, defaultdict

import boto3
from botocore.exceptions import ClientError

FAKE_ACCOUNT = "111111111111"
PARTITION = "aws"
TAGS = {"c7n-probe": "true"}
BATCH = 20          # tag-resources hard limit
THROTTLE_CODES = ("ThrottlingException", "Throttling", "RequestLimitExceeded",
                  "TooManyRequestsException", "ThrottledException")
MAX_ATTEMPTS = 6


def fill(template, idx):
    """Substitute every ${Placeholder}; keep account-shaped ones 12 digits."""
    token = "c7nprobe%04d" % idx

    def sub(m):
        name = m.group(1)
        if name == "Partition":
            return PARTITION
        if name == "Region":
            return REGION
        if "Account" in name:
            return FAKE_ACCOUNT
        return token

    return re.sub(r"\$\{(\w+)\}", sub, template)


def generate(path, guard_accounts):
    types = json.load(open(path))
    arns, skipped = {}, []
    idx = 0
    for service in sorted(types):
        for rtype in sorted(types[service]):
            idx += 1
            arn = fill(types[service][rtype], idx)
            for real in guard_accounts:
                if real and real in arn:
                    sys.exit("SAFETY ABORT: generated arn names a real account: %s" % arn)
            if not arn.startswith("arn:") or arn.count(":") < 5:
                skipped.append({"service": service, "type": rtype, "arn": arn,
                                "reason": "template is not a 6-field arn"})
                continue
            if arn in arns:
                skipped.append({"service": service, "type": rtype, "arn": arn,
                                "reason": "identical shape to %s/%s" % arns[arn]})
                continue
            arns[arn] = (service, rtype)
    return arns, skipped


def invoke(client, arn_list, results):
    """One tag_resources call. Returns the response, or None if a whole-call
    rejection was isolated by binary split and already recorded."""
    for attempt in range(MAX_ATTEMPTS):
        try:
            return client.tag_resources(ResourceARNList=arn_list, Tags=TAGS)
        except ClientError as e:
            err = e.response.get("Error", {})
            if err.get("Code") in THROTTLE_CODES:
                time.sleep(1.5 * 2 ** attempt)
                continue
            if len(arn_list) == 1:
                # isolated: this single arn is what the service rejects
                results[arn_list[0]] = {"ErrorCode": "CALL_%s" % err.get("Code"),
                                        "ErrorMessage": err.get("Message")}
                return None
            # one bad arn fails the whole call, so split to find it
            mid = len(arn_list) // 2
            call(client, arn_list[:mid], results)
            call(client, arn_list[mid:], results)
            return None
    sys.exit("gave up retrying throttled call")


def call(client, arn_list, results):
    """Record a verdict for every arn, retrying per-resource throttles."""
    pending = list(arn_list)
    for attempt in range(MAX_ATTEMPTS):
        resp = invoke(client, pending, results)
        if resp is None:
            return
        failed = resp.get("FailedResourcesMap", {})
        throttled = [a for a in pending
                     if (failed.get(a) or {}).get("ErrorCode") in THROTTLE_CODES]
        for arn in pending:
            if arn not in throttled:
                results[arn] = failed.get(arn)   # None == no failure reported
        if not throttled:
            return
        pending = throttled
        time.sleep(1.5 * 2 ** attempt)
    for arn in pending:                          # still throttled after backoff
        results[arn] = {"ErrorCode": "ThrottlingException",
                        "ErrorMessage": "still throttled after %d attempts" % MAX_ATTEMPTS}


def classify(err):
    if err is None:
        return "REPORTED_SUCCESS"
    code = err.get("ErrorCode") or ""
    msg = err.get("ErrorMessage") or ""
    if code == "InvalidParameterException" and "Unrecognized service or resource type" in msg:
        return "NOT_TAGGABLE"
    if code.startswith("CALL_"):
        return "CALL_REJECTED"
    return "ROUTED"


def main():
    global REGION
    path = sys.argv[1] if len(sys.argv) > 1 else "tests/data/arn-types.json"
    REGION = sys.argv[2] if len(sys.argv) > 2 else (
        os.environ.get("AWS_REGION") or boto3.Session().region_name or "us-east-1")

    session = boto3.Session(region_name=REGION)
    guard = {session.client("sts").get_caller_identity()["Account"]}
    print("region %s, guarding against account %s" % (REGION, ", ".join(guard)))

    arns, skipped = generate(path, guard)
    print("generated %d unique probe arns (%d templates skipped)" % (len(arns), len(skipped)))

    client = session.client("resourcegroupstaggingapi")
    ordered = list(arns)
    results = {}
    for i in range(0, len(ordered), BATCH):
        call(client, ordered[i:i + BATCH], results)
        print("\r  %d/%d" % (min(i + BATCH, len(ordered)), len(ordered)), end="", flush=True)
        time.sleep(0.2)
    print()

    rows = []
    for arn, (service, rtype) in arns.items():
        err = results.get(arn)
        rows.append({"service": service, "type": rtype, "arn": arn,
                     "verdict": classify(err),
                     "code": (err or {}).get("ErrorCode"),
                     "message": (err or {}).get("ErrorMessage")})
    out = "rgt-probe-results.json"
    json.dump({"region": REGION, "rows": rows, "skipped": skipped}, open(out, "w"), indent=2)

    print("\n=== verdicts ===")
    for v, c in Counter(r["verdict"] for r in rows).most_common():
        print("%6d  %s" % (c, v))

    by_svc = defaultdict(Counter)
    for r in rows:
        by_svc[r["service"]][r["verdict"]] += 1
    whole = sorted(s for s, c in by_svc.items() if set(c) == {"NOT_TAGGABLE"})
    print("\n=== services with no taggable arn type (%d) ===" % len(whole))
    print(", ".join(whole))

    region_only = sorted({r["service"] for r in rows
                          if "not supported in this region" in (r["message"] or "")})
    if region_only:
        print("\n=== global services: whole call fails outside us-east-1 (%d) ===" % len(region_only))
        print(", ".join(region_only))
        print("NOTE: one such arn fails its entire batch of %d, and the error is a "
              "ClientError rather than a FailedResourcesMap entry." % BATCH)

    print("\nwrote %s" % out)


if __name__ == "__main__":
    main()