jq - summarise Confluent Cloud service account topic access

summarise-acls.jq

(INDEX($sa[] ; .id) | map_values(.name)) as $dict
| [(.[] | select(.principal[:7] == "User:sa"))]
| group_by(.principal)  
| map(    
    {     
        (first.principal[5:] as $p     
            | ($dict[$p] // ("ACL-for-non-existant-service-account: " + $p))    
        ): map(    
            ((.resource_type | ascii_downcase) + "/"+ .resource_name)    
        )    
    }    
)    
|add  

We can use jq to quickly summarise Confluent Cloud service account information, grouping by service account id and using some simple service-account reference data to obtain the name of the account.

First thing is to obtain the list of ACLs for a given Confluent Cloud cluster, which we do via the confluent cli:

% confluent kafka acl list -o json > cluster-acls.json

The output looks like the following:

% jq '.[0:2]' cluster-acls.json                 
[
  {
    "principal": "User:sa-abcdef",
    "permission": "ALLOW",
    "operation": "READ",
    "resource_type": "GROUP",
    "resource_name": "*",
    "pattern_type": "LITERAL"
  },
  {
    "principal": "User:sa-abcdef",
    "permission": "ALLOW",
    "operation": "READ",
    "resource_type": "TOPIC",
    "resource_name": "*",
    "pattern_type": "LITERAL"
  }
]

Then we complement this with service account information, which includes the name of the account:

% confluent iam service-account list -o json > service-accounts.json

This looks like the following:

% jq '.[0:2]' service-accounts.json
[
  {
    "id": "sa-123abc",
    "name": "Application 1",
    "description": "App1 service account"
  },
  {
    "id": "sa-321xyz",
    "name": "Application 2",
    "description": "App2 service account"
  }
]

Line by line the attached script does the following:

1. Define a key/value index for the service-accounts.json file, relating the service account ID to its name
2. Filter the ACL file to ensure that only security principals that are service accounts are included. Security principals have a format of User:<sa|u>-... where u refers to a user (human) account and sa refers to an application.
3. We group the array of ACLs by principal name, creating individual arrays of ACLs for each principal. The output looks like

   [  
    [ 
       { sa1 acl info }, 
       { sa1 acl info },
       ...
    ],
    [ 
      { sa2 acl info }, 
      { sa2 acl info },
       ...
    ], 
    ... 
   ]
   

4. We then map the groups
5. Creating a JSON object: {
6. Using the principal field from the first item in the group... Given all members will have the same principal, this is safe. We store the principal name in variable $p for use later.
7. We look up the principal ($p) in our index, and if not found // we set a default value
8. This becomes the key of our JSON output from line (5) above, for which we use a map to obtain the details
9. Which are the resource type (e.g. TOPIC, which we downcase and suffix + "/" + with the actual name of the resource
10. Our output will be an array of JSON objects, which we can flatten into an array of keys/object pairs via applying the add function to the array.

Our final output looks like the following:

{
  "Application 1": [
    "topic/*"
  ],
  "Application 2": [
    "group/xya-consumer-group",
    "topic/xx1-topic",
    "topic/xx2-toipc"
  ]
}

To run this we need to configure inputs correctly:
jq --argfile sa service-accounts.json -f summarise-acls.jq cluster-acls.json

Firstly we will jq that the variable sa relates to the service-accounts.json file (our reference file). In the script on line 1 we refer to $sa.

We then pass in the jq filter script above and finally the ACL json file.