Skip to content

Instantly share code, notes, and snippets.

@shokoe

shokoe/Ext_iam.sh

Created April 15, 2023 15:04
Show Gist options
  • Save shokoe/7257d62473a476f7b4d452249353fde4 to your computer and use it in GitHub Desktop.
Save shokoe/7257d62473a476f7b4d452249353fde4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Ext_iam.sh
Syntax: Enfo_iam.sh
Options:
-c|--cache-dir <dir> Use a specifc report cache dir. Default is /var/cache/Enfo_iam/Enfo_iam_report_*.
-d|--debug Debug output
-E|--exit-on-error Stop script on aws error
-e|--error Error output
-f|--fresh-data Get a new report, automatically saved as last cache when finished
Works well with -q for cronjobs or with '-q -d' for debug
-F|--fields <comma separated fileds list>
Comma separated columns list (see below)
-h|--help Help
-P|--no-para Disables porfile parallelism (mainly for debug)
-q|--quiet Supress shell output
-N|--dont-save-report Do not save report (effective with -G)
-w|--col-width Max column width (trancated column end with _)
-G|--grep-entity Global grep on entity (user ot role)
-r|--retain-policies <dir>
Saves all policies json files into dir (for git)
Usage patterns:
Cron daily job
Enfo_iam.sh -f -q -E
Filtered (pre data collection) run without saving report (much faster than getting full report)
Enfo_iam.sh -f -G 'shoko' -N -E -F Entity,Type,Name,Group,Policy,Effect,Action,Resource,Profile
Non-parallel full run debug without printing or saving report
time Enfo_iam.sh -f -P -d -q -N -E
Columns:
Entity User or Role
Type Inline or Attach
Trust Role trusted entities
Name Username or role name
Time User or role creation time
Last User/Role last api access time
InsCount Role assigned instances count
Group User group
Policy Policy name
PoliARN Policy ARN
PoliOwn Policy owner (extracted from ARN)
Ptime Policy creation time
PoliVer Attached policy default version
Statement Policy statement index
Effect Statement effect - Allow/Deny
Namespace Action namespace
NSlast Namespace last api access time
Action Statement action
Resource Statement resource
Profile Profile
Account Account ID
Column full list:
Entity,Type,Trust,Name,Time,Last,InsCount,Group,Policy,PoliARN,PoliOwn,Ptime,PoliVer,Statement,Effect,Namespace,NSlast,Action,Resource,Profile,Account
Column filter operators:
Arithmetic > larger than
< less than
>= larger than or equal to
<= less than or equal to
note: also filter any non-numeric values as side effect)
Text = equal
~ regex comparison, case sensative, support | for or
Date Syntax: [column][<>][number][unit]
< date is after given value
> date is before given value
unit can be one of sec(ond)?s?|min(ute)?s?|hours?|days?|mo(nth)?s?|years?
Reverse glob ` match asterisk values
added specifically to find actions in policy actions and resources glob patterns
make sure you single qoute the field list!
Column prefix:
! Supress column output giving the option to filter on column without showing it.
Output files:
Detailed last access data /var/cache/Enfo_iam/Enfo_iam_report_YYMMDDHHmmss.<profile>.last
Base policy data /var/cache/Enfo_iam/Enfo_iam_report_YYMMDDHHmmss.<profile>.poli
Policy with last data /var/cache/Enfo_iam/Enfo_iam_report_YYMMDDHHmmss.<profile>.rich
Main merged report (cache file) /var/cache/Enfo_iam/Enfo_iam_report_YYMMDDHHmmss.rich
Usage Examples:
All entities that can delete object on a specific bucket
Ext_iam.sh -F 'Entity,Name,InsCount,Effect=Allow,Action`s3:DeleteObject,Resource`arn:aws:s3:::image-res-platform,Profile=default'
Unused ec2 roles
Ext_iam.sh -F 'Entity,Type,Trust~ec2,Name,Last,InsCount=None,Profile'
All policies that were changed in the last week
Ext_iam.sh -F 'Entity,Type,Trust,Name,Policy,Ptime<7day,Profile'
All the users that were not used (in any namespace) in the last 6 months
Ext_iam.sh -F 'Entity=User,Name,Last>6month,Profile'
All target services (namespace) included in trust from two specific sources
Ext_iam.sh -F 'Entity,Type,Trust~sage|robo,Name,Last,Profile,Namespace'
All in-use AWS managed policies with version
Ext_iam.sh -F 'Policy,!PoliOwn=aws,PoliVer,!Profile=default'
Notes:
- Last access data is per namespace (target service) and not a pecific action. Entity last access is derived
from the namespace data.
- 'NULL' values for namespace last access were returned from AWS report (generate-service-last-accessed-details).
'None' values mean the namespace was not found (should not happen).
- Multiple conditions on the same column is not supported.
root@jump:~# Ext_iam.sh | less
root@jump:~# Ext_iam.sh -w 20 | less
root@jump:~# Ext_iam.sh -w 8 | less
root@jump:~# view `which Ext_iam.sh `
[5]+ Stopped view `which Ext_iam.sh `
root@jump:~# cat `which Ext_iam.sh `
#!/bin/bash
export PS4='+[${SECONDS}s][${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }';
#set -xve;
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DBG=false
dbg_var(){ $DBG || return; [ -z "${!1}" ] && echo ">>>>>>>>>>> $1: NA!!!!!!!" >&2 && return; echo "${!1}" | sed "s#^#>>>>>>>>>>> $1: #" >&3; }
# policy cache efficency check:
# time /usr/local/bin/Enfo_iam.sh -w 120 -F 'Entity,Trust,Type=Attach,Name,Policy,Profile' 2>&1 | awk '{$4=""; print $0}' | column -t | sort | uniq -c | sort -nr | awk '{A+=$1; C+=$1-1}; END{print "ALL:"A, "CACHE:"C}'
# PATTERN EXAMPLE: multi colored stderr
exec 9>&2
exec 8> >( while IFS='' read -r line || [ -n "$line" ]; do echo -e "`date -Isec` -- \033[31m${line}\033[0m" >&2; done )
exec 3> >( while IFS='' read -r line || [ -n "$line" ]; do echo -e "`date -Isec` -- \033[33m${line}\033[0m" >&2; done )
exec 4> >( while IFS='' read -r line || [ -n "$line" ]; do echo -e "`date -Isec` -- \033[35mD4>> ${line}\033[0m" >&2; done )
exec 7> >( while IFS='' read -r line || [ -n "$line" ]; do echo -e "`date -Isec` -- \033[36mD7>> ${line}\033[0m" >&2; done )
exec 5> >( while IFS='' read -r line || [ -n "$line" ]; do echo "${line}" >&2; done )
exec 6> >( while IFS='' read -r line || [ -n "$line" ]; do echo "=== ${line}" >&3; echo "${line}"; done )
function undirect(){ exec 2>&9; }
function redirect(){ exec 2>&8; }
trap "redirect;" DEBUG
PROMPT_COMMAND='undirect;'
aws_cmd=`which aws`
CA="/dev/shm/shcnt_$$"
api_cache="/var/cache/Enfo_iam_api_cache_$$"
[ ! -d $api_cache ] && mkdir -p $api_cache
aws(){
$flag_exit_on_error && [ -f $cache_pref.err ] && exit
[ -e $CA ] && echo -n "$((`cat $CA`+1))" > $CA || echo -n 1 > $CA
aws_sig=`echo "$aws_cmd --profile $profile $@" | md5sum | sed 's# .*##'`
if [ -e $api_cache/${aws_sig}.cache ] && $flag_api_cache; then
$DBG && echo "[$(cat $CA)] [$aws_sig:cache] aws --profile $profile $*" >&4
O=`cat $api_cache/${aws_sig}.cache`
else
$DBG && echo "[$(cat $CA)] [$aws_sig:fresh] aws --profile $profile $*" >&4
O=$(timeout $aws_timeout $aws_cmd --profile $profile $@ 2>&1 | tee $api_cache/${aws_sig}.cache)
ec=$?
if [ $ec -ne 0 ];then
touch $cache_pref.err
echo "AWS ERR CMD (exitCode:$ec) [$aws_sig]: $aws_cmd --profile $profile $@" >&2
[ $ec -eq 124 ] && echo " aws command reached timeout of ${aws_timeout}sec (can be changed in the top of the script)"
echo -n "$O" | sed "s#^# [$aws_sig] #" >&2
fi
fi
[ -z "$O" ] && return
echo "$O"
#echo "aws $*" >&4
#$aws_cmd $@
}
export -f aws
Etul_fielder(){
local f=${1:-$fields_default};
read H;
#echo ">>> H: $H" >&7
local E=0;
local W=${col_width:-1000}
[[ ! $f =~ ^[Ff][Uu][Ll][Ll]$ ]] && for i in ${f//,/ }; do
local I=`echo $i | sed 's#\([A-Za-z]*\).*#\1#'`
if ! echo $H | egrep -q "\b$I\b"; then
echo "ERROR: wrong field '$i' (try -h)";
E=1;
fi;
done;
(($E)) && cat &> /dev/null && return 1;
[ -z "$f" ] || [[ "$f" =~ [Ff][Uu][Ll][Ll] ]] && ( echo "$H";
cat ) && return;
local f_sed=`echo "$H" | sed 's# *$##; s#\s\+#\n#g' | cat -n | awk '{printf "s#\\\b"$2"\\\b[^,]*#"$1"#g;"};'`;
local F=`echo $f | sed 's#,#\n#g;' | grep -v '^!' | xargs | sed 's# #,#g' | sed "$f_sed" | sed 's#^#$#; s#,#,$#g;'`;
local time_grep='sec(ond)?s?|min(ute)?s?|hours?|days?|mo(nth)?s?|years?'
local rev_regex_char='`'
# build standard (transitive) awk condition
local cond_f_sed=`echo "$H" | sed 's# *$##; s#\s\+#\n#g' | cat -n | awk '{printf "s#\\\b"$2"\\\b#"$1"#g;"};'`;
local cond=`echo $f | sed "$cond_f_sed" | sed 's#^#$#; s#,#,$#g;' | sed 's#$!#$#g; s#,#\n#g' | egrep -v "$time_grep" | grep -v "$rev_regex_char" | grep -v '^$[0-9]\+$' |\
sed 's#^\(\$[0-9]*\)\=\(.*\)#\1==\\\\"\2\\\\"#;
s#^\(\$[0-9]*\)!=\(.*\)#\1!=\\\\"\2\\\\"#;
s#^\(\$[0-9]*\)\(!\?~\)\(.*\)#\1\2/\3/#;
# adds filter of non numeric values for numeric comparison
s#^\(\$[0-9]*\)\([<>]\)\(.*\)#\1\2\3\&\&\1+0==\1#;
s#$#:::#' | xargs | sed 's#:::$##; s#::: #\&\&#g;'`
#set +x
[ -z "$cond" ] && cond="1"
#echo ">>> cond: $cond" >&4
# time conditions
local c o v C O V
while read c o v; do
#echo "$c $o $v" >&4
# turn it around so it's understandable
case $o in
'>') o='<';;
'<') o='>';;
esac
#echo "$c $o $v" >&4
local time_cond+=" iso2num($c)${o}V${c/$/}"
local time_vars+=" -v V${c/$/}=`date -d -$v -Isec | sed 's#+.*##; s#[^0-9]##g'`"
done < <(echo $f | sed "$cond_f_sed" | sed 's#^#$#; s#,#,$#g;' | sed 's#$!#$#; s#,#\n#g' | egrep "$time_grep" | sed 's#\([<>]\)# \1 #')
# revers glob condition
#set -x
while read c o v; do
C="tolower($c)"
local rev_cond+=" R${c/$/}~glb2re($c)"
local rev_vars+=" -v R${c/$/}=${v,,}"
done < <(echo $f | sed "$cond_f_sed" | sed 's#^#$#; s#,#,$#g;' | sed 's#$!#$#; s#,#\n#g' | grep "$rev_regex_char" | sed "s#\($rev_regex_char\)# \1 #")
#set +x
#local tmp=$(echo $f | sed "$cond_f_sed" | sed 's#^#$#; s#,#,$#g;' | sed 's#$!#$#; s#,#\n#g' | grep "$rev_regex_char" | sed "s#\($rev_regex_char\)# \1 #")
#echo ">>> rev input: $tmp" >&4
cond=`echo "$cond $time_cond $rev_cond" | sed 's#^ *##; s# *$##; s# *# #g; s# #\&\&#g;'`
$DBG && echo "fields: $F" >&4
# for limiting colmn width
F=`echo "$F" | sed 's#,#\n#g' | sed 's#\(.*\)#width(\1)#' | xargs | tr ' ' ','`
#F=`echo "$F" | sed 's#,#\n#g;' | sed 's#^\\\$##; s#\(.*\)#\\\"\1:\\\"width($\1)#' | xargs | tr ' ' ','`
$DBG && echo "All fields:" >&4
$DBG && echo "$H" | sed 's# *#\n#g' | cat -n | sed 's#^ *# $#' >&4
$DBG && echo "print: $F" >&4
$DBG && echo "cond: $cond" >&4
$DBG && echo "time_vars: $time_vars" >&4
$DBG && echo "rev_vars: $rev_vars" >&4
$DBG && echo "AWK: awk $time_vars $rev_vars 'function iso2num(iso){ gsub(/[^0-9]/,\"\",iso); }; function glb2re(glb){ gsub(\"*\",\".*\",glb); return (glb); }; function width(col){ pref=\"\"; if (length(col) > WIDTH) pref=\"_\"; return substr(col,1,WIDTH-1)pref; }; NR==1 || $cond {print $F}'" >&4
(
( echo "$H";
cat ) | gawk -v WIDTH=$W -M $time_vars $rev_vars '
function iso2num(iso){ gsub(/[^0-9]/,"",iso); return iso+0; }
function glb2re(glb){ gsub("*",".*",glb); return (tolower(glb)); }
function width(col){ pref=""; if (length(col) > WIDTH) pref="_"; return substr(col,1,WIDTH-1)pref; }
NR==1 || '$cond' {print '$F'};'
) | (read h; echo "$h" ; sort -u)
}
policy2tab2(){
$DBG && echo "$1" | jq -C . >&5
# retain policies
if $flag_ret_policy; then
local pfile="$policy_ret_dir/${profile}:${RoleType}:${AttachPoliARN/*\//}${InlinePoliName}:$PoliVer.json"
[ ! -f $pfile ] && echo "$1" > $pfile
fi
# even statment can be a simple val d not array (FFS)
local statment=`echo "$1" | jq '.Statement | if type!="array" then [.] else . end'`
local slen=`echo "$statment" | jq '. | length'`
dbg_var slen
for ((i=0; i<$slen; i++)); do
## Effect
local Effect=`echo "$statment" | jq -r --arg i $i '.[$i|tonumber].Effect'`
dbg_var Effect
## Action
local actions=`echo "$statment" | jq -r --arg i $i '.[$i|tonumber].Action | if type!="array" then . else .[] end'`
dbg_var actions
resources=`echo "$statment" | jq -r --arg i $i '.[$i|tonumber].Resource | if type!="array" then . else .[] end'`
dbg_var resources
echo "$actions" | while read Action; do
echo "$resources" | while read Resource; do
echo "$((i+1))/$slen $Effect ${Action/:*/} $Action $Resource"
done
done
done | column -t | sed "s#^#$2 #"
}
field_func(){
if [ ! -z "$fields" ]; then
cat | Etul_fielder $fields | column -t
else
cat
fi
}
get_report(){
echo "Entity Type Trust Name Time InsCount Group Policy PoliARN Ptime PoliVer Statement Effect Namespace Action Resource"
#### USERS
EntityType='User'
aws iam list-users --query 'Users[*].[UserName,CreateDate,Arn]' --output text | egrep "$grep_user_role" | while read UserName UserTime UserARN; do
$DBG || $ERR && echo ">> UserName:$UserName UserTime:$UserTime UserARN:$UserARN" >&3
#User_Header="EntityType RoleType -Trust UserName Utime -InsCount Group Policy Ptime PoliVer Statement Effect Action Resource"
#Role_Header="EntityType RoleType Trust RoleName Rtime InsCount -Group Policy Ptime PoliVer Statement Effect Action Resource"
Trust='None'
InsCount='None'
### Inline
RoleType='Inline'
Group='None'
PoliVer='None'
Ptime="None"
aws iam list-user-policies --user-name $UserName --query 'PolicyNames[*]' --output text | sed 's#[ \t]\+#\n#g' | while read InlinePoliName; do
$DBG || $ERR && echo ">>>> UserName:$UserName > InlinePoliName:$InlinePoliName" >&3
policy2tab2 "`aws iam get-user-policy --user-name $UserName --policy-name $InlinePoliName --query 'PolicyDocument'`" \
"$EntityType $RoleType $Trust $UserName $UserTime $InsCount $Group $InlinePoliName None $Ptime $PoliVer"
done
### Attached
RoleType='Attach'
Group='None'
aws iam list-attached-user-policies --user-name $UserName --query 'AttachedPolicies[].[PolicyArn]' --output text | while read AttachPoliARN; do
$DBG || $ERR && echo ">>>> UserName:$UserName > AttachPoliARN:$AttachPoliARN" >&3
PoliVer=`aws iam get-policy --policy-arn $AttachPoliARN --query 'Policy.DefaultVersionId' --output text`
Poli=`aws iam get-policy-version --policy-arn $AttachPoliARN --version-id $PoliVer`
Ptime=`echo "$Poli" | jq -r '.PolicyVersion.CreateDate'`
policy2tab2 "`echo "$Poli" | jq -r '.PolicyVersion.Document'`" \
"$EntityType $RoleType $Trust $UserName $UserTime $InsCount $Group ${AttachPoliARN/*\//} $AttachPoliARN $Ptime $PoliVer"
done
### Group
#aws iam list-groups-for-user --user-name $UserName >&5
aws iam list-groups-for-user --user-name $UserName --query 'Groups[].[GroupName]' --output text | while read Group; do
$DBG || $ERR && echo ">>>> UserName:$UserName > Group:$Group" >&3
### Group:Inline
RoleType='Inline'
PoliVer='None'
aws iam list-group-policies --group-name $Group --query 'PolicyNames[]' --output text | sed 's#[ \t]\+#\n#g' | while read InlinePoliName; do
policy2tab2 "`aws iam get-group-policy --group-name $Group --policy-name $InlinePoliName --query 'PolicyDocument'`" \
"$EntityType $RoleType $Trust $UserName $UserTime $InsCount $Group $InlinePoliName None $Ptime $PoliVer"
done
### Group:Attach
RoleType='Attach'
aws iam list-attached-group-policies --group-name $Group --query 'AttachedPolicies[].[PolicyArn]' --output text | while read AttachPoliARN; do
PoliVer=`aws iam get-policy --policy-arn $AttachPoliARN --query 'Policy.DefaultVersionId' --output text`
Poli=`aws iam get-policy-version --policy-arn $AttachPoliARN --version-id $PoliVer`
Ptime=`echo "$Poli" | jq -r '.PolicyVersion.CreateDate'`
policy2tab2 "`echo "$Poli" | jq -r '.PolicyVersion.Document'`" \
"$EntityType $RoleType $Trust $UserName $UserTime $InsCount $Group ${AttachPoliARN/*\//} $AttachPoliARN $Ptime $PoliVer"
done
done
done
#### ROLES
## Instance role counts
#. /opt/EC2ulz_20211005_cache.sh &>/dev/null
declare -A RoleCount
while read n r; do
RoleCount[$r]=$n
done < <(aws ec2 describe-instances --filters Name=instance-state-name,Values=running --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`].Value|[0],IamInstanceProfile.Arn]' --output text | sed 's#\([ \t]\)[^/]*/#\1#' | awk '{print $2}' | sort | uniq -c)
#done < <(ec2_profiles="default" Eins Name,State,Role | grep running | sed 's#[^ ]*/##' | grep -v '\bNone\b' | awk '{print $3}' | sort | uniq -c)
EntityType='Role'
aws iam list-roles |\
jq -r '.Roles[] | .RoleName as $r | .Arn as $a | .Description as $d | .CreateDate as $t | .AssumeRolePolicyDocument.Statement[0].Principal | .Federated as $f | .AWS as $w | .Service | "\($r) \(if type!="array" then [.] else . end | join(",")),\($f),\($w) \($t) \($a)"' | sed 's#\bnull\b##g; s#,,##; s#\([^a-zA-Z]\),#\1#; s#,\([^a-zA-Z]\)#\1#' |\
grep "$grep_user_role" |\
while read RoleName RolePrincipal RoleTime RoleArn; do
$DBG || $ERR && echo ">> RoleName:$RoleName RolePrincipal:$RolePrincipal RoleTime:$RoleTime RoleArn:$RoleArn" >&3
Header="EntityType RoleType Trust RoleName Statement Effect Action Resource"
Group='None'
#### Inline
RoleType='Inline'
PoliVer='None'
Ptime='None'
aws iam list-role-policies --role-name $RoleName --query "PolicyNames[*]" --output text | sed 's#[ \t]\+#\n#g' | while read InlinePoliName; do
$DBG || $ERR && echo ">>>> RoleName:$RoleName > InlinePoliName:$InlinePoliName" >&3
policy2tab2 "`aws iam get-role-policy --role-name $RoleName --policy-name $InlinePoliName --query 'PolicyDocument'`" \
"$EntityType $RoleType $RolePrincipal $RoleName $RoleTime ${RoleCount[$RoleName]:-None} $Group $InlinePoliName None $Ptime $PoliVer"
done
## [746] zenbusiness-zenbusinessSAMRole-1O3UF9LX8Y4L
#### Attach
RoleType='Attach'
aws iam list-attached-role-policies --role-name $RoleName --query 'AttachedPolicies[].[PolicyArn]' --output text | sed 's#[ \t]\+#\n#g' | while read AttachPoliARN; do
$DBG || $ERR && echo ">>>> RoleName:$RoleName > AttachPoliARN:$AttachPoliARN" >&3
PoliVer=`aws iam get-policy --policy-arn $AttachPoliARN --query 'Policy.DefaultVersionId' --output text`
Poli=`aws iam get-policy-version --policy-arn $AttachPoliARN --version-id $PoliVer`
Ptime=`echo "$Poli" | jq -r '.PolicyVersion.CreateDate'`
policy2tab2 "`echo "$Poli" | jq '.PolicyVersion.Document'`" \
"$EntityType $RoleType $RolePrincipal $RoleName $RoleTime ${RoleCount[$RoleName]:-None} $Group ${AttachPoliARN/*\//} $AttachPoliARN $Ptime $PoliVer"
done
done
}
get_last(){
echo "Entity Name Status Namespace NSlast"
(
aws iam list-users --query 'Users[*].[UserName]' --output text | sed 's#^#User #'
aws iam list-roles --query 'Roles[*].[RoleName]' --output text | sed 's#^#Role #'
) | egrep "$grep_user_role" | while read E N; do
$DBG && echo ">>> $E $N" >&7
jid_out=`aws iam generate-service-last-accessed-details --arn arn:aws:iam::$account:${E,,}/$N`
#$DBG && echo "aws iam generate-service-last-accessed-details --arn arn:aws:iam::479982568640:${E,,}/$N" >&4
echo "$jid_out" | jq -e . &>/dev/null
if [ ! $? -eq 0 ]; then
$DBG && echo "CRIT: Cant get damn jid" >&7
echo "$E $N ERROR ERROR ERROR"
$DBG && echo "$jidi_out" | sed "s#^# ERROR: #" >&7
continue
fi
jid=`echo "$jid_out" | jq -r '.JobId'`
if [[ ! $jid =~ ^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$ ]]; then
$DBG && echo "CRIT: Bad jid - $jid" >&7
continue
fi
flag_api_cache=false
local counter_get=0
while true; do
raw=`aws iam get-service-last-accessed-details --job-id "$jid"`
#$DBG && echo "aws iam get-service-last-accessed-details --job-id \"$jid\"" >&7
echo "$raw" | jq -e . &>/dev/null
if [ ! $? -eq 0 ]; then
$DBG && echo "CRIT: Cant get report (try $counter_get)" >&7
$DBG && echo "$raw" | sed "s#^# ERROR: #" >&7
if [ $((++counter_get)) -eq 10 ]; then
$DBG && echo "CRIT: Cant get report max retires (10) reached (try $counter_get)" >&7
echo "$E $N ERROR ERROR ERROR"
break
fi
continue
fi
[ `echo "$raw" | jq -r '.JobStatus'` = "COMPLETED" ] && break
$DBG && echo " wait for aws report (job-id:$jid)" >&7
sleep 1
done
flag_api_cache=true
#echo "$raw" | jq -C . >&7
echo "$raw" | jq -r '.JobStatus as $S | .ServicesLastAccessed[] | "\($S) \(.ServiceNamespace) \(.LastAuthenticated)"' | sed 's#null#NULL#g' | sed "s#^#$E $N #" | column -t
done #| tee ${cache_pref}.last
}
enrich_last(){
#local main="$1"
#local last="$2"
(
#head -1 $main | sed 's# Time # Time Last #; s# Namespace # Namespace NSlast #;'
echo "Entity Type Trust Name Time Last InsCount Group Policy PoliARN PoliOwn Ptime PoliVer Statement Effect Namespace NSlast Action Resource"
gawk '
function iso2num(iso){ gsub(/[^0-9]/,"",iso); return iso+0; }
NR==FNR{
S[$1" "$2" "$4]=$5
if (iso2num($5) > iso2num(E[$1" "$2])) E[$1" "$2]=$5
}
NR!=FNR{
LE=E[$1" "$4]
if (LE=="") LE="None"
if ($14!="*") {
LS=S[$1" "$4" "$14]
} else {
LS=LE
}
if (LS=="") LS="None"
# extract policy owner from policy ARN
split($9,arn,":")
POwn=arn[5]
if (POwn=="") POwn="None"
print $1,$2,$3,$4,$5,LE,$6,$7,$8,$9,POwn,$10,$11,$12,$13,$14,LS,$15,$16
}
END{
#for (e in E) print e, "--", E[e];
#for (s in S) print s, "--", S[s];
}' <(get_last | tee ${cache_pref}.$profile.last.tmp | sed 1d) <(get_report | tee ${cache_pref}.$profile.poli.tmp | sed 1d)
# PATTERN EXAMPLE: Handling two files with awk and implicit parallalism (with pipe size limit) without losing the output
# pipe effective buffer size - M=0; while printf A; do printf "\r$((++M)) B" >&2; done | sleep 2; echo
)
}
get_profile(){
export profile
export account=`aws sts get-caller-identity --query 'Account' | sed 's#"##g'`
enrich_last > ${cache_pref}.$profile.rich.tmp
for f in rich last poli; do
#column -t ${cache_pref}.$profile.${f}.tmp > ${cache_pref}.$profile.${f}
sed "1s#\$# Profile Account#; 2,\$s#\$# $profile $account#" ${cache_pref}.$profile.${f}.tmp | column -t > ${cache_pref}.$profile.${f}
rm ${cache_pref}.$profile.${f}.tmp
done
}
print_help(){
cat <<-HELP
Syntax: Enfo_iam.sh
Options:
-c|--cache-dir <dir> Use a specifc report cache dir. Default is /var/cache/Enfo_iam/Enfo_iam_report_*.
-d|--debug Debug output
-E|--exit-on-error Stop script on aws error
-e|--error Error output
-f|--fresh-data Get a new report, automatically saved as last cache when finished
Works well with -q for cronjobs or with '-q -d' for debug
-F|--fields <comma separated fileds list>
Comma separated columns list (see below)
-h|--help Help
-P|--no-para Disables porfile parallelism (mainly for debug)
-q|--quiet Supress shell output
-N|--dont-save-report Do not save report (effective with -G)
-w|--col-width Max column width (trancated column end with _)
-G|--grep-entity Global grep on entity (user ot role)
-r|--retain-policies <dir>
Saves all policies json files into dir (for git)
Usage patterns:
Cron daily job
Enfo_iam.sh -f -q -E
Filtered (pre data collection) run without saving report (much faster than getting full report)
Enfo_iam.sh -f -G 'shoko' -N -E -F Entity,Type,Name,Group,Policy,Effect,Action,Resource,Profile
Non-parallel full run debug without printing or saving report
time Enfo_iam.sh -f -P -d -q -N -E
Columns:
Entity User or Role
Type Inline or Attach
Trust Role trusted entities
Name Username or role name
Time User or role creation time
Last User/Role last api access time
InsCount Role assigned instances count
Group User group
Policy Policy name
PoliARN Policy ARN
PoliOwn Policy owner (extracted from ARN)
Ptime Policy creation time
PoliVer Attached policy default version
Statement Policy statement index
Effect Statement effect - Allow/Deny
Namespace Action namespace
NSlast Namespace last api access time
Action Statement action
Resource Statement resource
Profile Profile
Account Account ID
Column full list:
Entity,Type,Trust,Name,Time,Last,InsCount,Group,Policy,PoliARN,PoliOwn,Ptime,PoliVer,Statement,Effect,Namespace,NSlast,Action,Resource,Profile,Account
Column filter operators:
Arithmetic > larger than
< less than
>= larger than or equal to
<= less than or equal to
note: also filter any non-numeric values as side effect)
Text = equal
~ regex comparison, case sensative, support | for or
Date Syntax: [column][<>][number][unit]
< date is after given value
> date is before given value
unit can be one of sec(ond)?s?|min(ute)?s?|hours?|days?|mo(nth)?s?|years?
Reverse glob \` match asterisk values
added specifically to find actions in policy actions and resources glob patterns
make sure you single qoute the field list!
Column prefix:
! Supress column output giving the option to filter on column without showing it.
Output files:
Detailed last access data $rep_cache_dir/Enfo_iam_report_YYMMDDHHmmss.<profile>.last
Base policy data $rep_cache_dir/Enfo_iam_report_YYMMDDHHmmss.<profile>.poli
Policy with last data $rep_cache_dir/Enfo_iam_report_YYMMDDHHmmss.<profile>.rich
Main merged report (cache file) $rep_cache_dir/Enfo_iam_report_YYMMDDHHmmss.rich
Usage Examples:
All entities that can delete object on a specific bucket
${0//*\//} -F 'Entity,Name,InsCount,Effect=Allow,Action\`s3:DeleteObject,Resource\`arn:aws:s3:::image-res-platform,Profile=default'
Unused ec2 roles
${0//*\//} -F 'Entity,Type,Trust~ec2,Name,Last,InsCount=None,Profile'
All policies that were changed in the last week
${0//*\//} -F 'Entity,Type,Trust,Name,Policy,Ptime<7day,Profile'
All the users that were not used (in any namespace) in the last 6 months
${0//*\//} -F 'Entity=User,Name,Last>6month,Profile'
All target services (namespace) included in trust from two specific sources
${0//*\//} -F 'Entity,Type,Trust~sage|robo,Name,Last,Profile,Namespace'
All in-use AWS managed policies with version
${0//*\//} -F 'Policy,!PoliOwn=aws,PoliVer,!Profile=default'
Notes:
- Last access data is per namespace (target service) and not a pecific action. Entity last access is derived
from the namespace data.
- 'NULL' values for namespace last access were returned from AWS report (generate-service-last-accessed-details).
'None' values mean the namespace was not found (should not happen).
- Multiple conditions on the same column is not supported.
HELP
}
# MAIN
########
profiles='default'
time_stamp=`date -Isec | sed 's#+.*##; s#[^0-9]##g'`
rep_cache_dir='/var/cache/Enfo_iam'
[ ! -d ${rep_cache_dir} ] && mkdir -p ${rep_cache_dir}
if [ ! -d ${rep_cache_dir} ]; then
echo "Cache dir creation failed"
exit 1
fi
# indays
retain_old_rep=3
# in sec
aws_timeout=10
## DEFAULTS
fields='Entity,Type,Trust,Name,Time,Last,InsCount,Group,Policy,PoliARN,PoliOwn,Ptime,PoliVer,Statement,Effect,Namespace,NSlast,Action,Resource,Profile,Account'
flag_age=false
ERR=false
DBG=false
flag_fresh=false
flag_quiet=false
col_width=10000
flag_para=true
grep_user_role=''
flags_save_report=true
flag_exit_on_error=false
policy_ret_dir="/var/cache/Enfo_iam_ret_$$"
flag_ret_policy=false
unset cache_file
while [ ! -z "$*" ]; do
case "$1" in
-a|--report-age) flag_age=true;;
-f|--fresh-data) flag_fresh=true;;
-c|--cache-dir) shift; rep_cache_dir=$1;;
-F|--fields) shift; fields=$1;;
#-F|--fields) shift; fields=$1; field_func; exit;;
-d|--debug) DBG=true;;
-e|--error) ERR=true;;
-h|--help) print_help; exit 0;;
-w|--col-width) col_width=$2; shift;;
-P|--no-para) flag_para=false;;
-G|--grep-entity) grep_user_role="$2"; shift;;
-N|--dont-save-report) flags_save_report=false;;
-E|--exit-on-error) flag_exit_on_error=true;;
-r|--retain-policies) flag_ret_policy=true; shift; policy_ret_dir_target=$1; shift
$flag_ret_policy && [ ! -d $policy_ret_dir ] && mkdir $policy_ret_dir
[ -d $policy_ret_dir_target ] && echo "CRITICAL: Policy retention dir already exist" && exit 1;;
-q|--quiet) flag_quiet=true;;
esac
shift
done
# get report file $cache_file
if $flag_fresh || [ -z "`ls ${rep_cache_dir}/Enfo_iam_report_*.rich 2>/dev/null | egrep '[0-9][0-9].rich$' | tail -1`" ] ; then
cache_pref="${rep_cache_dir}/Enfo_iam_report_cache_${time_stamp}"
for profile in $profiles; do
# PATTERN EXAMPLE: quick parallel in a loop
### (
### export profile
### export account=`aws sts get-caller-identity --query 'Account' | sed 's#"##g'`
### enrich_last > ${cache_pref}.$profile.rich.tmp
### for f in rich last poli; do
### #column -t ${cache_pref}.$profile.${f}.tmp > ${cache_pref}.$profile.${f}
### sed "1s#\$# Profile Account#; 2,\$s#\$# $profile $account#" ${cache_pref}.$profile.${f}.tmp | column -t > ${cache_pref}.$profile.${f}
### rm ${cache_pref}.$profile.${f}.tmp
### done
### )&
### pids="$pids $!"
if $flag_para; then
get_profile&
pids="$pids $!"
else
get_profile
fi
done
$flag_para && wait $pids
$flag_exit_on_error && [ -f $cache_pref.err ] && exit
# merge rich report from all profiles
(
cat `ls ${cache_pref}.*.rich | head -1` | head -1
for r in ${cache_pref}.*.rich; do
cat $r | sed 1d
done
) | column -t > ${cache_pref}.rich.tmp
cache_file="${cache_pref}.rich.tmp"
else
echo NoCACHE
if [ -z "$cache_file" ]; then
#cache_file=`ls -tr ${rep_cache_dir}/Enfo_iam_report_*.rich | egrep '[0-9].rich$' | head -1`
cache_file=`ls -tr ${rep_cache_dir}/Enfo_iam_report_*.rich | egrep '[0-9][0-9].rich$' | tail -1`
fi
fi
if $flag_age; then
echo "report_file=$cache_file"
report_file_age_sec=$((`date +%s`-`stat -c %Y $cache_file`))
echo "report_file_age_sec=$report_file_age_sec"
echo "report_file_age=$(($report_file_age_sec/86400))d$(($report_file_age_sec%86400/3600)):$(($report_file_age_sec%3600/60)):$(($report_file_age_sec%60))"
echo "report_file_age_date=`stat -c %y $cache_file | sed 's# #T#; s#\..*##'`"
exit
fi
$DBG && echo "$cache_file" >&4
if $flags_save_report; then
# remove old reports
#rm_lst=`ls -altr /var/cache/Enfo_iam/Enfo_iam_report_* | sed "1,${retain_old_rep}d"`
rm_dates=`ls -altr ${rep_cache_dir}/Enfo_iam_report_* | awk -F'[_.]' '{print $5}' | sort -ur | sed '1,3d'`
for d in $rm_dates; do
rm ${rep_cache_dir}/Enfo_iam_report_${d}.*
done
#[ ! -z "$rm_lst" ] && rm $rm_lst &>/dev/null
# clean old tmp files
#find /var/cache/Enfo_iam/ -regex ".*Enfo_iam_report_.*.tmp" -mmin +60 -delete
fi
# main output
#echo PRINTING from $cache_file
$flag_quiet || cat $cache_file | field_func | column -t
$flags_save_report && [ -e ${cache_pref}.rich.tmp ] && mv ${cache_pref}.rich.tmp ${cache_pref}.rich
$flag_ret_policy && mv $policy_ret_dir $policy_ret_dir_target
[ ! -z "$cache_pref" ] && ! $flags_save_report && rm ${cache_pref}*
[ ! -z "$CA" ] && rm $CA &>/dev/null
[ ! -z "$api_cache" ] && [ -d $api_cache ] && rm -rf $api_cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment