Skip to content

Instantly share code, notes, and snippets.

@jlongman
Last active May 8, 2017 21:04
Show Gist options
  • Save jlongman/e1557e478245212694a98bfbbfe2ddcf to your computer and use it in GitHub Desktop.
Save jlongman/e1557e478245212694a98bfbbfe2ddcf to your computer and use it in GitHub Desktop.
jq for selecting key and nested value
# sample data:
echo '{"count": {}, "block":
{
"67.215.237.26": {
"paths": {
"/js/compiled.js": 1,
"/css/global-compiled.css": 1,
"/js/pace.min.js": 1
},
"max_req_per_min": 1,
"updated_at": "2017-05-08 13:59:40"
},
"70.186.138.162": {
"paths": {
"/includes/gallery.php": 2
},
"max_req_per_min": 2,
"updated_at": "2017-05-08 13:59:40"
}
}
}' | \
jq '.block| to_entries| .[] | select(.value.paths | keys | length ==1 ) | [.]| from_entries'
# or jq '[.block| to_entries| .[] | select(.value.paths | keys | length ==1 ) ] | from_entries'
# if you need them in an array
@jlongman
Copy link
Author

jlongman commented May 8, 2017

You're a crazy man, longman: jq '[.block| to_entries| .[] | select(.value.paths | keys | length ==1 ) ] | from_entries' /tmp/foo.json | jq '.| to_entries | .[] | select(.value.paths |keys|.[0] == "/")| (.key, .value.max_req_per_min)

@jlongman
Copy link
Author

jlongman commented May 8, 2017

This parses the output of https://aws.amazon.com/blogs/security/how-to-configure-rate-based-blacklisting-with-aws-waf-and-aws-lambda/ when parsed from a local file (my changes), which lists the pages accessed (my changes), then this code is used so that we can determine if an IP is only ever hitting one page (like we saw recently in a DDoS). The later addition searches for the path, "/" in this case.

Using this answer maybe we can merge the json outputs to get whether different pages were hit, just across log files.

Note these mostly serve as an independent verification from the main parser which is solving a slightly different problem (hitting any page too often rather than only hitting one page, or one specific page, too often)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment