How to inspect the packets used by AWS CLI, e.g. s3control AWSCore.jl POST content body

ReadMe.md

How to inspect the REST packets used by AWS CLI

Install git and virtualenv and python on your system.

On a mac, this would look like this:

brew install git
pip3 install virtualenv

Checkout the aws-cli source, setup the venv, and activate it, install the requirements and go to the bin folder.

git clone https://github.com/aws/aws-cli.git aws-cli && cd aws-cli
virtualenv venv
. venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -e .
python -m pip install awscli
cd bin
./aws s3 ls

This should have a functional command, and you should have your aws credentials setup already before coming to this page.

in venv/src/botocore/botocore/client.py add a simple print statement before the self.meta.events.emit_until_repsonse near line 652

Such as print("Peter was here")

Rerun the last command above:

./aws s3 ls

Before it lists your buckets it should now print your new line.

Viewing the SOAP calls, XML from the call

Now open up venv/src/botocore/botocore/hooks.py and add a new import and function to the file outside any of the classes:

from inspect import currentframe

def get_linenumber():
    cf = currentframe()
    return cf.f_back.f_lineno

Then find one of the functions (line 240) named emit_until_response and add in two print statements:

print(get_linenumber())
print(kwargs)

Save the file, and try your command again:

./aws s3 ls

The output now looks something like this:

(venv)
phyatt at Peters-MBP in ~/src/aws-cli/bin on develop
$ ./aws --profile dev1 s3 ls
Peter was here
366
248
{'model': OperationModel(name=ListBuckets), 'params': {'url_path': '/', 'query_string': '', 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/1.18.189 Python/3.9.0 Darwin/19.6.0 botocore/1.19.29'}, 'body': b'', 'url': 'https://s3.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}, 'request_signer': <botocore.signers.RequestSigner object at xxxxx>, 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
366
248
{'signing_name': 's3', 'region_name': 'us-east-1', 'signature_version': 's3v4', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': None}}}
Peter was here
366
248
{'model': OperationModel(name=AssumeRole), 'params': {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/1.18.189 Python/3.9.0 Darwin/19.6.0 botocore/1.19.29'}, 'body': {'Action': 'AssumeRole', 'Version': '2011-06-15', 'RoleArn': 'arn:aws:iam::0123456789012:role/MyRole', 'RoleSessionName': 'botocore-session-1234'}, 'url': 'https://sts.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None}}, 'request_signer': <botocore.signers.RequestSigner object at xxxxx>, 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None}}
366
248
{'signing_name': 'sts', 'region_name': 'us-east-1', 'signature_version': 'v4', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at xxxxx>, 'has_streaming_input': False, 'auth_type': None}}
2020-04-16 16:25:23 bucket-1
...
2020-04-16 16:25:22 bucket-n
(venv)

Now rerun the same command for something more exciting like

aws s3api create-job and you can see the real post packet converted from the json.

Hope that helps.