Debugging HTTP traffic with mitmproxy

README.md

This will let you see the request and response headers for traffic going through.

We're going to run this as a reverse proxy, rather than a usual proxy, so you don't get completely flooded with traffic.

Start the proxy

1. create a new VM
2. expose port 8080 to the public internet
3. SSH to the VM
4. make sure you have at least python 3.6
5. install mitmproxy with pip

   sudo -H pip3 install mitmproxy

6. create a python script that will be run for each request. Use the script.py file in this gist.
7. define an environmental variable with the endpoint you wish to proxy

   export THE_ENDPOINT=https://example.com:80 # change me

8. find out your public network interface address

   ifconfig

9. create an env var with the public IP:

   export THE_INTERFACE=11.22.33.44 # change me

10. start mitmproxy

    mitmdump \
      --listen-host $THE_INTERFACE \
      -p 8080 \
      --mode reverse:$THE_ENDPOINT \
      --set block_global=false \
      -s /path/to/script.py

Configure your client

Now you need to configure your client. Remember, this is not a proxy server, it's a reverse proxy so just change your code/config to use the IP and port of the VM instead of your endpoint.

Then you trigger your client and you should see see the traffic in the console on the VM as mitmproxy is running in the foreground.

Example output

==================================================
GET /preheat HTTP/1.1
------------------------- request headers -------------------------
ACCEPT-ENCODING               : identity
X-REQUESTED-WITH              : XMLHttpRequest
ACCEPT                        : application/json
USER-AGENT                    : Appcelerator Titanium/7.2.0 (Nexus 5; Android API Level: 27; en-AU;)
HOST                          : example.com
CONNECTION                    : Keep-Alive
------------------------- response headers -------------------------
CONTENT-TYPE                  : application/json; charset=utf-8
CONTENT-LENGTH                : 27
CONNECTION                    : keep-alive
DATE                          : Wed, 30 Jan 2018 03:05:42 GMT
VARY                          : Accept-Encoding
X-POWERED-BY                  : up
ETAG                          : W/"1b-SWSBjMAAv99BCA8zPdG9jaVeqEs"
------------------------- body (first 100 bytes) -------------------------
some body stuff...
118.210.255.140:47127: GET https://example.com/preheat
                    << 200 OK 27b

script.py

def response(flow):
    print("")
    print("="*50)
    print(flow.request.method + " " + flow.request.path + " " + flow.request.http_version)

    print("-"*25 + " request headers " + "-"*25)
    for k, v in flow.request.headers.items():
        print("%-30s: %s" % (k.upper(), v))

    print("-"*25 + " response headers " + "-"*25)
    for k, v in flow.response.headers.items():
        print("%-30s: %s" % (k.upper(), v))

    print("-"*25 + " body (first 100 bytes) " + "-"*25)
    print(flow.request.content[0:100])

Is there a way to use this to check what the docker API calls happening when we do a docker login and pull ?

@govindkailas there are two options I'd try.

The first is using mitmproxy in the normal usage: as a proxy. The instructions I have above are using a reverse-proxy instead. You can look at how to set up the docker CLI to use a proxy (https://docs.docker.com/network/proxy/) and then run mitmproxy on your local machine to intercept the traffic.

Alternatively, if the docker CLI can be configured to point to custom endpoints (container registries), which I'm pretty sure it can, then you can use the reverse proxy idea from this gist as a facade for the container registry. This way is harder, I'd try the regular proxy idea first.

thanks for the details, I tried the first option,

• added the localhost:8080 as http and https proxy to docker config as explained in the docker documentation.
• restarted docker service
• started mitmproxy in 8080 which is the default

I don't see any traffic being intercepted when I do a docker pull. I tried a couple of filters from the docs but that doesn't seem to work.