Get fiddler working with python requests module

fiddler_with_python.md

Get fiddlers base64 encoded root certificate

1. Install fiddler winget install -e --id Telerik.Fiddler
2. Open fiddler and go to Tools -> Options -> HTTPS
3. Enable Decrypt HTTPS traffic
4. Click the Actions button and select Export root certificate to desktop
5. Right click the FiddlerRoot.cer file on the desktop and click Open with -> Crypto Shell Extensions
6. In the Certificate window that opens up go to Details -> Copy to File
7. Click Next then select Base-64 encoded X.509 (.CER) then specify the file name (E.G. FiddlerRootBase64.cer)
8. Click Next to create the new file
9. Open the file and copy the contents to the clipboard

Get python to use root certificate

1. Find out where python is storing its certificate authority certs. For newer version of the python you can run the following commands

pip install certifi
python -c "import certifi; print(certifi.where())"

2. Open the file where the CA certs are and paste the fiddler base64 certificate contents to the end of the file

Set python code to use proxy server

The easiest way to do this is to have a block of code in the entry point of your application that sets some environment variables that the python requests module will use

import os

os.environ['HTTP_PROXY'] = 'http://127.0.0.1:8888'
os.environ['http_proxy'] = 'http://127.0.0.1:8888'
os.environ['HTTPS_PROXY'] = 'https://127.0.0.1:8888'
os.environ['https_proxy'] = 'https://127.0.0.1:8888'

You can also configure the proxy via code in the requests module if preferred

Troubleshooting

If you get any check_hostname requires server_hostname when using the proxy try updating changing the version of the urllib3 python module you're using https://stackoverflow.com/questions/66642705/why-requests-raise-this-exception-check-hostname-requires-server-hostname

It allows you to decrypt all HTTPS traffic in Fiddler that's coming to and from your Python application. It's useful if you want to get detailed information on why requests are failing

os.environ['HTTP_PROXY'] = 'http://127.0.0.1:8888'
os.environ['http_proxy'] = 'http://127.0.0.1:8888'
os.environ['HTTPS_PROXY'] = 'https://127.0.0.1:8888'
os.environ['https_proxy'] = 'https://127.0.0.1:8888'

This is wrong. The URL for the proxy should be the same in all cases, i.e. always http://, never https://. See Your proxy appears to only use HTTP and not HTTPS for more.

I was looking to sniff python requests in Fiddler. A fair bit of experience in python, but new to Fiddler. Intermediary applications like Fiddler are useful for inspecting request headers/cookies without having to rely on the server response; for instance, https://httpbin.org/headers will capitalize headers according to RFC outline, but what if you want to see the exact headers you sent to a website? What if some anti-bot is specifically looking for header order or case sensitivity?

I wanted to test if a specific HTTP Adapter in requests was ordering headers properly and sending with case sensitivity. Later, I wanted to test the same in Scrapy.

Initially, I was able to get this working by following a documentation guide here.

The problem was I kept having to add verify=False to every request I made, even if I wasn't using Fiddler proxy.

Using the steps outlined above, I was able to make requests without having verify=False. I wasn't able to get this to work on Ubuntu on Windows (WSL), but I was on macOS with no additional trouble shooting required (M1 chip, Apple Silicon).

Thank you!