HTTP, HTTPS and TLS: Part 13 — Intercepting HTTPS for debugging: DevTools, HAR files and mitmproxy

curl shows what you sent yourself, and tcpdump and Wireshark (“Network troubleshooting: tcpdump, sngrep and Wireshark”) show packets but without HTTPS decryption. Often you need to see what a third-party client sends: a browser when a form fails, a mobile app that cannot reach the API, a program that behaves differently from the documentation. For that traffic is intercepted: in the browser — with built-in developer tools; for other programs — with a proxy that decrypts HTTPS. Everything described applies to your own devices and programs or to systems you have permission to analyze.

Network tab in the browser

Developer tools open with F12 (on macOS — Cmd+Option+I). On the Network tab you can see all page requests: address, method, response code, type, size, time.

Useful settings at the top of the tab:

• Preserve log — don’t clear the list when navigating to another page. Without this you won’t see a form request that is followed by a redirect;
• Disable cache — load everything from the server while the tools are open;
• Throttling — emulate a slow network (Fast 4G, Slow 3G) and offline operation;
• filter Fetch/XHR — only JavaScript requests to APIs; the search box understands conditions like status-code:500, method:POST, domain:api.example.ru, larger-than:1M.

Clicking a request opens tabs:

• Headers — request and response headers, final URL, server address;
• Payload — query string parameters and form body or JSON;
• Preview / Response — response in parsed and raw form;
• Timing — breakdown of time: queueing, DNS, connect, TLS, server response wait (TTFB), download. The same information as curl -w;
• Initiator — which script or page element triggered the request.

Enable the Protocol column (right-click the table header): it shows h2, h3 or http/1.1.

You can copy a request to repeat it: Copy → Copy as cURL (as a curl command with all headers and cookies) or Copy as fetch (as JavaScript for the console).

CORS errors in the browser console look like network failures, and in Network the request status is CORS error. Check the preflight OPTIONS request next to it: most often it returned 401, 404 or a response without Access-Control-Allow-* headers.

Certificate and TLS. The Security tab (in Chrome) shows the TLS version, key exchange group, cipher and certificate for the current page, as well as resources loaded over HTTP.

HAR files

HAR (HTTP Archive) is a JSON file with all requests and responses of a session, including headers, bodies and timings. Support teams ask for it when they need to reproduce an issue on their side. Save it: on the Network tab — the down arrow button (Export HAR) or right-click → Save all as HAR.

HAR files contain passwords and sessions. The file includes cookies, Authorization headers, tokens in URLs and request bodies, including entered passwords. Sending such a file is equivalent to giving access to your account: in 2023 attackers gained access to Okta customers precisely via HAR files attached to support requests.

Since Chrome 130 the export is “sanitized” by default: without Cookie, Set-Cookie and Authorization. The full version is available only after enabling the Allow to generate HAR with sensitive data setting. But tokens in URLs, in request bodies and in custom headers (like X-Api-Key) are not removed by that sanitization. Before sending, open the file in an editor and search for token, key, password, session. After debugging, sign out so the session from the file becomes invalid.

You can view a HAR in the Network tab itself (drag the file into the window) or in the online viewer Google Admin Toolbox HAR Analyzer.

Intercepting proxy: how it works

For programs outside the browser you need a proxy that sits between the client and server and decrypts HTTPS. The most common free option is mitmproxy; there are also Charles, Proxyman (macOS), HTTP Toolkit, Burp Suite.

Scheme:

1. The client connects to the proxy instead of the server.
2. The proxy itself connects to the server over TLS and validates the server’s real certificate.
3. The proxy presents the client with a certificate for the same name, generated on the fly by its own certificate authority.
4. If the client trusts that CA, the proxy ends up with two decrypted connections and can see and modify everything between them.

The main requirement is to have the proxy’s root certificate installed on the device. Without it the client will see a fake certificate and terminate the connection — exactly how TLS is supposed to work.

mitmproxy

Install with brew install mitmproxy, pipx install mitmproxy or the Docker image mitmproxy/mitmproxy. Three interfaces:

• mitmproxy — interactive terminal interface;
• mitmweb — web interface at http://127.0.0.1:8081;
• mitmdump — no UI, for recording and scripts.

By default the proxy listens on port 8080. On first run mitmproxy creates its own CA in ~/.mitmproxy/.

Test with curl:

mitmweb &
curl -x http://127.0.0.1:8080 --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem https://example.ru/

The request will appear in the web interface with all headers and the body.

Installing the certificate. Configure a proxy on the device IP-of-computer:8080 and open http://mitm.it in a browser — a page served by mitmproxy with the certificate and instructions for each system.

• iOS: install the profile, then enable full trust in Settings → General → About → Certificate Trust Settings.
• Android: since version 7 apps do not trust user-installed certificates by default — only system ones. Chrome will trust them, most apps will not. For your own app, trust for user certificates is enabled in the debug build via network_security_config.xml; for third-party apps you need an emulator or a rooted device where the certificate is placed into the system store.
• Windows and macOS: import into the system root certificate store. Firefox uses its own store — you must add the certificate in its settings.

Modes of operation. Besides the usual proxy, mitmproxy can:

• --mode wireguard — launches a WireGuard server; the device connects to it as a VPN via a QR code from mitmweb, and all its traffic goes through mitmproxy without proxy configuration. Useful for phones and apps that ignore the system proxy;
• --mode local — intercept programs on the same machine without proxy settings: mitmproxy --mode local:curl — only curl, --mode local:!curl — everything except curl;
• --mode reverse:https://api.example.ru — reverse proxy in front of a specific server;
• --mode transparent — transparent interception together with firewall redirection rules.

Modifying requests. A Python script replaces requests and responses on the fly — convenient to test how an app survives a server error without touching the server:

# fail.py — return 503 for requests to /api/orders
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    if flow.request.pretty_url.startswith("https://api.example.ru/api/orders"):
        flow.response = http.Response.make(503, b'{"error":"maintenance"}',
                                           {"Content-Type": "application/json"})

mitmdump -s fail.py

Record a session and replay it later: mitmdump -w flows.mitm, then mitmproxy -r flows.mitm. Export to HAR — mitmdump -n -r flows.mitm --set hardump=flows.har (-n — don’t run the proxy, only process the file).

Full example: how an app handles an API failure

Task: check that a mobile app shows a user-friendly message instead of crashing when the orders API returns 503. The server must not be changed. Verified with mitmproxy 12.2.

1. Replacement script — the fail.py file from the section above, replace the URL with your API.

2. Start with session recording:

mitmweb -s fail.py -w flows.mitm --mode wireguard

mitmweb will open the web interface and show a QR code with WireGuard settings.

3. Phone. Install the WireGuard app, scan the QR code, enable the tunnel. Open http://mitm.it on the phone, install the certificate and enable trust for it (on iOS — in Certificate Trust Settings; on Android — only if this is the debug build of your app or an emulator).

4. Verify interception. Open any website in the phone’s browser — the request will appear in mitmweb. If the site does not open and shows a certificate error, the certificate trust is not enabled.

5. Scripted response. Open the orders list in the app. In mitmweb the request to /api/orders will get a 503 from the script, and the app should show an error message and offer to retry.

6. Review. After testing stop mitmweb (Ctrl+C) and inspect the recording:

mitmproxy -r flows.mitm                                # interactive
mitmdump -n -r flows.mitm --set hardump=flows.har      # to HAR — for example, attach to a ticket

7. Cleanup. Disable the WireGuard tunnel and remove the certificate profile from the phone. Files flows.mitm and flows.har contain app tokens — delete them or store them as secrets.

You can test the script without the phone from the command line:

mitmdump -s fail.py &
curl -x http://127.0.0.1:8080 --cacert ~/.mitmproxy/mitmproxy-ca-cert.pem https://api.example.ru/api/orders
# {"error":"maintenance"}

When interception doesn’t work

Pinning. The app verifies the server certificate not only against the system store but also against an embedded list of keys. The proxy’s certificate will not pass such a check, and the app will refuse to connect even if the root certificate is installed. Banking apps, messengers and apps from large services do this. For your own app pinning is disabled in the debug build. For third-party apps it’s bypassed with dynamic instrumentation tools like Frida — this is analysis of a protected app and is allowed only with the owner’s permission.

Protocols not using HTTP. Apps that use QUIC, custom protocols over TCP or UDP, gRPC with nonstandard settings, may not work through an HTTP proxy. For HTTP/3 browsers may fall back to TCP; newer mitmproxy versions can handle HTTP/3 but not in all modes.

App ignores the system proxy. Many programs don’t read system proxy settings: Go apps look only at HTTPS_PROXY environment variables, Java apps use their own JVM parameters. WireGuard mode or transparent mode helps — they intercept traffic at the network level.

TLS fingerprint. The server sees the proxy’s TLS fingerprint, not the app’s. If the server defends against bots by JA3/JA4 fingerprinting, requests via a proxy may get a captcha or 403, even though the app works directly.

Security after debugging

• Remove the proxy root certificate from the device when you finish. The private key of that CA is in ~/.mitmproxy/ on your computer, and anyone who gets it can decrypt traffic for devices that trust it.
• Don’t leave mitmproxy listening on all interfaces on a public network: an open proxy without authentication is a public proxy. The --set proxyauth=user:pass parameter sets a password.
• Recorded sessions and HAR files contain passwords and tokens — treat them as secrets and delete them.