Skip to content

Instantly share code, notes, and snippets.

@Efpophis
Last active June 22, 2026 12:44
Show Gist options
  • Select an option

  • Save Efpophis/d81a07bde9e61323a8ecdfd221daf848 to your computer and use it in GitHub Desktop.

Select an option

Save Efpophis/d81a07bde9e61323a8ecdfd221daf848 to your computer and use it in GitHub Desktop.
Possible packet corruption affecting large TLS uploads on Comcast backbone (AS7922) toward Chicago CDN edges (Cloudflare ORD / AWS CloudFront ORD)

TITLE

Possible packet corruption affecting large TLS uploads on Comcast backbone (AS7922) toward Chicago CDN edges (Cloudflare ORD / AWS CloudFront ORD)

EXECUTIVE SUMMARY

Large HTTPS uploads fail consistently with TLS error ERR_SSL_BAD_RECORD_MAC_ALERT when using a Comcast/Xfinity residential connection.

The identical uploads succeed immediately when routed through alternate backbone paths (Google Cloud WireGuard tunnel and corporate VPN).

Testing indicates the issue occurs specifically when traffic traverses Comcast backbone ASN AS7922 toward CDN edge infrastructure located in Chicago (ORD).

Uploads to services behind multiple CDN providers are affected, suggesting the issue may lie within Comcast backbone routing or peering rather than a single CDN provider.

ENVIRONMENT

ISP: Comcast / Xfinity
ISP ASN: AS7922

Gateway: Technicolor CGM4981COM (XB8)
Router: Linksys WRT3200ACM running OpenWRT
Client OS: Windows 11
Browser: Chrome

PROBLEM DESCRIPTION

Large HTTPS uploads fail with browser error:

ERR_SSL_BAD_RECORD_MAC_ALERT

This TLS alert indicates corrupted encrypted payload data was received by the server.

Failure characteristics:

  • occurs during upload (usually early or mid-transfer)
  • downloads and small requests succeed normally
  • reproducible consistently on Comcast path
  • disappears immediately when traffic is routed via VPN

NETWORK TESTING PERFORMED

MTU testing

ping -f -l 1472 files.bom.com β†’ success
ping -f -l 1473 files.bom.com β†’ fragmentation required

This confirms a standard 1500 byte MTU path.

Router testing

OpenWRT router flow offloading disabled. No change in behavior.

NIC testing

All NIC hardware offload features disabled:

  • TCP checksum offload
  • LSO / TSO
  • GSO

No change in behavior.

TCP MSS testing

Router forced TCP MSS reduction (~1300 bytes).

Uploads still fail identically.

DOCSIS SIGNAL QUALITY

Comcast technician recently replaced drop cable and repaired damaged upstream line segment.

Current modem metrics:

  • Downstream SNR: ~45–46 dB
  • Downstream power: +6 to +8 dBmV
  • Upstream power: ~37–39 dBmV

These are well within normal operating ranges.

General traffic shows no packet loss or instability.

PRIMARY FAILURE CASE

Example failing service: BOM / Arena Solutions upload portal

Destination:

files.bom.com.cdn.cloudflare.net
104.18.2.221

Cloudflare ASN: AS13335

Example response header:

CF-RAY: 9d9b14808a06c93a-ORD

Multiple observed CF-RAY values were different per request, but all ended in -ORD.

The "-ORD" suffix indicates traffic ingress through the Cloudflare Chicago POP.

Traceroute:

tracert -d files.bom.com

 1  <1 ms  <1 ms  <1 ms  10.95.29.1
 2   13 ms  12 ms   7 ms 100.92.46.2
 3   11 ms  13 ms  12 ms 69.139.243.81
 4   13 ms  12 ms  13 ms 96.110.94.18
 5    9 ms   8 ms   5 ms 162.151.127.97
 6   14 ms  15 ms  10 ms 96.108.20.93
 7     *    20 ms    *   96.110.42.181
 8   16 ms  25 ms  17 ms 96.110.33.214
 9     *      *      *   Request timed out
10   17 ms  18 ms  21 ms 141.101.73.218
11   21 ms  18 ms  20 ms 104.18.2.221

Earlier path inspection identified Comcast backbone routers including nodes at the Chicago internet exchange facility:

be-2211-pe11.350ecermak.il.ibone.comcast.net

Location:

350 E Cermak Rd
Chicago, IL

SECONDARY RELATED OBSERVATION

Intermittent upload failures also occur when uploading audio files to Audiomack.

Audiomack is hosted behind AWS CloudFront, not Cloudflare.

Example DNS resolution:

audiomack.com β†’ 18.154.110.13 / 18.154.110.44 / 18.154.110.84 / 18.154.110.103

Example HTTP headers:

Server: CloudFront
X-Amz-Cf-Pop: ORD58-P6

This indicates the request is entering AWS CloudFront infrastructure at a Chicago ORD edge.

The presence of upload failures affecting services behind both Cloudflare and CloudFront suggests the issue may not be CDN-specific.

CONTROL PATH TESTING

Uploads succeed reliably when routed through alternate backbone paths.

Example: Google Cloud WireGuard tunnel

Traceroute via Google backbone:

 1  pi.hole [10.66.66.1]
 2  74.125.251.128
 3  141.101.73.4
 4  141.101.73.74
 5  104.18.2.221

Path summary:

Client -> AS15169 (Google backbone)
   -> CDN edge
   -> destination

Uploads complete successfully via this route.

Packet Capture Evidence (Sanitized)

The following TCP segment window shows the exact moment the TLS upload fails.

Server sends an out-of-order packet, after which the client enters heavy retransmission and out-of-order recovery.

This coincides with the upload failure that the browser reports as:

net::ERR_SSL_BAD_RECORD_MAC_ALERT

Excerpt:

1378 104.18.3.221 β†’ CLIENT_IP  TCP Out-Of-Order
1379 CLIENT_IP β†’ 104.18.3.221  TCP Retransmission
1380 CLIENT_IP β†’ 104.18.3.221  TCP Retransmission
1381 CLIENT_IP β†’ 104.18.3.221  TCP Retransmission
...
1436 CLIENT_IP β†’ 104.18.3.221  TCP Retransmission
1441 CLIENT_IP β†’ 104.18.3.221  TCP Retransmission

INTERPRETATION

TLS bad_record_mac alerts occur when encrypted TLS records fail integrity validation. This normally occurs only when encrypted payload data has been modified or corrupted in transit.

Given that:

  • the problem occurs only on Comcast backbone path
  • uploads succeed via alternate backbone paths
  • MTU and MSS testing ruled out fragmentation issues
  • local hardware and NIC offloading ruled out
  • DOCSIS plant issues corrected and signal levels verified
  • failures occur across multiple CDN providers entering Chicago edges

the most plausible cause is packet corruption or unusual handling of sustained upload flows somewhere along the Comcast backbone path entering Chicago CDN infrastructure.

Possible causes include:

  • faulty backbone router line card
  • failing optical interface
  • forwarding ASIC issue
  • peering interface hardware errors
  • rare packet corruption in hardware forwarding path

REQUEST

Please investigate possible packet corruption or forwarding anomalies affecting traffic between Comcast backbone ASN AS7922 and CDN edge networks located in Chicago.

Because the issue affects multiple CDN providers (Cloudflare and AWS CloudFront), the problem may exist along a Comcast backbone segment or peering interface near the Chicago exchange facilities.

@Efpophis

Efpophis commented Mar 9, 2026

Copy link
Copy Markdown
Author

PCAP file not attachable. See the Packet Capture Evidence (sanitized) section.

@sourcec0de

sourcec0de commented Mar 12, 2026

Copy link
Copy Markdown

@Efpophis I'm reposting here in case the Cloudflare forum doesn't notify you.

We're seeing the same thing from Comcast to Cloudflare from Delaware.

Have you found a solution that doesn't involve using a VPN?

We're considering hardware replacement.
Working with Comcast, they replaced our hardware, which didn't resolve the issue.
The technician is escalating this; I'll report here as things develop.

Thanks for the diagnostics. πŸ™

@Efpophis

Copy link
Copy Markdown
Author

@sourcec0de I saw that and answered there as well.

Yeah, it's definitely not your gateway hardware, tho it doesn't hurt to troubleshoot your connection as well. Doing that, I found a damaged cable and now have a much healthier DOCSIS connection. But this issue still persists. It really needs to be escalated to Comcast's network engineering / edge team. The issue is convincing someone inside of that.

Hopefully you've got a good start - sounds encouraging.

@sourcec0de

sourcec0de commented Mar 17, 2026

Copy link
Copy Markdown

We've been escalated to their Tier II support team.
Hopefully, that will yield a better review of all the evidence.

Fingers crossed we eventually make it to their edge networking team.
The comcast support system is completely broken.

@sourcec0de

sourcec0de commented Mar 24, 2026

Copy link
Copy Markdown

@Efpophis just spoke to someone who's escalating this to the networking engineering team.
🀞 hopefylly we'll get some more eyes on this.

@kaushikt-skope

Copy link
Copy Markdown

@sourcec0de Have you had further updates on this please? We are observing a similar problem with TLS tunnel connectivity from primarily Comcast AS7922 networks.

@Efpophis

Copy link
Copy Markdown
Author

Nothing new, but the problem is still present. Everyone involved seems to want to blame the other guys rather than hunting it down πŸ‘Ž

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment