teksavvy_apple.md

iMessage broke on one ISP for 2+ hours. A Claude-assisted diagnosis.

Status as of 2026-05-12 13:02 EDT: still broken. Onset roughly 10:55–11:05 EDT; ~2h elapsed and counting.

TL;DR: Teksavvy (AS5645) has a stuck-up-but-dead BGP peering session with Apple (AS714) at the Toronto Internet Exchange. Apple's 17.0.0.0/8 is still in Teksavvy's RIB pointing at an unreachable next-hop on the IX peering LAN; Teksavvy's border router happily forwards 17.x-bound packets onto the IX fabric where they're dropped on the floor. No failover to Zayo transit, because from the BGP FSM's perspective the session is still Established. IPv6 to AS714 is unaffected — separate address family, separate session.

What follows is the protocol-level walkthrough that localized the problem from "iMessage is broken" to "a specific TCP/179 session at TorIX is wedged."

Timeline

Time (EDT)	Event
11:00	Last "Read" receipt on a working iMessage thread
~11:05	First "Not Delivered" red-exclamation message
11:35:53	First apsd/imagent log capture begun
11:36:32	Explicit APSErrorDomain Code=4 send-failure logged in apsd
12:14	Baseline traceroute panel on Teksavvy
12:17	First TCP-probe panel (Teksavvy): every 17.x Apple IP times out
12:21	TCP-probe panel on Bell 5G (A/B): every 17.x succeeds
12:27	Egress switched back to Teksavvy, re-confirmed
12:31	Probed Bell-DNS-resolved Apple IPs from Teksavvy: still timeout (rules out DNS)
12:46–12:48	sudo traceroute -P TCP + mtr -T ran past Teksavvy border; Apple-path dies at hop 7
13:02:48	Re-probed: still timeout on every Apple 17.x IP, controls fine

Confirming we're talking about APNS

iMessage rides APNS — a persistent TLS session from each Apple device to one of N-courier.push.apple.com (TCP/5223, fallback TCP/443, alt TCP/2197). All of those resolve into 17.0.0.0/8. apsd on macOS owns that socket. Log capture at 11:36:32 EDT was unambiguous:

2026-05-12 11:36:32.457  apsd[567]            APSErrorDomain Code=4
2026-05-12 11:36:32.458  identityservicesd    "Message retried too many times (2), not retrying anymore.
                                               Has received courier ack? NO"
2026-05-12 11:36:32.458  identityservicesd    [com.apple.IDS:SenderKeyDistribution]
                                              Failed to send sender key message
2026-05-12 11:36:32.458  identityservicesd    OUTGOING-PUSH_FULLY_SENT: ... SERVICE: com.apple.madrid

madrid is iMessage's internal service name on top of APNS. APS Code 4 is "courier didn't ack the outgoing send." Not auth, not crypto, not the APNS app layer — the underlying transport couldn't get a SYN/ACK out the door.

Relevant DNS records — note the absent AAAAs for the courier hostnames, which becomes important:

1-courier.push.apple.com   A    17.57.144.{135,136,137,138,...}   AAAA  <none>
courier.push.apple.com     A    17.57.144.{134,135,136,137,...}   AAAA  <none>
gs.apple.com               A    17.171.47.17                      AAAA  <none>
gsa.apple.com              A    17.157.64.66                      AAAA  <none>
init.push.apple.com        A    17.188.{171,172,182}.x            AAAA  2620:149:149:102e::a

APNS couriers are v4-only. There is no Happy Eyeballs fallback for the hostnames iMessage actually depends on. If your network can't reach 17.x over v4, you don't have iMessage.

12:17 EDT — TCP probing isolates the address family and the destination network

nc -zw4 17.57.144.244 5223        # APNS primary           timeout
nc -zw4 17.57.144.244 443         # APNS fallback          timeout
nc -zw4 17.57.144.244 2197        # APNS alt               timeout
nc -zw4 17.171.47.17 443          # gs.apple.com           timeout
nc -zw4 17.157.64.66 443          # gsa.apple.com          timeout
nc -zw4 17.188.182.132 443        # init.push.apple.com    timeout
nc -6 -zw4 init.push.apple.com 443  # IPv6 to AS714        SUCCEEDS
nc -zw4 www.apple.com 443         # 184.26.x — Akamai      succeeds
nc -zw4 1.1.1.1 443                                       # succeeds
nc -zw4 8.8.8.8 443                                       # succeeds
nc -zw4 github.com 443                                    # succeeds

Every Apple-hosted 17.x IP times out; every CDN-fronted Apple hostname works (because the CDN serves from an edge node our ISP can reach trivially); every non-Apple destination works; the v6 path into AS714 works. So the problem is specifically the IPv4 path to AS714.

Note the precision of "timed out" vs alternatives:

• Operation timed out — SYN went into the dark. Nothing replied with anything. Either a routing blackhole, a stateful firewall silently dropping, or the next-hop's interface is dead.
• Connection refused — got a TCP RST. The destination host is alive and reachable at L3; it's just not listening on that port.
• No route to host — your kernel has nothing in its FIB for the destination. Different problem entirely.

Confusing these will make you chase ghosts.

12:21 EDT — A/B against Bell 5G collapsed the search space

User tethered to Bell 5G (AS577, totally different egress). Same nc panel, four minutes after the Teksavvy run. Every 17.x Apple IP went from "timeout" to "Connection to ... succeeded!" Destinations are fine. Path from Teksavvy is not.

Bonus on that run: nc 17.57.144.244 2197 on Bell came back Connection refused. Refused is better than timeout — it means the SYN reached an Apple router that bothered to RST. Reachability proved by negation.

User then flipped back to Teksavvy at 12:27 EDT to continue. At 12:31 we probed the Bell-resolved Apple IPs from Teksavvy directly — still timed out. Whatever DNS handed out, the route was dead. DNS ruled out.

Apple 17.0.0.0/8 is healthy globally (~12:35 EDT)

check-host.net fires TCP probes from ~15 vantage points. Hit 17.57.144.244:5223 from 13 nodes (BR, DE, FR, IT, SE, JP, SG, IN, RS, LT, PL, RO, MD), four US nodes, and a CA node. 18/18 succeeded. The Canadian node connected in 51ms.

RIPEstat's routing-status for 17.0.0.0/8:

visibility.v4.ris_peers_seeing: 326 / 327
origins: [{origin: 714, route_objects: [LEVEL3, NTTCOM, RADB]}]

Apple is announcing 17/8 to essentially every BGP route collector on the planet. Origin AS714. Route objects registered with three RIRs. No withdrawal, no hijack, no flapping.

IIJ's Internet Health Report anomaly detectors agreed: zero disco events on AS5645, AS714, or AS6461 in the prior 7 days, zero forwarding-anomaly alarms, zero significant network-delay alarms. The aggregate health of either AS, observed from outside, is fine. Which is the signature of a peering-edge problem, not a network outage.

AS Hegemony pointed at the wrong suspect first

IHR's AS Hegemony snapshot for AS5645 at 2026-05-12T15:30 UTC (= 11:30 EDT):

Upstream	Hegemony	Name
AS6461	0.53	Zayo (primary transit)
AS1299	0.10	Arelion
AS3257	0.06	GTT

Hegemony 0.53 on Zayo means roughly half of Teksavvy's outbound paths to the wider internet ride Zayo's network. Plausible initial hypothesis: a Zayo↔Apple problem. The next test killed that theory.

12:47 EDT — TCP traceroute + per-hop AS labels

ICMP/UDP traceroute is useless past an ISP's edge — backbone routers filter the probes. TCP-mode traceroute (traceroute -P TCP -p 443) sends real SYNs with incrementing TTLs. Intermediate routers return ICMP Time Exceeded (Type 11) for each TTL=0 drop. Far less filtered.

Needs raw sockets, so sudo:

sudo traceroute -P TCP -p 443 -q 1 -w 1 -m 18 17.57.144.244
sudo traceroute -P TCP -p 443 -q 1 -w 1 -m 18 1.1.1.1
sudo mtr -T -P 443 -c 20 -r -n 17.57.144.244

Then label each hop via Team Cymru's whois service (TCP/43) — finicky from this environment, so we fell back to ip-api.com's HTTP endpoint:

echo 'begin verbose 198.48.240.193 ... end' | nc -w 5 whois.cymru.com 43
# AS | IP | BGP Prefix | CC | Registry | Allocated | AS Name

Labeled paths captured at 12:47:33 EDT:

HOP   → Cloudflare 1.1.1.1 (WORKS)              → 17.57.144.244 Apple APNS (BROKEN)
───   ───────────────────────────────────────── ─────────────────────────────────────
 1    192.168.0.1     RFC1918                   192.168.0.1     RFC1918
 2    198.48.240.193  AS5645 Teksavvy           198.48.240.193  AS5645 Teksavvy
 3    104.195.128.58  AS5645 Teksavvy           104.195.128.58  AS5645 Teksavvy
 4    104.195.128.57  AS5645 Teksavvy           104.195.128.201 AS5645 Teksavvy
 5    206.248.153.10  AS5645 Teksavvy           206.248.153.14  AS5645 Teksavvy
 6    206.248.153.7   AS5645 Teksavvy (border)  206.248.153.7   AS5645 Teksavvy (border)
 7    206.108.34.7    TorIX peering LAN         *** 100% loss, all subsequent hops
 8    108.162.239.22  AS13335 Cloudflare         ***
 9    1.1.1.1         AS13335 Cloudflare         ***

mtr -T over 12:48:01–12:48:43 EDT corroborated: hop 7 to Apple showed 100.0% loss across all 12 cycles, while the same hop on the Cloudflare path showed normal ECMP behavior (variable next-hop IPs across cycles, all in 206.108.34.0/23).

Both paths terminate hop 6 at the same Teksavvy border router. The Cloudflare-bound path's hop 7 is in 206.108.34.0/23 — registered to "Toronto Internet Exchange Community," i.e. the TorIX peering LAN. That hop has no AS owner of its own because peering-LAN IPs are allocated from the IX's address pool to participant routers' IX-facing interfaces. Cloudflare's TorIX-facing router responds with TTL-exceeded from 206.108.34.7, then the next hop is inside Cloudflare's network.

The Apple-bound path produces nothing past hop 6. Not "no route" — the packets are getting forwarded somewhere; they're just disappearing without any router along the way bothering to send a TTL-exceeded message. That's specifically what a broken next-hop on a shared L2 peering fabric looks like: packet egresses Teksavvy's IX-facing port into the VLAN, destination MAC isn't responding, L2 broadcast domain swallows the frame.

PeeringDB locks it in

peeringdb /netixlan?asn=714   →  TorIX  206.108.35.236  100000 Mbps
                              →  TorIX  206.108.35.237  100000 Mbps
peeringdb /netixlan?asn=5645  →  TorIX  206.108.34.67   100000 Mbps
                              →  TorIX  206.108.35.223  100000 Mbps

Both ASes are at TorIX with 100G connections. The shortest BGP path from Teksavvy customers to AS714 should be a single AS hop via TorIX peering — exactly the topology the Cloudflare path demonstrates. So when the Apple path dies at the IX boundary, the suspect is the BGP session between AS5645 and AS714 at TorIX.

The actual failure mode

Apple's 17.0.0.0/8 is still installed in Teksavvy's RIB/FIB with next-hop set to one of Apple's TorIX peering-LAN IPs (206.108.35.236 or 206.108.35.237). The BGP session that advertised that route is still in Established state from Teksavvy's perspective — keepalives every 30s over TCP/179 are still being acked by something — but Apple's router is no longer reachable on the L2 fabric. Teksavvy's border resolves the FIB entry, forwards the IP packet onto the IX VLAN, the destination MAC isn't there, frames drop.

The classic "BGP session stuck up while the underlying datapath is broken" failure mode. The session and the datapath share an L2 fabric but BGP itself only validates layer 4 — TCP keepalives running over the same VLAN can keep getting acked even when the destination route's data plane via that same VLAN has degraded. BFD (Bidirectional Forwarding Detection) was invented to solve this exact problem; it appears to not be in use on this session.

No failover to the Zayo transit path because, from BGP's perspective, nothing has changed. The route's still there. Apple is preferred over transit because of standard BGP path selection (shorter AS_PATH via peering, higher LOCAL_PREF, customer-route preference, whatever Teksavvy's policy is — all of which still match). The path-selection algorithm doesn't second-guess Established sessions.

IPv6 to AS714 unaffected because BGP runs separate address families. The v6 session at TorIX (or via transit) is independent of the v4 session. init.push.apple.com has an AAAA record and v6 worked. Almost everything else iMessage needs is v4-only at Apple.

What would have fixed it on Teksavvy's side

• Clear the stuck BGP session (clear ip bgp neighbor 206.108.35.236 or equivalent). The router would re-establish, re-learn the route, and if Apple's L2 is still actually broken, the new session wouldn't reach Established and the route would fall through to the Zayo transit fallback.
• Enable BFD on the IX peering sessions so layer-2 reachability is continuously validated independent of TCP keepalives.
• A static blackhole or null-route for 17.0.0.0/8 would have been a blunt but effective workaround; Teksavvy's transit would have picked it up via Zayo's announcement and packets would have flowed.

None are user-fixable. Local options: tether to cellular, Cloudflare WARP, or wait.

Ticket text

Connectivity from my line to all Apple AS714 prefixes (17.0.0.0/8) is broken on IPv4 starting ~11:05 EDT 2026-05-12, still broken as of 13:02 EDT (~2h). traceroute -P TCP -p 443 to 17.57.144.244 matches my working path to 1.1.1.1 through your network exactly through hop 6 (206.248.153.7), then 100% loss. Cloudflare via your TorIX peering: fine. IPv6 to AS714: fine. Apple's 17/8 globally visible (RIPEstat 326/327 RIS peers), reachable from ca1.node.check-host.net in 51ms. Suspect a stuck BGP session with AS714 at TorIX (Apple's TorIX IPs per PeeringDB: 206.108.35.236, 206.108.35.237). Please bounce the session or check L2 reachability on the peering LAN.