Skip to content

Instantly share code, notes, and snippets.

@ParkWardRR

ParkWardRR/engenius-ecs2512fp-failure-report.md

Last active May 4, 2026 15:35
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ParkWardRR/2b2e2a0231511e41d22175dfa9aecd62 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ParkWardRR/2b2e2a0231511e41d22175dfa9aecd62 to your computer and use it in GitHub Desktop.
Download ZIP
EnGenius ECS2512FP hardware failure forensic report: Realtek RTL9302C ASIC lockup diagnosis with serial console capture, NAND bad block analysis, and load-dependent crash reproduction. Documents progressive management plane failure on an 8-port 2.5GbE PoE+ managed switch with 10G SFP+ uplink — crash intervals degraded from 40 minutes to 13 secon…
Raw
engenius-ecs2512fp-failure-report.md

EnGenius ECS2512FP — Hardware Failure Forensic Report

Date: May 2026
Status: 🔴 HARDWARE FAILURE CONFIRMED — Unit Condemned
Replacement: TP-Link SG3210XHP-M2 (Omada) — deployed and stable

Verdict

The EnGenius ECS2512FP is condemned. The Realtek RTL9302C switching ASIC has a progressive fault that causes the internal bus between the forwarding engine and the MIPS management core to lock up under electrical load. The failure is:

  • Deterministic — 13-second crash timing under full load, 100% reproducible across 50+ boot cycles
  • Load-dependent — stable when unloaded, crashes the moment 2.5GbE PHYs begin passing traffic
  • Progressive — intervals shortened from 40 minutes to 13 seconds over 48 hours
  • Firmware-independent — reproduced across both runtime partitions and rescue kernel

The bad NAND block in the config partition contributes to data corruption but is not the primary failure — the ASIC lockup occurs even with a clean factory-default configuration.

Key Findings

1. Load-Dependent ASIC Lockup

An automated watchdog tested different load conditions across 50+ boot cycles:

Condition Load Result
All 6 APs connected (PoE + 2.5G data) ~51W PoE, 6x 2.5GbE links Crash in exactly 13 seconds, 100% reproducible
All APs disconnected (no PoE, no data) 0W, 0 links Immediately stable — pings indefinitely at ~2ms
1–2 APs reconnected ~18W PoE, 2x 2.5GbE Crash within ~60 seconds
1 AP on 1G (forced speed) ~10W PoE, 1x 1GbE Stable >10 min, eventually crashed

The 13-second crash corresponds to 2.5GbE PHY auto-negotiation completing on all ports. The RTL9302C locks at the moment the Broadcom BCM50998 PHYs begin passing traffic at line rate.

2. Console Proves ASIC Bus Failure

Post-crash serial console session:

Switch> enable
Switch# show interfaces status
Gi0/1  connected  2.5G  full
Gi0/2  connected  2.5G  full
...
Gi0/9  connected  10G   full
Switch# ping <gateway>
PING: 100% packet loss

Management CPU is alive (console works, commands execute, ports report "connected") but zero traffic passes. The MIPS core can read its own interface state registers, but the data path between the switching fabric and the CPU's network stack is severed. No software bug produces this state.

3. Zero UART Output During 57 Crashes

57 consecutive automated crash captures showed the identical pattern: boot completes normally, login prompt appears, then 13 seconds of absolute silence. No kernel panic, no oops, no watchdog bark, no partial output. The lockup occurs below the Linux kernel's panic handler — ASIC silicon fault.

4. Progressive Degradation

# Uptime Before Crash Recovery
1–3 33–40 min Power cycle
4–5 ~3 min Power cycle
6 0 min (won't boot) Factory reset
7 18 hours (post-reset) Power cycle
8 ~5 min Console capture
9–58+ 13 seconds Automated reboot

Factory reset extended the interval from 40 min → 18 hours by wiping corrupted NAND config. The ASIC fault re-asserted regardless.

Evidence Matrix

# Evidence Rules Out Proves
1 Console produces zero UART output during crash Kernel panic, watchdog, OOM ASIC-level lockup below kernel
2 Console CLI responds post-crash, network dead Total CPU failure Internal bus between ASIC and CPU severed
3 PoE stays active during crash PSU / board failure BCM591xx MCU independent, main SoC is fault
4 Crash interval degrades (40m→5m→13s) Deterministic firmware bug Progressive hardware degradation
5 Identical crash on both firmware partitions Corrupt firmware image Not a firmware bug
6 Identical crash on rescue kernel Partition-specific corruption Fault is in hardware
7 Bad NAND block at PEB 418 Software-only config corruption Physical flash damage
8 Binary garbage in config text fields User misconfiguration Storage-level corruption
9 4 port-flap cycles per boot (normal: 0–1) Normal ASIC init Forwarding plane instability
10 Load-dependent: stable unloaded, 13s under load Environmental causes ASIC/power delivery fault
11 Factory reset extends but doesn't fix Config corruption as root cause ASIC fault masked by clean config

Remediation Attempts

# Action Result
1 Factory Reset (10s button hold) Crash interval: 40 min → 18 hours. Did not fix
2 Disable SNTP client via CLI No effect — firmware reloads pool.ntp.org default at every boot
3 Disable Cloud controller (standalone mode) No effect
4 External syslog (UDP 514) Captured normal ops, nothing during crash — CPU locks first
5 Block NTP at upstream firewall Prevented NTP traffic, did not prevent crashes
6 Firmware upgrade to partition 2 Crashed after 5 min — identical failure
7 Boot from rescue partition (SPI NOR) Config corruption visible, crash not reproducible (no managed switching in rescue)

Device Identification

Field Value
Model EnGenius ECS2512FP
SoC Realtek RTL9302C (MIPS 4KEc, 800 MHz)
RAM 512 MB DDR3 (600 MHz bus)
2.5GbE PHYs 2x Broadcom BCM50998 (quad-port, 8 copper ports total)
10G SFP+ 1x cage, port Gi0/9
PoE Controller Broadcom BCM591xx (802.3bt Type 3), I2C bus, MCU: Nuvoton M05xx (firmware v0x4008)
PoE Budget 240W total, ~51W consumed at time of failure
Hardware Revision 1.0.0

Firmware Stack

All version strings captured from serial console boot output and CLI show system information.

Layer Version Build Date Notes
U-Boot 2011.12 (3.6.1.1) Jul 07 2020 Realtek SDK fork. Board ID: RTL9300. Cannot be field-upgraded
Loader 03.02.01 — Stored in SPI NOR at 0x000000–0x0E0000 (896 KB)
Firmware (runtime) 3.03.192 2026-01-05 17:18:17 UTC Image: IMG-3.03.192. MIPS Linux kernel, LZMA compressed, 20.6 MB
Cloud/Management 1.2.125 — File: ECS15XX-25XX_firmware_v1.2.125-192.imag
Protocol 3.03.572 — Internal protocol version
PHY Firmware B0-v02-02-12 (0xC102) — BCM50998 microcode, loaded per quad-PHY at boot
PoE MCU v0x4008 — Nuvoton M05xx, operates independently of main SoC

Note

Firmware v1.2.125 / 3.03.192 (built 2026-01-05) is newer than the latest version listed on the EnGenius public download page. This build was obtained directly from EnGenius cloud-managed auto-update. The public download portal has not been updated to reflect it.

Kernel Boot Parameters

Board: RTL9300 CPU:800MHz LX:175MHz DDR:600MHz
DRAM:  512 MB
Image Type:   MIPS Linux Kernel Image (lzma compressed)
Load Address: 80000000
Entry Point:  802c4f80
Calibrating delay loop... 531.66 BogoMIPS (lpj=2658304)
Serial: 8250/16550 driver, ttyS0 at MMIO 0x0 (irq=47, base_baud=10764700)

Firmware Upgrade Path

  • Dual-partition boot: runtime1 (partition 0) and runtime2 (partition 1) in SPI NAND
  • Active partition selectable via Web UI or U-Boot autoboot index
  • Upload .imag to inactive partition via Web UI (System → Firmware Upgrade)
  • Config stored separately in ubifs — survives partition switch
  • Rescue kernel on SPI NOR (0x200000, 14 MB) — does not run managed switching
  • show activepartition to check current boot partition
  • Both partitions tested with v1.2.125 / 3.03.192 — identical crash on both

SNTP Firmware Bug

The firmware loads pool.ntp.org as primary NTP server at every boot before applying saved config. set sntp client disable only takes effect after the default is already loaded. 0.0.0.0 and 127.0.0.1 are rejected as invalid server addresses. The only mitigation is blocking UDP/123 from the switch at the upstream firewall.

Topology at Time of Failure

[Gateway/Router] ── 1G copper
    │
    ▼
[10G SFP+ Core Switch] ── STP Root Bridge
    │ sfp+ (10G)
    ▼
[EnGenius ECS2512FP] ── THIS SWITCH
    ├── Gi0/1  2.5G  → Proxmox hypervisor (NVR)
    ├── Gi0/2  2.5G  → EWS377AP (PoE, 9.3W)
    ├── Gi0/3  2.5G  → EWS377AP (PoE, 8.5W)
    ├── Gi0/4  2.5G  → EWS377AP (PoE, 8.7W)
    ├── Gi0/5  2.5G  → EWS377AP (no PoE)
    ├── Gi0/6  1G    → EWS377AP (PoE, 10.1W)
    ├── Gi0/7  —     → Empty
    ├── Gi0/8  2.5G  → EWS377AP (PoE, 9.5W)
    └── Gi0/9  10G   → SFP+ uplink to core
  • Total PoE: ~51W of 240W budget (21%)
  • STP: No loops detected. Core switch is root bridge
  • Upstream FCS errors: 10 total over weeks — negligible

Detailed Evidence

Crash Behavior (Identical Every Time)

  1. PoE continues delivering power — BCM591xx MCU operates independently of RTL9302C
  2. SFP+ link stays physically UP — upstream switch shows link running, no carrier loss
  3. All L2/L3 forwarding stops — upstream RX byte counter freezes at zero
  4. Management CPU unresponsive over network — SSH, HTTP, HTTPS, SNMP, ICMP all fail simultaneously
  5. Serial console: zero UART output — confirmed via FTDI USB-to-RJ45 at 115200 8N1
  6. ~30 wireless clients lose internet — complete L2 black hole

Pre-Reset SSH Readings

CPU:   7.25% usr, 2.63% sys, 0.0% nic, 90.0% idle, 0.1% io, 0.0% irq, 0.0% sirq
Temp:  38°C (local sensor), 34°C (remote/junction sensor)
Mem:   Total 512MB, Used ~180MB, Free ~330MB
  • CPU idle at 90% — not overloaded
  • Temperature 38°C — within spec (max 50°C)
  • STP normal, no topology changes
  • PoE delivering rated wattage, no faults

Memory/Config Corruption

Before factory reset, SNTP config contained binary garbage in text-only hostname fields:

SNTP/NTP Primary Server update to [0x00 0xFF 0xAB ... non-ASCII binary data]
SNTP/NTP Secondary Server update to [0x00 0xFF 0xAB ... non-ASCII binary data]

Direct evidence of storage-level corruption in ubifs config partition — consistent with bad NAND block at PEB 418.

Boot Console Anomalies

# Console Message Significance
1 UBI: bad PEB detected at PEB 418 NAND bad block in config partition, every boot
2 sched: RT throttling activated CPU pegged at 100% during BCM50998 PHY init
3 open: No such file or directory TAP/TUN module init failure
4 SNTP/NTP Primary Server update to pool.ntp.org Hardcoded default loaded before saved config
5 4x port-flap cycles in 50 seconds All copper ports DOWN→UP simultaneously — ASIC forwarding plane resets

Flash Layout

Chip 1 — SPI NOR Flash (16 MB, Macronix MX25L12835F, JEDEC ID C22018)

0x000000-0x0E0000  LOADER      (U-Boot, 896 KB)
0x0E0000-0x0F0000  BDINFO      (MAC, serial, hardware rev)
0x0F0000-0x100000  SYSINFO
0x100000-0x180000  RESCUE_JFFS2_CFG
0x180000-0x200000  RESCUE_JFFS2_LOG
0x200000-0x1000000 RESCUE_KERNEL     (14 MB)

Chip 2 — SPI NAND Flash (128 MB, Macronix MX35LF1GE4AB, JEDEC ID C212)

0x0000000-0x4000000  ubifs      (64 MB) — config, logs, user data [UBI-managed]
0x4000000-0x5E00000  runtime1   (30 MB) — Firmware partition 1
0x5E00000-0x7C00000  runtime2   (30 MB) — Firmware partition 2

NAND Defect (UBI Attach Output)

UBI: attaching mtd1
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: number of good PEBs:        511
UBI: number of bad PEBs:         1
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of PEBs reserved:    20
UBI: max erase counter:          773
UBI: bad PEB detected at PEB 418

Bad block at PEB 418 (0x3440000) in the ubifs config partition. Max erase counter 773 — moderate wear, not end-of-life. Manufacturing defect or early block failure.

CLI Reference

# SSH (requires legacy ciphers — OpenSSH 8.8+ disabled these by default)
ssh -o HostKeyAlgorithms=+ssh-rsa \
    -o Ciphers=+aes128-cbc \
    -o KexAlgorithms=+diffie-hellman-group14-sha1 \
    admin@<switch-ip>

# Serial: 115200 8N1, no flow control
# Cable: RJ45 console (Cisco pinout) → USB FTDI adapter

# Diagnostics
show system information          # firmware, uptime, serial
show activepartition             # current boot partition
show interfaces status           # link state, speed, duplex
show ip interface                # VLAN IP, MTU
show spanning-tree               # STP state, root bridge
show power inline                # PoE per port
show process cpu                 # CPU utilization
show environment temperature     # thermal sensors
show logging buffered            # syslog ring buffer

# SNTP (cannot fully disable — defaults reload at boot)
configure → sntp → set sntp client disable → exit → exit → save

# Factory reset: hold reset button 10 seconds

Crash Timeline

Day 1 00:28  — Link flap #1 on SFP+ uplink (self-recovered)
Day 1 01:02  — Link flap #2 (recovered)
Day 1 06:30  — Link flap #3 (recovered)
Day 1 ~08:10 — First management lockup. Ping/SSH dead, PoE alive. Power cycle.
Day 1 09:33  — Recovery. SSH diagnostics: CPU 7%, temp 38°C, STP normal.
Day 1 ~10:20 — Second lockup after 33 min. Power cycle.
Day 1 ~14:42 — Third lockup after ~40 min. Power cycle.
Day 1 15:04  — Fourth crash during boot, 3 min uptime. Power cycle.
Day 1 15:10  — Fifth crash. SNTP disabled. Crashed after 3 min. Power cycle.
Day 1 15:15  — Won't boot at all. FACTORY RESET.
Day 1 ~22:50 — Reset complete. Config hardened: static IP, SNTP off, cloud off, syslog on.
Day 2 12:02  — Health check: 15h uptime, 8,882 pings, 0 drops.
Day 2 15:05  — Silent lockup after 18 hours. Power cycle.
Day 2 15:15  — Serial console attached via USB FTDI through network KVM.
Day 2 15:49  — First full boot console captured. Bad NAND, RT throttle, 4x port flap.
Day 2 15:55  — Firmware v1.2.125 uploaded to partition 2. Reboot.
Day 2 16:05  — CRASH after 5 min. Console: ZERO BYTES. CPU alive, network dead.
               HARDWARE FAILURE CONFIRMED.
Day 2 16:27  — Terminal Death Spiral: 13-second crash loop begins.
Day 2 16:28  — Automated watchdog deployed. Serial reboots every ~4 min.
Day 2 17:05  — APs unplugged → IMMEDIATELY STABLE. Load-dependent failure confirmed.
Day 2 17:11  — 2 APs reconnected → crash within 60 seconds.
Day 2 17:24  — Watchdog v1.2: added path validation before rebooting.
Day 2 17:24–18:14 — 57 consecutive crashes captured. All zero UART output.
Day 2 18:14  — Unit powered off. Condemned. Replacement ordered.
Day 3        — TP-Link SG3210XHP-M2 deployed. Network stable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment