Egocentric hand-tracking annotation platform with difficulty-aware frame classification.
Built an egocentric hand-tracking annotation platform. Core challenge: automatically classify first-person video frames into 5 categories — no_hands, low_lighting, occluded, dexterous_pose, easy.
"Started simple: uniform 2fps sampling + single MediaPipe detector in IMAGE mode. Got 70% no_hands, 20% dexterous, 10% easy — but zero occlusion or low-lighting detection. Clear failure: occlusion, lighting, and complex poses were invisible."
Insight: MediaPipe's BlazePalm fails entirely under occlusion (no partial landmarks), so a single frame can't distinguish no_hands from occluded.
"Switched to VIDEO mode for temporal tracking, added a dual-threshold ensemble (primary at 0.5 confidence, secondary at 0.15) and a hybrid sampler — uniform base rate + event-driven bursts on motion/skin-presence spikes. Occlusion finally appeared at 3.6%, but still way too low."
"Added CLAHE enhancement for dark frames, blur scoring via Laplacian variance, a dark-first classification priority. Low-light jumped to 30.8%. But occlusion still only 10% — the fundamental problem remained."
"Realized the only way to separate occlusion from no-hands is temporal context. Added a two-pass pipeline: Pass 1 detects everything and caches results. Pass 2 computes nearest_conf_dt — seconds to the nearest confident neighbor detection — and uses bidirectional bridging (0.7s window). If a confident detection exists nearby, a frame without hands is occluded, not no_hands. This tripled occlusion detection to 25.6%."
sequenceDiagram
autonumber
participant Video as MP4 Video<br/>S3 / Local
participant Sampler as HybridSampler<br/>Pass 1: Coarse Scan
participant Events as Event Detector<br/>_detect_events()
participant Plan as Pass 2: Sample Plan<br/>uniform + event bursts
participant Ensemble as DisagreementEnsemble<br/>Primary 0.5 / Secondary 0.15
participant Feats as Feature Extractor<br/>image_stats + hand_features
participant Cache as Frame Cache<br/>_Record[] (JPEG + signals)
participant Tracker as Bidirectional Tracker<br/>_track_support()
participant Classifier as FrameClassifier<br/>Stage 1: Presence<br/>Stage 2: Difficulty
participant Output as Output Writer<br/>JPEG dirs + report.json
rect rgb(30, 50, 70)
Note over Video,Plan: PHASE 1: HYBRID SAMPLING (two-pass)
Video->>+Sampler: scan(video_path)
Sampler->>Sampler: Read at ~1 fps, resize to 160×90
Sampler->>Sampler: Compute brightness, motion (frame diff),<br/>skin mask (HSV+YCrCb)
Sampler-->>-Events: ScanRow[] (presence, brightness, motion)
Events->>Events: Detect presence cross, brightness drop,<br/>motion spike → timestamped events
Events-->>Plan: events: (hand_event | confidence_drop, ts)
Plan->>Plan: Uniform 1 fps across whole clip
Plan->>Plan: Event bursts at 5 fps (±3s window)<br/>Confidence drops at 10 fps
Plan->>Plan: Dedup by frame index,<br/>priority: confidence_drop > hand_event > transition > uniform
Plan->>Plan: Cap to max_frames, thin uniform if over budget
Plan-->>Feats: Sample[] (frame_index, timestamp, reason, rate)
end
rect rgb(40, 60, 80)
Note over Ensemble,Cache: PHASE 2: DETECTION PASS (per planned frame)
Feats->>Feats: Compute ImageStats<br/>(mean_lum, p10_lum, contrast, blur, hist_spread)
alt mean_luminance < 0.32
Feats->>Feats: CLAHE on LAB L-channel<br/>brighten shadows without blowing highlights
end
Feats->>+Ensemble: detect(frame, timestamp_ms)
Ensemble->>Ensemble: Primary HandDetector (conf=0.5)<br/>Secondary HandDetector (conf=0.15)
Ensemble->>Ensemble: Both run MediaPipe HandLandmarker<br/>(BlazePalm + 21-keypoint model)
Ensemble-->>-Feats: r_primary, r_secondary, disagreement, n1, n2
Feats->>Feats: Merge hands: keep primary,<br/>add secondary if IoU < 0.3 with primary
Feats->>Feats: Extract HandFeatures per hand<br/>(bbox, handedness, articulation,<br/>finger_spread, out_of_frame, border_clipping)
Feats->>Feats: Compute local_strength<br/>(0.6×confidence + 0.25×completeness + 0.15×size)<br/>×0.5 if secondary-only detection
Feats->>Feats: Encode JPEG
Feats-->>Cache: _Record (sample, feats, local_strength,<br/>disagreement, n_primary, n_secondary,<br/>best_box, jpeg, used_clahe)
end
rect rgb(50, 70, 50)
Note over Tracker,Classifier: PHASE 3: CLASSIFICATION PASS (bidirectional temporal)
Tracker->>Tracker: For each _Record at index i:
Tracker->>Tracker: Scan neighbours backward + forward<br/>within ±temporal_window_s (2.0s)
Tracker->>Tracker: Compute track_support = max neighbour local_strength
Tracker->>Tracker: Compute nearest_conf_dt = seconds to<br/>nearest neighbour with strength ≥ track_strong (0.55)
Tracker-->>Classifier: (track_support, nearest_conf_dt)
Classifier->>Classifier: Compute jitter = 1 - mean(IoU with adjacent frame boxes)
Classifier->>Classifier: Compute scores:<br/>s_low = f(mean_lum, p10_lum, contrast)<br/>s_occ = f(partial_loss, border_clip, blur, low_conf, disagreement)<br/>s_dex = f(articulation, finger_spread, self_overlap, jitter)
Note over Classifier: Stage 1: PRESENCE DECISION
Classifier->>Classifier: local_present = local_strength ≥ 0.50
Classifier->>Classifier: short_gap = nearest_conf_dt ≤ 0.7s
Classifier->>Classifier: hinted_gap = has_hint AND nearest_conf_dt ≤ 1.4s
alt local_present
Classifier->>Classifier: present=true, lost=false
else short_gap or hinted_gap
Classifier->>Classifier: present=true, lost=true<br/>╴ temporal bridge: dropout inside live track
else
Classifier->>Classifier: present=false, lost=false
end
Note over Classifier: Stage 2: DIFFICULTY ATTRIBUTION
alt not present
Classifier->>Classifier: → no_hands
else s_low ≥ 0.35
Classifier->>Classifier: → low_lighting
else lost (temporal dropout, not dark)
Classifier->>Classifier: → occluded
else s_dex ≥ 0.30 AND s_dex ≥ s_occ
Classifier->>Classifier: → dexterous_pose
else s_occ ≥ 0.30
Classifier->>Classifier: → occluded
else strict_gates_pass (handedness≥0.7, blur≤0.25,<br/>complete landmarks, scores below thresholds)
Classifier->>Classifier: → easy
else fallback to nearest hard class
Classifier->>Classifier: → occluded or dexterous_pose
end
Classifier-->>Output: FrameResult (label, scores, hand_evidence,<br/>features dict with all signals)
end
rect rgb(70, 50, 50)
Note over Output: PHASE 4: OUTPUT
Output->>Output: Write JPEG to output/<stem>/frames/<label>/*.jpg
Output->>Output: Accumulate label_counts, reason_counts
Output->>Output: Return VideoStats
Note over Output: Outputs report.json with<br/>per-frame features + scores<br/>for ingestion into backend DB
end
| Layer | Language | Role |
|---|---|---|
| Pipeline | Go | S3 download → subprocess classifier → upload |
| Classifier | Python (uv) | Hybrid sampling → MediaPipe ensemble → temporal classification |
| Backend | Go | HTTP API, PostgreSQL, JWT auth, annotation CRUD |
| Frontend | React/TS | Annotation editor (canvas), admin dashboard |
| Infra | Terraform | AWS EC2, Docker, security groups |