Based on TrackOSC by @JGL. Compatible with VisionOSC by @lingdong-. All messages are sent unbundled, one per enabled detector per processed frame, including when nothing is detected (header-only). Coordinates are as described above.
Everything on the wire uses one convention — the same one VisionOSC uses:
(0,0) ──────────► x width
│ ┌───────────────────────────────┐
│ │ │
▼ │ pixels, y down │ height
y │ │
└───────────────────────────(w,h)
- Pixels, not normalised: divide by the frame width/height from the
message header (or
/camerainfo) to normalise. - Origin top-left, y grows downward (screen convention, not math convention).
- Never mirrored. The selfie-mirror option only flips the phone's display; wire coordinates are always the unmirrored scene.
- Dimensions follow orientation: portrait sends 720×1280, landscape
1280×720. Listen to
/camerainfoand your mapping code needs no special-casing:
// Processing: map a TrackOSC point into your sketch window
float sx = x / frameW * width; // frameW/frameH from the message header
float sy = y / frameH * height;/camerainfo (Not mandatory; VisionOSC receivers ignore it) — sent with every processed frame:
| # | Type | Value |
|---|---|---|
| 0 | int32 | frame width in pixels |
| 1 | int32 | frame height in pixels |
| 2 | int32 | orientation: 0 = landscape, 90 = portrait, 180 = landscape flipped, 270 = portrait upside-down |
| 3 | int32 | camera facing: 0 = back, 1 = front |
Every message begins with the same header:
| # | Type | Value |
|---|---|---|
| 0 | int32 | frame width (oriented pixels, e.g. 720) |
| 1 | int32 | frame height (e.g. 1280) |
| 2 | int32 | number of detections n (capped at 32) |
Then, per detection:
/poses/arr — float confidence, then 17 joints × (float x, float y,
float confidence). Joint order (PoseNet order): nose, leftEye, rightEye,
leftEar, rightEar, leftShoulder, rightShoulder, leftElbow, rightElbow,
leftWrist, rightWrist, leftHip, rightHip, leftKnee, rightKnee, leftAnkle,
rightAnkle.
/hands/arr — float confidence, then 21 joints × (x, y, confidence).
Order: wrist; thumb CMC, MP, IP, tip; index MCP, PIP, DIP, tip; middle …;
ring …; pinky … .
/faces/arr — float confidence, then 76 landmark points ×
(x, y, precisionEstimate), in Vision's constellation order.
/texts/arr — float confidence, float left, float top,
float width, float height, string recognised text.
/animals/arr — float confidence, float left, float top,
float width, float height, string label ("Cat" or "Dog").
A joint/point that wasn't detected is sent as x=0, y=frameHeight, confidence=0 (VisionOSC's convention) — filter on confidence == 0.
VisionOSC's /faces/arr carries only the 76 landmark points, no face
boundary. TrackOSC adds two messages (VisionOSC receivers ignore them; the
five messages above are untouched). Both start with the standard
width/height/n header, and both list the same faces in the same order,
so index i in one matches index i in the other. (/faces/arr applies a
stricter landmark check and can, in rare cases, contain fewer faces — don't
assume its indices line up with these.)
/faces/box — Per face, 8 floats (fixed stride: face i starts at
argument 3 + i×8):
| Type | Value |
|---|---|
| float | confidence |
| float × 4 | bounding box: left, top, width, height (pixels, origin top-left) |
| float × 3 | head rotation: roll, yaw, pitch in degrees (0 when unavailable) |
/faces/contour — per face: float confidence, int32 m (contour point count), then m × (float x, float y). The contour is the jawline, an open polyline from ear to chin to ear; don't close it. m varies by OS version (typically 17) and is 0 when Vision reports no contour for that face. (Always loop on m, never hardcode it.)