Mouse Coordinate Fix for Canvas Zoom

Date: October 31, 2025 Status: ✅ SOLVED Issue: Mouse clicks in xterm.js terminals were inaccurate at non-100% canvas zoom levels

The Problem

When terminals were rendered inside a CSS transform: scale() canvas (for zoom), mouse clicks registered at incorrect positions:

At 50% zoom: Clicks registered ~50% higher than cursor position
At 150% zoom: Clicks registered ~50% lower than cursor position
At 100% zoom: Clicks were accurate (no transform applied)

Why This Happened

xterm.js doesn't support CSS transforms on parent elements.

When a parent element has transform: scale():

The visual rendering is scaled (what you see)
The coordinate system is NOT scaled (what JavaScript sees)
This creates a mismatch between visual position and coordinate position

Example at 120% canvas zoom:
┌─────────────────────────────────────┐
│ offsetWidth: 1000px (layout size)   │
│ boundingRect.width: 1200px (visual) │
│ User clicks at visual position 600px│
│ → Browser reports clientX: 600      │
│ → xterm expects position in 1000px  │
│ → xterm calculates: 600/1000 = 60%  │
│ → But visually it's: 600/1200 = 50% │
│ → Click is off by 10%! ❌           │
└─────────────────────────────────────┘

The Failed Attempts

Attempt 1: Divide by canvasZoom (Oct 31, 2025 - Commit 0897aad, 345339c)

// ❌ FAILED: Made accuracy worse
const transformedX = visualX / canvasZoom;

Why it failed: This approach double-compensated because getBoundingClientRect() already returns scaled dimensions. It also didn't account for browser zoom.

Attempt 2: FitAddon Recalculation (Commit 3a89f42)

// ❌ FAILED: Only recalculates cell dimensions, not mouse coordinates
fitAddonRef.current.fit();

Why it failed: FitAddon only adjusts terminal rows/cols based on container size. It doesn't fix mouse coordinate transformation.

Attempt 3: Internal xterm.js API Calls (Initial attempt)

// ❌ FAILED: APIs don't exist or have changed
xterm._core._renderService.onResize();

Why it failed: Internal xterm.js APIs are not stable and caused runtime errors.

The Working Solution ✅

Key Insight from Diagnostics

The critical discovery came from comparing browser zoom (which works) vs canvas zoom (which didn't):

Browser Zoom (Ctrl+MouseWheel):

devicePixelRatio: 1.0 → 1.25 ✓
innerWidth: 3392 → 2714 ✓
offsetWidth: 3392 → 2714 ✓
boundingRect.width: 3392 → 2714 ✓

All dimensions scale proportionally!
The coordinate system itself changes.

Canvas Zoom (CSS transform):

devicePixelRatio: 1.25 (unchanged) ✗
innerWidth: 2714 (unchanged) ✗
offsetWidth: 2714 (unchanged) ✗
boundingRect.width: 2714 → 3256.8 (1.2x visual scaling) ✓

Only visual rendering scales!
The coordinate system stays the same.

The Correct Formula

// Calculate the ratio between visual size and layout size
const visualToLayoutRatioX = rect.width / offsetWidth;
const visualToLayoutRatioY = rect.height / offsetHeight;

// Transform visual coordinates to layout coordinates
const layoutX = visualX / visualToLayoutRatioX;
const layoutY = visualY / visualToLayoutRatioY;

Why this works:

Accounts for both browser zoom AND canvas zoom
offsetWidth is the layout size (accounts for browser zoom)
rect.width is the visual size (accounts for both zooms)
The ratio gives us the exact visual scaling factor

Example:

At 125% browser zoom + 120% canvas zoom:
offsetWidth: 2714px (3392 ÷ 1.25 browser zoom)
rect.width: 3256.8px (2714 × 1.2 canvas zoom)
ratio: 3256.8 ÷ 2714 = 1.2 (exactly the canvas zoom!)

Click at visual 600px:
layoutX = 600 ÷ 1.2 = 500px ✓

Implementation Details

Location

File: frontend/src/components/Terminal.tsx Lines: ~333-428

Code Structure

// 1. Use WeakSet to track processed events (prevents infinite recursion)
const processedEvents = new WeakSet<Event>();

const mouseTransformHandler = (e: MouseEvent) => {
  // 2. Skip already-processed events
  if (processedEvents.has(e)) return;
  processedEvents.add(e);

  // 3. Calculate visual-to-layout ratio
  const rect = terminalRef.current.getBoundingClientRect();
  const offsetWidth = terminalRef.current.offsetWidth;
  const visualToLayoutRatioX = rect.width / offsetWidth;

  // 4. Only transform if there's a mismatch (zoom != 100%)
  if (Math.abs(visualToLayoutRatioX - 1) > 0.01) {
    e.stopImmediatePropagation(); // Block original event

    // 5. Transform coordinates
    const visualX = e.clientX - rect.left;
    const layoutX = visualX / visualToLayoutRatioX;

    // 6. Create new event with corrected coordinates
    const transformedEvent = new MouseEvent(e.type, {
      clientX: rect.left + layoutX,
      clientY: rect.top + layoutY,
      // ... copy all other properties
    });

    processedEvents.add(transformedEvent); // Mark transformed event

    // 7. Dispatch to xterm internal element (not wrapper)
    const xtermViewport = terminalRef.current.querySelector('.xterm-viewport');
    xtermViewport?.dispatchEvent(transformedEvent);
  }
};

// 8. Add in capture phase (intercepts BEFORE xterm sees it)
const mouseEventTypes = ['mousedown', 'mouseup', 'mousemove', 'click', 'dblclick', 'contextmenu', 'wheel'];
mouseEventTypes.forEach(eventType => {
  terminalRef.current.addEventListener(eventType, mouseTransformHandler, { capture: true });
});

Critical Implementation Details

Capture Phase: Events are intercepted in the capture phase (before bubbling) so we see them BEFORE xterm.js
WeakSet Tracking: Using WeakSet instead of modifying event objects is cleaner and prevents memory leaks (events auto garbage-collect)
Dispatch to Child: Transformed events are dispatched to .xterm-viewport (inside xterm), not the wrapper element, to avoid re-triggering our handler
Threshold Check: Only transform when |ratio - 1| > 0.01 to avoid floating-point errors at exactly 100% zoom
All Mouse Events: Transform all mouse events (mousedown, mouseup, mousemove, click, dblclick, contextmenu, wheel) for complete coverage

Testing Results

Test Matrix

Browser Zoom	Canvas Zoom	Click Accuracy	Notes
100%	50%	✅ Perfect	Previously too high
100%	100%	✅ Perfect	No transformation applied
100%	150%	✅ Perfect	Previously too low
100%	200%	✅ Perfect	Previously very low
125%	100%	✅ Perfect	Browser zoom handled correctly
125%	120%	✅ Perfect	Compound zoom handled!
75%	150%	✅ Perfect	Complex compound case

Verified Functionality

✅ Click accuracy at all zoom levels (25% - 300%)
✅ Text selection works correctly
✅ Drag selection accurate
✅ Right-click context menu positioning
✅ Mouse wheel scrolling in terminals
✅ Double-click text selection
✅ No performance degradation
✅ No infinite recursion
✅ Works with locked terminals (no transformation needed)
✅ Works with maximized terminals

Why This Fix is Better Than Alternatives

Alternative 1: Portal All Terminals Outside Canvas

Pros: Clean, no transformation needed Cons: Complex positioning calculations, loses canvas integration, significant refactoring

Alternative 2: Remove Canvas Zoom, Use Browser Zoom Only

Pros: No coordinate issues, leverages native browser features Cons: Can't programmatically control zoom, loses zoom buttons UI, sidebar/header also zoom

Alternative 3: Fork xterm.js

Pros: Could fix at source Cons: Maintenance burden, rebasing on updates, xterm.js team unlikely to accept PR (this is a rare use case)

Our Solution: Event Interception ✅

Pros:

✅ Works with existing architecture
✅ No xterm.js modifications needed
✅ Handles both browser and canvas zoom
✅ Minimal code (~100 lines)
✅ No performance impact
✅ Easy to maintain

Cons:

⚠️ Relies on event interception (could break if event flow changes)
⚠️ Requires understanding of coordinate systems

Lessons Learned

CSS transform: scale() only affects rendering, not coordinates
- Visual appearance ≠ Coordinate system
- Third-party libraries (like xterm.js) don't expect transformed parents
Browser zoom changes the coordinate system itself
- devicePixelRatio changes
- All dimensions scale proportionally
- No coordinate transformation needed
The diagnostic approach was key
- Created ZoomDiagnostic.tsx to compare browser vs canvas zoom
- Logged offsetWidth vs boundingRect.width difference
- This revealed the exact ratio needed for transformation
Event interception requires careful handling
- Use WeakSet to track processed events
- Dispatch to child elements to avoid re-triggering handlers
- Use capture phase to intercept before library sees events
Previous attempts failed because:
- Dividing by canvasZoom alone doesn't account for browser zoom
- It double-compensated when getBoundingClientRect() already returns scaled values
- The visual-to-layout ratio is the correct measurement

Related Files

frontend/src/components/Terminal.tsx - Main implementation
frontend/src/components/ZoomDiagnostic.tsx - Diagnostic tool (can be removed if needed)
frontend/src/components/DraggableTerminal.tsx - Passes canvasZoom prop to Terminal
frontend/src/App.tsx - Canvas zoom state management
docs/archive/DEBUG_MOUSE_ZOOM_ISSUE.md - Earlier debugging notes

Git History

0897aad - First attempt: mouse coordinate transformation (failed)
24b3a00 - Dimension refresh strategy (partial)
345339c - Improved coordinate transformation (failed worse)
faca3b0 - Revert to clean baseline
3a89f42 - FitAddon recalculation approach (didn't fix mouse)
[This commit] - ✅ Working solution: visual-to-layout ratio transformation

If This Breaks in the Future

Symptoms

Mouse clicks miss their target at non-100% zoom
Text selection starts at wrong position
Right-click context menu appears at wrong location

Debug Steps

Check if transformation is running:

// Add this to mouseTransformHandler
console.log('Transform ratio:', visualToLayoutRatioX);

Verify the ratio calculation:

console.log({
  rectWidth: rect.width,
  offsetWidth: offsetWidth,
  ratio: rect.width / offsetWidth,
  canvasZoom: canvasZoom
});

Ratio should equal canvasZoom at single zoom level.

Check event flow:
- Ensure capture phase listener is still first
- Verify WeakSet is tracking events correctly
- Check that transformed events dispatch to child element
Verify xterm.js version:
- This fix assumes xterm.js doesn't change internal element structure
- Check that .xterm-viewport or .xterm-screen still exist

Rollback Plan

If this breaks, the cleanest fallback is:

Remove the mouseTransformHandler entirely
Implement "Browser Zoom Only" approach
Remove canvas zoom buttons
Add instruction: "Use Ctrl+MouseWheel to zoom"

Acknowledgments

This fix was discovered through:

Extensive git history research (5 previous attempts documented)
Creating diagnostic tools to compare browser vs canvas zoom
Understanding the difference between visual rendering and coordinate systems
Trial and error with event interception approaches

The key breakthrough was realizing that rect.width / offsetWidth gives us the exact scaling factor that accounts for BOTH browser and canvas zoom simultaneously.

GGPrompts/MOUSE_COORDINATE_FIX.md

Select an option

No results found