Proposal for User Interaction Indirection

(formerly "Proposal for Enhanced Keyboard Interaction")

"Abstraction of user interaction by introduction command primitives"

originally I thought this would only mean something to the keyboard. But having an abstract concept of a "command" enables us to access the same actions via keyboard, voice, touch-gesture, etc. Probably the main problem is going to be "context". Which command is applicable to what kind of an element and in which situation is it even possible to execute the action.

TODO

See IndieUI: Events and parts of IndieUI: User Context

Requirements

There are some tricky ones like copy/paste or undo/redo (e.g. what happens when undo is called while focused on a textarea?) - https://twitter.com/tobie/status/420610454046519296
That standard might also tackle the cmd [ problem (on my keyboard "[" is alt 5) - and thus useless) - https://twitter.com/rodneyrehm/status/420627964091695105
They need to be high-level enough to work across-OS (so an undo event rather than cmd shift z or cmd y). - https://twitter.com/tobie/status/420628830274199552
make the browser the source of truth when it comes to keyboard mappings

Annoyances

backspace may trigger history.back() - not on my system?!
cmd left may trigger history.back - unless activeElement is <input> - possibly failing in contenteditable="true"

Questions

There is a Standard for the Clipboard defining what it does - event.dataTransfer?
How do accessibility tools currently work in this space?
- We don't want to overwrite their combos
- is the accessibility API invovled?
How is this affected by OSX Keyboard Remapping?
Can commands be "exported" to the browser-menu?
- can this be made available to accessibility tools (such as screen readers)
- can this be made available to Web Speech API?
- can this be mapped to gestures for touch devices?
- can this be
Is this best declared in HTML or JS/WebIDL-only
What about AccessKey attribute?
would event.preventDefault() on keyup/keydown prevent the command execution?
what kind of a micro-syntax is best used for describing key-combos?
What about context?
- within , within contenteditable=true
- will document.activeElement help?
should key sequences (e.g. the infamous konami code) be supported?
Can we define all keyboard interaction "defaultEvents" currently common in browser this way?
- tab, shift tab - iterate activeElement
- space, shift space - scroll documt by viewport-height
- return - submit form (if activeElement is <input>)
- backspace - history.back()
should the eninge maintain a set of default actions?
- e.g. copy, paste, help, new, add, remove, enable, disable, toggle, …
how best to handle context?
- "new" might be available for one element, but not for another
giving user control is primary objective
- disable commands (global, per origin)
- assign/overwrite key combos (global, per origin)
- visualize list of available commands, global and per origin
- should a user be prompted for permission? (I don't think so)

Problems in JS

browser's language navigator.language is not necessarily the system's language (my OS is en-US, Firefox is de-DE, Firefox accept-lang is en-US, de-DE)
cannot identify the keyboard layout (which key is where, [ is alt 5 on a de-DE OSX)
don't know the browser's/system's/user's preference for undo/redo (cmd y vs. cmd shift z)
cannot reliably detect capslock
cannot destinguish numpad (or can you?)
OS specific shortcuts like cmd , on OSX

look at github's keyboard interaction
https://delicious.com/rodneyrehm/keyboard
http://blog.rodneyrehm.de/archives/23-Reclaim-Your-Keyboard-Shortcuts-in-Firefox.html

API banter

// declaring support for a command
// <commandName>, <defaultKeyCombination>, <defaultCommandName>, <specificCommandNameCallback:optional>
navigator.registerCommand("foobar", "cmd shift k", "Do Foobar", function(activeElement) {
  // callback so you an app can be more specific about the command's actual name
  return {
    // the name to be shown for the command in context of activeElement
    name: "Do Foobar with " + activeElement.title,
    // signal the command is un/specified for the given activeElement
    available: true,
    // signal the command is un/usable on the given activeElement
    disabled: false
  };
});
// alternate for callback function could be hooking into activeElement changes?

// registering for a command
window.addEventListener('command', function(event) {
  event.type = 'command';
  event.keyCombination = 'cmd shift k';
  event.command = 'foobar';
  event.name = 'Do Foobar with "title"';
  
  // this is system wide, context can only be activeElement
  event.bubbles = false;
  // you cannot abort cmd+c (copy) or "new tab" or "quit application"
  event.cancelable = false;
  // prevent other listeners from executing
  event.preventImmediatePropagation();
});
// list currently registered commands
navigator.commands {
  foobar: {
    command: 'foobar',
    defaultKeyCombination: 'cmd shift k',
    defaultName: 'Do Foobar',
    
    
    // userDefinedKeyCombination || defaultKeyCombination || null if userDisabled
    keyCombination: 'cmd shift k',
    // context evaluated
    name: 'Do Foobar with "title",
    available: true,
    disabled: false
  }
}

rodneyrehm/user-interaction-indirection.md