thoughts.md

Problem

The current process of timing the launching of Firefox OS applications is tricky. At the time of writing, most tests rely on the firing of the window onload event in order to determine when it is ready to use. Unfortunately with the state of dynamic script and view loading, it's impossible to tell when an application is truly ready to be interacted with. The loading of these assets can be deferred, sometimes even to the point where it occurs after window onload. If we want to hold an application accountable to recommended launch times, engineers need only defer more loading until the test measurement completes in order to make metrics look attractive. Couple this with the fact that the event isn't truly indicative of user interactivity and you can see that the current state is unsustainable.

Interim Solution

I suggested that if we were to get an accurate pulse on when launch times occur, it was going to take being told by the application itself. There are techniques we can use to try and infer when this occurs, but it will most likely be inaccurate, latent, or not verbose enough to outline the many stages of the application launch lifecycle.

I then proposed a series of application-raised events that would cover the launch lifecycle that would be generic between all applications. Suggestions by others brought to my attention the need for these events to also be consumed by the platform itself as indicators for usage beyond performance testing (at the time I was only looking to improve the situation for gathering metrics on the performance lifecycle). At the present time, these events are outlined and raised as follows (and outlined in bug 996038):

// This particular event denotes an app has loaded
// and bound the events needed to interact with its core UI
window.dispatchEvent(new CustomEvent('moz-content-interactive'));

In order for the event to be useful for performance testing, each application includes a script which listens for these events so they can be re-triggered and picked up by the testing harness (script abbreviated for clarity):

window.addEventListener('moz-content-interactive', function () {
  if (!window.mozPerfHasListener) {
    return;
  }
  
  var now = window.performance.now();
  
  setTimeout(function () {
    window.dispatchEvent(new CustomEvent('x-moz-perf', {
      detail: {
        name: 'moz-content-interactive',
        timestamp: now
      }
    });
  });
});

So we listen for the custom events, capture the timestamp, and re-broadcast to the generic event the testing harness is listening to. While this works as an interim solution, I still believe it has limitations:

1. In order to catch the performance event, the script which listens for the event must be included prior to the firing of the event. This has the potential to miss events if fired out of order and causes undue dependencies, violating Inversion of Control.
2. The inclusion of the listener script itself will affect the launch timings.
3. The listener script must be individually included for every concerned application, causing either duplication concerns or maintainability problems when copied and revisions are made.
4. Applications wanting timing metrics for custom events not generic to all applications must bypass the intuitiveness of the window event dispatching and must trigger the low-level event manually.
5. The performance listener script is not efficient and will need modification for new standardized launch metrics.

Longer-term Idea

In wanting to make this process more intuitive and re-usable, I came up with the following idea for expanding the web APIs which I believe solve many of the previous concerns.

1. Introduce PerformanceEvent

A new event constructor of PerformanceEvent will give us a proper API for triggering events concerned with the timing of any performance indicator. Dispatching a PerformanceEvent can then be utilized by the underlying engine (e.g. Gecko) to perform other actions. For example, maybe dispatching a PerformanceEvent('mozAppVisuallyComplete') can be utilized by Gecko to perform deterministic first-paints for Firefox OS apps, or relaying these to the Firefox OS System app for other purposes.

// Constructor for PerformanceEvent
[Partial]
[Constructor(DOMString timing, optional EventInit eventInitDict)]
interface PerformanceEvent : Event {
  ...
}

// Usage
new PerformanceEvent('mozContentInteractive');

new PerformanceEvent('mozAppLoaded', {
  detail: { example: 'property' }
});

2. Introduce performancetiming event

With a standard performancetiming event, an application can have a decoupled mechanism for being aware of the exact occurrence of launch or performance events.

// Usage
window.addEventListener('performancetiming', function (e) {
  // e is a PerformanceEvent
  
  e.timing; // e.g. mozContentInteractive, mozAppLoaded, etc.
  
  // we still have access to standard properties, and importantly,
  // we have access to when the event was triggered, saving us from
  // having to manually capture when the event occurred
  e.timeStamp;
});

3. Propose inclusion of performance events on window.performance.timing

In order to save from having to bind event handlers up front, it would be better for testing if we could attach after the fact and still get the valued performance launch data. This is, in fact, how the window.performance.timing hash currently works. Throughout the load lifecycle of an application, several standardized timings are captured and populated by the engine into window.performance.timing, each property the name of a timing event, which is mapped to a timestamp (Unix epoch in Gecko, high-resolution timestamp in Blink). I would like to see us able to automatically publish abitrary keys into window.performance.timing which contain relevant and useful timings for later retrieval. With a standard cache for containing all aggregated performance events that have occurred with the current application, we have an easy place to calculate deltas of the duration of these important events.

I propose that new properties are added to window.performance.timing every time a new PerformanceEvent is dispatched.

window.performance.timing.navigationStart; // 1402552218684
window.performance.timing.mozAppVisuallyComplete; // 1402555613020

window.dispatchEvent(new PerformanceEvent('mozAppVisuallyComplete'));

...

window.performance.timing.mozAppVisuallyComplete; // 1402555613020

If you are concerned about adding vendor specific timings here, note that the W3C recommendation technically allows these extensions provided they follow the established conventions and are prefixed with moz.

4. Allow all members of window.performance.timing to fire performancetiming

Coming full circle, it would then make sense for any entry in window.performance.timing to automatically create a PerformanceEvent resulting in a performancetiming event. This would include standardized timings such as navigationStart, requestStart, and responseEnd to fire the performancetiming event in addition to the custom ones needed by applications.

window.addEventListener('performancetiming', function (e) {
  if (!e.timing === 'mozAppLoaded') {
    return;
  }
  
  console.log(
    window.performance.timing.mozAppVisuallyComplete -
    window.performance.timing.navigationStart
  ); // 3394336
});

Applications can get the data about their own loading lifecycle, and can be informed throughout the process as well (for stages that are relevant and available at the time of loading) for built-in timings as well as custom timings.

Benefits

Attempting to solve the previously outlined limitations:

1. No external script is needed to capture the events, meaning no missed values. Data is cached by the platform and stored in window.performance.timing. No dependency qualms.
2. With no listener script, launch timings are no longer affected by the capturing of metrics.
3. No listener script means no duplication or maintainability concerns.
4. Applications that want to time performance events outside of the standard platform launch events can do so easily by just dispatching a PerformanceEvent.
5. Efficiency is improved by pushing logic into a standardized system which can be re-used from the platform level, e.g. usage with Gecko or relaying to the System app.

Conclusion

Our current methodology for conducting automated performance testing is functional, but can be improved and expanded on. By opening up and leveraging existing and familiar web APIs, we can improve the intuitiveness of the testing, and hence encouage more engineers to outfit their applications for performance evaluation. These changes will also give a richer indicator to the platform for making further optimizations to the loading of applications. I also believe that through these efforts we can improve the tooling and ease its implementation, creating better value for the web.