DevTools as System Addon tests

devtools_addon_tests.txt

(DSA = DevTools as System Addon)

Setup: The version of DevTools as system addon that I am testing is:
- the patches from ship as system addon https://bugzilla.mozilla.org/show_bug.cgi?id=1369801
- the patches from move key-shortcut https://bugzilla.mozilla.org/show_bug.cgi?id=1386616
- the "workaround" from system addons sideloaded don't start https://bugzilla.mozilla.org/show_bug.cgi?id=1386295

Test scenarios
- S1: 
  1 - Open Firefox with DSA with a clean profile
  2 - Check that devtools can be opened
- S2:
  1 - Open Firefox without DSA with a clean profile
  2 - Check that devtools can be opened
  3 - With the same profile, open Firefox with DSA
  4 - Check that DevTools can be opened
  5 - With the same profile, open Firefox without DSA again
  6 - Check that DevTools can still be opened
- S3:
  1 - Open Firefox with DSA with a clean profile
  2 - Check that devtools can be opened
  3 - With the same profile, open Firefox without DSA
  4 - Check that DevTools can be opened
  5 - With the same profile, open Firefox with DSA again
  6 - Check that DevTools can still be opened

A: Run the scenarios above with packaged Firefox builds.

  A.S1: OK
  A.S2: OK
  A.S3: OK

B: Rerun the same scenarios with local builds, switching between the current central and my local branch, clobber before each build.

  B.S1: OK
  B.S2: FAILED at step 4 same issue as below, but with devtools-shim locale.
        Devtools don't open using keyboard shortcuts. 
        Using "Inspect Element" works. WebDeveloper menus are empty.
        Restarting the browser with the same profile doesn't fix it.
        Enabling dumps: 
          JavaScript error: file:///Users/jdescottes/Development/hg/fx-team/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/Resources/browser/components/devtools-startup.js, line 65: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIStringBundle.GetStringFromName]
  B.S3: FAILED at step 4 same issue as above, but with devtools locale.

C: Rerun the same scenarios with local builds, switching between the current central and my local branch, clobber before each scenario.

  C.S1: OK
  C.S2: FAILED at step 6, but works again after restarting. 
        On first startup, devtools don't open, see logs at [1]
        On second startup, devtools open successfully but we still get error logs [2]
  C.S3: FAILED at step 4, same issue as above.

D: Rerun the same scenarios with local builds, switching between central+key-shortcut and my local branch, clobber before each build.

  D.S1: OK
  D.S2: OK
  D.S3: OK

E: Rerun the same scenarios with local builds, switching between central+key-shortcut and my local branch, clobber before each scenario.

  E.S1: OK
  E.S2: FAILED at step 6, same as C.S2
  E.S3: FAILED at step 6, same as C.S3

LOGS:

[1] Logs from C.S2 first startup

1502284707687 addons.xpi  WARN  Error loading bootstrap.js for [email protected]: Error opening input stream (invalid filename?): file:///Users/jdescottes/Development/hg/fx-team/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/Resources/browser/features/[email protected]/bootstrap.js
        1502284707688 addons.xpi  WARN  Add-on [email protected] is missing bootstrap method startup
        2017-08-09 15:18:28.551 plugin-container[73413:4873216] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x974f, name = 'com.apple.tsm.portname'
        See /usr/include/servers/bootstrap_defs.h for the error codes.
        2017-08-09 15:18:28.553 plugin-container[73413:4873216] *** CFMessagePort: bootstrap_register(): failed 1100 (0x44c) 'Permission denied', port = 0x9a0f, name = 'com.apple.CFPasteboardClient'
        See /usr/include/servers/bootstrap_defs.h for the error codes.
        2017-08-09 15:18:28.553 plugin-container[73413:4873216] void __CFPasteboardSetup() : Failed to allocate communication port for com.apple.CFPasteboardClient; this is likely due to sandbox restrictions
        JavaScript error: file:///Users/jdescottes/Development/hg/fx-team/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/Resources/browser/components/devtools-startup.js, line 65: NS_ERROR_FILE_NOT_FOUND: Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIStringBundle.GetStringFromName]
        1502284708800 addons.xpi-utils  WARN  updateMetadata: Add-on [email protected] is invalid: Error: Directory /Users/jdescottes/Development/hg/fx-team/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/Resources/browser/features/[email protected] does not contain a valid install manifest (resource://gre/modules/addons/XPIInstall.jsm:828:11) JS Stack trace: 
          [email protected]:828:11 
            < async*[email protected]:974:10 
            < [email protected]:957:3 
            < [email protected]:1331:21 
            < [email protected]:1549:26 
            < [email protected]:3235:7 
            < async*[email protected]:3101:12 
            < [email protected]:56:28 
            < async*[email protected]:52:5 
            < async*[email protected]:1073:5 
            < [email protected]:341:9 
            < [email protected]:1167:9 
            < onBeforeBrowserWindowShown/<@SessionStore.jsm:1312:9 
            < promise callback*[email protected]:1297:5 
            < [email protected]:764:9 
            < [email protected]:1351:5 
            < [email protected]:1:1
        1502284708801 addons.xpi-utils  WARN  Could not uninstall invalid item from locked install location
        JavaScript error: resource://gre/modules/addons/XPIProvider.jsm -> resource://gre/modules/addons/XPIProviderUtils.js, line 1569: TypeError: newAddon is null



[2] Logs from C.S2 second startup

addons.xpi-utils  WARN  addMetadata: Add-on [email protected] is invalid: Error: Directory /Users/jdescottes/Development/hg/fx-team/obj-x86_64-apple-darwin15.6.0/dist/Nightly.app/Contents/Resources/browser/features/[email protected] does not contain a valid install manifest (resource://gre/modules/addons/XPIInstall.jsm:828:11) JS Stack trace: 
        [email protected]:828:11 
          < async*[email protected]:974:10 
          < [email protected]:957:3 
          < [email protected]:1235:21 
          < [email protected]:1599:23 
          < [email protected]:3235:7 
          < async*[email protected]:3101:12 
          < [email protected]:56:28 
          < async*[email protected]:52:5 
          < async*[email protected]:1073:5 
          < [email protected]:341:9 
          < [email protected]:1167:9 
          < onBeforeBrowserWindowShown/<@SessionStore.jsm:1312:9 
          < promise callback*[email protected]:1297:5 
          < [email protected]:764:9 
          < [email protected]:1351:5 
          < [email protected]:1:1
1502284388064 addons.xpi-utils  WARN  Could not uninstall invalid item from locked install location