gistfile1.txt

Aki and I spent a ton of time trying to get this done today. We did manage to sign it, but it crashed on startup on Aki's laptop for reasons that aren't clear. Here's my best attempt at a recap what we did as cltbld@mac-v3-signing1:
* Baku provided a .pkg that was previously signed with a Distribution certificate.
* Hacked the scriptworker code to sign `Frameworks` and handle signing the embedded WireGuard app (more on that later):
```
diff --git a/iscript/src/iscript/mac.py b/iscript/src/iscript/mac.py
index 7bbc764..4aeb2bc 100644
--- a/iscript/src/iscript/mac.py
+++ b/iscript/src/iscript/mac.py
@@ -206,13 +206,14 @@ async def sign_app(key_config, app_path, entitlements_path):
         IScriptError: on error.
 
     """
-    SIGN_DIRS = ("MacOS", "Library")
+    SIGN_DIRS = ("MacOS", "Library", "Frameworks")
     parent_dir = os.path.dirname(app_path)
     app_name = os.path.basename(app_path)
     await run_command(["xattr", "-cr", app_name], cwd=parent_dir, exception=IScriptError)
     identity = key_config["identity"]
     keychain = key_config["signing_keychain"]
     sign_command = _get_sign_command(identity, keychain)
+    log.debug("BHEARUSM: signing %s" % app_name)
 
     if key_config.get("sign_with_entitlements", False):
         sign_command.extend(["-o", "runtime", "--entitlements", entitlements_path])
@@ -227,7 +228,8 @@ async def sign_app(key_config, app_path, entitlements_path):
                 log.debug("Skipping %s because it's not in SIGN_DIRS.", abs_dir)
                 dirs.remove(dir_)
                 continue
-            if dir_.endswith(".app"):
+            log.debug("BHEARUSM DEBUGGING DO NOT RUN IN PRODUCTION")
+            if dir_.endswith(".app") or dir_.endswith(".appex"):
                 await sign_app(key_config, abs_dir, entitlements_path)
         if top_dir == contents_dir:
             log.debug("Skipping file iteration in %s because it's the root directory.", top_dir)
@@ -236,7 +238,7 @@ async def sign_app(key_config, app_path, entitlements_path):
         for file_ in files:
             abs_file = os.path.join(top_dir, file_)
             # Deal with inner .app's above, not here.
-            if top_dir[app_path_len:].count(".app") > 0:
+            if top_dir[app_path_len:].count(".app") > 0 or top_dir[app_path_len:].count(".app") > 0:
                 log.debug("Skipping %s because it's part of an inner app.", abs_file)
                 continue
             # app_executable gets signed with the outer package.
@@ -376,7 +378,7 @@ def get_app_dir(parent_dir):
         UnknownAppDir: if there is no single app dir
 
     """
-    apps = glob("{}/*.app".format(parent_dir)) + glob("{}/*/*.app".format(parent_dir))
+    apps = glob("{}/*.app*".format(parent_dir)) + glob("{}/*/*.app*".format(parent_dir))
     if len(apps) != 1:
         raise UnknownAppDir("Can't find a single .app in {}: {}".format(parent_dir, apps))
     return apps[0]
@@ -920,7 +922,7 @@ async def create_pkg_files(config, key_config, all_paths):
     for app in all_paths:
         # call set_app_path_and_name because we may not have called sign_app() earlier
         set_app_path_and_name(app)
-        app.pkg_path = app.app_path.replace(".app", ".pkg")
+        app.pkg_path = app.app_path.replace(".app", ".pkg").replace("pkgex", "pkg")
         app.pkg_name = os.path.basename(app.pkg_path)
         pkg_opts = []
         if key_config.get("pkg_cert_id"):
diff --git a/iscript/src/iscript/mac.py b/iscript/src/iscript/mac.py
index 7bbc764..4aeb2bc 100644
--- a/iscript/src/iscript/mac.py
+++ b/iscript/src/iscript/mac.py
@@ -206,13 +206,14 @@ async def sign_app(key_config, app_path, entitlements_path):
         IScriptError: on error.
 
     """
-    SIGN_DIRS = ("MacOS", "Library")
+    SIGN_DIRS = ("MacOS", "Library", "Frameworks")
     parent_dir = os.path.dirname(app_path)
     app_name = os.path.basename(app_path)
     await run_command(["xattr", "-cr", app_name], cwd=parent_dir, exception=IScriptError)
     identity = key_config["identity"]
     keychain = key_config["signing_keychain"]
     sign_command = _get_sign_command(identity, keychain)
+    log.debug("BHEARUSM: signing %s" % app_name)
 
     if key_config.get("sign_with_entitlements", False):
         sign_command.extend(["-o", "runtime", "--entitlements", entitlements_path])
@@ -227,7 +228,8 @@ async def sign_app(key_config, app_path, entitlements_path):
                 log.debug("Skipping %s because it's not in SIGN_DIRS.", abs_dir)
                 dirs.remove(dir_)
                 continue
-            if dir_.endswith(".app"):
+            log.debug("BHEARUSM DEBUGGING DO NOT RUN IN PRODUCTION")
+            if dir_.endswith(".app") or dir_.endswith(".appex"):
                 await sign_app(key_config, abs_dir, entitlements_path)
         if top_dir == contents_dir:
             log.debug("Skipping file iteration in %s because it's the root directory.", top_dir)
@@ -236,7 +238,7 @@ async def sign_app(key_config, app_path, entitlements_path):
         for file_ in files:
             abs_file = os.path.join(top_dir, file_)
             # Deal with inner .app's above, not here.
-            if top_dir[app_path_len:].count(".app") > 0:
+            if top_dir[app_path_len:].count(".app") > 0 or top_dir[app_path_len:].count(".app") > 0:
                 log.debug("Skipping %s because it's part of an inner app.", abs_file)
                 continue
             # app_executable gets signed with the outer package.
@@ -376,7 +378,7 @@ def get_app_dir(parent_dir):
         UnknownAppDir: if there is no single app dir
 
     """
-    apps = glob("{}/*.app".format(parent_dir)) + glob("{}/*/*.app".format(parent_dir))
+    apps = glob("{}/*.app*".format(parent_dir)) + glob("{}/*/*.app*".format(parent_dir))
     if len(apps) != 1:
         raise UnknownAppDir("Can't find a single .app in {}: {}".format(parent_dir, apps))
     return apps[0]
@@ -920,7 +922,7 @@ async def create_pkg_files(config, key_config, all_paths):
     for app in all_paths:
         # call set_app_path_and_name because we may not have called sign_app() earlier
         set_app_path_and_name(app)
-        app.pkg_path = app.app_path.replace(".app", ".pkg")
+        app.pkg_path = app.app_path.replace(".app", ".pkg").replace("pkgex", "pkg")
         app.pkg_name = os.path.basename(app.pkg_path)
         pkg_opts = []
         if key_config.get("pkg_cert_id"):
```
* Our tooling requires a .app or a .dmg, so to create that we did:
```
pkgutil --expand Mozilla\ VPN.pkg extracted
cd extracted/org.mozilla.macos.FirefoxVPN.pkg
mv Payload Payload.gz
gunzip Payload
cpio -iv < Payload
```
...at which point we now have a `Mozilla VPN.app`.

* Our tooling also requires the entitlements to be in their own file. To extract those, we pulled the `Entitlements` section from `Mozilla VPN.app/Contents/embedded.provisionprofile`, and put it in a gist (not ideal): https://gist.githubusercontent.com/mozbhearsum/24a737f1175d5f0ca1ba5371a6827ea9/raw/e460de20ab24dd50ac0d2db739c3dc00988a1e0e/gistfile1.txt
* We did the same for the embedded WireGuard app (more below on that), and its entitlements are at https://gist.githubusercontent.com/mozbhearsum/8d7f981c36588fc604fd40ded84b06ec/raw/9c5ce449b58771a21f2b85892ae636a5452fa1e8/gistfile1.txt

* We prepped for signing by using `create_test_workdir` to prep the workarea:
```
cd /builds/scriptworker
source virtualenv/bin/activate
create_test_workdir --overwrite TDclMiOXTTSvmAyp44FPrg # this is the taskid of a recent part1 notorization job on mozilla-beta
```
* We discovered early on that we had to resign the embedded WireGuard separately, because it has its own entitlements. To sign that, we packaged that .appex in a tarball and put it in the right place for signing:
```
cd "/tmp/extracted/org.mozilla.macos.FirefoxVPN.pkg/Mozilla VPN.app/Contents/PlugIns"
tar -zvcf /tmp/wireguard.tar.gz .
cp /tmp/wireguard.tar.gz /builds/scriptworker/work/cot/dX3iiBheQ2au9x_9OLOL4Q/public/build/target.tar.gz
```
* Edited the `task.json` to update the `entitlements-url` to the wireguard gist above.
* Ran iscript to do the signing & notarization:
```
cd /builds/scriptworker
iscript script_config.yaml 
```
* Took the resulting package and re-embedded into the Mozilla VPN app:
```
cd "/tmp/extracted/org.mozilla.macos.FirefoxVPN.pkg/Mozilla VPN.app/Contents/PlugIns"
tar -zvxf /builds/scriptworker/artifacts/public/build/target.tar.gz
```
* Packaged the .app in a tarball and put it the right place for signing:
```
cd /tmp/extracted/org.mozilla.macos.FirefoxVPN.pkg/
tar -zvcf /builds/scriptworker/work/cot/dX3iiBheQ2au9x_9OLOL4Q/public/build/target.tar.gz Mozilla\ VPN.app/
```
* Edited the `task.json` to update the `entitlements-url` to the gist above.

* Ran iscript to do the signing & notarization:
```
cd /builds/scriptworker
iscript script_config.yaml 
```

This signed & notarization correctly, and dropped the signed pkg and tarball contained a .app into `/builds/scriptworker/artifacts/public/build`. As noted above, the resulting build seems to crash on start up.


Other things we tried:
* Resigning without resigning the WireGuard .app. This failed notarization.
* Signing the .pkg provided directly. We couldn't make this work for reasons I don't fully recall now (I think because the .app inside was signed with a Distribution cert instead of a Developer ID?)

My best guess on what's wrong is that the previous signing that Baku did has left bits somewhere inside the built that somehow break it even after we resign. Or possibly just that WireGuard's signature was overwritten (I'm presuming this had its own signature originally). I think the next best step forward is to get our hands on a fully _unsigned_ version of Mozilla VPN.app, and try the procedure above with that.