Encrypted Realm with secret in keychain in app supporting background fetch intermittently causes realm to fail to initialize and crash app at try! - SOLVED

Encrypted Realm with secret in keychain in app supporting background fetch intermittently causes realm to fail to initialize and crash app at try! - SOLVED

This is like a drop-in replacement for realms example swift project enabling encryption using the function getKey().

https://github.com/realm/realm-cocoa/blob/master/examples/ios/swift/Encryption/ViewController.swift

The following swift function is a drop in replacement, it documents the painful experience of an issue
which has been unsolved for years till crashlytics revealed a very useful hint, which lead to the root cause
and fix.

Comments included for education on what the symptoms were, the cause and fix and other notes.

private static func getEncryptionKey() -> NSData {
		/*
			PLI @2019-07-01
			identified from crashlytics the "Realm file decryption failed" error...
			https://github.com/realm/realm-cocoa/issues/5615

			https://forums.developer.apple.com/thread/114159 - KeyChain SecItemAdd return -25308 when app is launched
			https://forums.developer.apple.com/thread/78372 - Access Keychain When iPhone Locked
		
			in short, add kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock
		
			Disgnosis Explained:
			====================
			Symptoms: intermittent/unexplained crashes (force unwrap try! Realm()) when realm for some reason would
		              not properly initialise, when it should initialise perfectly fine.
		
			Thoery of contributing factors:
				- encrypted realm
				- application supports background updates
				- device is in locked state
				- *** application is not running, but is launched by OS to initiate a background update ***
					- This fact alone is VERY hard, if not impossible to replicate, as the simulator only allows
					  simulate background fetch when the app is being debugged... but what we need is
					  app not running, and device locked, and OS launches our app!
					  Even if you did replicate it by not running app and leaving overnight or for a long duration
					  one would not know when the OS calculated 'launch app to do fetch' will occur, as the algorithm
					  of the OS is not published. So one could be waiting for ages...
		
			Theory of sequence of events to cause crash:
				- when the device is locked, keychain requests if not configured correctly will likely
				  return errSecInteractionNotAllowed (-25308), which indicates a UI cannot be presented to prompt to unlock
				  and thus gain access. (unless kSecAttrAccessibleAfterFirstUnlock attribute set)
				- as a result the code would fall through (as it didn't check for errSecInteractionNotAllowed error)
				  and only checked for errSecSuccess, and run the code assuming the entry did not exist
				  and thus create a new key and add to the keychain.
				- the code would have asserted, as it would have got a duplicate entry error, however being production
				  code asserts are not compiled in. And since this bug was very intermittent, we never saw it
				  in the debug code version where asserts were compiled in.
				- because in production code the asserts was compiled out, the result would be a new key was created
				  and returned to the caller, which would of course be a different encryption key as that
				  originally used to encrypt the realm DB, and thus realm would correctly thrown an
				  exception "Realm file decryption failed".
				- thus resulting in a Realm object not being instantiated, and hence forcing to resolve try! would crash.
		
			Solution:
				- in order to resolve the problem we need to tag the entry with accessible attribute:
					kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock
					We dont want to limit to only current device, as user may backup and restore, and we
					would still like the DB to work.
					We dont want to have only accessible on unlock, as then it will not work when performing
					background tasks.
				- for compatibility, we need to delete/update the entry with the missing attribute to then include
					the required attribute, to avoid crashes moving forwards and not require the user to delete
					and re-install the app (since keychain persists per app items for different durations when
					app deleted depending on iOS version)
				- a query for item with or without kSecAttrAccessibleAfterFirstUnlock attribute will find item
				  originally created with kSecAttrAccessibleAfterFirstUnlock attribute set. BUT a query will only
				  find an item created without kSecAttrAccessibleAfterFirstUnlock if the query does not have
				  the attribute kSecAttrAccessibleAfterFirstUnlock set.
				  UPDATE: OS automatically adds attribute default of kSecAttrAccessibleWhenUnlocked if none
					  aupplied by caller on adding to keychain. Thus if you query/fetch with attribute
					  not supplied or as kSecAttrAccessibleWhenUnlocked, then a match will result.

		
			Notes:
				- interestingly in realms keychain_helper.cpp there is a is a function build_search_dictionary()
				  which conveniently does pass in the appropriate attributes:

					CFDictionaryAddValue(d.get(), kSecAttrAccessible, kSecAttrAccessibleAlways);
		
				  We do NOT pass kSecAttrAccessibleAlways, as this is deprecated in iOS 13, and is probably considered
				  less secure, so we use the next best thing, kSecAttrAccessibleAfterFirstUnlock
		
		*/
		
		// Identifier for our keychain entry - should be unique for your application
		let keychainIdentifier = kKeychainIdentifier /* REPLACE THIS WITH YOUR OWN IDENTIFIER */
		let keychainIdentifierData = keychainIdentifier.data(using: String.Encoding.utf8, allowLossyConversion: false)!
		
		// First check in the keychain for an existing key
		var query: [NSString: AnyObject] = [
			kSecClass: kSecClassKey,
			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
			kSecAttrKeySizeInBits: 512 as AnyObject,
			kSecReturnData: true as AnyObject,
			kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock	/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
		]
		
		// To avoid Swift optimization bug, should use withUnsafeMutablePointer() function to retrieve the keychain item
		// See also: http://stackoverflow.com/questions/24145838/querying-ios-keychain-using-swift/27721328#27721328
		var dataTypeRef: AnyObject?
		var status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(query as CFDictionary, UnsafeMutablePointer($0)) }
		if status == errSecSuccess {
			return dataTypeRef as! NSData
		}
		
		////////////////////////////////////////////////
		////// BUG FIX COMPATIBILITY CODE MI v5.2 //////
		////////////////////////////////////////////////
		let queryWithCompatibility: [NSString: AnyObject] = [
			kSecClass: kSecClassKey,
			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
			kSecAttrKeySizeInBits: 512 as AnyObject,
			kSecReturnData: true as AnyObject
			/* kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock - OLD APP DIDNT HAVE THIS KEY */
		]

		status = withUnsafeMutablePointer(to: &dataTypeRef) { SecItemCopyMatching(queryWithCompatibility as CFDictionary, UnsafeMutablePointer($0)) }
		if status == errSecSuccess {
			// delete old entry
			status = SecItemDelete(queryWithCompatibility as CFDictionary)
			
			// add newly updated entry... oddly can't use SecItemUpdate() as always getting -50 error... maybe did something wrong.
			query = [
				kSecClass: kSecClassKey,
				kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
				kSecAttrKeySizeInBits: 512 as AnyObject,
				kSecValueData: dataTypeRef as! NSData,
				kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock		/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
			]
			
			status = SecItemAdd(query as CFDictionary, nil)
			assert(status == errSecSuccess, "Failed to insert/update the new key in the keychain")
			
			return dataTypeRef as! NSData
		}

		////////////////////////////////////////////////////
		////// END BUG FIX COMPATIBILITY CODE MI v5.2 //////
		////////////////////////////////////////////////////

		// No pre-existing key from this application, so generate 64 bytes of random data to serve as the encryption key
		let keyData = NSMutableData(length: 64)!
		let result = SecRandomCopyBytes(kSecRandomDefault, 64, keyData.mutableBytes.bindMemory(to: UInt8.self, capacity: 64))
		assert(result == 0, "Failed to get random bytes")
		
		// Store the key in the keychain
		query = [
			kSecClass: kSecClassKey,
			kSecAttrApplicationTag: keychainIdentifierData as AnyObject,
			kSecAttrKeySizeInBits: 512 as AnyObject,
			kSecValueData: keyData,
			kSecAttrAccessible: kSecAttrAccessibleAfterFirstUnlock		/* Key required to access keychain when app in background - stop the realm crashes on app startup in background */
		]
		
		status = SecItemAdd(query as CFDictionary, nil)
		assert(status == errSecSuccess, "Failed to insert the new key in the keychain")
		
		return keyData
	}

Thanks for this gist Peter! This problem seems to happen more often on iOS 15 betas, and the fix described here has been very helpful to us 🙇

Thanks for the gist, @PeterLi !
I was actually wondering what is your proposal for the generation of the keychain entry (keychainIdentifier)?

Previously in the app that I am developing, we were using the identifier that was based on the app's bundle id.
But now when the users started to install the app on iP14/iP14 Pro devices we've noticed an increased amount of crashes caused by Realm file encryption. The code we're using is basically the same as here.

But I was wondering, maybe we should also add the identifierForVendor to the keychainIdentifier so we're sure that the encryption key will be different when the same keychain user installs the app on a different device. What are your thoughts?