Richard Newman rnewman

Designing bookmark sync… again

For the second time, I'm building an implementation of bookmark sync on a new platform. The last one, in early 2012, was Firefox for Android. That turned out relatively well, considering the circumstances, but it didn't go far enough in avoiding some ancient errors baked into the desktop codebase.

For Firefox on iOS we have a new set of platform constraints and a strong desire to avoid some of the known pitfalls. This post sets out some of the considerations. A future post will set out the proposed solution.

Our goal: keep a tree of bookmarks in sync across multiple devices.

Syncs occur to and from a server. Devices don't communicate directly.

Implementing bookmarks sync… again

Since our first installment we've implemented batched downloading for bookmarks: Bug 1201108.

This incrementally populates a local snapshot of the server (initially termed the ‘mirror’, now more accurately renamed the ‘buffer’), validating each record and fitting the contents into a relational schema. As it processes folders it also fills a table with the top-down structure of the downloaded records. We can be confident that when the downloader finishes, we'll have everything the server currently contains, minus any records that aren't well-formed.

What's next?

We're not done with schema changes: Bug 1201110. The buffer is a blind and forgetful one: it truly represents the current state of the server. We need more than that to sync — in order to find out what changed on each end since we last synced we n

	#ifndef OpenInFirefox_Bridging_Header_h
	#define OpenInFirefox_Bridging_Header_h

	#import <Foundation/Foundation.h>
	#endif

	{"app_name":"Client","app_cohort":"2\|date=1436805000000&sf=143441&tid=a36a22a188ce5cc3adb70f186eb338cdd65cfec2d928ee91bdb79f26fe5031a5&ttype=e","app_version":"1.0","slice_uuid":"6504f5c7-a763-31bc-990d-6a7b7ae5789e","adam_id":989804926,"build_version":"24","bundleID":"org.mozilla.ios.Firefox","share_with_app_devs":false,"is_first_party":false,"bug_type":"109","os_version":"iPhone OS 8.4 (12H143)","is_beta":true,"name":"Client"}
	Incident Identifier: 9724E92E-4A2C-42A5-867B-C8EAD3F54808
	Beta Identifier: C2F6A688-5A2D-4436-9E18-363ED2FFFBD1
	Hardware Model: iPhone7,2
	Process: Client [9471]
	Path: /private/var/mobile/Containers/Bundle/Application/D1827542-0DFA-45B4-A744-96B2B1BB5505/Client.app/Client
	Identifier: org.mozilla.ios.Firefox
	Version: 24 (1.0)
	Beta: YES
	Code Type: ARM-64 (Native)

	$ jar tvf fennec-40.0b8.en-US.android-arm.apk \| fgrep omx
	Picked up _JAVA_OPTIONS: -Xmx1024m -Xms512m
	7752 Mon Jul 27 21:32:00 PDT 2015 assets/armeabi-v7a/libomxplugin.so
	7236 Mon Jul 27 21:32:00 PDT 2015 assets/armeabi-v7a/libomxplugingb.so
	7241 Mon Jul 27 21:32:00 PDT 2015 assets/armeabi-v7a/libomxplugingb235.so
	7030 Mon Jul 27 21:32:00 PDT 2015 assets/armeabi-v7a/libomxpluginhc.so
	9479 Mon Jul 27 21:32:00 PDT 2015 assets/armeabi-v7a/libomxpluginkk.so

	2015-08-05 19:13:13.859 [Info] [Profile.swift:177] init(localName:app:): New profile. Removing old account data.
	2015-08-05 19:13:13.868 [Debug] [BrowserDB.swift:73] init(filename:secretKey:files:): Initializing BrowserDB.
	2015-08-05 19:13:13.869 [Debug] [BrowserDB.swift:142] createOrUpdate: Create or update tableList version 1.
	2015-08-05 19:13:13.874 [Debug] [BrowserDB.swift:92] createTable(_:table:): Try create tableList version 1
	2015-08-05 19:13:13.875 [Debug] [SwiftData.swift:125] transaction: Op in transaction succeeded. Committing.
	2015-08-05 19:13:13.876 [Debug] [BrowserDB.swift:142] createOrUpdate: Create or update favicons version 0.
	2015-08-05 19:13:13.877 [Debug] [BrowserDB.swift:92] createTable(_:table:): Try create favicons version 0
	2015-08-05 19:13:13.878 [Debug] [SwiftData.swift:125] transaction: Op in transaction succeeded. Committing.
	2015-08-05 19:13:13.879 [Info] [BrowserTable.swift:73] init(version:): SQLite version: 3.8.8.3 (3008008).
	2015-08-05 19:13:13.879 [Debug] [BrowserDB.swift:14

	historyID url title guid domain_id domain localVisitDate remoteVisitDate localVisitCount remoteVisitCount frecency
	---------- ---------------------------------------------------- ------------- ------------ ---------- ------------- -------------------- --------------- --------------- ---------------- ----------------
	8055 http://www.ericajong.com/poems/autumnperspective.htm 404 Not Found VfZh4ANR4Eab 516 ericajong.com 1.43949352420582e+15 0 7 0 8365.95843605489
	4684 https://irccloud.mozilla.com/#!/ircs://irc1.dmz.scl3 * #mobile \| M kjktO6fWWlek 857 irccloud.mozi 1.43553051130267e+15 1.4394982251373 1 521 5679.44351381141
	8230 http://www.thekitchenwhisperer.net/2012/07/15/better Better than r TKOaEazO7Dzz 1035 thekitchenwhi 1.43939192547637e+15 0 4 0

	19:50 <@rnewman> well, this might be a problem.
	19:50 <@rnewman> 0:01.90 Sync.Engine.Bookmarks DEBUG Mapped: Bookmarks Toolbar,fTaggy tag,fake-guid-1,false
	19:50 <@rnewman> 0:01.90 Sync.Engine.Bookmarks DEBUG Finding mapping: , fBookmarks Toolbar
	19:50 <@rnewman> 0:01.90 Sync.Engine.Bookmarks DEBUG Mapped dupe: toolbar
	19:50 <@rnewman> 0:01.90 Sync.Engine.Bookmarks DEBUG zzzzzzzzzzzz mapped to toolbar 0:01.90 Sync.Engine.Bookmarks DEBUG Local item toolbar is a duplicate for incoming item zzzzzzzzzzzz 0:01.90 Sync.Store.Bookmarks DEBUG Number of rows matching GUID toolbar: 0
	19:50 <@rnewman> trivial test
	19:50 <@rnewman> create a folder on a client somewhere
	19:50 <@rnewman> name it "Bookmarks Toolbar"
	19:51 <@rnewman> any desktop client that applies that record is gonna have a bad time.
	19:51 * rnewman tries it for real

	/*
	I'm trying to define a method in a protocol that's generic,
	then implement it in a class and allow the class to also have
	a member that refers to that type.
	*/

	class AAA {
	let a: String
	init(a: String) {
	self.a = a

	2015-10-06 20:02:38.834 [Debug] [Profile.swift:675] syncSeveral > Syncing bookmarks…
	2015-10-06 20:02:38.834 [Debug] [Profile.swift:619] mirrorBookmarksWithDelegate(_:prefs:ready:) > Mirroring server bookmarks to storage.
	2015-10-06 20:02:38.834 [Info] [Synchronizer.swift:130] init(scratchpad:delegate:basePrefs:collection:) > Synchronizer configured with prefs 'synchronizer.bookmarks.'.
	2015-10-06 20:02:38.835 [Debug] [Profile.swift:761] greenLight() > Checking green light. Backgrounded: false.
	2015-10-06 20:02:38.835 [Info] [BookmarksDownloader.swift:176] init(collectionClient:basePrefs:collection:) > Downloader configured with prefs 'downloader.bookmarks.'.
	2015-10-06 20:02:38.835 [Debug] [BookmarksDownloader.swift:127] go(_:greenLight:) > Downloading up to 100 records.
	2015-10-06 20:02:38.836 [Debug] [BookmarksDownloader.swift:239] go(_:limit:) > Modified: 1444186946620; last 1444186729270.
	2015-10-06 20:02:38.836 [Debug] [BookmarksDownloader.swift:259] downloadNextBatchWithLimit(_:advancingOnCompletionTo: