Synchronization has been on my mind recently. I have to build a sync engine, primarily on Apple platforms. While the ultimate goal is to make sure that a user can access and modify data on any device, there are many implementation details which have different tradeoffs.

For instance, how do various different devices communicate changes between each other? The general assumption is a network connection will transmit some form of state. Is the state transmitted via HTTP or some other protocol? Is the complete state of the device transmitted or only changes made on the device? Should data be synchronized in real-time or can the data be synchronized with a delay?

Beyond how the state is transmitted, is each device a complete view of the entire data set? Does each device even have the capacity for all of the data? If each device is only a partial view of the data, is there a single source of truth? How does a device determine what is relevant data if it only has a partial view, and how does it obtain the relevant data?

Do devices have constant and reliable network connectivity? If they do not, how does data get merged together when a device cannot communicate? While the device is unreachable, it may make changes and other devices could make changes as well. If there are conflicting changes made, how do the conflicts get resolved?

These questions and decisions are only a few of the considerations which will have major effects on how the synchronization is made.

To clarify which choices to make, there are other desired goals when synchronizing data:

  • The data transmitted to and from a device should only be what is absolutely necessary. Network resources can be fairly scarce especially on mobile devices. Whether on Wi-Fi, on cellular networks, or on wired ethernet, using the network should be considered relatively expensive. Furthermore, any data sent or received requries power to process, so it is better to only process what is necessary.

  • Synchronization data should be batchable. With network resources being considered scarce, the number of network requests should be limited. Therefore, any data changes should be able to be batched together as one network request (whether data is transmitted to or from a device).

  • Devices should be able to receive a partial set of the synchronization data and make progress towards the latest state. If a device received only half of the data updates required to transition the device to the latest state, it should still be able to process the updates received and transition the local device state. The synchronization updates could be large enough either in quantity or size that a device may need to process multiple batches.

  • The processing of synchronization data should be streamable. Whether there is one other device or hundreds of other devices, a device could need to synchronize a massive number of updates made since the last time the device synced. In a mobile device world, the devices have varying degrees of processing power with limited amounts of memory, so the synchronization data needs to be able to be processed as a stream of data versus processing all of the data at once.

With the above desired goals, a few other properties become desirable:

  • The processing of the synchronization data should be idempotent. A device could receive the same synchronization data multiple times. Either the device needs to be able to identify that the updates have already been processed or processing the data should be idempotent. Using specialized data structures such as conflict-free replicated data types may help simplify processing.

  • Any device should be able to track what changes have been made since a local point in relative time. By keeping track of changes, a device can identify what changes were made locally which need to be transmitted to another device. Furthermore, a device could synchronize with multiple devices (or services) which requires multiple points in relative time to be able to be tracked. Note that relative time may not be a traditional clock time but could be as simple as a counter or a change token.

  • As a corollary, a local device should be able to inform another service or device what data has already been synchronized. Whether it is using (change) tokens or other means, a local device can inform another device what it has already processed to reduce the number of requests and data required to get the local device up to date.

  • Any device (including services) which have the data should have ACID transactions. ACID-compliance make synchronization easier by ensuring updates are actually processed when an operation says it is processed.

There are many other considerations for synchronization, but hopefully these thoughts give an idea on what are some of the possible complexities and desired properties when synchronizing data.