SourceGear Blogs

Handling Simultaneous Updates in Zumero: There Is No Step 2

2013-05-01T14:51:00+00:00

When we talk about the advantages of always-local data for mobile apps, the typical first reaction is “that sounds awesome”.

The next step is to start thinking about your apps, and how mobile users interact with your data, and how those users aren’t going to sit down and coordinate with each other:

“It’s my turn to update the inventory records for store #42”

“Done with those yet?”

“Yep, go ahead and sync.”

“Cool, thanks. Now I can update the ordered-quantity data.”

(a) That’s not how work gets done, (b) computers are supposed to take care of picky details for us, and (c) I don’t want to hang out with those guys at all.

Dealing with this scenario in a Zumero-enabled app boils down to:

Users update and add data as they see fit.
At some point, probably in the background, the app syncs to the server.
Up-to-date, merged data now lives on the device.

You don’t have to write any code that knows conflicting changes happened. Seriously. That’s our problem.

Even when the changes are tricky.

Consider the Zumero wiki sample app. A wiki is a collaborative tool. People can, will, should edit the same pages. Sometimes at the same time.

What happens if they do? Andy creates a page on his iPad:

Colin pulls up the app on his iPhone, syncs, and gets his own copy. While offline, he adds a line to the page:

At the same time, Andy completes one of the items on his list, and marks that line as completed.

Colin comes back online, his app checks in and syncs, and he sees this:

Here’s the code that caused their apps to reconcile the conflicting updates:

Sync a Zumero database link

ok = [_db sync:scheme user:username password:password error:&err];

There’s no step 2.

To be fair, there were a couple of lines of code that ran when we initially set up the database, which boiled down to:

If there are conflicting edits on the ‘text’ field of a wiki page, try to merge them, Zumero. Thanks.

No step 2.

Keep your CRUD off the Internet

2013-04-23T15:00:01+00:00

Whenever I explain database sync for mobile, somebody asks, "What kinds of apps would benefit from that?"

The correct answer is: "ALL apps".

Actually, that's an exaggeration. I see no reason for my alarm clock or calculator apps to sync data with a server. Let me rephrase:

Sync is the best architecture
for apps where a database and a server are involved.

CRUD over REST is an Anti-Pattern

Many apps today keep the data only on the server and access it using REST. This approach has some big negative consequences:

Performance: Your app is less responsive.

A round-trip from a mobile device to a server takes a LONG time. Under good network conditions, it could be a quarter of a second. That's enough time for the user to perceive a lag.

Sometimes the round-trip will take longer, more like a full second, or more. That's an eternity, a one-star review in the App Store.

Numerous studies show that when it comes to the responsiveness of a user interface, fractions of a second matter a LOT. And people don't necessarily complain about your app. They just stop using it.

If you're going to make a round-trip to the server in response to a user action, your app is going to feel sluggish.
Reliability: Your app doesn't work offline.

WiFi is great, but not everything is a coffee shop.

Cellular data networks have dead spots, sometimes in surprising places. Even in areas with solid 3G or LTE coverage, the latency varies wildly.

If your app is going to stop working when the wireless network is unavailable or unreliable, then your app is going to stop working a lot.
Complexity: Your app's code has to deal with networking issues everywhere.

People wrap their REST calls in pretty wrappers to make them look like networking is not involved. But it is. A robust app needs to deal with all the things that can go wrong.

If you're going to do all your basic database operations over a REST API, all your code is networking code.

The real question is "What kinds of apps would benefit from having user interaction dependent on a wireless network with inconsistent coverage and random latency spikes?"

Your app does lots of basic database operations which are often referred to as CRUD (Create, Read, Update, Delete). These operations do not want to happen over a network. They want to happen in a database which is local, on the mobile device.

And so you need sync. In the background (so the user doesn't have to wait on it). With automatic conflict resolution.

And that's hard.

Solutions that make sync easy

Do you like SQL? If so, I invite you to look at Zumero, our database sync platform based on SQLite, the relational database software preinstalled on over a billion mobile devices.

Do you prefer NoSQL? Then I'm sure our worthy competitors over at Couchbase would be happy for you to look at Couchbase Lite.

Either way, keep your CRUD off the Internet.

Announcing Zumero SQLite for Xamarin

2013-04-14T17:00:00+00:00

I am pleased to announce that Zumero SQLite is now available in the Xamarin Component Store. It's free, and it supports Android and iOS.

Zumero SQLite for Xamarin starts with the core "replicate and sync" functionality from the Zumero Client SDK and adds:

ADO.NET -- Familiar C# API for SQL developers. (provided by System.Data.SQLite, which is maintained by the same folks who develop SQLite itself)
Full-database encryption -- Critical piece of the "replicate and sync" story. Mobile devices get lost. (provided by the excellent SQLCipher library)
Painless setup -- Install the component and you've got everything you need. (provided by the neato component store feature of Xamarin Studio)

Comfy

In some ways, the core of the Zumero Client SDK is not very easy to use. That is intentional. It is designed to be the sort of SDK that leaves many things as the responsibility of the app (or the next layer up).

It doesn't actually include SQLite itself, since the app might care about which version of SQLite it wants, or how it needs to be compiled.
It doesn't include any page-level encryption support, since it doesn't know whether the app wants to use SQLCipher or the SQLite Encryption Extension or something else.
It plugs into SQLite using C, because that's the native language that SQLite uses for its API.

Power users and control freaks need this kind of SDK. And it comes first, since everything else is built on top of it. But most users want an experience that is more "comfy".

They don't want to think about versions of SQLite or compile options for same.
They don't want to choose an encryption library and figure out how to integrate it with their app.
They don't want to write their mobile app in C.

The Zumero Client SDK actually does include some carpeting and furniture on top of that cold, hard C floor. For Android developers using Java, it presents the android.database.sqlite API. And for iOS developers using Objective-C, it offers a wrapper that lets you work with things that have NS prefixes.

(Which reminds me, check out our iOS wiki app sample on GitHub.)

Anyway, these efforts are just the beginning. Our direction is toward "comfy", and we are moving fast.

This component for Xamarin is a step forward, and I think it is an important one.

Samples

This new component includes three sample apps, all of which are modified versions of samples provided by Xamarin.

Data (iOS, based on https://github.com/xamarin/monotouch-samples/tree/master/Data)
Notepad (Android, based on https://github.com/xamarin/monodroid-samples/tree/master/NotePad-Mono.Data.Sqlite)
Tasky (cross-platform, based on https://github.com/xamarin/mobile-samples/tree/master/Tasky)

The Data and Notepad samples have simply been ported to System.Data.SQLite (which involved very few changes).

The Tasky sample goes a bit further and adds a very simple ability to sync the task list.

Evolve

I am in Austin for Xamarin's Evolve conference, which starts today. Xamarin is doing some of the coolest stuff in mobile development today. We are thrilled to be a part of it.

Zumero: Background Sync in Objective-C

2013-04-09T14:31:00+00:00

Mobile offline RSS reader, Part 5

In the second “crossover episode” of our series (previous episodes begin on Eric’s blog), we look at background operations.

Previously, we built Zumero databases full of RSS lists and feed contents. The ZumeroReader sample app pulls, reads and displays those feeds on iOS devices. In action, it looks like this:

At startup, it pulls the feed list like so:

ZumeroDB *db = [[ZumeroDB alloc]
              initWithName:@"all_feeds"
              folder:path host:host];
db.delegate = self;

NSError *err = nil;
BOOL ok = YES;

if (! [db exists])
  ok = [db createDB:&err];

ok = ok && ([db isOpen] || [db open:&err]);

NSDictionary *scheme = nil;
NSString *uname = nil;
NSString *pwd = nil;

ok = ok && [db sync:scheme user:uname password:pwd error:&err];

We’re creating a Zumero database file named “all_feeds”. It will sync with a similarly-named dbfile on the server side.

We don’t need any authentication info, since (as previously discussed) the feed lists and feed databases are pullable by anyone.

The ZumeroDB::sync method always performs its network activity on a background thread, calling delegate methods when the action is completed (hence db.delegate = self).

In this case, on sync success, we grab the latest copies of the individual feed databases:

ok = [db selectSql:@"select feeds.feedid, url, title from feeds, about "
    "where (feeds.feedid = about.feedid)" values:nil
    rows:&rows error:&err];
// ...
  
for (NSDictionary *row in rows)
{
  NSNumber *id = [row objectForKey:@"feedid"];
  
  [self syncFeed:id];
}

// ...

- (void) syncFeed:(NSNumber *)feedid
{
  // ...
  NSString *dbname = [NSString stringWithFormat:@"feed_%@", feedid];
  ZumeroDB *db = [[ZumeroDB alloc] initWithName:dbname folder:path host:host];
  // ...
  db.delegate = self;
  ok = [db sync:nil user:nil password:nil error:&err];
  // ...
}

But what if feeds are updated while the app is running? There are seemingly infinite strategies for launching background tasks in iOS, and you’ll use the one that fits your application best. In this case, I cribbed a simple plan from a StackOverflow post - every time a touch is detected, we restart a timer. If that timer actually manages to expire, then we’ve seen no touches in 5 seconds.

- (void)sendEvent:(UIEvent *)event {
    [super sendEvent:event];
  
    NSSet *allTouches = [event allTouches];
    if ([allTouches count] > 0) {
        UITouchPhase phase = ((UITouch *)[allTouches anyObject]).phase;
        if (phase == UITouchPhaseBegan || phase == UITouchPhaseEnded)
            [self resetIdleTimer:maxIdleTime];
    }
}

- (void)resetIdleTimer:(NSTimeInterval)secs
{
    if (idleTimer) {
        [idleTimer invalidate];
      idleTimer = nil;
    }
  
    idleTimer = [NSTimer scheduledTimerWithTimeInterval:secs
               target:self selector:@selector(idleTimerExceeded)
               userInfo:nil repeats:NO];
}

When that timer fires, we check to see if its been 5 minutes since our last sync, and that a sync is wanted. If so, we sync again:

- (void)idleTimerExceeded {
  if (wantToSync)
  {
      NSTimeInterval since = [nextSync timeIntervalSinceNow];
      
      if (since <= 0)
      {
          wantToSync = FALSE;
          
          BOOL ok = FALSE;

          // the view controller's sync method from earlier          
          if (mvc)
              ok = [mvc sync];
          
          if (ok)
              self.networkActivityIndicatorVisible = YES;
          else
              // the sync call failed; try again later
              [self waitForSync:(10 * 60)];
      }
      else
      {
          // nope, check again next idle time
          [self resetIdleTimer:since];
      }
      
      return;
  }
  
  [self resetIdleTimer:maxIdleTime];
}

We also kill our timers when exiting, restart them when waking up or activating, etc.

Finally, when we go into the background, we try for one last sync before our process is suspended:

- (void)applicationDidEnterBackground:(UIApplication *)application
{
  // ...
  bgtask = [application beginBackgroundTaskWithExpirationHandler:^{
        [self endBackgroundTask:bgtask];
        bgtask = UIBackgroundTaskInvalid;
    }];
    dispatch_async(dispatch_get_global_queue(
      DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
      [mvc sync];
      // finishBackgroundTask will be called by the sync handlers
    });
}

Like any good sample app, there’s so much that this app doesn’t do. You’re invited to experiment with adding them.

We don’t maintain any read/unread information, either at the feed level or for individual items. This would be a good place to try creating/syncing new databases.
We naïvely grab the first <link> element we see and assume that it’s our permalink, without ever checking its rel attribute or content type.
A feed’s contents are displayed as one big chunk of HTML. Individual table cells might be nice.
We don’t make any attempt to become a long-running background process, since we don’t actually play audio, collect location information, etc. Might be fun to play with that.
Configuring the app is done by editing the source and rebuilding. That seems rude.
If you did update some sort of last-read database, wouldn’t that be a great moment to kick off a background sync? (The Wiki app included in the Zumero SDK does this when a page is created or saved)

Altering a Zumero Table

2013-04-09T14:26:00+00:00

Mobile offline RSS reader, Part 4

In part 3 of this series, Eric discussed three programs that do the back-end work of RSS aggregation - scanning feed lists, caching summaries and IDs, creating data files as we go.

Before we get into using that information in part 5, we need to adjust one of the schemas a bit.

The items table in each feed’s database is set up as:

CREATE VIRTUAL TABLE items
USING zumero
(
  id TEXT PRIMARY KEY NOT NULL,
  title TEXT NOT NULL,
  summary TEXT NOT NULL,
  pubdate_unix_time INTEGER NOT NULL
);

Where the id field comes from the feed items’s id or guid element, or from the first link found in the entry. In many RSS feeds, id will also be the permalink to the entry’s “destination”, e.g.

<item>
  <title>Enterprise mobile will have a lot of SQL going on</title>
  <guid>http://www.ericsink.com/entries/mobile_sql.html</guid>
  ...

But that’s not mandatory. For example:

<entry>
  <id>http://zumero.com/2013/04/04/just-released-the-zumero-development-server</id>
  <link type="text/html" rel="alternate"
   href="http://zumero.com/2013/04/04/just-released-the-zumero-development-server.html"/>
  <title>Just Released: the Zumero Development Server</title>
  ...

We still want that unique id, but if a link is available, we’ll want to store that, too.

Easy enough when creating a new items table. We’ll alter z_rss_create.cs to read:

db.Execute(
        @"CREATE VIRTUAL TABLE 
        cur.items 
        USING zumero
        (
          id TEXT PRIMARY KEY NOT NULL, 
          title TEXT NOT NULL,
          summary TEXT NOT NULL,
          pubdate_unix_time INTEGER NOT NULL,
          permalink TEXT
        );"
        );

But what about items tables that already exist? Unfortunately, SQLite does not allow alter table ... add column to run against a virtual table like Zumero’s. We’ll use the Zumero-provided alternative, zumero_alter_table_add_column().

In z_rss_update.cs, we’ll see if the new column already exists:

var query = "pragma cur.table_info(\"items\")";
List<SQLiteConnection.ColumnInfo> cols = db.Query<SQLiteConnection.ColumnInfo> (query);

foreach (var c in cols) {
    found = (string.Compare ("permalink", c.Name, StringComparison.OrdinalIgnoreCase) == 0);
    if (found)
        break;
}

If not…

db.Execute("BEGIN TRANSACTION;");
db.ExecuteScalar<string>(@"select zumero_alter_table_add_column(
   'cur', 'items', 'permalink TEXT');");
db.Execute("COMMIT TRANSACTION;");

// after altering a zumero table, the dbfile connection must be reopened

db.Execute("DETACH cur;", dbfile_name_for_this_feed);
db.Execute("ATTACH ? AS cur;", dbfile_name_for_this_feed);

See this fork of z_rss for all the necessary code.

In part 5, we’ll see a simple iOS RSS reader that pulls these databases in the background, and lets you browse the feed summaries and link through to the full articles.