Using the Cloud APIs to Sync Email

 

How Nylas syncs mail, contacts, and calendars

When an account is connected to Nylas, the sync engine starts pulling in all mail for the account, prioritizing recent messages first (though mail isn’t strictly added in reverse chronological order). As new email arrives in the user’s mailbox, it is also added in parallel.

For contacts and calendar syncing, events and contacts are downloaded from newest to oldest. New events or contacts added after the start of the sync are not downloaded until the initial sync is complete. Since the volume of data for contacts and calendar is typically not large, this isn’t generally a problem.

Nylas continues to keep its data up-to-date with the backend provider through a number of mechanisms: keeping an IMAP IDLE connection open, using Exchange ActiveSync’s “ping” notifications, using webhooks from calendar providers, or, in the case that no better mechanism is available, simply polling the provider.

Syncing Strategies

Overview

There are two main ways to pull in email information from an account:

  • The Threads and Messages endpoints allows you retrieve all messages and threads, or filter for a certain subset of them based on various constraints.
  • Deltas allow you to process new data quickly without having to fetch an index of the user’s mailbox or perform a large number of API calls.

You shouldn’t make any assumptions about the sync progress of an account. We recommend using both mechanisms to ensure you’re able to quickly provide data to a user when the account starts syncing (through the threads endpoint), and to quickly update the user when new mail comes through (through deltas). There are other caveats you should keep in mind that are discussed below.

Threads and Messages

In some cases, it’s easiest to request data from the Nylas API as it’s needed for display, without ever needing to store it locally. The Threads and Messages endpoints are great for this use case.

For example, if your application provides a custom “Inbox” view, you could fetch data from the /threads endpoint and use it to render HTML or provide data to a single page JavaScript application. Making /threads requests on-demand rather than caching the data in a local database makes it easier to ensure the data is always current, and it’s easy to filter messages too.

Note: When you first connect an account and fetch data from the /threads endpoint you won’t receive all of a user’s email if Nylas is still syncing it. You can’t make any assumptions about whether an account has finished syncing on Nylas’ servers, and it can take anywhere from a few minutes to several days depending on the account size.

Tip: You could use the delta streaming purely as a notification method. New delta comes in? Just request the thread endpoint again for new messages. (Don’t forget to stay within our rate limits!)

Deltas

The Nylas Sync Engine builds a transaction log that records every change as it synchronizes users’ mailboxes. Your application can use this transaction log, exposed through the Delta APIs, to build email applications that process incoming changes quickly, without fetching an index of the user’s mailbox, polling, or making a large number of API requests.

The Delta API documentation describes the endpoint, request parameters, and response objects in detail. It also describes the difference between streaming and long polling and when you would typically use each strategy.

If your application wants to ingest emails as they are processed by the Nylas Sync Engine, the delta stream is an efficient solution.

 

Deltas + Threads

By themselves, the Threads and Deltas APIs aren’t sufficient to build a full, realtime view of a user’s mailbox, since:

  • Threads only return mail that has already been processed by Nylas. If a user has just connected their account, a request to /threads might only return a few Thread objects that Nylas has processed. A few hours later, /threads might return many thousand objects since more mail has been processed.
  • Deltas only return mail as it’s being processed by Nylas. Listening to deltas tells your application that mail is available which was was not previously synced or has been modified.

To build a robust application that maintains an up-to-date cache of a user’s entire mailbox, you should leverage both Threads and Deltas. Be sure to:

  • Begin listening to deltas (changes to the mailbox) at the same time you start paginating /threads (existing data in the mailbox.)
  • Always “upsert” when you retrieve objects. A delta about a thread should create or update that thread in your cache.

If your application is only interested in caching a subset of mail—for example, mail in the last month, or mail in a particular folder—you should:

  • Limit pagination using threads filter parameters
  • Listen for deltas, but ignore deltas about mail you do not care about.

In general, you should obtain the cursor first before paginating through /threads to ensure you don’t miss any emails.

  1. Authorize and account
  2. Obtain a cursor
  3. Start listening for deltas
  4. Paginate /threads (remember to account for duplicates)

Managing multiple account syncs

The Delta API documentation goes into detail on when to use long polling versus streaming to call the delta endpoint. Depending on what strategy you use and your software stack, the number of parallel processes or open connections when syncing a large number of accounts may be a concern.

In our experience, clients have been successful using the Delta API with large numbers of accounts using the following asynchronous frameworks:

 

Using Deltas - Ruby SDK Example

The delta sync API allows fetching all the changes that occurred after a specific time. Read this for more details about the API.

# Get an API cursor. Cursors are API objects identifying an individual change.
# The latest cursor is the id of the latest change which was applied
# to an API object (e.g: a message got read, an event got created, etc.)
cursor = nylas.latest_cursor

last_cursor = nil
nylas.deltas(cursor) do |event, object|
  if event == "create" or event == "modify"
    if object.is_a?(Nylas::Contact)
      puts "#{object.name} - #{object.email}"
    elsif object.is_a?(Nylas::Event)
      puts "Event!"
    end
  elsif event == "delete"
    # In the case of a deletion, the API only returns the ID of the object.
    # In this case, the Ruby SDK returns a dummy object with only the id field
    # set.
    puts "Deleting from collection #{object.class.name}, id: #{object}"
  end
  last_cursor = object.cursor
end

# Don't forget to save the last cursor so that we can pick up changes
# from where we left.
save_to_db(last_cursor)

Using the Delta sync streaming API

The streaming API will receive deltas in real time, without needing to repeatedly poll.

MRI

MRI uses EventMachine for async IO.

cursor = nylas.latest_cursor

EventMachine.run do
  nylas.delta_stream(cursor) do |event, object|
    if event == "create" or event == "modify"
      if object.is_a?(Nylas::Contact)
        puts "#{object.name} - #{object.email}"
      elsif object.is_a?(Nylas::Event)
        puts "Event!"
      end
    elsif event == "delete"
      # In the case of a deletion, the API only returns the ID of the object.
      # In this case, the Ruby SDK returns a dummy object with only the id field
      # set.
      puts "Deleting from collection #{object.class.name}, id: #{object}"
    end
  end
end

To receive streams from multiple accounts, call delta_stream for each of them inside an EventMachine.run block.

api_handles = [] # a list of Nylas::API objects

EventMachine.run do
  api_handles.each do |a|
    cursor = a.latest_cursor()
    a.delta_stream(cursor) do |event, object|
      puts object
    end
  end
end

JRuby

The JRuby implementation uses the Simple JSON Streaming gem instead of EventMachine and YAJL. No need for the EventMachine.run block.

cursor = nylas.latest_cursor

nylas.delta_stream(cursor) do |event, object|
  if event == "create" or event == "modify"
    if object.is_a?(Nylas::Contact)
      puts "#{object.name} - #{object.email}"
    elsif object.is_a?(Nylas::Event)
      puts "Event!"
    end
  elsif event == "delete"
    # In the case of a deletion, the API only returns the ID of the object.
    # In this case, the Ruby SDK returns a dummy object with only the id field
    # set.
    puts "Deleting from collection #{object.class.name}, id: #{object}"
  end
end

To receive streams from multiple accounts, call delta_stream for each of them inside an EventMachine.run block.

api_handles = [] # a list of Nylas::API objects

api_handles.each do |a|
  cursor = a.latest_cursor()
  a.delta_stream(cursor) do |event, object|
    puts object
  end
end

Exclude changes from a specific type — get only messages

nylas.deltas(cursor, exclude=[Nylas::Contact,
                              Nylas::Event,
                              Nylas::File,
                              Nylas::Tag,
                              Nylas::Thread]) do |event, object|
  if ['create', 'modify'].include? event
    puts object.subject
  end
end

Expand Messages from the Delta stream

It’s possible to ask the Deltas and delta stream API to return expanded messages directly:

nylas.deltas(cursor, exclude=[Nylas::Contact,
                              Nylas::Event,
                              Nylas::File,
                              Nylas::Tag,
                              Nylas::Thread], expanded_view=true) do |event, object|
  if ['create', 'modify'].include? event
    if obj.is_a?(Nylas::Message)
      puts obj.subject
      puts obj.message_id
    end
  end
end

 

Using Deltas - Node SDK Example

var DELTA_EXCLUDE_TYPES = ['contact', 'calendar', 'event', 'file', 'tag'];
var nylas = Nylas.with(accessToken);

nylas.deltas.latestCursor(function(error, cursor) {

  // Save inital cursor.
  persistCursor(cursor);

  // Start the stream and add event handlers.
  var stream = nylas.deltas.startStream(cursor, DELTA_EXCLUDE_TYPES);

  stream.on('delta', function(delta) {
    // Handle the new delta here.
    console.log('Received delta:', delta);
    // Save new cursor so this delta doesn't need to be re-fetched for future streams.
    persistCursor(delta.cursor);

  }).on('error', function(err) {
    // Handle errors here, such as by restarting the stream at the last cursor.
    console.error('Delta streaming error:', err);

  });

  // Closing the stream explicitly, if needed
  stopButton.addEventListener('click', function() {
    stream.close();
  });
})
Have more questions? Submit a request

Comments