My server times are out of sync!

When you are dealing with an app that uses several different servers, and which relies on timestamps for workflow activities, it’s very hard to understand what’s going on when the server times are out of sync, as they so often are. It’s even worse when you move from one environment to another, so you have to figure out a way of compensating for these time sync differences.

In my Ephemeral Exchange cross platform cache, push notification of changes is managed by a server that is not the same as the one used by the API to create subscription notifications. This means that you can’t be sure if a notification is required for events that happen very close to the registration activity.

How events are detected

I use Redis as a back end, and the API writes data to a Redis store from the regular API endpoint. If a request for subscription to push notification is requested, and entry is made in the Redis store that subscriber ‘A’ wants to know it anything happens to item ‘X’.

Every time an item is updated, expires or is deleted is creates an entry with various details in a log in a different Redis store. The push server is subscribed to that Log store and is woken up every time a log event happens. It then checks to see which Effex clients are interested in knowing about that item and pushes the notification using whichever notification message it has specified as its preference, and also keeps track of which notifications it has already been sent. Like this, a client can ask for repeat historical information on events before it actually started watching that item.

The problem

If the effex server is out of sync (and I’ve found that it can be as much as 3 seconds out) with the push server, then the log event of the item being updated can seem to happen (according to their timestamps) before the item is updated, or before the item started to be watched. This will also compensate for logging delays or some network delays.

The workaround

There are a number of solid solutions out there for syncing time between servers, but if you are using a shared environment over which you don’t have control, a simple way is just to compensate for the observed differences, applying a little more weight to the more recent events. This is not 100% perfect, but works just fine.

var timeOffset = 0, timeNobs = 0, smooth = .1;
  // when the log event is detected
       var now = new Date().getTime();
          
  // get the actual item    
  redisWatchable_.get (key)
    .then (function(ob) {
       // when it was created
       var then = ob.created;

       // versus when it was logged
       // adjust the average difference between the times, but give more weight to the latest observations
        timeOffset = timeNobs ? (then-now)*smooth + timeOffset *(1 - smooth) : then - now;
        timeNobs ++;
    });

You can then use timeOffset to apply to the logtime to get the approximate creation time. You can see how it smooths out variable results below
The amount of smoothing is controlled by the smooth variable – to keep the trend fairly free of outliers, I use a small value (0.1).
To follow the ups and downs a little more closely, using a bigger number like 0.5
A smoothing factor of 1 does no smoothing at all, and of course a number > 1 exaggerates. Math is a wonderful thing.

For more like this, see React, redux, redis, material-UI and firebase. Why not join our forum, follow the blog or follow me on twitter to ensure you get updates when they are available.