Use firebase instead of socket.io


In Ephemeral Exchange I wanted to create a push notification capability so that subscribing apps could be informed immediately a cache item had changed or expired without having to poll for changes. The back end for cache is Redis, and that has a pub.sub capability, so one option would have been to open up redis to efx clients and have them listen for changes in the redis keyspace. 

However I have Redis locked down, and didn't like the idea of exposing it externally so I quickly canned that idea. Plan B was to create a push server that listened to changes by subscribing to the redis keyspace, and somehow informed subscribing clients that something had changed. I decided to go with Plan B.

Socket.io

This is a way to have apps talking to each other in real time, using websockets. I got that working fairly well, with the push server running on a Google Compute engine, and subscribing clients opening a socket to the push server. However, the production version of the push server runs on Google App Engine, which doesn't (easily) support socket.io, so after many wasted hours, I gave up on Socket.io.

Firebase

Firebase uses websockets to synchronize its database clients, and works well on App Engine, so I decided to use Firebase in a very ephemeral way to flag events from the push server, and for subscribing clients to listen to Firebase events on a key that identified their effex event subscription.

How it works


1. An app makes a request to the API like this to watch a particular item, passing the item id and a key that's authorized to access it.
        efx.on("update", keys.item, keys.updater, function (id, packet) {

            efx.read (keys.item)
            .then (function (result) {
                // do something - a change has been detected
            });
          }, {
          type: "push"
        });

2. The API  creates a key and makes an entry against this combination.

3. The push server monitors the Redis keyspace, and writes a log event for changes of interest. It also monitors for logevents, so that when a log item is successfully committed, it can check to see if there are any active entries that care about a specific item being changed.
// set up keyspace event monitoring - watching out for del, expired and set on data items and time syncing redisSubscribe_.config('set', 'notify-keyspace-events', 'Exg$z') .then(function() { // subscribe to datbase changes return redisSubscribe_.psubscribe('__keyevent@[' + st.db.client + st.db.log + st.db.ts + ']__:*'); }) .then(function() { redisSubscribe_.on('pmessage', function(pattern, channel, message) { var method = channel.replace(/.*__:/, ''); var db = parseInt(channel.replace(/.*@/, "").replace(/__.*/, ""), 10); var now = new Date().getTime(); // I'm logging all interesting events -- written data // but relying on watching for the logevent to be done before doing a push // an item changes if (db === st.db.client && st.watchable.events[method] && message.slice(0, st.itemPrefix.length) === st.itemPrefix) { ns.logEvent(method, message, "redis", now + timeOffset ); } // a log file records an event else if (db === st.db.log && method === "zadd" && message.slice(0, st.logPrefix.length) === st.logPrefix) { ns.pushEvent(message);

}

// a watchable expires - i use this to keep firebase clean
else if (db === st.db.watchable && method === "expired" && message.slice(0, st.watchablePrefix.length) === st.watchablePrefix) {
ns.cleanFirebase (message);
}

}); }); };
4. When a log event is detected, an active watch subscription item (that would have been created by the API in response to a .on request) that matches the logged event triggers the push notification process (this can be a webhook or various other methods but I'm looking at the real time push here). When found it makes a very small update to a firebase item. Actually just a key and a timestamp.
       if (sx.options.type === "push") {
              
              // fb doesnt allow $
              var fbKey = sxKey.replace ("$","___");
              // push to firebase and let him worry about it
              fb.set (fbKey + "/" + sx.options.uq, JSON.stringify(packet.value))
etc....

5. Here's what Firebase sees.


6. Back in the API, the .on action is triggered by the change in the Firebase item. 

// set listener for a specific key
ns.setOn = (key, func) => {
  // listen on any key
  var ref = key ? ns.baseRef.child(key) : ns.baseRef;

  // call the user function
  ref.on("value", function(snapshot) {
    var value = snapshot.val();
    if (value)func(value);
  });
};


7. Which  uses the keys it already knows to retrieve the metadata about which item has changed, and pass it to the user function

fb.setOn(fbKey + "/" + watch.options.uq, (value) => {
  // for security, the public firebase db only contains a watchable key, and the values that provoked it
  // so i need to get that watch item
  ns.getWatchable(result.data.watchable, key)
    .then((w) => {
      if (!w.data.ok) {
        throw JSON.stringify(w.data);
      }
      else {
        var p = w.data.value;
        // fb doesnt like arrays, so we've stingified it
        p.value = JSON.parse(value);
        callback(result.data.watchable, p);
      }
    })});
Here's a summary of the whole thing.

Expiring


One of the reasons I use Redis as the back end for the cache is that it has an expiration mechanism. If Firebase had that, then I may have decided to use Firebase for the whole thing. But now I have some update notification small items in Firebase that won't got away unless I delete them. To deal with that I simply watch for the original on request expiring in Redis , and delete them from firebase when they do.
// a watchable expires - i use this to keep firebase clean else if (db === st.db.watchable && method === "expired" && message.slice(0, st.watchablePrefix.length) === st.watchablePrefix) { ns.cleanFirebase (message); }

And here's a video of 5 platforms talking to each other using this technique.








For more like this, see React, redux, redis, material-UI and firebase. Why not join our forum, follow the blog or follow me on twitter to ensure you get updates when they are available.
Comments