Use firebase instead of socket.io

In Ephemeral Exchange I wanted to create a push notification capability so that subscribing apps could be informed immediately a cache item had changed or expired without having to poll for changes. The back end for cache is Redis, and that has a pub.sub capability, so one option would have been to open up redis to efx clients and have them listen for changes in the redis keyspace.

However I have Redis locked down, and didn’t like the idea of exposing it externally so I quickly canned that idea. Plan B was to create a push server that listened to changes by subscribing to the redis keyspace, and somehow informed subscribing clients that something had changed. I decided to go with Plan B.

This is a way to have apps talking to each other in real time, using websockets. I got that working fairly well, with the push server running on a Google Compute engine, and subscribing clients opening a socket to the push server. However, the production version of the push server runs on Google App Engine, which doesn’t (easily) support socket.io, so after many wasted hours, I gave up on Socket.io.

Firebase uses websockets to synchronize its database clients, and works well on App Engine, so I decided to use Firebase in a very ephemeral way to flag events from the push server, and for subscribing clients to listen to Firebase events on a key that identified their effex event subscription.

efx.on("update", keys.item, keys.updater, function (id, packet) {

            efx.read (keys.item)
            .then (function (result) {
                // do something - a change has been detected
            });
          }, {
          type: "push"
        });

// set up keyspace event monitoring - watching out for del, expired and set on data items and time syncing
    redisSubscribe_.config('set', 'notify-keyspace-events', 'Exg$z')
      .then(function() {
        // subscribe to  datbase changes
        return redisSubscribe_.psubscribe('__keyevent@[' + st.db.client + st.db.log + st.db.ts + ']__:*');
      })
      .then(function() {

        redisSubscribe_.on('pmessage', function(pattern, channel, message) {

          var method = channel.replace(/.*__:/, '');
          var db = parseInt(channel.replace(/.*@/, "").replace(/__.*/, ""), 10);
          var now = new Date().getTime();

          // I'm logging all interesting events -- written data
          // but relying on watching for the logevent to be done before doing a push

          // an item changes
          if (db === st.db.client && st.watchable.events[method] && message.slice(0, st.itemPrefix.length) === st.itemPrefix) {
            ns.logEvent(method, message, "redis", now + timeOffset );
          }
          
          // a log file records an event
          else if (db === st.db.log && method === "zadd" && message.slice(0, st.logPrefix.length) === st.logPrefix) {
            ns.pushEvent(message);
          }
 

         // a watchable expires - i use this to keep firebase clean
         else if (db === st.db.watchable && method === "expired" && message.slice(0, st.watchablePrefix.length) === st.watchablePrefix) {
           ns.cleanFirebase (message);
         }

          
         
        });
      });

  };

4. When a log event is detected, an active watch subscription item (that would have been created by the API in response to a .on request) that matches the logged event triggers the push notification process (this can be a webhook or various other methods but I’m looking at the real time push here). When found it makes a very small update to a firebase item. Actually just a key and a timestamp.

 if (sx.options.type === "push") {
              
              // fb doesnt allow $
              var fbKey = sxKey.replace ("$","___");
              // push to firebase and let him worry about it
              fb.set (fbKey + "/" + sx.options.uq, JSON.stringify(packet.value))
etc....

5. Here’s what Firebase sees.

6.Back in the API, the .on action is triggered by the change in the Firebase item.

// set listener for a specific key
ns.setOn = (key, func) => {
  // listen on any key
  var ref = key ? ns.baseRef.child(key) : ns.baseRef;

  // call the user function
  ref.on("value", function(snapshot) {
    var value = snapshot.val();
    if (value)func(value);
  });
};

7.Which  uses the keys it already knows to retrieve the metadata about which item has changed, and pass it to the user function

b.setOn(fbKey + "/" + watch.options.uq, (value) => {
  // for security, the public firebase db only contains a watchable key, and the values that provoked it
  // so i need to get that watch item
  ns.getWatchable(result.data.watchable, key)
    .then((w) => {
      if (!w.data.ok) {
        throw JSON.stringify(w.data);
      }
      else {
        var p = w.data.value;
        // fb doesnt like arrays, so we've stingified it
        p.value = JSON.parse(value);
        callback(result.data.watchable, p);
      }
    })});

Here’s a summary of the whole thing.

 

// a watchable expires - i use this to keep firebase clean
          else if (db === st.db.watchable && method === "expired" && message.slice(0, st.watchablePrefix.length) === st.watchablePrefix) {
            ns.cleanFirebase (message);
          }