Upstash as an Apps Script cache platform

Upstash is a brand new service offering a serverless redis over https via a GraphQL API. Previously redis was hard to use along with Apps Script since we can’t naturally get to it with UrlFetchApp. In 3 Favourite things in one article: redis, apps script and graphQl I introduce a library to use Upstash from Apps Script, and now here’s a plugin for Apps script library with plugins for multiple backend cache platforms so we can use Upstash as a back end for caching large objects across platforms.

Benefits of using Upstash

GraphQL makes the interface really straightforward to implement and extend, and of course since redis is at the backend we can access our Upstash database from anything that can access GraphQL or a redis client. It has a pretty decent free tier to get started, and the only issue I’ve found so far is that the maximum object size it can write is 400k on the free plan, and 1mb on the paid plan. This is where bmCrusher comes in as it compresses and spreads objects across multiple records.

How to use

I like Upstash a lot, so I’ve actually added this plugin to the standard plugins for cacheservice, propertyservice and Drive that are already built in to bmCrusher, but I’ll go through how it’s built here in any case.

Just as in the other examples, it starts with a store being passed to the crusher library to be initialized. The plugin is used in exactly the same way as the Google platform plugins – like this. For details on how to sign up to upstash and get an access key for GraphQL see 3 Favourite things in one article; redis, apps script and graphQl

  const crusher = new bmCrusher.CrusherPluginUpstashService().init({
    tokenService: () => PropertiesService.getUserProperties().getProperty('usrwkey'),
    prefix: '/crusher/store',
    fetcher: UrlFetchApp.fetch
  })

Initialize the crusher

The plugin

We’ll be using the bmUpstash library so there’s not much to do. This plugin is already implemented in the bmCrusher library so you don’t need to do any of this, but I reproduce it here in case you are interested in seeing how to write a bmCrusher plugin

// plugins for Squeeze service 
function CrusherPluginUpstashService() {

  const self = this;

  // these will be specific to your plugin
  let _settings = null;


  // prefixs on redis can be any string but
  // make sure we start and end with a single slash for consistency
  const fixPrefix = (prefix) => ((prefix || '')   '/').replace(/^\/ /, '/').replace(/\/ $/, '/')

  // standard function to check store is present and of the correct type
  const checkStore = () => {
    if (!_settings.chunkSize) throw "You must provide the maximum chunksize supported";
    if (!_settings.prefix) throw 'The prefix must be the path of a folder eg /crusher/store';
    if (!_settings.tokenService || typeof _settings.tokenService !== 'function') throw 'There must be a tokenservice function that returns an oauth token';
    if (!_settings.fetcher || typeof _settings.fetcher !== 'function') throw 'There must be a fetch function that can do a urlfetch (url,options)';
    return self;
  }

  /**
   * @param {object} settings these will vary according to the type of store
   */
  self.init = function (settings) {

    _settings = settings || {};

    // the settings are the same as the crusher settings
    _settings.store = {
      ug: bmUpstash.gqlRedis({
        fetcher: _settings.fetcher,
        tokenService: _settings.tokenService,
      })
    }

    // make sure we start and end with a single slash
    _settings.prefix = fixPrefix(_settings.prefix)

    // upstash supports value sizes of up to 1mb - but actually it doesn't work above 400k for now.
    // see - https://github.com/upstash/issues/issues/3
    _settings.chunkSize = _settings.chunkSize || 400000;

    // respect digest can reduce the number of chunks read, but may return stale
    _settings.respectDigest = Utils.isUndefined(_settings.respectDigest) ? false : _settings.respectDigest;

    // must have a cache service and a chunksize, and the store must be valid
    checkStore();

    // now initialize the squeezer
    self.squeezer = new Squeeze.Chunking()
      .setStore(_settings.store)
      .setChunkSize(_settings.chunkSize)
      .funcWriteToStore(write)
      .funcReadFromStore(read)
      .funcRemoveObject(remove)
      .setRespectDigest(_settings.respectDigest)
      .setCompressMin(_settings.compressMin)
      .setPrefix(_settings.prefix);

    // export the verbs
    self.put = self.squeezer.setBigProperty;
    self.get = self.squeezer.getBigProperty;
    self.remove = self.squeezer.removeBigProperty;
    return self;

  };

  // return your own settings
  self.getSettings = () => _settings;


  /**
   * remove an item
   * @param {string} key the key to remove
   * @return {object} whatever you  like
   */
  const remove = (store, key) => {
    checkStore();
    return store.ug.execute('Del', key)
  }

  /**
   * write an item
   * @param {object} store whatever you initialized store with
   * @param {string} key the key to write
   * @param {string} str the string to write
   * @param {number} expiry time in secs .. ignored in drive
   * @return {object} whatever you like
   */
  const write = (store, key, str, expiry) => {
    checkStore();
    const result = !expiry ? store.ug.execute('Set', key, str) : store.ug.execute('SetEX', key, str, expiry)
    if (result !== 'OK') throw new Error('failed to set value for key', key)
    return result
  }

  /**
   * read an item
   * @param {object} store whatever you initialized store with   
   * @param {string} key the key to write
   * @return {object} whatever you like
   */
  const read = (store, key) => {
    checkStore();
    return store.ug.execute('Get', key)
  }


}

upstash bmcrusher plugin

We’ll be using the exact same methods introduced in Apps script library with plugins for multiple backend cache platforms will now operate on Upstash instead of the other platforms. Here’s a refresher

Writing

All writing is done with a put method.

crusher.put(key, data[,expiry])

write some data

put takes 3 arguments

key – a string with some key to store this data against
data – It automatically detects converts to and from objects, so there’s no need to stringify anything.
expiry – optionally provide a number of seconds after which the data should expire.

Reading

const data = crusher.get (key)

retrieving data

get takes 1 argument

key – the string you put the data against, and will restore the data to it’s original state

Removing

crusher.remove(key)

removing an item

Expiry

Upstash has expiry built in so if you add an expiry value when you create an item it will be automatically removed in time.

Here’s what some store entries look like in the Upstash GraphQL explorer. You can see that one of the entries has been distributed across multiple keys to deal with the maximum value size in Upstash. Getting the value will restore it to its original state.

Fingerprint optimization

Since it’s possible that an item will spread across multiple physical records, we want a way of avoiding rewriting (or decompressing) them if nothing has changed. Crusher keeps a fingerprint of the contents of the compressed item. When you write something and it detects that the data you want to write has the same fingerprint as what’s already stored, it doesn’t bother to rewrite the item.

However if you’ve specified an expiry time, then it will be rewritten so as to update its expiry. There’s a catch though. If your chosen store supports its own automatic expiration (as in the CacheService), then the new expiration wont be applied. Sometimes this behavior is what you want, but it does mean a subtle difference between different stores.

You can disable this behavior altogether when you initialize the crusher.


 const crusher = new bmCrusher.CrusherPluginUpstashService().init({
    tokenService: () => PropertiesService.getUserProperties().getProperty('usrwkey'),
    prefix: '/crusher/store',
    fetcher: UrlFetchApp.fetch,
	respectDigest: false
  })

Always rewrite store even if the data has not changed

Formats

Crusher writes all data as zipped base64 encoded compressed, so the mime type will be text, and will need to be read by bmCrusher to make sense of it.