I’ve been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things Server side. All of these modes need to share state, and it always ends up rather messy. I though I’d take some inspiration from Redux and Vuex and apply those techniques to centralize and simplify state management from the business going on in each of these separate but co-operating environments

Background

This Add-on is actually a Sheets Add-on. It takes documents (images, pdfs etc) and runs them through the DocumentAI API, using ML to decipher invoices and other kinds of material and load the results into a sheet. It’s neither finished nor published yet, but I’m going to take this opportunity to get my thoughts about state management down while I’m still working on it.

Here’s an approximation of what the Add-on looks like in operation at this point.

State management complexities

Its always been a little challenging sharing state cleanly between HTMLService based Add-ons and Server side functions, but the added dimension of Cardservice needs some special attention. When the contents of a card changes you need to recreate the card (as opposed to just a component of it), so all your state is going to be messed up. People usually use Property service to manage this, but Property service calls peppered around the card service code is error prone and mind bending.

In this Add-on, I also needed a filePicker, and CardService doesn’t have one. Luckily I had a pre-baked one from another Add-on (slidesmerge) so I just needed to resurrect that and fire it off as a webapp from a CardService component. However, how to cleanly get the results from the filepicker back over to the re-rendered card service without adding to complexity?

State management needs

The types of state I need to manage and share are

Add-on global settings – Script property service to persist across sessions for all users
Add-on user settings – User property service to persist across sessions for each user
CardService component values – User propety service to persist across sessions for each user
Results from HTML service – User Cache service to persist for a little while, and usually provoked as the result of changes to Card Service components
Large scale results from Server Side API calls – User Cache to persist for a little while, and usually provoked from the combination of HTML service results and CardService component values
Archived results – Cloud Storage – persist forever to avoid duplicate processing of the same file with the same parameters that woudl give the same result as a previous run

Another 2 challenges specific to this API, is that calling the DocumentAI is neither free, nor especially cheap – so I wanted to ensure that if any call was made to the documentAI it was truly for a new document I’d never seen before – so it’s more cost effective (and efficient) to store all results to Cloud storage, indexed by the md5Hash of a file along with its processing parameters, in addition to the short term version held in cache. (as an aside, these results can be quite hefty – so I’m using Apps script caching with compression and enhanced size limitations to write any size of data to Apps Script cache). It’s also used in SuperFetch caching: How does it work?

Debugging

Another requirement is to make debugging these environments a little easier. It occurred to me that if i centralized all state management and wrapped each call to these various stores not only could I write a script to examine exactly what was in store at any given time, even when the Add-on was actually active, but i could also log and keep a record of state changes.

State management solution

First of all, a simple Store Class to wrap property store gets and sets

/**
 * @Class Store
 * GWAO uses property store a lot to persist values through instantiation
 * this class provides a simpel interface for that
 */
class Store {
  /**
   * @constructor
   * @param {PropertiesStore} [store=userProperties] the store to use
   * @param {string} [prefix=''] prefix to segregate entries from anything else in the prop store if required
   * @return {Store} 
   */
  constructor({store,prefix= '',log= false}) {
    this.store = store || PropertiesService.getUserProperties()
    this.prefix = prefix
    this.log = log
  }

  /**
   * to make stuff in property store more versatile we'll convert it to an object and stringify it
   * @param {*} ob
   * @return string
   */
  stringify(ob) {
    return JSON.stringify({
      ob 
    })
  }

  /**
   * to make stuff in property store more versatile we'll convert it to an object and stringify it
   * this undoes that
   * @param {string} value the value from store
   * @return {*}
   */
  unstringify(value) {
    try {
      const {ob} = JSON.parse(value)
      return typeof ob === typeof undefined ? value: ob
    }
    catch (err) {
      return value
    }
  }

  /**
    * make a key with the prefix
    * @param {string} key store agains this key
    * @return {string} prefixed key
    */
  getKey (key) {
    return this.prefix ? (this.prefix   '_'   key) : key
  }

  /**
    * put to property store
    * @param {string} key store agains this key
    * @param {*} value thing to write
    */
  set (key, value) {
    const k = this.getKey(key)
    if (this.log) console.log('storelog','setting',k, value)
    return this.store.setProperty(k, this.stringify(value))
  }

  /**
   * @param {string} key stored agains this key
   * @return {string} the value
   */
  get (key) {
    const k = this.getKey(key)
    const value = this.unstringify(this.store.getProperty(k))
    if (this.log) console.log('storelog','getting',k, value)
    return value
  } 

  /**
   * @param {string} key stored agains this key
   */
  delete(key) {
    const k = this.getKey(key)
    if (this.log) console.log('storelog','removing',k)
    return this.store.deleteProperty(k)
  } 
}

Store class

Next an AppStore, where all state is handled

/**
 * this the state management central for this app
 * since it uses server, html service and cardservice
 * all state management and communication between them is handled here
 */
var AppStore = {

  // set this to true for debugging to see state interactions being logged
  get log() {
    return false
  },

  // use for large, ephemeral state values such as immediate results
  get userCache() {
    const log = this.log
    return new Cacher({
      cachePoint: CacheService.getUserCache(),
      expiry: 60 * 60 * 12,
      stale: false,
      staleKey: 'docai-stale',
      log
    })
  },

  // properties stores to use for small, persistent state values
  get user() {
    const log = this.log
    return new Store({
      store: PropertiesService.getUserProperties(),
      log
    })
  },
  get script() {
    const log = this.log
    return new Store({
      store: PropertiesService.getScriptProperties(),
      log
    })
  },

  // any specific user settings can be stored here
  set settings(value) {
    this.user.set('settings', value)
  },
  get settings() {
    return this.user.get("settings")
  },

  // select the location of documentAI processor to use
  get locationId() {
    return this.user.get('locationId')
  },
  set locationId(value) {
    this.user.set('locationId', value)
  },

  // select the type of documentAI processor to use
  get type() {
    return this.user.get('type')
  },
  set type(value) {
    this.user.set('type', value)
  },

  // select the documentAI processor to use
  get displayName() {
    return this.user.get('displayName')
  },
  set displayName(value) {
    this.user.set('displayName', value)
  },

  // these are the locations currently the addon supports from the documentAI api
  get locationList() {
    return ["us", "eu"].map(f => ({ locationId: f }))
  },

  // the folder at which to start picking
  get pickerRoot() {
    return this.user.get('pickerRoot') || 'root'
  },
  set pickerRoot(id) {
    this.user.set('pickerRoot', id)
  },

  // the apps script token needs to be scoped for all the functions of the add-on, server side functions, and picker
  get token() {
    return ScriptApp.getOAuthToken()
  },

  // this will need to be set in the project script properties manually and is the api key for the picker
  get pickerDeveloperKey() {
    return this.script.get('pickerDeveloperKey')
  },

  // this will need to be set in the project script properties manually and is the current project number
  // dont have a way to pick this up automatically
  // eventually this can be enhanced to step outside the current apps script project, but will need IAM teaking for proxy token
  get projectNumber() {
    return this.script.get('projectNumber')
  },


  // this is the p ublished webapp required to host the filepicker
  get serviceUrl() {
    return ScriptApp.getService().getUrl()
  },

  // for reporting where we're up to
  set workingOn(file) {
    this.userCache.set('workingOn', file.md5)
  },
  get workingOn() {
    const picked = this.pickedFiles
    if (!picked) return null

    const md5 = this.userCache.get('workingOn')
    if (!md5) return null

    // find in currently picked files
    return picked.find(f => f.md5 === md5)
  },


  // picked files are by selected processor type
  // this allows a change of procoessor withot needing to repick the files
  get pickedFilesKeyOptions() {
    return this.type
  },
  get pickedFiles() {
    const options = this.pickedFilesKeyOptions
    return options ? this.userCache.get('pickedFiles', options) : null
  },
  set pickedFiles(value) {
    const options = this.pickedFilesKeyOptions
    if (options) {
      this.userCache.set('pickedFiles', value, {
        options
      })
    }
  },
  delPickedFiles() {
    const options = this.pickedFilesKeyOptions
    if (options) {
      this.userCache.remove('pickedFiles', options)
    }
  },


  // the key for the results is based on the md5 of the picked files   the selected processor
  get processedResultsKeyOptions() {
    // if any of these are missing there can be no valid entry
    const files = this.pickedFiles
    if (!files) return null

    const selectedProcessor = this.selectedProcessor
    if (!selectedProcessor) return null

    // this will be added to the key to make sure we pick up the results that match the current selections
    return {
      md5s: files.map(f => f.extras.md5),
      selectedProcessor
    }
  },
  get processedResults() {
    // this'll make sure we get the results for the correct state
    const options = this.processedResultsKeyOptions
    if (!options) return null
    return this.userCache.get('processedResults', options)
  },
  set processedResults(value) {
    const options = this.processedResultsKeyOptions
    if (options) {
      this.userCache.set('processedResults', value, {
        options
      })
    }
  },

  // the current processor can be discovered from the list of known processors and the selected displayname
  set processors(value) {
    this.userCache.set('processors', value)
  },
  get processors() {
    return this.userCache.get('processors')
  },
  get selectedProcessor() {
    const { processors, displayName } = this
    return processors && displayName ? processors.find(f => f.displayName === displayName) : null
  }
}

App Store

Finally, a client side Provoker to get and set state all the state values from within html service. This is an enhancement of the technique i use in How to use Vue.js, Vuex and Vuetify to create Google Apps Script Add-ons

/**
 * this should contain a list of all the functions authorized
 * to be run from the client 
 * for V8, it's important that it's a namespace ief to ensure its parsed in the correct order
 * and is available throughout the project on initialization
 */
const WhitelistedActions = (() => {
  const ks = new Map()
  // whitelist everything that can be run
  ks.set('get-picker-developerKey', () => AppStore.pickerDeveloperKey)
  ks.set('get-token', () => AppStore.token)
  ks.set('set-picked-files', (value) => AppStore.pickedFiles = value)
  ks.set('set-picker-root', (value) => AppStore.pickerRoot = value)
  ks.set('get-picker-root', () => AppStore.pickerRoot)
  ks.set('get-picked-files', () => AppStore.pickedFiles)
  ks.set('del-picked-files', () => AppStore.delPickedFiles)
  return ks
})();
/**
 * run something from the whitelist
 * will generall be invoked by google.script.run
 * @param {string} name the key of the action to run in the whilelistedActions store
 * @param {...*} args any arguments for the action
 * @return {*} the result
 */
const runWhitelist = (name, ...args) => {
  // get what to run from the store
  const action = WhitelistedActions.get(name)
  if (!action) {
    throw new Error(`${name} is not in the list of actions that can be run from the client`)
  }
  return action(...args)
}

provoke

Using the AppStore

Now I don’t need to worry about the details of how state is maintained.

For example, to see what files have been selected by the latest pick action, I can just run this – even while the Add-on is active

 console.log(AppStore.pickedFiles)

In the card service, I can do this kind of thing –

Getting a currently selected value

    const decoratedText = CardService.newDecoratedText()
      .setText(AppStore.projectNumber)
      .setTopLabel("Project number containing DocumentAI processors");

getting a value

Setting the result of an api call

    // get a new document AI - locationId & projectNumber are in the store
    const dai = new DocumentAI({ locationId: AppStore.locationId, projectNumber: AppStore.projectNumber })
    const { processors, nextPageToken } = dai.listProcessors({ noCache: true })

    const result = processors.filter(f => f.state === 'ENABLED')
    AppStore.processors = result

setting the result of an api call

Mixing settings, tokens and component values

  const newDai = () => {
    const token = AppStore.token
    return new DocumentAI({
      tokenService: () => token,
      projectNumber: AppStore.projectNumber,
      locationId: AppStore.locationId,
      gcsOutputPath: AppStore.gcsOuputPath
    })
  }

mixing it up

Dealing with a value change in a component

  /**
   * called when a selection is modified
   * @param {CardAction.event} e
   * @param {object} e.parameters the parameters passed to the handler
   * @param {object} e.formInput the other values in the form
   * @param {string} e.parameters.key the forminput key that caused the action
   * @return {CardService.ActionResponse}
   */
  const handleSelection = ({ parameters, formInput }) => {
    // store the updated value in property store
    const { key } = parameters
    AppStore.user.set(key, formInput[key])
    return rootCard()
  }

generalized change selected value state

Displaying results from file picker

  const pickedFilesSection = () => {
    const section = CardService.newCardSection().setHeader("Files to process")
    // these files would have been picked by a picker session
    const pickedFiles = AppStore.pickedFiles
    if (!pickedFiles || !pickedFiles.length) {
      return section.addWidget(CardService.newTextParagraph()
        .setText('...no files selected yet')
      )
    }
    // Add a row for each picked file with thumbnail etc
    pickedFiles.forEach((file) => {
      const { extras, url } = file
      section.addWidget(CardService.newDecoratedText()
        .setOpenLink(CardService.newOpenLink()
          .setUrl(url)
          .setOpenAs(CardService.OpenAs.OVERLAY)
          .setOnClose(CardService.OnClose.NOTHING))
        .setText(extras.name)
        .setStartIcon(CardService.newIconImage().setIconUrl(extras.thumbnailLink).setAltText(`updated:${extras.modifiedTime}`))
      )
    })
    return section

filepicker results

In the client side html service code

Setting file picker results

  UsePicker.imageDialog()
    .then(packet => {
      const {docs} = packet
      if(docs.length) {
        const pickerRoot =  docs[docs.length-1].extras.parents[0].id
        return Promise.all([

          // this is the array of picked files
          Provoke.run("set-picked-files", docs),

          // this is the folder the last one came from - make it default for future picks
          Provoke.run("set-picker-root", pickerRoot)
          
        ]).then(() => window.top.close())
      } else {
        return Provoke.run ("del-picked-files")
      }
    })

    .catch(err => {
        App.showNotification("File pick error", err);
    })

setting file picker results

Getting filePicker parameters

  let prep = null
  const appStore = {}
  // Initialize the picker
  // get a token & make sure gapi gets loaded for later
  const init = () => {

    // get all persisted data from server side
    prep = Promise.all([
      Provoke.run('get-picker-root'),
      Provoke.run('get-picker-developerKey'),
      Provoke.run('get-token'),
      Provoke.run('get-picked-files'),
      new Promise(resolve => {
        gapi.load('picker', { callback: resolve });
      })]).then((results) => {
        const [pickerRoot, pickerDeveloperKey, token, pickedFiles] = results
        appStore.pickerRoot = pickerRoot
        appStore.pickerDeveloperKey = pickerDeveloperKey
        appStore.token = token
        appStore.pickedFiles = pickedFiles
      })
      .catch(err => {
        App.showNotification("prepping keys", err);
      });

  };

getting file picker results

Server side functions

  const {pickedFiles, processedResults} = AppStore
  Sheetify.make({pickedFiles,processedResults,fiddler})

This approach reduces the complexity of the main code and centralizes the state logic in one place, irrespective of whether you are using it server side, client side or in the card service avoiding all that timewasting of accidental mistreatments of state values and their side effects.

It also means you can easily start off where you left off – so useful not only for Add-ons but also for spltting up runs that migh otherwise go over execution time.

The best bit though is the ability to log all state changes with one setting and to be able to run tests to inspect state values while the Add-on is active, or even after it’s finished and exited.

I’ll publish the full code of this Add-on in the future. Ping me if you’d like to know morein the meantime.