Motivation

I’ve been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things Server side. All of these modes need to share state, and it always ends up rather messy. I though I’d take some inspiration from Redux and Vuex and apply those techniques to centralize and simplify state management from the business going on in each of these separate but co-operating environments

Background

This Add-on is actually a Sheets Add-on. It takes documents (images, pdfs etc) and runs them through the DocumentAI API, using ML to decipher invoices and other kinds of material and load the results into a sheet. It’s neither finished nor published yet, but I’m going to take this opportunity to get my thoughts about state management down while I’m still working on it.

Here’s an approximation of what the Add-on looks like in operation at this point.

document AI add-on

State management complexities

Its always been a little challenging sharing state cleanly between HTMLService based Add-ons and Server side functions, but the added dimension of Cardservice needs some special attention. When the contents of a card changes you need to recreate the card (as opposed to just a component of it), so all your state is going to be messed up. People usually use Property service to manage this, but Property service calls peppered around the card service code is error prone and mind bending.

In this Add-on, I also needed a filePicker, and CardService doesn’t have one. Luckily I had a pre-baked one from another Add-on (slidesmerge)  so I just needed to resurrect that and fire it off as a webapp from a CardService component. However, how to cleanly get the results from the filepicker back over to the re-rendered card service without adding to complexity?

State management needs

The types of state I need to manage and share are

  • Add-on global settings – Script property service to persist across sessions for all users
  • Add-on user settings – User property service to persist across sessions for each user
  • CardService component values – User propety service to persist across sessions for each user
  • Results from HTML service – User Cache service to persist for a little while, and usually provoked as the result of changes to Card Service components
  • Large scale results from Server Side API calls – User Cache to persist for a little while, and usually provoked from the combination of HTML service results and CardService component values
  • Archived results – Cloud Storage – persist forever to avoid duplicate processing of the same file with the same parameters that woudl give the same result as a previous run

Another 2 challenges specific to this API, is that calling the DocumentAI is neither free, nor especially cheap – so I wanted to ensure that if any call was made to the documentAI it was truly for a new document I’d never seen before – so it’s more cost effective (and efficient) to store all results to Cloud storage, indexed by the md5Hash of a file along with its processing parameters, in addition to the short term version held in cache. (as an aside, these results can be quite hefty – so I’m using Apps script caching with compression and enhanced size limitations to write any size of data to Apps Script cache). It’s also used in SuperFetch caching: How does it work?

Debugging

Another requirement is to make debugging these environments a little easier. It occurred to me that if i centralized all state management and wrapped each call to these various stores not only could I write a script to examine exactly what was in store at any given time, even when the Add-on was actually active, but i could also log and keep a record of state changes.

State management solution

First of all, a simple Store Class to wrap property store gets and sets

/**
* @Class Store
* GWAO uses property store a lot to persist values through instantiation
* this class provides a simpel interface for that
*/
class Store {
/**
* @constructor
* @param {PropertiesStore} [store=userProperties] the store to use
* @param {string} [prefix=''] prefix to segregate entries from anything else in the prop store if required
* @return {Store}
*/
constructor({store,prefix= '',log= false}) {
this.store = store || PropertiesService.getUserProperties()
this.prefix = prefix
this.log = log
}

/**
* to make stuff in property store more versatile we'll convert it to an object and stringify it
* @param {*} ob
* @return string
*/
stringify(ob) {
return JSON.stringify({
ob
})
}

/**
* to make stuff in property store more versatile we'll convert it to an object and stringify it
* this undoes that
* @param {string} value the value from store
* @return {*}
*/
unstringify(value) {
try {
const {ob} = JSON.parse(value)
return typeof ob === typeof undefined ? value: ob
}
catch (err) {
return value
}
}

/**
* make a key with the prefix
* @param {string} key store agains this key
* @return {string} prefixed key
*/
getKey (key) {
return this.prefix ? (this.prefix '_' key) : key
}

/**
* put to property store
* @param {string} key store agains this key
* @param {*} value thing to write
*/
set (key, value) {
const k = this.getKey(key)
if (this.log) console.log('storelog','setting',k, value)
return this.store.setProperty(k, this.stringify(value))
}

/**
* @param {string} key stored agains this key
* @return {string} the value
*/
get (key) {
const k = this.getKey(key)
const value = this.unstringify(this.store.getProperty(k))
if (this.log) console.log('storelog','getting',k, value)
return value
}

/**
* @param {string} key stored agains this key
*/
delete(key) {
const k = this.getKey(key)
if (this.log) console.log('storelog','removing',k)
return this.store.deleteProperty(k)
}
}
Store class

Next an AppStore, where all state is handled

/**
* this the state management central for this app
* since it uses server, html service and cardservice
* all state management and communication between them is handled here
*/
var AppStore = {

// set this to true for debugging to see state interactions being logged
get log() {
return false
},

// use for large, ephemeral state values such as immediate results
get userCache() {
const log = this.log
return new Cacher({
cachePoint: CacheService.getUserCache(),
expiry: 60 * 60 * 12,
stale: false,
staleKey: 'docai-stale',
log
})
},

// properties stores to use for small, persistent state values
get user() {
const log = this.log
return new Store({
store: PropertiesService.getUserProperties(),
log
})
},
get script() {
const log = this.log
return new Store({
store: PropertiesService.getScriptProperties(),
log
})
},

// any specific user settings can be stored here
set settings(value) {
this.user.set('settings', value)
},
get settings() {
return this.user.get("settings")
},

// select the location of documentAI processor to use
get locationId() {
return this.user.get('locationId')
},
set locationId(value) {
this.user.set('locationId', value)
},

// select the type of documentAI processor to use
get type() {
return this.user.get('type')
},
set type(value) {
this.user.set('type', value)
},

// select the documentAI processor to use
get displayName() {
return this.user.get('displayName')
},
set displayName(value) {
this.user.set('displayName', value)
},

// these are the locations currently the addon supports from the documentAI api
get locationList() {
return ["us", "eu"].map(f => ({ locationId: f }))
},

// the folder at which to start picking
get pickerRoot() {
return this.user.get('pickerRoot') || 'root'
},
set pickerRoot(id) {
this.user.set('pickerRoot', id)
},

// the apps script token needs to be scoped for all the functions of the add-on, server side functions, and picker
get token() {
return ScriptApp.getOAuthToken()
},

// this will need to be set in the project script properties manually and is the api key for the picker
get pickerDeveloperKey() {
return this.script.get('pickerDeveloperKey')
},

// this will need to be set in the project script properties manually and is the current project number
// dont have a way to pick this up automatically
// eventually this can be enhanced to step outside the current apps script project, but will need IAM teaking for proxy token
get projectNumber() {
return this.script.get('projectNumber')
},


// this is the p ublished webapp required to host the filepicker
get serviceUrl() {
return ScriptApp.getService().getUrl()
},

// for reporting where we're up to
set workingOn(file) {
this.userCache.set('workingOn', file.md5)
},
get workingOn() {
const picked = this.pickedFiles
if (!picked) return null

const md5 = this.userCache.get('workingOn')
if (!md5) return null

// find in currently picked files
return picked.find(f => f.md5 === md5)
},


// picked files are by selected processor type
// this allows a change of procoessor withot needing to repick the files
get pickedFilesKeyOptions() {
return this.type
},
get pickedFiles() {
const options = this.pickedFilesKeyOptions
return options ? this.userCache.get('pickedFiles', options) : null
},
set pickedFiles(value) {
const options = this.pickedFilesKeyOptions
if (options) {
this.userCache.set('pickedFiles', value, {
options
})
}
},
delPickedFiles() {
const options = this.pickedFilesKeyOptions
if (options) {
this.userCache.remove('pickedFiles', options)
}
},


// the key for the results is based on the md5 of the picked files the selected processor
get processedResultsKeyOptions() {
// if any of these are missing there can be no valid entry
const files = this.pickedFiles
if (!files) return null

const selectedProcessor = this.selectedProcessor
if (!selectedProcessor) return null

// this will be added to the key to make sure we pick up the results that match the current selections
return {
md5s: files.map(f => f.extras.md5),
selectedProcessor
}
},
get processedResults() {
// this'll make sure we get the results for the correct state
const options = this.processedResultsKeyOptions
if (!options) return null
return this.userCache.get('processedResults', options)
},
set processedResults(value) {
const options = this.processedResultsKeyOptions
if (options) {
this.userCache.set('processedResults', value, {
options
})
}
},

// the current processor can be discovered from the list of known processors and the selected displayname
set processors(value) {
this.userCache.set('processors', value)
},
get processors() {
return this.userCache.get('processors')
},
get selectedProcessor() {
const { processors, displayName } = this
return processors && displayName ? processors.find(f => f.displayName === displayName) : null
}
}
App Store

Finally, a client side Provoker to get and set state all the state values from within html service. This is an enhancement of the technique i use in How to use Vue.js, Vuex and Vuetify to create Google Apps Script Add-ons

/**
* this should contain a list of all the functions authorized
* to be run from the client
* for V8, it's important that it's a namespace ief to ensure its parsed in the correct order
* and is available throughout the project on initialization
*/
const WhitelistedActions = (() => {
const ks = new Map()
// whitelist everything that can be run
ks.set('get-picker-developerKey', () => AppStore.pickerDeveloperKey)
ks.set('get-token', () => AppStore.token)
ks.set('set-picked-files', (value) => AppStore.pickedFiles = value)
ks.set('set-picker-root', (value) => AppStore.pickerRoot = value)
ks.set('get-picker-root', () => AppStore.pickerRoot)
ks.set('get-picked-files', () => AppStore.pickedFiles)
ks.set('del-picked-files', () => AppStore.delPickedFiles)
return ks
})();
/**
* run something from the whitelist
* will generall be invoked by google.script.run
* @param {string} name the key of the action to run in the whilelistedActions store
* @param {...*} args any arguments for the action
* @return {*} the result
*/
const runWhitelist = (name, ...args) => {
// get what to run from the store
const action = WhitelistedActions.get(name)
if (!action) {
throw new Error(`${name} is not in the list of actions that can be run from the client`)
}
return action(...args)
}
provoke

Using the AppStore

Now I don’t need to worry about the details of how state is maintained.

For example, to see what files have been selected by the latest pick action, I can just run this – even while the Add-on is active

 console.log(AppStore.pickedFiles)

In the card service, I can do this kind of thing –

Getting a currently selected value
    const decoratedText = CardService.newDecoratedText()
.setText(AppStore.projectNumber)
.setTopLabel("Project number containing DocumentAI processors");
getting a value
Setting the result of an api call
    // get a new document AI - locationId & projectNumber are in the store
const dai = new DocumentAI({ locationId: AppStore.locationId, projectNumber: AppStore.projectNumber })
const { processors, nextPageToken } = dai.listProcessors({ noCache: true })

const result = processors.filter(f => f.state === 'ENABLED')
AppStore.processors = result
setting the result of an api call
Mixing settings, tokens and component values
  const newDai = () => {
const token = AppStore.token
return new DocumentAI({
tokenService: () => token,
projectNumber: AppStore.projectNumber,
locationId: AppStore.locationId,
gcsOutputPath: AppStore.gcsOuputPath
})
}
mixing it up
Dealing with a value change in a component
  /**
* called when a selection is modified
* @param {CardAction.event} e
* @param {object} e.parameters the parameters passed to the handler
* @param {object} e.formInput the other values in the form
* @param {string} e.parameters.key the forminput key that caused the action
* @return {CardService.ActionResponse}
*/
const handleSelection = ({ parameters, formInput }) => {
// store the updated value in property store
const { key } = parameters
AppStore.user.set(key, formInput[key])
return rootCard()
}
generalized change selected value state
Displaying results from file picker
  const pickedFilesSection = () => {
const section = CardService.newCardSection().setHeader("Files to process")
// these files would have been picked by a picker session
const pickedFiles = AppStore.pickedFiles
if (!pickedFiles || !pickedFiles.length) {
return section.addWidget(CardService.newTextParagraph()
.setText('...no files selected yet')
)
}
// Add a row for each picked file with thumbnail etc
pickedFiles.forEach((file) => {
const { extras, url } = file
section.addWidget(CardService.newDecoratedText()
.setOpenLink(CardService.newOpenLink()
.setUrl(url)
.setOpenAs(CardService.OpenAs.OVERLAY)
.setOnClose(CardService.OnClose.NOTHING))
.setText(extras.name)
.setStartIcon(CardService.newIconImage().setIconUrl(extras.thumbnailLink).setAltText(`updated:${extras.modifiedTime}`))
)
})
return section
filepicker results

In the client side html service code

Setting file picker results
  UsePicker.imageDialog()
.then(packet => {
const {docs} = packet
if(docs.length) {
const pickerRoot = docs[docs.length-1].extras.parents[0].id
return Promise.all([

// this is the array of picked files
Provoke.run("set-picked-files", docs),

// this is the folder the last one came from - make it default for future picks
Provoke.run("set-picker-root", pickerRoot)

]).then(() => window.top.close())
} else {
return Provoke.run ("del-picked-files")
}
})

.catch(err => {
App.showNotification("File pick error", err);
})
setting file picker results
Getting filePicker parameters
  let prep = null
const appStore = {}
// Initialize the picker
// get a token & make sure gapi gets loaded for later
const init = () => {

// get all persisted data from server side
prep = Promise.all([
Provoke.run('get-picker-root'),
Provoke.run('get-picker-developerKey'),
Provoke.run('get-token'),
Provoke.run('get-picked-files'),
new Promise(resolve => {
gapi.load('picker', { callback: resolve });
})]).then((results) => {
const [pickerRoot, pickerDeveloperKey, token, pickedFiles] = results
appStore.pickerRoot = pickerRoot
appStore.pickerDeveloperKey = pickerDeveloperKey
appStore.token = token
appStore.pickedFiles = pickedFiles
})
.catch(err => {
App.showNotification("prepping keys", err);
});

};
getting file picker results

Server side functions

  const {pickedFiles, processedResults} = AppStore
Sheetify.make({pickedFiles,processedResults,fiddler})

Next

This approach reduces the complexity of the main code and centralizes the state logic in one place, irrespective of whether you are using it server side, client side or in the card service avoiding all that timewasting of accidental mistreatments of state values and their side effects.

It also means you can easily start off where you left off – so useful not only for Add-ons but also for spltting up runs that migh otherwise go over execution time.

The best bit though is the ability to log all state changes with one setting and to be able to run tests  to inspect state values while the Add-on is active,  or even after it’s finished and exited.

I’ll publish the full code of this Add-on in the future. Ping me if you’d like to know morein the meantime.

Related

Superfetch plugin

Caching, property stores and pre-caching

I've written many times about various Apps Script caching techniques such as how to deal with size limits and use ...
Read More
document AI add-on

State management across CardService, HtmlService and Server side Add-ons

Motivation I've been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things ...
Read More
Secret Manager

SuperFetch Plugin: Cloud Manager Secrets and Apps Script

Smg is a SuperFetch plugin to access the Google Cloud Secrets API. SuperFetch is a proxy for UrlFetchApp with additional ...
Read More
SuperFetch

Apps script caching with compression and enhanced size limitations

Motivation Caching is a great way to improve performance, avoid rate limit problems and even save money if you are ...
Read More

12 Years and 1000 pages in Office,Google (Docs,Gsuite) Workspace, and other stuff

1000 pages and counting Most years I do a post on 'a year in Apps Script', looking back over the ...
Read More
import add-on

Import, export and mix container bound and standalone Apps Script projects

This article covers how to pull scripts from multiple projects and import them into another project. You can even use ...
Read More

Apps Script server side to client side htmlservice progress reporting using CacheService

I've published a few examples of this in other projects (for example Implementing a client side progress bar reporting on ...
Read More
crusher files on drive

Sharing data between Apps Script and Node using Google Drive back end – 5 minute example

Another quick demo of data sharing Here's a challenge that shares the data in a spreadsheet with node, set up ...
Read More
admin.google.com

The nodejs Drive API, service account impersonation and async iterators

This is part of the series on sharing data between Apps Script and Node on various backends, Apps script library with ...
Read More