Smg is a SuperFetch plugin to access the Google Cloud Secrets API. SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.

This is another in my series on SuperFetch plugins.

Page Content hide

1.1 When to use the secret manager instead of Property Service

1.2 Secret versions

1.3 Cloud console

1.4 Smg plugin for SuperFetch

1.5 Secrets format

1.6 Script Preparation

1.7 Instantiation

1.7.1 Smg instance

1.7.1.1 noCache property (optional)

1.8 Usage

1.8.1 Creating a secret

1.8.2 Add version

1.8.3 Get a secret

1.8.4 Getting a specific version

1.8.5 Dealing with native secrets

1.8.6 Adding labels to secrets

1.8.7 Setting aliases for versions

1.8.8 Listing Secrets

1.8.9 Listing versions

1.8.10 Filter queries

1.13.2 Caching, property stores and pre-caching

1.13.3 State management across CardService, HtmlService and Server side Add-ons

1.13.4 SuperFetch Plugin: Cloud Manager Secrets and Apps Script

1.13.5 SuperFetch caching: How does it work?

1.13.6 SuperFetch plugins: Tank events and appending

1.13.7 SuperFetch Plugins: Apps Script streaming with Tank and Drive

1.13.8 SuperFetch Tank Plugin: Streaming for Apps Script

1.13.9 SuperFetch plugin – Google Drive client for Apps Script – Part 1

1.13.10 SuperFetch – Twitter plugin for Apps Script – Get Follows, Mutes and blocks

1.13.11 SuperFetch plugin – Twitter client for Apps Script – Counts

1.13.12 OAuth2 and Twitter API – App only flow for Apps Script

1.13.13 SuperFetch plugin – Twitter client for Apps Script – Search and Get

1.13.14 Apps Script Oauth2 library Goa: tips, tricks and hacks

1.13.15 Apps Script Oauth2 – a Goa Library refresher

1.13.16 SuperFetch plugin – Firebase client for Apps Script

1.13.17 SuperFetch plugin – iam – how to authenticate to Cloud Run from Apps Script

1.13.18 SuperFetch – a proxy enhancement to Apps Script UrlFetch

1.13.19 Apps script caching with compression and enhanced size limitations

1.13.20 Simple but powerful Apps Script Unit Test library

Motivation

Normally Apps Script developers use the Property service to store the secrets they need to run their scripts, and for most cases this is fine. However, Google cloud platform does have a Cloud Secret Manager you can use to store and manage secrets.

When to use the secret manager instead of Property Service

The main motivation would be if you need to share secrets across multiple Apps Script projects. There are a couple of workarounds to enable this with vanilla Apps Script services, but they are prone to potential security and management fails. A couple of examples would be.

Store secrets in some shared Drive file
Expose a script containing secrets as a library so that co-operating scripts can get the secrets from that library’s property store

Using the secret manager instead means sharing a Google Cloud Project across multiple Apps Script projects each of which will have access to the Project’s secret manager, with different levels of access to them controlled by IAM. You could also share across multiple cloud projects, but let’s leave that for another day.

If you already use Kubernetes, you’ve probably used its secret manager and will be familiar with this whole concept.

Note that the use of the Secret Manager is potentially not free, but it does have a free tier.

Secret versions

The secret manager has many features unavailable in the property store – in particular versioning, version aliases and labeling, which will allow you to use different versions of a secret in differetn modes – eg test/dev/prod etc.

Cloud console

Secrets are manageable via the cloud console and look like this

Secret manager also keeps a log so you can keep track of changes.

Smg plugin for SuperFetch

The smg plugin uses the Secrets Manager JSON API to provide a simple to use SuperFetch plugin to access and manager Secrets directly from Apps Script.

Secrets format

Secrets created via the console are held in base64 format in the secret manager. The Smg plugin can read and write these base64 content, but it also allows you to commit strings, objects, numbers, booleans and blobs, which it will automatically restore to their original state when you access them. More of that when we get into some examples.

Script Preparation

You’ll need the bmSuperFetch library and if you want to run some tests, the bmUnitTest library – details at the end of the article.

You need to use a regular cloud project rather than the Apps Script managed one (you can change it in Project settings), and you’ll need at least these scopes enabled in your manifest file (appsscript.json), as well as the Secret Manager API enabled

  "oauthScopes": [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/script.external_request"
  ]

scopes

You’ll also may need to assign IAM roles to access secrets – See this configuration instructions for how, to those that need to access the secrets. Things get a little more complex if you are writing a script for someone else to use, but the secrets belong to you. I’ll deal with that scenario in a future article.

Instantiation

First you need a SuperFetch instance. If you don’t need caching, then just omit the cacheService property.


  const { Plugins, SuperFetch } = bmSuperFetch

  // use this basic drv setup
  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: ScriptApp.getOAuthToken,
    cacheService: CacheService.getUserCache()
  })

superfetch instance

Smg instance

This will be the handle for accessing Smg


    const {Smg} = Plugins
    const smg = new Smg({ 
      superFetch,
      projectId: 'your-project-id',
    })

smg instance

There are various other standard SuperFetch Plugin parameters that I’ll deal with later, but the one shown is of special interest to this article. The project id you can get from the cloud console dashboard.

noCache property (optional)

You can turn off caching by setting this to true, or by omitting the CacheService poperty in the SuperFetch instantiation.

Usage

There are 2 kinds of secrets that Smg can handle

Those created in the Cloud console
Those created and managed by Smg

Smg will automatically figure out which is which, and if it’s a secret that it created, it will be able to restore the data to the original state. If it’s one created in the console or by some other means, you’ll get back a standard base64 value, as well as decoded string version.

Creating a secret

A secret has an id of your choice. When you create a secret, you are not yet adding any value to it – just setting it up. The response is a standard SuperFetch response (see SuperFetch – a proxy enhancement to Apps Script UrlFetch for what those responses look like). The secret metadata will be found in the .data property.

  
const secret = smg.secret({id: 'mysecret'}).create()

/**
data: 
   { name: 'projects/xxxxx/secrets/mysecret',
     replication: { automatic: {} },
     createTime: '2022-09-11T15:52:23.762750Z',
     etag: '"15e868c3dbdf3e"' }
*/

create secret

Add version

Secrets can have multiple versions. To assign data to secret you simply add a version. These version numbers will be incremented automatically. In this example, we’re just adding a string.

const {data} = mySecret.addVersion ({data: 'some string data'})

/**
data: 
{ name: 'projects/xxxxx/secrets/mysecret/versions/1',
  createTime: '2022-09-11T16:58:41.313653Z',
  state: 'ENABLED',
  replicationStatus: { automatic: {} },
  etag: '"15e869b0f07b75"' }
*/

create secret

Get a secret

Getting a secret also returns the data for a version you select (or the latest by default)

console.log(mySecret.get().data)
/**
{ version: '1',
  name: 'projects/xxxxx/secrets/mysecret/versions/1',
  value: 'some string data',
  valueType: 'string' }
 */

version data

Getting a specific version

console.log(mySecret.get({version: 2}).data)
/**
{ version: '2',
  name: 'projects/xxxxx/secrets/mysecret/versions/2',
  value: 'some string data',
  valueType: 'string' }
 */

specific version

Dealing with native secrets

You’ll notice that the response from smg includes .value and .valueType properties. This is syntactic sugar added by smg. If you retrieve a secret that wasn’t created by smg, you’ll just get a value64 and value response – which is how the secrets are stored natively. As a help, smg attempts to return a decoded string in the .value property, but also returns the .value64 property in case you need to decode some non- string value.

 console.log(smg.secret({id: 'uistring'}).get().data)
 /**
 { version: '1',
  name: 'projects/xxx/secrets/uistring/versions/1',
  valueType: 'b64',
  value: 'uistring',
  value64: 'dWlzdHJpbmc=' }
  */

native

Adding labels to secrets

You can add labels to secrets when you create them. Labels are useful for filtering queries when listing secrets. Notice I’ll adding the .throw() method in most of the following examples – like all SuperFetch plugins, this method will throw an error if one is detected.

    const labs = smg.secret({id: 'labeltest'})
    console.log(labs.create ({ labels: {app: 'foo', creds: 'passwords'}}).throw().data)
	/**
	{ name: 'projects/xxxx/secrets/labeltest',
  replication: { automatic: {} },
  createTime: '2022-09-12T14:21:24.737788Z',
  labels: { creds: 'passwords', app: 'foo' },
  etag: '"15e87b9c5122fc"' }
  */

labelling secrets

Setting aliases for versions

Instead of juggling version numbers, you can create aliases for secret versions then use those aliases to get the values.

    // create v1-3
    console.log(labs.addVersion ({data: {user: 'd1', password: '1111'}}).throw().data)
    console.log(labs.addVersion ({data: {user: 't1', password: '2222'}}).throw().data)
    console.log(labs.addVersion ({data: {user: 'p1', password: '333'}}).throw().data)
    
    // assign aliases
    console.log(labs.setAliases({aliases: { dev: 1, test: 2 , prod: 3}}).throw().data)
	/**
	{ name: 'projects/xxx/secrets/labeltest',
  replication: { automatic: {} },
  createTime: '2022-09-12T14:29:11.710647Z',
  labels: { app: 'foo', creds: 'passwords' },
  etag: '"15e87bb83d3508"',
  versionAliases: { dev: '1', test: '2', prod: '3' } }
  */

    // fetching by alias
    console.log(labs.get({version: 'dev'}).throw().data)
	/**
	{ version: '1',
  name: 'projects/xxx/secrets/labeltest/versions/1',
  value: { user: 'd1', password: '1111' },
  valueType: 'object' }
  */
    console.log(labs.get({version: 'test'}).throw().data)
	/**
	{ version: '2',
  name: 'projects/xxx/secrets/labeltest/versions/2',
  value: { user: 't1', password: '2222' },
  valueType: 'object' }
  */
    console.log(labs.get({version: 'prod'}).throw().data)
	
	/**
	{ version: '3',
  name: 'projects/xxx/secrets/labeltest/versions/3',
  value: { user: 'p1', password: '333' },
  valueType: 'object' }
  */

aliases

Listing Secrets

You can list all secrets like this

console.log(smg.list().throw().data)
/**
{ items: 
   [ { name: 'projects/xxx/secrets/aliasTest',
       replication: [Object],
       createTime: '2022-09-12T11:15:01.713372Z',
       etag: '"15e87901e1170c"',
       versionAliases: [Object] },
     { name: 'projects/xxx/secrets/bar',
       replication: [Object],
       createTime: '2022-09-12T14:01:03.791415Z',
       labels: [Object],
       etag: '"xxx"' },
     { name: 'projects/xxx/secrets/blo',
       replication: [Object],
       createTime: '2022-09-12T14:01:07.762830Z',
       labels: [Object],
       etag: '"15e87b53c7928e"' },
     { name: 'projects/xxx/secrets/byt',
       replication: [Object],
       createTime: '2022-09-12T14:01:09.595054Z',
       labels: [Object],
       etag: '"15e87b53e387ae"' }, ....
	   
	*/

listing secrets

Listing versions

Here’s how to list all the versions of a given

    const labs = smg.secret({id: 'labeltest'})
    console.log(labs.list().throw().data)
	/**
	{ items: 
   [ { name: 'projects/xxx/secrets/labeltest/versions/3',
       createTime: '2022-09-12T14:45:18.291689Z',
       state: 'ENABLED',
       replicationStatus: [Object],
       etag: '"15e87bf1c36ee9"' },
     { name: 'projects/xxx/secrets/labeltest/versions/2',
       createTime: '2022-09-12T14:45:17.579126Z',
       state: 'ENABLED',
       replicationStatus: [Object],
       etag: '"15e87bf1b88f76"' },
     { name: 'projects/xxx/secrets/labeltest/versions/1',
       createTime: '2022-09-12T14:45:16.886129Z',
       state: 'ENABLED',
       replicationStatus: [Object],
       etag: '"15e87bf1adfc71"' } ] }
	   */

list versions of a secret

Filter queries

You can add filters to the .list() method – there is a filter language described here.

console.log(smg.list({ query: "labels.app=foo" }).throw().data)
/**
{ items: 
   [ { name: 'projects/xxxxx/secrets/labeltest',
       replication: [Object],
       createTime: '2022-09-12T15:00:15.514619Z',
       labels: [Object],
       etag: '"15e87c275bc2e5"',
       versionAliases: [Object] } ] }
*/

querying

Paging

Smg supports paging, but it’s quite unlikely you’ll have enough secrets to make this required. Without a page closure with a max parameter specified, it will return all matching secrets. It also supports a pageToken parameter if you have interupted a list with a max parameter. It works the same as described in SuperFetch plugin: Google Drive client for Apps Script : Part 1 where the mechanism is fully described.

labs.page({max: 2}).list()

page

Ref

You can create a new instance of smg base on an existing instance. To change some of the characteristics of the instance (eg switch off caching).

 const smgNocache = smg.ref({noCache: true})

ref

Error handling

All Superfetch plugins use the same method of signalling or optionally, reacting to an error.

You can detect an error by checking the result.error property or you can force an automatic throw if an error is detected. Here’s examples of each of these.

// handle the error yourself
const result = smg.list()
if (result.error) {
	//... handle the error
	throw result.error
} else {
// results are in result.data
}

// throw automatically on error
const result = smg.list().throw()

// results are in result.data

error handling

Unit testing

I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls. These tests demonstrate each of the topics mentioned in this article, as well as some I’ll be covering in a later article, and could serve as a useful crib sheet for the plugin – it’s a long read so you may want to just come back and refer to this.


const testSmg = ({
  force = false,
  unit } = {}) => {
  // control which tests to skip
  const skipTest = {
    basic: true && !force,
    smgvalue: true && !force,
    listing: true && !force,
    alias: true && !force
  }
  // get a testing instance
  unit = unit || new bmUnitTester.Unit({
    showErrorsOnly: true
  })


  // import required modules
  const { Plugins, SuperFetch } = bmSuperFetch
  const { Smg, ValuePacker, Helpers } = Plugins

  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: ScriptApp.getOAuthToken,
    cacheService: CacheService.getUserCache()
  })

  unit.section(() => {
    
    const smg = new Smg({
      superFetch,
      projectId: 'devshere',
      showUrl: true
    })
    const devData = {
      apiKey: '1234dev',
      name: 'dev'
    }
    const prodData = {
      apiKey: '1234prod',
      name: 'prod'
    }
    const testData = {
      apiKey: '1234test',
      name: 'test'
    }
    const secret = smg.secret({id: 'testAlias'})
    secret.delete()
    secret.create().throw().data
    secret.addVersion ({data: testData}).throw()
    secret.addVersion ({data: prodData}).throw()
    secret.addVersion ({data: devData}).throw()
    secret.setAliases({aliases: {test: 1, prod: 2, dev: 3}}).throw()

    unit.is (testData, secret.get ({version:'test'}).throw().data.value, {
      description: 'test alias'
    })
    
    unit.is (prodData, secret.get ({version:'prod'}).throw().data.value, {
      description: 'prod alias'
    })

    unit.is (devData, secret.get ({version:'dev'}).throw().data.value, {
      description: 'dev alias'
    })

    unit.is(3, secret.list().throw().data.items.length, {
      description: 'listing versions works'
    })

    secret.delete()
  }, {
    description: 'aliases',
    skip: skipTest.alias
  })

 
  unit.section(() => {
    // create an iam instance
    const smg = new Smg({
      superFetch,
      projectId: 'devshere',
      showUrl: true
    })
    const foo = smg.secret({ id: 'foo' })
    const bar = smg.secret({ id: 'bar' })
    const chi = smg.secret({ id: 'chi' })
    const blo = smg.secret({ id: 'blo' })
    const byt = smg.secret({ id: 'byt' })
    const nul = smg.secret({ id: 'nul' })
    const nat = smg.secret({ id: 'nat' })

    const fooData = 'foodata'
    const barData = { barData: 'barData' }
    const chiData = true
    const bloData = Utilities.newBlob('{"a":1}', "application/json", "f3")
    const bytData = bloData.getBytes()
    const nulData = null
    const natData = 'a string to make native b64'

    const tests = [{
      s: foo,
      d: fooData,
      labels: {
        type: 'dev',
        test: 'easy'
      }
    }, {
      s: bar,
      d: barData,
      labels: {
        test: 'harder'
      }
    }, {
      s: chi,
      d: chiData,
      labels: {
        type: 'dev'
      }
    }, {
      s: blo,
      d: bloData,
      labels: {
        type: 'prod',
        test: 'hard'
      }
    }, {
      s: byt,
      d: bytData,
      labels: {
        test: 'easy'
      }
    }, {
      s: nul,
      d: nulData
    }, {
      s: nat,
      d: natData,
      b64: true,
      labels: {
        test: 'unmanaged'
      }

    }]

    tests.forEach(t => {
      t.s.delete()
      t.s.create({ labels: t.labels }).throw()
      t.s.addVersion({ data: t.d, b64: t.b64 })
      const data = t.s.get().throw().data
      if (Utils.isBlob(t.d)) {
        unit.is(t.d.getBytes(), data.value.getBytes(), {
          description: t.s.id   ' checked blob value'
        })
      } else {
        unit.is(t.d, data.value, {
          description: t.s.id   ' checked value'
        })
      }
    })

    // ui generatedsecrets are b64encoded
    const ui = smg.secret({ id: 'uistring' }).get().throw().data
    unit.is('b64', ui.valueType, {
      description: 'ui generated vtypes are b64'
    })
    unit.is('string', typeof ui.value, {
      description: 'ui generated types are strings'
    })

    const { actual: list } = unit.not(
      null, smg.list().throw(), {
      description: 'list them all'
    })

    unit.is(list.data.items, smg.page({ pageSize: 3 }).list().throw().data.items, {
      description: 'pagesizes'
    })


    const { actual: max5 } = unit.not(null, smg.page({ pageSize: 3, max: 5 }).list().throw(), {
      description: 'ps 3/max 5 still gets 5'
    })

    unit.is(list.data.items.slice(0, 5).length, max5.data.items.length, {
      description: 'max listing combinaton'
    })

    unit.is(
      list.data.items.length,
      smg.page({ pageToken: max5.pageToken }).list().throw().data.items.length   max5.data.items.length, {
      description: 'pagetoken'
    })
    const { actual: easy } = unit.not(null, smg.list({ query: "labels.test=easy" }).throw().data.items, {
      description: 'select easy test'
    })

    unit.is(
      true,
      easy.every(f => f.labels && f.labels.test && f.labels.test === 'easy'), {
      description: 'easy labels match'
    })

    const { actual: dev } = unit.not(null, smg.list({ query: "labels.type=dev" }).throw().data.items, {
      description: 'select easy test'
    })

    unit.is(
      true,
      dev.every(f => f.labels && f.labels.type && f.labels.type === 'dev'), {
      description: 'type labels match'
    })

  }, {
    description: 'listing',
    skip: skipTest.listing
  })

  unit.section(() => {
    const packer = new ValuePacker()
    const a = packer.pack('a')
    const blob = Utilities.newBlob('xxx', 'text/plain', 'blobby')
    const fix1 = ['a', 1, 'b', { a: 'a' }, [1, 2, 3], blob.getBytes()]
    fix1.forEach(r => {
      unit.is(r, packer.unpack(packer.pack(r)).value, {
        description: `pack and unpack ${Helpers.whichType(r)}`
      })
    })

    const fix2 = [blob]
    fix2.forEach(r => {
      unit.is(r, packer.unpack(packer.pack(r)).value, {
        description: `pack and unpack ${Helpers.whichType(r)}`,
        compare: (e, a) => {
          return e.name === a.name && e.contentType === a.contentType && unit.deepEquals(e.getBytes(), a.getBytes())
        }
      })
    })

  }, {
    description: 'test value construction',
    skip: skipTest.smgvalue
  })
  // test all methods 
  unit.section(() => {


    const smg = new Smg({
      superFetch,
      projectId: 'devshere',
      showUrl: true
    })

    // target secret
    const { actual: foo } = unit.not(null, smg.secret({ id: "foo" }), {
      description: 'get a path to a secret'
    })

    unit.is(404, foo.delete().data, {
      description: 'removed any existing and get either a 404, or a success',
      compare: (e, a) => !e.error || (e.error && e.responseCode === a)
    })

    unit.is(foo.id, foo.create(), {
      description: 'create a secret- check is part of name',
      compare: (e, a) => a.data.name.match(new RegExp(e   '$'))
    })

    const f1 = {
      bar: 'foo1'
    }
    const f2 = 'foo2'
    const f3 = Utilities.newBlob('{"a":1}', "application/json", "f3")

    unit.is(foo.id, foo.addVersion({ data: f1 }), {
      description: 'check version 1',
      compare: (e, a) => a.data.name.match(new RegExp(e   '\\/versions\\/1$'))
    })

    const { actual: latest } = unit.not(null, foo.get(), {
      description: 'get latest'
    })

    unit.is(false, latest.cached, {
      description: 'wasnt cached'
    })

    unit.is(f1, latest.data.value, {
      description: 'value matches for first version'
    })

    unit.is(true, foo.get().cached, {
      description: 'cached 2nd attempt'
    })


    unit.is(foo.get().data, foo.get({ version: 1 }).data, {
      description: 'latest shortcut works'
    })

    unit.is(foo.id, foo.addVersion({ data: f2 }), {
      description: 'check version 2',
      compare: (e, a) => a.data.name.match(new RegExp(e   '\\/versions\\/2$'))
    })

    unit.is(f1, foo.get({ version: 1 }).data.value, {
      description: 'first version ok'
    })

    unit.is(f2, foo.get({ version: 2 }).data.value, {
      description: 'second version ok'
    })

    const { actual: s3 } = unit.not(null, foo.addVersion({ data: f3 }), {
      description: 'add a blob'
    })

    unit.is(
      smg.secret().get({ id: foo.id }).data.value.getBytes(),
      foo.get().data.value.getBytes(), {
      description: 'check closure -vs- direct'
    }
    )
    unit.is(f3.getName(), smg.secret().get({ id: foo.id }).data.value.getName(), {
      description: 'check blob name and id passed in get works'
    })

    unit.is(f3.getContentType(), foo.get().data.value.getContentType(), {
      description: 'check blob content and id passed in get works'
    })

    unit.is(null, foo.destroy({ version: 1 }).throw().error, {
      description: 'destroy a version'
    })

    unit.is(400, foo.get({ version: 1 }).responseCode, {
      description: 'destroyed version state is 400'
    })

    const { actual: lastestData } = unit.not(null, foo.get().data, {
      description: 'get latest'
    })


    unit.is(null, foo.destroy({ version: lastestData.version }).throw().error, {
      description: 'destroy latest version'
    })

    unit.is(400, foo.get().responseCode, {
      description: 'latest responseCode shows its in a destroyed state'
    })

    // restore a previous version
    unit.not(null, foo.addVersion({ data: foo.get({ version: 2 }).data.value }), {
      description: 'adding a prev version'
    })

    unit.is('4', foo.get().data.version, {
      description: 'new version created'
    })

    unit.is(null, foo.delete().error, {
      description: 'no remove error'
    })

    unit.is(404, foo.get().responseCode, {
      description: 'nothing there'
    })

    const bar = smg.secret({ id: 'bar' })
    unit.is(404, bar.delete().data, {
      description: 'removed any existing and get either a 404, or a success',
      compare: (e, a) => !e.error || (e.error && e.responseCode === a)
    })

    const labels = {
      donkey: "kong",
      space: "invaders"
    }
    unit.not(null, bar.create({
      labels
    }))

    unit.is(labels, bar.get().data.labels, {
      description: 'labels worked'
    })



  }, {
    description: 'basic secrets',
    skip: skipTest.basic
  })

  unit.report()

}

tests

This example shows how to use the Apps Script Oauth token to get secrets, which is fine if the script owner is accessing the secret. We’ll need to do something special for token processing if you are writing a script that will be run in the context of someone else who normally wouldn’t have the IAM rights to access secrets.