I’ve written a few articles about JavaScript proxying on here, and I’m a big fan. I also use a lot of APIS, and it can be time consuming to keep on checking the REST documentation for how to call them and deal with the UrlFetch responses. SuperFetch is a proxy for UrlFetchApp to help.

Page Content hide

1.2 test API

1.3 A SuperFetch instance

1.4 Test Framework

1.5 A minimal Superfetch

1.5.1 A fetcher

1.5.2 A fetcher response

1.5.2.1 throw method

1.5.2.2 Testing for thrown error

1.6 Caching

1.6.1 Fake HttpResponse response

1.7 Throttling

1.7.1 Sub Libraries

1.7.2 Throttling delay demo

1.7.3 Rate limiting

1.8 Complete test set

1.9 Making an API class

1.9.1 OmdApi class

1.9.1.1 Using the OmdApi

1.10 What’s next

1.11 Links

1.11.1 Convert any file with Apps Script

1.11.2 Caching, property stores and pre-caching

1.11.3 State management across CardService, HtmlService and Server side Add-ons

1.11.4 SuperFetch Plugin: Cloud Manager Secrets and Apps Script

1.11.5 SuperFetch caching: How does it work?

1.11.6 SuperFetch plugins: Tank events and appending

1.11.7 SuperFetch Plugins: Apps Script streaming with Tank and Drive

1.11.8 SuperFetch Tank Plugin: Streaming for Apps Script

1.11.9 SuperFetch plugin – Google Drive client for Apps Script – Part 1

1.11.10 SuperFetch – Twitter plugin for Apps Script – Get Follows, Mutes and blocks

1.11.11 SuperFetch plugin – Twitter client for Apps Script – Counts

1.11.12 OAuth2 and Twitter API – App only flow for Apps Script

1.11.13 SuperFetch plugin – Twitter client for Apps Script – Search and Get

1.11.14 Apps Script Oauth2 library Goa: tips, tricks and hacks

1.11.15 Apps Script Oauth2 – a Goa Library refresher

1.11.16 SuperFetch plugin – Firebase client for Apps Script

1.11.17 SuperFetch plugin – iam – how to authenticate to Cloud Run from Apps Script

1.11.18 SuperFetch – a proxy enhancement to Apps Script UrlFetch

1.11.19 Apps script caching with compression and enhanced size limitations

1.11.20 Simple but powerful Apps Script Unit Test library

What is SuperFetch

SuperFetch is a proxy for UrlFetchApp which has these extra capabilities – most of which I use for all APIS

Built in caching and compression – it uses Apps script caching with compression and enhanced size limitations
A standard reponse format and error handling
Inbuilt JSON parsing
Automatic token insertion
One liner capability to make Unit Testing less of a chore
Plug-in enhancements such as automatic throttling and rate limit management

You’ll need the bmSuperFetch and bmUnitTest libraries – details at the end.

Testing

Throughout this article I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls

test API

I originally create superFetch to use as a plugin replacement for UrlFetch to my api library (I have one library that I use to access all the APIS I use regularily – mainly Google ones). For this article, though I’ll use the Open Movie Database API (omd) and move onto some more complex APIS in later articles.

The omd needs an api key and a query specifed as url parameters.

A SuperFetch instance

Before we start, let’s set up a few constants to use throughought the test

  // marker for missing arguments eg func (_, arg)
  const _ = undefined
  
  // the OMDB api endpoint
  const omdEndPoint = 'http://www.omdbapi.com'
  
  // omd api key -- get your own!
  const omdApiKey = 'xxxxxxxx'
  
  // we'll use this query throughout
  const query = 't=Casino'
  
  // make up the relative url params
  const url = `?${query}&${apiKey}`
  
  // and the title property of the expected response
  const title = "Casino"

some consts for the tests

Test Framework

This demo will all happen with a unit test section that looks like this

// get a unit test instance
const unit = new bmUnitTester.Unit({ showErrorsOnly: true })

// section housing many tests
unit.section (()=> {
    .... demo tests

}, {
	description: 'open mdb using vanilla superfetch',
})

test framework

A minimal Superfetch

SuperFetch is dependency free, so you’ll need to give it UrlFetchApp to fiddle with. This instance will have nothing fancy like caching yet.

    // a minimal superfetch
    let superFetch = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp
    })

minimal superfetch

A fetcher

Now we need to create a fetcher specific to the API we want to access. The same SuperFetch can be used with multiple APIS. This fetcher is a proxy for UrlFetch, but with added features

    // make a fetcher
    let fetcher = superFetch.proxy({
      endPoint: omdEndPoint
    })

fetcher

A fetcher response

Much of the faffing around with UrlFetch is converting to JSON, checking for and handling failure etc. SuperFetch does all that and returns something that can generally be used as a oneLiner as you’ll see in this demo. The response always looks like this.

/**
 * This is the response from the apply proxied function
 * @typedef PackResponse
 * @property {boolean} cached whether item was retrieved from cache
 * @property {object||null} data parsed data
 * @property {number} age age in ms of the cached data
 * @property {Blob|| null} blob the recreated blob if there was one
 * @property {boolean} parsed whether the data was parsed
 * @property {HttpResponse} response fetch response
 * @property {function} throw a function to throw on error
 * @property {Error || string || null} the error if there was one
 * @property {number} responseCode the http response code 
 */

PackResponse

The regular HttpResponse from UrlFetch is a property of the response should you need to reference it, but since most APIS will be returning JSON, you’ll find the useful stuff already parsed in the data property. We’ll get to the other properties as we go along.

These first 2 tests check that the call worked, and that the data.Title is as expected

    unit.is(200, fetcher(url).responseCode, {
      description: 'check we code a good response code'
    })

    unit.is(title, fetcher(url).data.Title, {
      description: 'check we found the film'
    })

simple fetches

throw method

There’s an additional method returned from superFetch – .throw(). If you add this to the fetch request it’ll throw an error if it detects one, otherwise it’ll just return the response as usual. This allows for easy one liner testing.

    unit.is(title, fetcher(url).throw().data.Title, {
      description: 'an invisible throw when no errors'
    })

pass through throw() method

Testing for thrown error

Unit test can also check for an error being thrown. In this example, we’ll send an invalid api key and check that an error was thrown and it was the error we expected.

    unit.is('Invalid API key!',
      unit.threw(() => fetcher(url   'rubbish').throw()).Error, {
      description: 'should throw'
    })

testing for a throw

Caching

SuperFetch has caching built in using Apps script caching with compression and enhanced size limitations.

The current instance doesn’t have caching enabled, so a fetch will not be found in cache.

    unit.is(false, fetcher(url).cached, {
      description: 'no caching turned on'
    })

no caching yet

Here’s how to create a caching instance – passing the Apps Script CacheService of your choice.

    // add caching
    superFetch = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      cacheService: CacheService.getUserCache()
    })

    // make a cached fetcher
    fetcher = superFetch.proxy({
      endPoint: omdEndPoint
    })

fetcher with caching

Fetches using this fetcher will write and read from cache in preference to going to to the API

    unit.is(title, fetcher(url).data.Title, {
      description: 'check we found the film'
    })

    unit.is(true, fetcher(url).cached, {
      description: 'caching now turned on'
    })

caching is now on

Fake HttpResponse response

If an item has been retrieved to cache it of course won’t have an httpResponse available (should you need to access the headers etc), but SuperFetch creates a fake one based on the values of the original httpResponse

    unit.is(200, fetcher(url).responseCode, {
      description: 'even though its from cache we still get a reponse code'
    })


    unit.is('object', typeof fetcher(url).response.getHeaders(), {
      description: 'even though its from cache we still have access to some response functions'
    })

fake reponse

Throttling

Some APIS need you to throttle calls. SuperFetch uses Rate limit handler and helper: Test and manage rate limiting APIS to throttle calls if you need this.

Let’s say an API doesn’t want to you make calls more than every 2 seconds. Here’s how.

    // with throttling && nocaching
    let delay = 2000
    fetcher = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      rottler: bmSuperFetch.Libraries.bmRottler.newRottler({
        delay
      })
    }).proxy({
      endPoint: omdEndPoint
    })

throttling

Sub Libraries

You’ll notice that there is a bmSuperFetch.Libraries property. This allows you to access any of the libraries used by SuperFetch directly – so you don’t have to bother adding a reference to them in your own script. In other words, in the fragment of code above, you’re able to access the bmRottler library via bmSuperFetch.Libraries.bmRottler.

Throttling delay demo

Here there should be at least 2000ms between each call

    let start = new Date().getTime()
    unit.is(false, fetcher(url).cached, {
      description: 'no caching turned on'
    })
    unit.is(true, (() => {
      fetcher(url)
      return (new Date().getTime() - start) > delay
    })(), {
      description: `should be at least ${delay}ms since last call`
    })

throttle

Rate limiting

This is a bit different that delay. This is to deal with the case where an API says it will accept only ‘n’ calls in a period. For example 100 calls an hour.

Delay can also be combined with rate limiting – so in this example we’re allowing 2 calls every 4 seconds, with each call at lease 500ms apart. I’ve also overridden the sleep function, so we can see exactly how long it is waiting till it makes the next call

    let period = 4000
    let rate = 2
    delay = 500

    fetcher = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      rottler: bmSuperFetch.Libraries.bmRottler.newRottler({
        delay,
        period,
        rate,
        sleep: (ms) => {
          // override the default sleeper and watch how long we're sleeping for
          console.log('sleeping for', ms)
          Utilities.sleep(ms)
        }
      })
    }).proxy({
      endPoint: omdEndPoint
    })

rate limiting and delay

Here’s the tests

    start = new Date().getTime()
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting 1'
    })
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting 2'
    })
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting3'
    })
    unit.is(true, new Date().getTime() - start > period, {
      description: `should be at least ${period delay}ms start to finish`
    })

rate and delay demo

Complete test set

For convenience, here’s all of those tests wrapped in a test section

const _ = undefined
  const omdEndPoint = 'http://www.omdbapi.com'
  const omdApiKey = 'xxxx'
  unit.section(() => {

    // a minimal superfetch
    let superFetch = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp
    })
    // make a fetcher
    let fetcher = superFetch.proxy({
      endPoint: omdEndPoint
    })

    const apiKey = `apiKey=${omdApiKey}`
    const query = 't=Casino'
    const url = `?${query}&${apiKey}`
    const title = "Casino"

    unit.is(200, fetcher(url).responseCode, {
      description: 'check we code a good response code'
    })

    unit.is(title, fetcher(url).data.Title, {
      description: 'check we found the film'
    })

    unit.is(title, fetcher(url).throw().data.Title, {
      description: 'an invisible throw when no errors'
    })
    unit.is('Invalid API key!',
      unit.threw(() => fetcher(url   'rubbish').throw()).Error, {
      description: 'should throw'
    })

    unit.is(false, fetcher(url).cached, {
      description: 'no caching turned on'
    })

    // add caching
    superFetch = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      cacheService: CacheService.getUserCache()
    })

    // make a cached fetcher
    fetcher = superFetch.proxy({
      endPoint: omdEndPoint
    })

    unit.is(title, fetcher(url).data.Title, {
      description: 'check we found the film'
    })

    unit.is(true, fetcher(url).cached, {
      description: 'caching now turned on'
    })



    unit.is(200, fetcher(url).responseCode, {
      description: 'even though its from cache we still get a reponse code'
    })


    unit.is('object', typeof fetcher(url).response.getHeaders(), {
      description: 'even though its from cache we still have access to some response functions'
    })

    // with throttling && nocaching
    let delay = 2000
    fetcher = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      rottler: bmSuperFetch.Libraries.bmRottler.newRottler({
        delay
      })
    }).proxy({
      endPoint: omdEndPoint
    })
    let start = new Date().getTime()
    unit.is(false, fetcher(url).cached, {
      description: 'no caching turned on'
    })
    unit.is(true, (() => {
      fetcher(url)
      return (new Date().getTime() - start) > delay
    })(), {
      description: `should be at least ${delay}ms since last call`
    })

    // with rate limiting
    let period = 4000
    let rate = 2
    delay = 500

    fetcher = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      rottler: bmSuperFetch.Libraries.bmRottler.newRottler({
        delay,
        period,
        rate,
        sleep: (ms) => {
          // override the default sleeper and watch how long we're sleeping for
          console.log('sleeping for', ms)
          Utilities.sleep(ms)
        }
      })
    }).proxy({
      endPoint: omdEndPoint
    })
    start = new Date().getTime()
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting 1'
    })
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting 2'
    })
    unit.is(title, fetcher(url).data.Title, {
      description: 'starting rate limiting3'
    })
    unit.is(true, new Date().getTime() - start > period, {
      description: `should be at least ${period delay}ms start to finish`
    })

  }, {
    description: 'open mdb using vanilla superfetch'
  })

complete test set

Making an API class

For illustration, let’s make a simple API class based on SuperFetch to access the omd api. In later articles I’ll demonstrate more complex APIs to access the Google REST APIS for gcs, drv, iam and others. They’ll all follow the same pattern as this simple demo, but of course with more methods and capabilities.

OmdApi class

This class can substitute for the fetcher in the previous examples


/**
 * @typedef OmdOptions
 * @property {_SuperFetch} superFetch a superfetch instance
 * @property {boolean} noCache whether to cache
 * @property {boolean} showUrl whether to showUrls when fetching
 * @property {object[]} extraParams always add these params
 */
class _OmdApi {
  /**
   * @param {OmdOptions} 
   * @return {_OmdApi}
   */
  constructor({
    superFetch,
    // caching can be used here if you don't want to keep calling the iam service for each call
    // but of course they do expire
    noCache = false,
    showUrl,
    extraParams = []
  }) {
    this.extraParams = Utils.arrify(extraParams)
    this.proxy = superFetch.proxy({
      endPoint: `http://www.omdbapi.com`,
      noCache,
      showUrl
    })
  }

  makePath({ path = '', params }) {
    return Utils.makeUrl({
      url: Utils.makepath({ path, base: '' }),
      params: params.concat(this.extraParams)
    })
  }

  get({path=''} = {}, ...params) {
    return this.proxy(this.makePath({path, params }))
  }
}

var OmdApi = _OmdApi

OmdApi

Using the OmdApi

Here’s a few examples

unit.section(() => {

    let superFetch = new bmSuperFetch.SuperFetch({
      fetcherApp: UrlFetchApp,
      cacheService: CacheService.getUserCache()
    })

    // make an api
    const omd = new OmdApi({
      superFetch,
      extraParams: {
        apiKey: omdApiKey
      }
    })
    const title = "Casino"
    const query = {t:title}

    unit.is(200, omd.get(_,query).responseCode, {
      description: 'check we code a good response code'
    })

    unit.is(title, omd.get(_,query).data.Title, {
      description: 'check we found the film'
    })

    unit.is(title, omd.get(_,query).throw().data.Title, {
      description: 'an invisible throw when no errors'
    })

    unit.is(true, omd.get(_,query).throw().cached, {
      description: 'caching turned on'
    })

    // lets make a noncache version of a fetcher
    const omdNoCache = new OmdApi({
      noCache: true,
      superFetch,
      extraParams: {
        apiKey: omdApiKey
      }
    })
    
    unit.is(false, omdNoCache.get(_,query).cached, {
      description: 'now no cache'
    })

    unit.is(false, omd.get(_,query).cached, {
      description: 'it also cleared the cache from the other cacher'
    })

    unit.is(true, omd.get(_,query).cached, {
      description: 'but now its back in cache again'
    })

    unit.is(true, omd.get(_,query).age < 1000, {
      description: 'you can see even how long ago the cache entry was written'
    })

  }, {
    description: 'open mdb using omdApi'
  })

some example for omdApi

What’s next

In future articles we’ll look at other Apis built on SuperFetch, as well as built in recovery from Rate Limit errors, authentication and impersonation. But that’s it for now