What is SuperFetch
SuperFetch is a proxy for UrlFetchApp which has these extra capabilities – most of which I use for all APIS
- Built in caching and compression – it uses Apps script caching with compression and enhanced size limitations
- A standard reponse format and error handling
- Inbuilt JSON parsing
- Automatic token insertion
- One liner capability to make Unit Testing less of a chore
- Plug-in enhancements such as automatic throttling and rate limit management
You’ll need the bmSuperFetch and bmUnitTest libraries – details at the end.
Throughout this article I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls
I originally create superFetch to use as a plugin replacement for UrlFetch to my api library (I have one library that I use to access all the APIS I use regularily – mainly Google ones). For this article, though I’ll use the Open Movie Database API (omd) and move onto some more complex APIS in later articles.
The omd needs an api key and a query specifed as url parameters.
A SuperFetch instance
Before we start, let’s set up a few constants to use throughought the test
This demo will all happen with a unit test section that looks like this
A minimal Superfetch
SuperFetch is dependency free, so you’ll need to give it UrlFetchApp to fiddle with. This instance will have nothing fancy like caching yet.
Now we need to create a fetcher specific to the API we want to access. The same SuperFetch can be used with multiple APIS. This fetcher is a proxy for UrlFetch, but with added features
A fetcher response
Much of the faffing around with UrlFetch is converting to JSON, checking for and handling failure etc. SuperFetch does all that and returns something that can generally be used as a oneLiner as you’ll see in this demo. The response always looks like this.
The regular HttpResponse from UrlFetch is a property of the response should you need to reference it, but since most APIS will be returning JSON, you’ll find the useful stuff already parsed in the data property. We’ll get to the other properties as we go along.
These first 2 tests check that the call worked, and that the data.Title is as expected
There’s an additional method returned from superFetch – .throw(). If you add this to the fetch request it’ll throw an error if it detects one, otherwise it’ll just return the response as usual. This allows for easy one liner testing.
Testing for thrown error
Unit test can also check for an error being thrown. In this example, we’ll send an invalid api key and check that an error was thrown and it was the error we expected.
SuperFetch has caching built in using Apps script caching with compression and enhanced size limitations.
The current instance doesn’t have caching enabled, so a fetch will not be found in cache.
Here’s how to create a caching instance – passing the Apps Script CacheService of your choice.
Fetches using this fetcher will write and read from cache in preference to going to to the API
Fake HttpResponse response
If an item has been retrieved to cache it of course won’t have an httpResponse available (should you need to access the headers etc), but SuperFetch creates a fake one based on the values of the original httpResponse
Some APIS need you to throttle calls. SuperFetch uses Rate limit handler and helper: Test and manage rate limiting APIS to throttle calls if you need this.
Let’s say an API doesn’t want to you make calls more than every 2 seconds. Here’s how.
You’ll notice that there is a bmSuperFetch.Libraries property. This allows you to access any of the libraries used by SuperFetch directly – so you don’t have to bother adding a reference to them in your own script. In other words, in the fragment of code above, you’re able to access the bmRottler library via bmSuperFetch.Libraries.bmRottler.
Throttling delay demo
Here there should be at least 2000ms between each call
This is a bit different that delay. This is to deal with the case where an API says it will accept only ‘n’ calls in a period. For example 100 calls an hour.
Delay can also be combined with rate limiting – so in this example we’re allowing 2 calls every 4 seconds, with each call at lease 500ms apart. I’ve also overridden the sleep function, so we can see exactly how long it is waiting till it makes the next call
Here’s the tests
Complete test set
For convenience, here’s all of those tests wrapped in a test section
Making an API class
For illustration, let’s make a simple API class based on SuperFetch to access the omd api. In later articles I’ll demonstrate more complex APIs to access the Google REST APIS for gcs, drv, iam and others. They’ll all follow the same pattern as this simple demo, but of course with more methods and capabilities.
This class can substitute for the fetcher in the previous examples
Using the OmdApi
Here’s a few examples
In future articles we’ll look at other Apis built on SuperFetch, as well as built in recovery from Rate Limit errors, authentication and impersonation. But that’s it for now