Custom checking for exponential backoff

If you are using my exponential backoff  function from the cUseful library you will be aware that an apps script error that qualifies as a something worth retrying will provoke retries. The benefit of this is that you can transparently deal with rate limiting issues. 

It's used like this - you simply wrap your function 

var result = cUseful.rateLimitExpBackoff(function() {
    // some function that might be rate limitded
    // for example .... return UrlFetchApp.fetch(url, finalOptions);
});

Of course it relies on two things for detecting whether it should enter a round of exponential backoff
  • Identifying the error messages that would benefit from retrying.
  • Apps script throwing an error in the first place

Non -apps script errors

When you are using external APIS it's impossible to predict all retryable errors, since different APIS would have entirely different responses that could be described a retryable.

Exposing exponential functionality

This post is about enabling a capability to signal remaining within the exponential backoff wrapper if an external API error is detected that does not provoke an apps script error

Example

When developing Getting your apps scripts to Github I came across a bug in the Github API that was a perfect candidate for this treatment - described here. To cut a long story short, if you throw too many things at github too quickly it randomly returns a consistency error message if it hasn't caught up. This is the kind of thing that the advice usually given is to wait a few seconds between requests - but why wait, when 90% of the cases don't fail and dont need to wait. In other words a perfect candidate for exponential backoff. 

Here's my commit function -

return cUrlResult.urlPost (repoObject.contents_url.replace("{+path}",path), options,
    "PUT", self.accessToken, self.contentOptions(),
    function (expResult) {
       // this is a checker to force an exponential backoff even if there isnt an http error thrown.
       // specifically to address this github http://stackoverflow.com/questions/19576601/github-api-issue-with-file-upload
       var code = expResult.getResponseCode();
       if (409 === code) {
         throw cUseful.TRYAGAIN;
       }
       return expResult; 
    }
);


Let's look at what curlResult is doing


var result = cUseful.rateLimitExpBackoff(function() {
  return UrlFetchApp.fetch(url, finalOptions);
}, undefined , undefined, undefined , undefined , optChecker);


You'll see that there is a new argument - optChecker - its purpose is to be able to signal back to rateLimitBackoff() that this is a retryable error and it shouls continue to work through the normal exponential back off process.

In the case of the gitHub example, the optChecker function was this.

function (expResult) {
  var code = expResult.getResponseCode();
  if (409 === code) {
    throw cUseful.TRYAGAIN;
  }
  return expResult;
}

If you pass an optChecker function to rateLimitBackoff() it will call it after executing the main function (usually a urlFetch), passing to it the result of the main function. 

In the example above, the gutHub retryable error is indicated by a response code 409. optChecker signals back to rateLimitBackoff() that it should try again by throwing the special error cUseful.TRYAGAIN, otherwise it should simply return the passed result.

Like this, you have full control over retrying errors that wouldn;y normally provoke a retry cycle


Many of the snippets in this section of the site are part of the cUseful library. You can find the details below.

https://script.google.com/macros/s/AKfycbwZ2Hht93wTNzvRmYINYF7obaOHciBXWcP_wAiEtyGq70_x3cI/exec?list=cUseful



For more like this, see  Google Apps Scripts snippets. Why not join our forum,follow the blog or follow me on twitter to ensure you get updates when they are available. 

You want to learn Google Apps Script?

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, available All formats are available now from O'Reilly,Amazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.






Comments