Custom checking for exponential backoff

If you are using my exponential backoff function from the cUseful library you will be aware that an apps script error that qualifies as a something worth retrying will provoke retries. The benefit of this is that you can transparently deal with rate limiting issues.

It’s used like this – you simply wrap your function

var result = cUseful.rateLimitExpBackoff(function() {
    // some function that might be rate limited
    // for example .... return UrlFetchApp.fetch(url, finalOptions);
	});

Of course it relies on two things for detecting whether it should enter a round of exponential backoff

Non -apps script errors

When you are using external APIS it’s impossible to predict all retryable errors, since different APIS would have entirely different responses that could be described a retryable.

Exposing exponential functionality

This post is about enabling a capability to signal remaining within the exponential backoff wrapper if an external API error is detected that does not provoke an apps script error

Example

When developing Getting your apps scripts to Github I came across a bug in the Github API that was a perfect candidate for this treatment – described here. To cut a long story short, if you throw too many things at github too quickly it randomly returns a consistency error message if it hasn’t caught up. This is the kind of thing that the advice usually given is to wait a few seconds between requests – but why wait, when 90% of the cases don’t fail and dont need to wait. In other words a perfect candidate for exponential backoff.

Here’s my commit function –

return cUrlResult.urlPost (repoObject.contents_url.replace("{+path}",path), options,
    "PUT", self.accessToken, self.contentOptions(),
    function (expResult) {
       // this is a checker to force an exponential backoff even if there isnt an http error thrown.
       // specifically to address this github http://stackoverflow.com/questions/19576601/github-api-issue-with-file-upload
       var code = expResult.getResponseCode();
       if (409 === code) {
         throw cUseful.TRYAGAIN;
       }
       return expResult; 
    }
	);

Let’s look at what curlResult is doing

var result = cUseful.rateLimitExpBackoff(function() {
  return UrlFetchApp.fetch(url, finalOptions);
  }, undefined , undefined, undefined , undefined , optChecker);

You’ll see that there is a new argument – optChecker – its purpose is to be able to signal back to rateLimitBackoff() that this is a retryable error and it shouls continue to work through the normal exponential back off process.

In the case of the gitHub example, the optChecker function was this.

function (expResult) {
  var code = expResult.getResponseCode();
  if (409 === code) {
    throw cUseful.TRYAGAIN;
  }
  return expResult;
  }

If you pass an optChecker function to rateLimitBackoff() it will call it after executing the main function (usually a urlFetch), passing to it the result of the main function.

In the example above, the gutHub retryable error is indicated by a response code 409. optChecker signals back to rateLimitBackoff() that it should try again by throwing the special error cUseful.TRYAGAIN, otherwise it should simply return the passed result.

Like this, you have full control over retrying errors that wouldn;y normally provoke a retry cycle

Many of the snippets in this section of the site are part of the cUseful library. You can find the details below.

Why not join our forum,follow the blog or follow me on twitter to ensure you get updates when they are available.