Using exponential backoff with github api - dealing with data "in preparation"

Some of the responses from the GitHub API rely on the answers being in its cache. If you make a request and there is no available cache result, it returns an HTTP code (202) to let you know that you need to check back later (not very much later, but shortly). This is pretty standard and quite a few APIS work this way.

In Apps Script, one way would be to sleep for a while, and try again until you get a 200 code (or some failure), which means the data is ready. A better way is to use exponential backoff, which will wait increasingly longer periods of time and finally give up. 

Exponential backoff is usually used for dealing with rate limiting, and of course the GitHub API also has rate limiting, so it's ideal to use it for both purposes. You'll find a write up on my exponential backoff here Custom checking for exponential backoff, along with where to get it.

Custom checking with exponential backoff


One of the the options to exponential backoff is a lookahead function. Here's a simple example to detect a rate limit error for an API that signals such an error in its JSON response (as opposed to via an HTTP code). The lookahead functions returns true if it recognizes the response as a rate limit error, and false if it doesn't recognize the response

    var response = cUseful.Utils.expBackoff (
      function () {
        return UrlFetchApp.fetch(url , {
          muteHttpExceptions:true,                     
        });
      }, {
        lookahead:function (response,attempt) {
          // the api doesnt fail on rate limiting in any case
          // so we need to parse the content (if parseable)
          try {
            var r = JSON.parse(response.getContentText());
            return r && r.error && r.error.indexOf('code: 429') !== -1;
          }
          catch(err) {
            return false;
          }
        }

GitHub API 


Some of the requests to the github API (for example commit information), returns a 202 while it is preparing a response and its up to you to keep trying till you get a 200 or some error.  In my main scripts, I can't be bothered to worry about this so I've built it into the library I use for GitHub access and can simply forget all about it.

The github API is a little more complex that this as there is pagination to deal with, along with reading information from the response headers, so for my bulk github/apps script implementation, Getting your apps scripts to Github, I use my library cGitJsonAPI which knows how to deal with all of that

1_4RfsIW57fdzWh7T38O9IfGdTRgYbOSyC5PvsOm3a4GU1sxllw8blEUl

along with cUrlResult  which uses exponential backoff behind the scenes

1NtAiJulZM4DssyN0HcK2XXTnykN_Ir2ee2pXV-CT367nKbdbTvRX4pTM


How to get commit data

Of course - this is pretty easy with the libraries (I use goa for managing access tokens - see Github service for Goa examples), so here's the entire code for getting commit information on a given repo.

  var git = new cGitJsonApi.cGitJsonApi(SETTINGS).setAccessToken( getAccessToken('git'));
  Logger.log (JSON.stringify(git.getCommitActivity  (SETTINGS.git.userAgent , "cUseful" ) ));


Which gives me this
{
"success": true,
"data": [{
"days": [0, 4, 0, 0, 0, 0, 0],
"total": 4,
"week": 1472947200
}, {
"days": [0, 0, 0, 0, 0, 0, 0],
"total": 0,   etc....


Adding lookahead function to my cGitJsonApi,meant just a small change to provide a lookahead function to be passed to exponential backoff in the method that receives data from the JSON API and unpaginates them, like this

    var result = cUrlResult.urlGet(url,accessToken,options,function (response,attempt) {
      try {
        return response && response.getResponseCode() === 202;
      }
      catch(err) {
        return false;
      }
    });

Of course there's a lot going on around this, but the key point is that you can use exponential backoff to take care of many API quirks like this one and get the complication out of your main code and into utilities like this one. This lookahead simply checks for a 202 code  - if it is, exponential backoff knows how to retry, how long to wait and how many times to attempt. If it's not a 202, then processing will continue normally.

The full updated method looks like this
/**
  * get intercept to deal with pagination
  * @param {string} url
  * @param {string} accessToken
  * @param {object} options
  * @param {Array.object} data so far
  * @return {object} standard result object
  */
  self.getUnpaged = function (url,accessToken,options,data) {
    
    // a 202 for the github api means the result isn't quite ready yet
    
    data = data || [];
    var result = cUrlResult.urlGet(url,accessToken,options,function (response,attempt) {
      try {
        return response && response.getResponseCode() === 202;
      }
      catch(err) {
        return false;
      }
    });
    
    // need to recurse for multiple pages
    if (result.success) {
      result.data = cUseful.arrayAppend(data,result.data);
      var h = result.headers.Link;
      
      if(h) {
        var link = /<([^>]*)>;\s?rel="next"/.exec(h);
        if(link) {
          var newUrl = link[1].toString();
          self.getUnpaged ( newUrl , accessToken, options , result.data);
        }
      }  
      
    }
    
    return result;
    
  } 

and is called like this
  /**
  * get comments
  * @param {string} owner
  * @param {string} repo
  * @return {object} standard result object
  */
  self.getCommitActivity = function (owner , repo ) {
    return self.getUnpaged (self.apiBase() + "/repos/" + owner + "/" + repo + "/stats/commit_activity", self.accessToken , self.apiOptions());
  };

For more like this, see Google Apps Scripts snippets. Why not join our forum, follow the blog or follow me on twitter to ensure you get updates when they are available.

You want to learn Google Apps Script?

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, available All formats are available now from O'Reilly,Amazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.






Comments