In Apps Script, one way would be to sleep for a while, and try again until you get a 200 code (or some failure), which means the data is ready. A better way is to use exponential backoff, which will wait increasingly longer periods of time and finally give up.
Exponential backoff is usually used for dealing with rate limiting, and of course the GitHub API also has rate limiting, so it’s ideal to use it for both purposes. You’ll find a write up on my exponential backoff here Custom checking for exponential backoff, along with where to get it.
Custom checking with exponential backoff
One of the options to exponential backoff is a lookahead function. Here’s a simple example to detect a rate limit error for an API that signals such an error in its JSON response (as opposed to via an HTTP code). The lookahead functions return true if it recognizes the response as a rate limit error, and false if it doesn’t recognize the response
var response = cUseful.Utils.expBackoff (
function () {
return UrlFetchApp.fetch(url , {
muteHttpExceptions:true,
});
}, {
lookahead:function (response,attempt) {
// the api doesnt fail on rate limiting in any case
// so we need to parse the content (if parseable)
try {
var r = JSON.parse(response.getContentText());
return r && r.error && r.error.indexOf('code: 429') !== -1;
}
catch(err) {
return false;
}
}
GitHub API
Some of the requests to the github API (for example commit information), returns a 202 while it is preparing a response and its up to you to keep trying till you get a 200 or some error. In my main scripts, I can’t be bothered to worry about this so I’ve built it into the library I use for GitHub access and can simply forget all about it.
The github API is a little more complex that this as there is pagination to deal with, along with reading information from the response headers, so for my bulk github/apps script implementation, Getting your apps scripts to Github, I use my library cGitJsonAPI which knows how to deal with all of that
1_4RfsIW57fdzWh7T38O9IfGdTRgYbOSyC5PvsOm3a4GU1sxllw8blEUl
along with cUrlResult which uses exponential backoff behind the scenes
1NtAiJulZM4DssyN0HcK2XXTnykN_Ir2ee2pXV-CT367nKbdbTvRX4pTM
How to get commit data
Of course – this is pretty easy with the libraries (I use goa for managing access tokens – see Github service for Goa examples), so here’s the entire code for getting commit information on a given repo.
var git = new cGitJsonApi.cGitJsonApi(SETTINGS).setAccessToken( getAccessToken('git'));
Logger.log (JSON.stringify(git.getCommitActivity (SETTINGS.git.userAgent , "cUseful" ) ));
Which gives me this
{
"success": true,
"data": [{
"days": [0, 4, 0, 0, 0, 0, 0],
"total": 4,
"week": 1472947200
}, {
"days": [0, 0, 0, 0, 0, 0, 0],
"total": 0, etc....
Adding lookahead function to my cGitJsonApi,meant just a small change to provide a lookahead function to be passed to exponential backoff in the method that receives data from the JSON API and unpaginates them, like this
var result = cUrlResult.urlGet(url,accessToken,options,function (response,attempt) {
try {
return response && response.getResponseCode() === 202;
}
catch(err) {
return false;
}
});
Of course there’s a lot going on around this, but the key point is that you can use exponential backoff to take care of many API quirks like this one and get the complication out of your main code and into utilities like this one. This lookahead simply checks for a 202 code – if it is, exponential backoff knows how to retry, how long to wait and how many times to attempt. If it’s not a 202, then processing will continue normally.
The full updated method looks like this
/**
* get intercept to deal with pagination
* @param {string} url
* @param {string} accessToken
* @param {object} options
* @param {Array.object} data so far
* @return {object} standard result object
*/
self.getUnpaged = function (url,accessToken,options,data) {
// a 202 for the github api means the result isn't quite ready yet
data = data || [];
var result = cUrlResult.urlGet(url,accessToken,options,function (response,attempt) {
try {
return response && response.getResponseCode() === 202;
}
catch(err) {
return false;
}
});
// need to recurse for multiple pages
if (result.success) {
result.data = cUseful.arrayAppend(data,result.data);
var h = result.headers.Link;
if(h) {
var link = /]*)>;\s?rel="next"/.exec(h);
if(link) {
var newUrl = link[1].toString();
self.getUnpaged ( newUrl , accessToken, options , result.data);
}
}
}
return result;
}
and is called like this
/**
* get comments
* @param {string} owner
* @param {string} repo
* @return {object} standard result object
*/
self.getCommitActivity = function (owner , repo ) {
return self.getUnpaged (self.apiBase() + "/repos/" + owner + "/" + repo + "/stats/commit_activity", self.accessToken , self.apiOptions());
};