Running things in parallel using HTML service was a brief intro on how to run a number of things at once, orchestrating executing using Google Apps Script HTML service. In Some hints on setting up parallel running profiles I showed how to set up some complex profiles. We’ll use them as the basis for this example, so you should read that first.
The profiles
function logEmailsProfile() { var profile = []; // get the matching threads var profileThreads = [ { "name": "GET THREADS", "functionName": "getTheThreads", "skip": false, "options": { "searchText": "The Excel Liberation forum has moved to a Google+ community" } } ];
Now we need to split the threads into some number of chunks to be worked on in parallel. We’ll write the getMessages() function later
// get and process all the messages var CHUNKS = 3; var profileMessages = []; for (var i =0; i <CHUNKS;i++ ) { profileMessages.push ({ "name": "MESSAGES-"+i, "functionName":"getMessages", "skip":false, "options":{ index: i, threads:CHUNKS, } }); }
Next, a reduction to bring all the results together. As usual we’ll use the common function reduceTheResults(), that we’ve used in all the examples for that
// next reduce the messages to one var profileReduction = []; profileReduction.push({ "name": "reduction", "functionName":"reduceTheResults", "options":{ } });
Finally we’ll log all the results in a spreadsheet. We can use a common logTheResults() function for that too
// finally log the results var profileLog = [{ "name": "LOG", "functionName": "logTheResults", "skip": false, "options": { "driver": "cDriverSheet", "clear": true, "parameters": { "siloid": "emails", "dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk", "peanut": "bruce" } } }];
Now we create a single profile to sequence all of those components
// put it all together profile.push ( profileThreads, profileMessages, profileReduction, profileLog ); return profile; }
The only change we need to make now is to call this function to set up the run profile
function showSidebar() { // kicking off the sidebar executes the orchestration libSidebar('asyncService',ADDONNAME, logEmailsProfile () ); }
The executors
function getTheThreads(options) { return cUseful.rateLimitExpBackoff( function() { return GmailApp.search(options.searchText).map(function(d) { return d.getId(); }); }); }
and this one will be run in parallel, processing chunks of the total messages
/** * do a chunk of message processing * @param {object} options describes what to do * @param {object} reduceResults this would contain results from a previous stage if present * @return {object} test data to pass on to next stage */ function getMessages (options,reduceResults) { // we'll only do a section of data in this thread var data = reduceResults[0].results; var start = Math.round(options.index/options.threads * data.length); var finish = Math.round((options.index+1)/options.threads * data.length) ; // work with that slice of messages return data.slice (start, finish).reduce ( function (p,c) { // for later decrypt testing, we'll include everything cUseful.rateLimitExpBackoff(function () { GmailApp.getThreadById(c).getMessages().forEach(function(d) { cUseful.arrayAppend(p, d.getTo().split(",").map(function(e) { return {to:e,subject:d.getSubject(),dateSent:d.getDate().toString(),from:d.getFrom()}; })); }); },1000); return p; },[]); }
Here’s a snap of the run – We got 1026 seconds of processing over 371 seconds, and more importantly – managed to process something we wouldn’t have been able to inside of a 6 minute limit
Less is sometimes more
Authorization
function smallEmailTest() { var messages = getTheThreads({searchText:'something bizarre'}); Logger.log(getMessages ({index:0,threads:2},[{results:messages}])); }