Running things in parallel using HTML service was a brief intro on how to run a number of things at once, orchestrating executing using Google Apps Script HTML service. In Some hints on setting up parallel running profiles I showed how to set up some complex profiles. We’ll use them as the basis for this example, so you should read that first.
The profiles
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
function logEmailsProfile() { var profile = []; // get the matching threads var profileThreads = [ { "name": "GET THREADS", "functionName": "getTheThreads", "skip": false, "options": { "searchText": "The Excel Liberation forum has moved to a Google+ community" } } ]; |
Now we need to split the threads into some number of chunks to be worked on in parallel. We’ll write the getMessages() function later
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
// get and process all the messages var CHUNKS = 3; var profileMessages = []; for (var i =0; i <CHUNKS;i++ ) { profileMessages.push ({ "name": "MESSAGES-"+i, "functionName":"getMessages", "skip":false, "options":{ index: i, threads:CHUNKS, } }); } |
Next, a reduction to bring all the results together. As usual we’ll use the common function reduceTheResults(), that we’ve used in all the examples for that
1 2 3 4 5 6 7 8 |
// next reduce the messages to one var profileReduction = []; profileReduction.push({ "name": "reduction", "functionName":"reduceTheResults", "options":{ } }); |
Finally we’ll log all the results in a spreadsheet. We can use a common logTheResults() function for that too
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
// finally log the results var profileLog = [{ "name": "LOG", "functionName": "logTheResults", "skip": false, "options": { "driver": "cDriverSheet", "clear": true, "parameters": { "siloid": "emails", "dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk", "peanut": "bruce" } } }]; |
Now we create a single profile to sequence all of those components
1 2 3 4 5 6 7 8 9 |
// put it all together profile.push ( profileThreads, profileMessages, profileReduction, profileLog ); return profile; } |
The only change we need to make now is to call this function to set up the run profile
1 2 3 4 5 6 |
function showSidebar() { // kicking off the sidebar executes the orchestration libSidebar('asyncService',ADDONNAME, logEmailsProfile () ); } |
The executors
1 2 3 4 5 6 7 |
function getTheThreads(options) { return cUseful.rateLimitExpBackoff( function() { return GmailApp.search(options.searchText).map(function(d) { return d.getId(); }); }); } |
and this one will be run in parallel, processing chunks of the total messages
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
/** * do a chunk of message processing * @param {object} options describes what to do * @param {object} reduceResults this would contain results from a previous stage if present * @return {object} test data to pass on to next stage */ function getMessages (options,reduceResults) { // we'll only do a section of data in this thread var data = reduceResults[0].results; var start = Math.round(options.index/options.threads * data.length); var finish = Math.round((options.index+1)/options.threads * data.length) ; // work with that slice of messages return data.slice (start, finish).reduce ( function (p,c) { // for later decrypt testing, we'll include everything cUseful.rateLimitExpBackoff(function () { GmailApp.getThreadById(c).getMessages().forEach(function(d) { cUseful.arrayAppend(p, d.getTo().split(",").map(function(e) { return {to:e,subject:d.getSubject(),dateSent:d.getDate().toString(),from:d.getFrom()}; })); }); },1000); return p; },[]); } |
Here’s a snap of the run – We got 1026 seconds of processing over 371 seconds, and more importantly – managed to process something we wouldn’t have been able to inside of a 6 minute limit
Less is sometimes more
Authorization
1 2 3 4 |
function smallEmailTest() { var messages = getTheThreads({searchText:'something bizarre'}); Logger.log(getMessages ({index:0,threads:2},[{results:messages}])); } |