In Running GmailApp in parallel we found that throwing additional parallel executors at a rate limited service doesn’t help much – it will only let you do so much at the same time. However Running things in parallel using HTML service is a sequential scheduler as well as a parallel orchestrator. So we can split the job up and run sequential sets of parallel threads to eventually get the job done. A head and tail janitor process carry forward the work that has already been done, or needs to be done to a reduction process ready for the next set. In this way we get through all the work, as long as we keep each thread to below 6 minutes.

As usual we’ll create profiles and some new .map executors for this.

The profiles

The first task is to get all the threads matching the search text in your email.

Next, a reduction to bring all the results together. As usual we’ll use the common function reduceTheResults(), that we’ve used in all the examples for that. The reason for a reduction is that we might later split the getTheThreads() function into parallel execution tasks, and we’d need to combine the results.

 

Now we need to split the threads into some number of sets, and within each set, a number of threads to be worked on in parallel. First the head janitor function. His role is to carry forward any work that has already been done in previous sets.

 

 

Now the threads within each set

 

And then the tail janitor – his job will be to carry forward the work that needs to be done in subsequent sets

 

We need a reduction between each set.

 

Now we need a special reduction that will blow out cases where we have multiple recipients into separate records

 

Finally we’ll need to log the results

 

The only change we need to make now is to call this function to set up the run profile

 

The executors

Now we need to write the couple of new functions mentioned in these profiles that will be called by the htmlservice
This one will search the email for threads that match the search term – this is the same as the one used in Running GmailApp in parallel
 

and this one will be run in parallel, processing chunks of the total messages. It differs from the one in Running GmailApp in parallel, since we now have the concept of ‘sets’. We’re also postponing the duplication of records with multiple recipients in the to: list until the final reduction, rather than doing it here.

 

Here’s the modified final reduction – expanding the to: recipient field

 

Here’s a snap of the run – We got 725 seconds of processing over 386 seconds, and managed to process something we wouldn’t have been able to inside of a 6 minute limit, even with parallel execution.

Remember these tips:

  • it doesn’t help to throw lots of executors at a rate limited service – most of the time will be spent waiting. Running scheduled sets will eventually get it done.
  • Your execution processes should never change the number of records. The end of a set should always have the same as it started with – execution threads are .map() operations
  • If your final result does need to change the number of records, it should be done in a final reduction.

For the code you need to set this all up, see Parallel implementation and getting started

Authorization

GmailApp needs authorization, so its worth running a small test first to force an authorization dialog, as well as to test your executor functions. That can easily be done by emulating a couple of execution steps in a simple function like this.
 

For more snippets like this see Google Apps Scripts snippets