Using GmailApp in parallel processes

Running things in parallel using HTML service was a brief intro on how to run a number of things at once, orchestrating executing using Google Apps Script HTML service. You may have already tried Parallel implementation and getting started or even Running things in parallel that need oAUTH2 - datastore. Now here's something a little more complicated.

In CryptoJS libraries for Google Apps Script I mentioned that they were kind of slow under Apps Script - depending on the cryptography - up to a second for each one to encrypt/decrypt. Testing this kind of thing is pretty tricky when you run out of quota all the time. Here's how I tested thousands at once with no quota problems.

The process


I was able to orchestrate the parallel running of all this using Parallel process orchestration with HtmlService

Here's a snap of the run - We got 40 minutes of uninterrupted processing over 13 minutes, and processed 16,000 encrypt/decrypt pairs. To speed it up even more all I need to do is to create additional processing threads in the profile.


Orchestration profile

The action is controlled by an orchestration profile.  But there are some wrinkles in this one, because I wanted to make it dynamic, depending on the size of the run. Here's how it's built up. Creating the orchestration is a good first step. We don't even have to write the functions that do the work at this stage.

Create the test data

I'm creating 4000 test items. 2 threads will do for this trivial process.
  // first stage - create test data 
  var SCALE = 2000, CHUNKS = 2;
  var profileTestData = [];
  for (var i =0; i <CHUNKS;i++ ) {
    profileTestData.push ({
        name: 'testdata-'+i,
        functionName:'cryptoTestData',
        options:{
          scale:SCALE
        }
    });
  } 

Reduce the results

After a parallel map operation, you always need to reduce the results. This ensures you have removed any dependencies between how the previous step was executed. I use a standard function for all reduce functions.
  // next reduce the test data to one
  var profileReduction = [];
  profileReduction.push({
    name: 'reduction',
    functionName:'reduceTheResults',
    options:{}
  });

Do the encryption

I'm applying 4 different kind of encryption to each of the randomly generated messages and passwords that were created in the first process. I'm going to give 2 threads to each encryption method, so there will be 8 threads, each of which will handle 2000 encryptions. I'll explain later what we're going to use the options for.
  // next,execute the cryptotests
  var CRYPTOTHREADS =2,CIPHERS=['tripledes','aes','rabbit','des'];

  var profileCrypto = [];
  
  for (var i = 0; i < CRYPTOTHREADS ; i++) {
    CIPHERS.forEach (function (d) {
      profileCrypto.push ({
        name:'crypto-'+d+'-'+i,
        functionName:'cryptoEncrypt',
        options: {
          index: i,
          threads:CRYPTOTHREADS,
          cipher:d
        }
      });
    });
  }

Reduce the results

Again we need to reduce the results. We can just use the same reduction profile as previously

Check that it worked

Decrypting everything and checking against the original messages will show that the operation worked. This time I'll go for 6 threads. I'll explain the option values later.
  // next we'll test the results by decrypting and check against original
  var DECRYPTOTHREADS =  6;
  var profileDecrypto = [];
  for (var i=0; i < DECRYPTOTHREADS ;i++ ) {
    profileDecrypto.push ({
      name:'decrypto-'+i,
      functionName:'cryptoDecryptCheck',
      options: {
        stopOnFail:true,
        index:i,
        threads:DECRYPTOTHREADS
      }
    });
  }

Reduce the results

Again we need to reduce the results. We can just use the same reduction profile as previously.

Log the results

We want to make sure that we ended up with the same number of items that we started with, so do a simple summary  of the reduced data. Most of the options here are about Database abstraction with google apps script, which I always use to simplify writing data to spreadsheets.

  // finally log the results
  var profileLog = [{
    "name": "LOG",
    "functionName": "cryptoLog",
    "skip": false,
    "options": {
      "driver": "cDriverSheet",
      "clear": true,
      "parameters": {
        "siloid": "logencrypt",
        "dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
        "peanut": "bruce"
      }
    }
  }];

Here's what was logged

Put it all together

Now we've created a profile for each section, we can put it all together like so. I'll repeat the whole function at the end of this page
  // put it all together
  profile.push (
    profileTestData,
    profileReduction,
    profileCrypto,
    profileReduction,
    profileDecrypto,
    profileReduction,
    profileLog
  );


The executor functions

These are called to do the work by the profiles we set up above. Each functionName property should point to one of these. In all cases executor functions are passed 2 parameters
  • The options you set up in your profile
  • The data from the previous stage (if there was any)

Creating the test data.

This creates a chunk of random test messages and passwords - volume as defined in options.scale.
/**
 * create some test data for encryption tests
 * @param {object} options describes what to do 
 * @param {object} reduceResults this would contain results from a previous stage if present
 * @return {object} test data to pass on to next stage
 */
function cryptoTestData (options, reduceResults) {
  
  var testData = [];
 //generate some test data
   for (var i=0; i < options.scale ; i++ ) {
     testData.push ( {
       message: cUseful.generateUniqueString(Math.round(Math.random()*100)) ,
       password: cUseful.generateUniqueString(Math.round(Math.random()*10))
     });
   }
  
   return testData;
}

Reduce the data

Used in multiple steps - takes the results of the previous map processes, and reduces them to a single result.
/**
 * reduce the results from a previous mapping excercise
 * @param {object} options describes what to do 
 * @param {object} mapResults this would contain results from a previous stage if present
 * @return {array.*} test data to pass on to next stage
 */
function reduceTheResults(options, mapResults) {

  // we'll have all the results here so consolidate

  var results = mapResults.reduce ( function (p,c) {
      (Array.isArray (c.results) ? c.results : [c.results]).forEach (function(d) {
        p.push (d);
      })
      return p;
  },[]);
  
  return results;

}

Do some encryptions

A number of these will be run in parallel threads. Since the data it will be passed will be all the test data, it needs to have a mechanism of deciding which part of the test data it will work on. In options we passed index and threads properties. From this it can select a section of the testdata unique to this this thread. A more sophisticated algorithm might include adjusting the number of items for this thread according to the estimated complexity, but I'm just doing a simple equal distribution between threads. It will return the encrypted message, plus everything a future decrypt function will need to validate that it worked.
/**
 * do a chunk of encryption testing
 * @param {object} options describes what to do 
 * @param {object} reduceResults this would contain results from a previous stage if present
 * @return {object} test data to pass on to next stage
 */
function cryptoEncrypt (options,reduceResults) {
  
  // get testdata and encrypt
  // we'll only do a section of data in this thread
  var data = reduceResults[0].results;
  var start = options.index/options.threads * data.length;
  var finish = (options.index+1)/options.threads * data.length ;

  // encrypt that slice
  return data.slice (start, finish).map ( function (d) {
    // for later decrypt testing, we'll include everything
    return {
      encrypted:new cCryptoGS.Cipher(d.password,options.cipher).encrypt(d.message),
      password:d.password,
      message:d.message,
      cipher:options.cipher
    }
  });
  
}

Decrypt and check

This will receive the reduced encrypted data, decrypt it, and check against the original message. Is uses the same allocation method as the encryption function for deciding which data to work on.
/**
 * do a chunk of decryption testing
 * @param {object} options describes what to do 
 * @param {object} reduceResults this would contain results from a previous stage if present
 * @return {object} test data to pass on to next stage
 */
function cryptoDecryptCheck (options,reduceResults) {
  
  // check that it all decrypts properly
  
  // we'll only do a section of data in this thread
  var data = reduceResults[0].results;
  var start = options.index/options.threads * data.length;
  var finish = options.threads === options.index + 1 ? 
        data.length : (options.index+1)/options.threads * data.length ;
  
  return data.slice(start,finish).map ( function (d) {
    var decrypt = new cCryptoGS.Cipher(d.password,d.cipher).decrypt(d.encrypted);
    if (options.stopOnFail) {
      if(decrypt !== d.message) throw 'decrytped does not match encrypted' + d.cipher;
    }
    return {cipher:d.cipher, error: decrypt !== d.message};
  });
  

}

Logging the results

I'm using the very useful crossFilter library to summarize the number of items by cipher that were checked, and outputting the result to a sheet using Database abstraction with google apps script
/**
 * summarize the results of the orchestration
 * @param {object} options describes what to do 
 * @param {object} reduceResults this would contain results from a previous stage if present
 * @return {object} test data to pass on to next stage
 */
function cryptoLog (options,reduceResults) {

    // load it to crossFilter
  var cf = cCrossFilter.crossfilter(reduceResults[0].results);

  // create a cipher dimension
  var ciphers = cf.dimension ( function (d) { return d.cipher });
 
// write a summary 
  var cipherSummary = ciphers.filter(null).group().all();
  
  var handler = new cDbAbstraction.DbAbstraction ( eval(options.driver),  options.parameters ); 
  assert(handler.isHappy(), 'unable to get handler',options.driver);

  if (options.clear) {
    var result = handler.remove();
    if (result.handleCode < 0) {
      throw result.handleError;
    }
  }
  
  var result = handler.save(cipherSummary);
  if (result.handleCode < 0) {
    throw result.handleError;
  }
  return reduceResults.results;
}

The whole profile

You don't need to change any of the stuff you have already set up in Running things in parallel using HTML service, except to load this profile in Code.gs

/**
 * Shows a custom HTML user interface in a sidebar 
 */
function showSidebar() {
   
   // kicking off the sidebar executes the orchestration
   libSidebar('asyncService',ADDONNAME, cryptoProfile () );
 
}

Here's the code for creating the profile for this all together.  

function cryptoProfile() {

  // profile for testing crypto functions
  var profile = [];
  
  // first stage - create test data 
  var SCALE = 2000, CHUNKS = 2;
  var profileTestData = [];
  for (var i =0; i <CHUNKS;i++ ) {
    profileTestData.push ({
        name: 'testdata-'+i,
        functionName:'cryptoTestData',
        options:{
          scale:SCALE
        }
    });
  } 
  
  // next reduce the test data to one
  var profileReduction = [];
  profileReduction.push({
    name: 'reduction',
    functionName:'reduceTheResults',
    options:{}
  });

  // next,execute the cryptotests
  var CRYPTOTHREADS =2,CIPHERS=['tripledes','aes','rabbit','des'];

  var profileCrypto = [];
  
  for (var i = 0; i < CRYPTOTHREADS ; i++) {
    CIPHERS.forEach (function (d) {
      profileCrypto.push ({
        name:'crypto-'+d+'-'+i,
        functionName:'cryptoEncrypt',
        options: {
          index: i,
          threads:CRYPTOTHREADS,
          cipher:d
        }
      });
    });
  }
  // do another reduction - we can use the same reduction profile, so nothing to do here
  
  // next we'll test the results by decrypting and check against original
  var DECRYPTOTHREADS =  6;
  var profileDecrypto = [];
  for (var i=0; i < DECRYPTOTHREADS ;i++ ) {
    profileDecrypto.push ({
      name:'decrypto-'+i,
      functionName:'cryptoDecryptCheck',
      options: {
        stopOnFail:true,
        index:i,
        threads:DECRYPTOTHREADS
      }
    });
  }
  // do another reduction - we can use the same reduction profile, so nothing to do here
  
  // finally log the results
  var profileLog = [{
    "name": "LOG",
    "functionName": "cryptoLog",
    "skip": false,
    "options": {
      "driver": "cDriverSheet",
      "clear": true,
      "parameters": {
        "siloid": "logencrypt",
        "dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
        "peanut": "bruce"
      }
    }
  }];
  
  // put it all together
  profile.push (
    profileTestData,
    profileReduction,
    profileCrypto,
    profileReduction,
    profileDecrypto,
    profileReduction,
    profileLog
  );
  return profile;
}

and here is the profile it produces
[
    [
        {
            "name": "testdata-0",
            "functionName": "cryptoTestData",
            "options": {
                "scale": 2000
            }
        },
        {
            "name": "testdata-1",
            "functionName": "cryptoTestData",
            "options": {
                "scale": 2000
            }
        }
    ],
    [
        {
            "name": "reduction",
            "functionName": "reduceTheResults",
            "options": {}
        }
    ],
    [
        {
            "name": "crypto-tripledes-0",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 0,
                "threads": 2,
                "cipher": "tripledes"
            }
        },
        {
            "name": "crypto-aes-0",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 0,
                "threads": 2,
                "cipher": "aes"
            }
        },
        {
            "name": "crypto-rabbit-0",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 0,
                "threads": 2,
                "cipher": "rabbit"
            }
        },
        {
            "name": "crypto-des-0",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 0,
                "threads": 2,
                "cipher": "des"
            }
        },
        {
            "name": "crypto-tripledes-1",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 1,
                "threads": 2,
                "cipher": "tripledes"
            }
        },
        {
            "name": "crypto-aes-1",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 1,
                "threads": 2,
                "cipher": "aes"
            }
        },
        {
            "name": "crypto-rabbit-1",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 1,
                "threads": 2,
                "cipher": "rabbit"
            }
        },
        {
            "name": "crypto-des-1",
            "functionName": "cryptoEncrypt",
            "options": {
                "index": 1,
                "threads": 2,
                "cipher": "des"
            }
        }
    ],
    [
        {
            "name": "reduction",
            "functionName": "reduceTheResults",
            "options": {}
        }
    ],
    [
        {
            "name": "decrypto-0",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 0,
                "threads": 6
            }
        },
        {
            "name": "decrypto-1",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 1,
                "threads": 6
            }
        },
        {
            "name": "decrypto-2",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 2,
                "threads": 6
            }
        },
        {
            "name": "decrypto-3",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 3,
                "threads": 6
            }
        },
        {
            "name": "decrypto-4",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 4,
                "threads": 6
            }
        },
        {
            "name": "decrypto-5",
            "functionName": "cryptoDecryptCheck",
            "options": {
                "stopOnFail": true,
                "index": 5,
                "threads": 6
            }
        }
    ],
    [
        {
            "name": "reduction",
            "functionName": "reduceTheResults",
            "options": {}
        }
    ],
    [
        {
            "name": "LOG",
            "functionName": "cryptoLog",
            "skip": false,
            "options": {
                "driver": "cDriverSheet",
                "clear": true,
                "parameters": {
                    "siloid": "logencrypt",
                    "dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
                    "peanut": "bruce"
                }
            }
        }
    ]
] 

For more on this topic, see Running things in parallel using HTML service. For more snippets like this see Google Apps Scripts snippets
For help and more information join our forum,follow the blog or follow me on twitter .

You want to learn Google Apps Script?

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, available All formats are available now from O'Reilly,Amazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.




Comments