Running things in parallel using HTML service was a brief intro on how to run a number of things at once, orchestrating executing using Google Apps Script HTML service. You may have already tried Parallel implementation and getting started or even Running things in parallel that need oAUTH2 – datastore. Now here’s something a little more complicated.
In CryptoJS libraries for Google Apps Script I mentioned that they were kind of slow under Apps Script – depending on the cryptography – up to a second for each one to encrypt/decrypt. Testing this kind of thing is pretty tricky when you run out of quota all the time. Here’s how I tested thousands at once with no quota problems.
Page Content
hide
The process
I was able to orchestrate the parallel running of all this using Parallel process orchestration with HtmlService
Here’s a snap of the run – We got 40 minutes of uninterrupted processing over 13 minutes, and processed 16,000 encrypt/decrypt pairs. To speed it up even more all I need to do is to create additional processing threads in the profile.
Orchestration profile
The action is controlled by an orchestration profile. But there are some wrinkles in this one, because I wanted to make it dynamic, depending on the size of the run. Here’s how it’s built up. Creating the orchestration is a good first step. We don’t even have to write the functions that do the work at this stage.
Create the test data
I’m creating 4000 test items. 2 threads will do for this trivial process.
// first stage - create test data
var scale = optScale || 2000, CHUNKS = 2;
var profileTestData = [];
for (var i =0; i <CHUNKS;i++ ) {
profileTestData.push ({
name: 'testdata-'+i,
functionName:'cryptoTestData',
options:{
scale:scale
}
});
}
Reduce the results
After a parallel map operation, you always need to reduce the results. This ensures you have removed any dependencies between how the previous step was executed. I use a standard function for all reduce functions.
// next reduce the test data to one
var profileReduction = [];
profileReduction.push({
name: 'reduction',
functionName:'reduceTheResults',
options:{}
});
Do the encryption
I’m applying 4 different kind of encryption to each of the randomly generated messages and passwords that were created in the first process. I’m going to give 2 threads to each encryption method, so there will be 8 threads, each of which will handle 2000 encryptions. I’ll explain later what we’re going to use the options for.
// next,execute the cryptotests
var CRYPTOTHREADS =2,CIPHERS=['tripledes','aes','rabbit','des'];
var profileCrypto = [];
for (var i = 0; i < CRYPTOTHREADS ; i++) {
CIPHERS.forEach (function (d) {
profileCrypto.push ({
name:'crypto-'+d+'-'+i,
functionName:'cryptoEncrypt',
options: {
index: i,
threads:CRYPTOTHREADS,
cipher:d
}
});
});
}
Reduce the results
Again we need to reduce the results. We can just use the same reduction profile as previously
Check that it worked
Decrypting everything and checking against the original messages will show that the operation worked. This time I’ll go for 6 threads. I’ll explain the option values later.
// next we'll test the results by decrypting and check against original
var DECRYPTOTHREADS = 6;
var profileDecrypto = [];
for (var i=0; i < DECRYPTOTHREADS ;i++ ) {
profileDecrypto.push ({
name:'decrypto-'+i,
functionName:'cryptoDecryptCheck',
options: {
stopOnFail:true,
index:i,
threads:DECRYPTOTHREADS
}
});
}
Reduce the results
Again we need to reduce the results. We can just use the same reduction profile as previously.
Log the results
We want to make sure that we ended up with the same number of items that we started with, so do a simple summary of the reduced data. Most of the options here are about Database abstraction with google apps script, which I always use to simplify writing data to spreadsheets.
// finally log the results
var profileLog = [{
"name": "LOG",
"functionName": "cryptoLog",
"skip": false,
"options": {
"driver": "cDriverSheet",
"clear": true,
"parameters": {
"siloid": "logencrypt",
"dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
"peanut": "bruce"
}
}
}];
Here’s what was logged
Put it all together
Now we’ve created a profile for each section, we can put it all together like so. I’ll repeat the whole function at the end of this page
// put it all together
profile.push (
profileTestData,
profileReduction,
profileCrypto,
profileReduction,
profileDecrypto,
profileReduction
);
if (cUseful.applyDefault(optLog, true) ) {
profile.push(profileLog);
}
The executor functions
These are called to do the work by the profiles we set up above. Each functionName property should point to one of these. In all cases executor functions are passed 2 parameters
- The options you set up in your profile
- The data from the previous stage (if there was any)
Creating the test data.
This creates a chunk of random test messages and passwords – volume as defined in options.scale.
/**
* create some test data for encryption tests
* @param {object} options describes what to do
* @param {object} reduceResults this would contain results from a previous stage if present
* @return {object} test data to pass on to next stage
*/
function cryptoTestData (options, reduceResults) {
var testData = [];
//generate some test data
for (var i=0; i < options.scale ; i++ ) {
testData.push ( {
message: cUseful.generateUniqueString(Math.round(Math.random()*100)) ,
password: cUseful.generateUniqueString(Math.round(Math.random()*10))
});
}
return testData;
}
Reduce the data
Used in multiple steps – takes the results of the previous map processes, and reduces them to a single result.
/**
* reduce the results from a previous mapping excercise
* @param {object} options describes what to do
* @param {object} mapResults this would contain results from a previous stage if present
* @return {array.*} test data to pass on to next stage
*/
function reduceTheResults(options, mapResults) {
// we'll have all the results here so consolidate
var results = mapResults.reduce ( function (p,c) {
(Array.isArray (c.results) ? c.results : [c.results]).forEach (function(d) {
p.push (d);
})
return p;
},[]);
return results;
}
Do some encryptions
A number of these will be run in parallel threads. Since the data it will be passed will be all the test data, it needs to have a mechanism of deciding which part of the test data it will work on. In options we passed index and threads properties. From this it can select a section of the testdata unique to this this thread.
A more sophisticated algorithm might include adjusting the number of items for this thread according to the estimated complexity, but I’m just doing a simple equal distribution between threads. It will return the encrypted message, plus everything a future decrypt function will need to validate that it worked.
/**
* do a chunk of encryption testing
* @param {object} options describes what to do
* @param {object} reduceResults this would contain results from a previous stage if present
* @return {object} test data to pass on to next stage
*/
function cryptoEncrypt (options,reduceResults) {
// get testdata and encrypt
// we'll only do a section of data in this thread
var data = reduceResults[0].results;
var start = options.index/options.threads * data.length;
var finish = (options.index+1)/options.threads * data.length ;
// encrypt that slice
return data.slice (start, finish).map ( function (d) {
// for later decrypt testing, we'll include everything
return {
encrypted:new cCryptoGS.Cipher(d.password,options.cipher).encrypt(d.message),
password:d.password,
message:d.message,
cipher:options.cipher
}
});
}
Decrypt and check
This will receive the reduced encrypted data, decrypt it, and check against the original message. Is uses the same allocation method as the encryption function for deciding which data to work on.v
/**
* do a chunk of decryption testing
* @param {object} options describes what to do
* @param {object} reduceResults this would contain results from a previous stage if present
* @return {object} test data to pass on to next stage
*/
function cryptoDecryptCheck (options,reduceResults) {
// check that it all decrypts properly
// we'll only do a section of data in this thread
var data = reduceResults[0].results;
var start = options.index/options.threads * data.length;
var finish = options.threads === options.index + 1 ?
data.length : (options.index+1)/options.threads * data.length ;
return data.slice(start,finish).map ( function (d) {
var decrypt = new cCryptoGS.Cipher(d.password,d.cipher).decrypt(d.encrypted);
if (options.stopOnFail) {
if(decrypt !== d.message) throw 'decrytped does not match encrypted' + d.cipher;
}
return {cipher:d.cipher, error: decrypt !== d.message};
});
}
Logging the results
I’m using the very useful crossFilter library to summarize the number of items by cipher that were checked, and outputting the result to a sheet using Database abstraction with google apps script
/**
* summarize the results of the orchestration
* @param {object} options describes what to do
* @param {object} reduceResults this would contain results from a previous stage if present
* @return {object} test data to pass on to next stage
*/
function cryptoLog (options,reduceResults) {
// load it to crossFilter
var cf = cCrossFilter.crossfilter(reduceResults[0].results);
// create a cipher dimension
var ciphers = cf.dimension ( function (d) { return d.cipher });
// write a summary
var cipherSummary = ciphers.filter(null).group().all();
var handler = new cDbAbstraction.DbAbstraction ( eval(options.driver), options.parameters );
assert(handler.isHappy(), 'unable to get handler',options.driver);
if (options.clear) {
var result = handler.remove();
if (result.handleCode < 0) {
throw result.handleError;
}
}
var result = handler.save(cipherSummary);
if (result.handleCode < 0) {
throw result.handleError;
}
return reduceResults.results;
}
The whole profile
You don’t need to change any of the stuff you have already set up in Running things in parallel using HTML service, except to load this profile in Code.gs
/**
* Shows a custom HTML user interface in a sidebar
*/
function showSidebar() {
// kicking off the sidebar executes the orchestration
libSidebar('asyncService',ADDONNAME, cryptoProfile () );
}
Here’s the code for creating the profile for this all together.
function cryptoProfile() {
// profile for testing crypto functions
var profile = [];
// first stage - create test data
var SCALE = 2000, CHUNKS = 2;
var profileTestData = [];
for (var i =0; i <CHUNKS;i++ ) {
profileTestData.push ({
name: 'testdata-'+i,
functionName:'cryptoTestData',
options:{
scale:SCALE
}
});
}
// next reduce the test data to one
var profileReduction = [];
profileReduction.push({
name: 'reduction',
functionName:'reduceTheResults',
options:{}
});
// next,execute the cryptotests
var CRYPTOTHREADS =2,CIPHERS=['tripledes','aes','rabbit','des'];
var profileCrypto = [];
for (var i = 0; i < CRYPTOTHREADS ; i++) {
CIPHERS.forEach (function (d) {
profileCrypto.push ({
name:'crypto-'+d+'-'+i,
functionName:'cryptoEncrypt',
options: {
index: i,
threads:CRYPTOTHREADS,
cipher:d
}
});
});
}
// do another reduction - we can use the same reduction profile, so nothing to do here
// next we'll test the results by decrypting and check against original
var DECRYPTOTHREADS = 6;
var profileDecrypto = [];
for (var i=0; i < DECRYPTOTHREADS ;i++ ) {
profileDecrypto.push ({
name:'decrypto-'+i,
functionName:'cryptoDecryptCheck',
options: {
stopOnFail:true,
index:i,
threads:DECRYPTOTHREADS
}
});
}
// do another reduction - we can use the same reduction profile, so nothing to do here
// finally log the results
var profileLog = [{
"name": "LOG",
"functionName": "cryptoLog",
"skip": false,
"options": {
"driver": "cDriverSheet",
"clear": true,
"parameters": {
"siloid": "logencrypt",
"dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
"peanut": "bruce"
}
}
}];
// put it all together
profile.push (
profileTestData,
profileReduction,
profileCrypto,
profileReduction,
profileDecrypto,
profileReduction,
profileLog
);
return profile;
}
and here is the profile it produces
[
[
{
"name": "testdata-0",
"functionName": "cryptoTestData",
"options": {
"scale": 2000
}
},
{
"name": "testdata-1",
"functionName": "cryptoTestData",
"options": {
"scale": 2000
}
}
],
[
{
"name": "reduction",
"functionName": "reduceTheResults",
"options": {}
}
],
[
{
"name": "crypto-tripledes-0",
"functionName": "cryptoEncrypt",
"options": {
"index": 0,
"threads": 2,
"cipher": "tripledes"
}
},
{
"name": "crypto-aes-0",
"functionName": "cryptoEncrypt",
"options": {
"index": 0,
"threads": 2,
"cipher": "aes"
}
},
{
"name": "crypto-rabbit-0",
"functionName": "cryptoEncrypt",
"options": {
"index": 0,
"threads": 2,
"cipher": "rabbit"
}
},
{
"name": "crypto-des-0",
"functionName": "cryptoEncrypt",
"options": {
"index": 0,
"threads": 2,
"cipher": "des"
}
},
{
"name": "crypto-tripledes-1",
"functionName": "cryptoEncrypt",
"options": {
"index": 1,
"threads": 2,
"cipher": "tripledes"
}
},
{
"name": "crypto-aes-1",
"functionName": "cryptoEncrypt",
"options": {
"index": 1,
"threads": 2,
"cipher": "aes"
}
},
{
"name": "crypto-rabbit-1",
"functionName": "cryptoEncrypt",
"options": {
"index": 1,
"threads": 2,
"cipher": "rabbit"
}
},
{
"name": "crypto-des-1",
"functionName": "cryptoEncrypt",
"options": {
"index": 1,
"threads": 2,
"cipher": "des"
}
}
],
[
{
"name": "reduction",
"functionName": "reduceTheResults",
"options": {}
}
],
[
{
"name": "decrypto-0",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 0,
"threads": 6
}
},
{
"name": "decrypto-1",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 1,
"threads": 6
}
},
{
"name": "decrypto-2",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 2,
"threads": 6
}
},
{
"name": "decrypto-3",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 3,
"threads": 6
}
},
{
"name": "decrypto-4",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 4,
"threads": 6
}
},
{
"name": "decrypto-5",
"functionName": "cryptoDecryptCheck",
"options": {
"stopOnFail": true,
"index": 5,
"threads": 6
}
}
],
[
{
"name": "reduction",
"functionName": "reduceTheResults",
"options": {}
}
],
[
{
"name": "LOG",
"functionName": "cryptoLog",
"skip": false,
"options": {
"driver": "cDriverSheet",
"clear": true,
"parameters": {
"siloid": "logencrypt",
"dbid": "1yTQFdN_O2nFb9obm7AHCTmPKpf5cwAd78uNQJiCcjPk",
"peanut": "bruce"
}
}
}
]
]
or more snippets like this see Google Apps Scripts snippets