Tank is a SuperFetch plugin to emulate streaming with Apps Script. SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.

This is another in my series on SuperFetch plugins.

Page Content hide
1 Why streaming ?

Why streaming ?

You may have come across streaming if you have experience outside of Apps Script, for example in Node JS. In Apps Script, you’d typically ingest all the data from a data source (eg a file), possibly transform it, and then write the copy you’ve made somewhere else – for example from Drive to Drive or Drive to cloud storage.

apps script copying

This is what the download/upload/export/convert methods in the drv and gcs Superfetch plugins do by default. (tldr; Actually they use a technique called resumable upload which splits the upload into chunks – allowing it to bypass the Drive and UrlFetch payload size limits)

However, what happens if the file size is just too big for Apps Script to handle all in one go, or if the data is arriving in chunks from a network service? In platforms that support it, you’d use a streaming technique – chunks arrive as input and are immediately piped to the output – so you never need to worry about having the complete data in memory at any one time.

One way is to use Html Service to handle client side streaming – but that would mean bouncing the data down to your client, so we probably don’t want to do that.  Another would be to step outside Apps Script and use cloud functions or cloud run (see Blistering fast file streaming between Drive and Cloud Storage using Cloud Run). But if we want to stick to plain Server side Apps Script – the Tank Plugin is a pseudo streamer for Apps Script to emulate that behavior.

How Tank works

Data is fed to the input tank in chunks with a filler function that you provide. When using Tank with Superfetch, that would typically be the drv or gcs plugin, but any chunked data can be handled by Tank should you want to use it in some other context.

When the input tank is full, it is emptied into to the output tank. In the meantime, if the output tank detects it is full, it will keep emptying itself in chunks using an emptier function you provide. SuperFetch plugins use resumable uploading to handle theses chunked uploads and will generate filler and emptier functions specific to the API the plugin is using.

tank works

The tank capacities and the chunk sizes are all completely independent of each other allowing you to optimize each for the specific case. Bigger tank sizes mean more memory but less fetching. Smaller tank sizes mean extra fetching. Tank sizes that are multiples of the maximum filler and emptier post sizes might marginally improve performance.

Normally of course, streaming is asynchronous allowing uploading and downloading simultaneously, but Apps Script doesn’t support that (easily anyway) – so this solution as described here is purely synchronous.

Getting started with Tank

For the purposes of this article, I’ll mock up input and output data handling. In a future article you’ll see how the Drv and GCS plugins use Tank to be able to handle very large file transfers.

Script Preparation

You’ll need the bmSuperFetch library and if you want to run some tests, the bmUnitTest library – details at the end of the article.

First you need the Tank plugin.


// import required modules
const { Plugins } = bmSuperFetch
const { Tank } = Plugins

import modules

Filler and emptier functions

Tank needs a filler function (what to do when the input tank is empty) and an emptier function (what to do when the output tank is full).  These are mock filler and emptiers that simulate getting and putting data from and to an API, with a random delay.

  // just some mock data
const fixture = [
Array.from({ length: 9001 }, (_, i) => 'a' i),
Array.from({ length: 1243 }, (_, i) => 'b' i),
Array.from({ length: 372 }, (_, i) => 'c' i),
Array.from({ length: 8701 }, (_, i) => 'd' i)
]

// use this to simulate a random wait between calls
const randomWait = () => Utilities.sleep(Math.floor(Math.random() * 100))

// instead of streaming the result, we'll just store it in an array
let fixUsed = 0
const fixResult = []

// mock return chunks of data to fill up the input tank
// this is used to pipe initial data into the tank
// the filler will receive the tank instance in case yiou need it
// it should return null if there's no more data
// or {items: [arrayofitems], error: 'if there was an error'}
const filler = (tank) => {
randomWait()
return fixUsed < fixture.length ? {
items: fixture[fixUsed ]
} : {
items: null
}
}


// mock empty the tank
const emptier = (tank, items) => {
randomWait()
Array.prototype.push.apply(fixResult, items)
}
mock filler and emptier

Tank Instances

Each of the input and output tanks typically have their own tank instance

  // the tanks can be different sizes
const inTank = new Tank({
filler,
capacity: 6203,
name: 'in'
})

const outTank = new Tank({
emptier,
capacity: 7007,
name: 'out'
})
instantiate

The tank constructor takes these argument properties

filler (tank)

A function that can get data to write to the input tank. It should return null if there’s no more data or an object with these properties

  • items – an array of items to write to the tank
  • error – if there’s an error put it here

emptier (tank, items)

A function that can write ‘items’ to the target. If there’s are error it should return an object with an error property

capacity

The number of items that a tank should hold – For streaming operations this is typically going to be number of bytes, and the item array will be an array of bytes – but it can refer to any kind of item.

Usage

Piping

There are various tank methods, but the one you’ll need for this kind of usage is .pipe()

Piping is joining multiple tanks together. It’s this simple.

  // copy from the source to the target inchunks
inTank.pipe(outTank)
piping

Events

There are a number of events you may want to track. You can use  tank.on(eventName, action) to callback an action function when a given event happens. tank.off(eventName) will cancel that event callback.

These events can all be tracked with the .on() method


'data',
'filler-start',
'filler-end',
'emptier-start',
'emptier-end',
'done',
'empty',
'full',
'error'
events

For convenience, eventName can also be an array of eventNames so you can track multiple events that have the same action with one call.

Action callback

Your action function will receive these arguments

action  ({ readings, tank})

where tank is the tank instance, and readings is an object with these properties

/**
* @typedef TankReadings
* property {string} name the tank name given at the construction
* @property {number} createdAt the time the instance was created
* @property {number} capacity the number of items the tank can hold
* @property {number} timeStamp when the event was emitted
* @property {string} eventName the name of the event
* @property {number} creationOffset how many ms since the item was created
* @property {number} the current level of the tank content
* @property {number} in total number of items ever input to the tank
* @property {number} out total number of items ever output from the tank
* @property {number} itemsFilled how many items were received at the last fill up
* @property {number} itemsEmptied how many items were emptied by the last emptier
* @property {boolean} isDone the input is exhausted
* @property {*} error any error preventing the tank from operating
*/
TankReadings

Visualizing readings

It might be handy to look at the movement in levels of the the tanks during the pipe operation. For that, we’ll use the .on() method to log filling and emptying events, then write the log to a spreadsheet.

Since we’re in Apps Script here, we as may well write it all to a sheet. I’m going to use the bmPreFiddler library for this, as it makes sheet manipulation a bit easier. ID details at end of post.

 // we can log different events
const inLog = []
const outLog = []
inTank.on(['empty', 'full'], ({ readings }) => inLog.push(readings))
outTank.on(['empty', 'full'], ({ readings }) => outLog.push(readings))

// copy one to the other
inTank.pipe(outTank)
// write the log to a sheet
bmPreFiddler.PreFiddler().getFiddler({
sheetName: `intank-track`,
id: '1UU6t01SRssYQhZGSSo62HD7u7rVw6NtZ3jCPadmY7uA',
createIfMissing: true
})
.setData(inLog)
.dumpValues()

bmPreFiddler.PreFiddler().getFiddler({
sheetName: `outtank-track`,
id: '1UU6t01SRssYQhZGSSo62HD7u7rVw6NtZ3jCPadmY7uA',
createIfMissing: true
})
.setData(outLog)
.dumpValues()
logging

Here’s how the tanks contents and levels looked throughout the pipe operation

visualize tank

Unit testing

I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls. These tests demonstrate each of the topics mentioned in this article, as well as some I’ll be covering in a later article, and could serve as a useful crib sheet for the plugin

const testTank = ({
force = false,
unit } = {}) => {
// control which tests to skip
const skipTest = {
basic: false && !force,
}

// get a testing instance
unit = unit || new bmUnitTester.Unit({
showErrorsOnly: true,
showValues: false
})

// import required modules
const { Plugins, SuperFetch } = bmSuperFetch
const { Tank } = Plugins

const fixture = [
Array.from({ length: 9001 }, (_, i) => 'a' i),
Array.from({ length: 1243 }, (_, i) => 'b' i),
Array.from({ length: 372 }, (_, i) => 'c' i),
Array.from({ length: 8701 }, (_, i) => 'd' i)
]



const randomWait = () => Utilities.sleep(Math.floor(Math.random() * 100))
unit.section(() => {
// mock return chunks of data
// this is used to pipe initial data into the tank
const fixResult = []
let fixUsed = 0
const filler = (tank) => {
randomWait()
return fixUsed < fixture.length ? {
items: fixture[fixUsed ]
} : {
items: null
}
}

// mock empty the tank
const emptier = (tank, items) => {
randomWait()
Array.prototype.push.apply(fixResult, items)
}
const inTank = new Tank({
filler,
capacity: 6203,
name: 'in'
})

const outTank = new Tank({
emptier,
capacity: 7007,
name: 'out'
})
inTank.pipe(outTank)

unit.is(fixResult, fixture.flat(), {
description: 'was reconstituted'
})
}, {
description: 'basic test',
skip: skipTest.basic
})
}
tests

Next

That’s a basic intro into pseudo streaming in Apps Script. In a later article I’ll cover how this is used to by SuperFetch to handle files that would otherwise be too big for Apps Script. I’ll also describe how some kind of asynchronicity can be achieved in Apps Script.

Links

bmSuperFetch: 1B2scq2fYEcfoGyt9aXxUdoUPuLy-qbUC2_8lboUEdnNlzpGGWldoVYg2

IDE

GitHub

bmUnitTester: 1zOlHMOpO89vqLPe5XpC-wzA9r5yaBkWt_qFjKqFNsIZtNJ-iUjBYDt-x

IDE

GitHub

bmPreFiddler (13JUFGY18RHfjjuKmIRRfvmGlCYrEkEtN6uUm-iLUcxOUFRJD-WBX-tkR)

IDE

GitHub

Related

file conversion

Convert any file with Apps Script

The Drive API offers a whole range of conversions between mimeTypes, but it's a little fiddly to figure out exactly ...
Superfetch plugin

Caching, property stores and pre-caching

I've written many times about various Apps Script caching techniques such as how to deal with size limits and use ...
document AI add-on

State management across CardService, HtmlService and Server side Add-ons

Motivation I've been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things ...
Secret Manager

SuperFetch Plugin: Cloud Manager Secrets and Apps Script

Smg is a SuperFetch plugin to access the Google Cloud Secrets API. SuperFetch is a proxy for UrlFetchApp with additional ...
Superfetch plugin

SuperFetch caching: How does it work?

SuperFetch is a proxy for UrlFetchApp with additional features such as built-in caching – see SuperFetch – a proxy enhancement ...
superfetch tank drive

SuperFetch plugins: Tank events and appending

Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
superfetch tank drive

SuperFetch Plugins: Apps Script streaming with Tank and Drive

Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
superfetch tank apps script streaming

SuperFetch Tank Plugin: Streaming for Apps Script

Tank is a SuperFetch plugin to emulate streaming with Apps Script. SuperFetch is a proxy for UrlFetchApp with additional features ...
superfetch drive plugin logo

SuperFetch plugin – Google Drive client for Apps Script – Part 1

Drv is a SuperFetch plugin to access the Google Drive API. SuperFetch is a proxy for UrlFetchApp with additional features ...
Superfetch plugin twitter

SuperFetch – Twitter plugin for Apps Script – Get Follows, Mutes and blocks

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Superfetch plugin twitter

SuperFetch plugin – Twitter client for Apps Script – Counts

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
goa twitter oauth2 apps script

OAuth2 and Twitter API – App only flow for Apps Script

I covered how to handle the somewhat more complex OAUTH2 authorization flow for the Twitter v2 API (OAuth 2.0 Authorization ...
Superfetch plugin twitter

SuperFetch plugin – Twitter client for Apps Script – Search and Get

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Goa Oauth2 for Apps Script

Apps Script Oauth2 library Goa: tips, tricks and hacks

Motivation Goa is a library to support OAuth2 for Apps Script connecting to a variety of services, using a variety ...
goa twitter oauth2 apps script

Apps Script Oauth2 – a Goa Library refresher

It's been a few years since I first created the Goa library. Initially it was mainly to provide OAuth2 authorization ...
SuperFetch

SuperFetch plugin – Firebase client for Apps Script

Frb is a SuperFetch plugin to easily access a Firebase Real time database. SuperFetch is a proxy for UrlFetchApp with ...
SuperFetch

SuperFetch plugin – iam – how to authenticate to Cloud Run from Apps Script

SuperFetch is a proxy for UrlFetchApp with additional features - see SuperFetch - a proxy enhancement to Apps Script UrlFetch ...
SuperFetch

SuperFetch – a proxy enhancement to Apps Script UrlFetch

I've written a few articles about JavaScript proxying on here, and I'm a big fan. I also use a lot ...
SuperFetch

Apps script caching with compression and enhanced size limitations

Motivation Caching is a great way to improve performance, avoid rate limit problems and even save money if you are ...

Simple but powerful Apps Script Unit Test library

Why unit testing? There are many test packages for Node (my favorite is ava) and there are also a few ...