Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script.

SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.

This is another in my series on SuperFetch plugins.

Page Content hide

Tank events

You can see more about Pseudo streaming for Apps Script (it doesn’t support real streaming natively), at SuperFetch Tank Plugin: Streaming for Apps Script and how to use it to copy very large files using the drv plugin at SuperFetch Plugins: Apps Script streaming with Tank and Drive

This article will cover how to use events to tweak data passing through the pipes between tanks on its streaming journey.

Events

You can set up event callbacks by the tank.on method (and remove them with with tank.off). You provide a function to be triggered when any of these events happen – this is not an exhaustive list as more will be added from time to time – so check the library code (details at the bottom) for which events are supported.

      'data',
'filler-start',
'filler-end',
'emptier-start',
'emptier-end',
'transformer-start',
'transformer-end',
'done',
'empty',
'full',
'error',
'level',
'pipe-done',
'stream-end',
'before-stream-end'
tank events that can be monitored

Callback actions

You set up an event callback like this

tank.on (eventNames, action[ ,context])

where

  • eventNames – a single or an array of eventNames from the list above that will trigger your action function when detected
  • action – a function to handle the event when it triggers
  • context – an optional argument that will be passed to your action function

action function

An action function looks like this

const action = ({readings , tank , context}) => { … do something }

where

Example

In previous articles I’ve shown how to log movement of data through tanks and visualize them using various event callbacks and the readings they return.

This time, we’re going to tweak the behavior of the tank, specifically to append new data while copying a file. Drive doesn’t have an append method, so you always have to copy the content, add new content, then make a completely new copy. Even an update/patch operation requires that you write the entire content again.

If your file is too big for apps script to handle in one go, you can use streaming with a call back on the ‘before-stream-end’ event to append the extra data.

Preparation

Enable the Drive API in your cloud project and set the required scopes. If you need help on how to do this – see SuperFetch plugin: Google Drive client for Apps Script – Part 1

Import the plugins

  // import required modules
const { Plugins, SuperFetch } = bmSuperFetch
const { Drv } = Plugins

// use this basic drv setup - don't need caching for this
const superFetch = new SuperFetch({
fetcherApp: UrlFetchApp,
tokenService: ScriptApp.getOAuthToken
})

// create a drv instance
const drv = new Drv({
superFetch
})
import

The copy and append code

    // the input file  
const bigPath = drv.files.path({ id: '11qAslAJmLCLpjkQZPz3G45DmYCHtzl7J' })

// the folder the result will go to
const tankPath = drv.files.path({ path: 'test/tanked' })

// create a readstream to get the input file data in chunks
const bigReadStream = bigPath.readStream().create()

// this is some data we want to append - you need to convert it to a byte array
const extraBytes = Utilities.newBlob('some extra data to add to the end').getBytes()

// create a writestream
const bigWriteStream = tankPath.writeStream({

// inherit the content type from the input stream
contentType: bigReadStream.contentType,

// derive the ourput file name from the input files name
name: `big-stream-upload-${bigReadStream.name}`,

// the size will be the original size the extrabytes
size: bigReadStream.size extraBytes.length
}).create()

// this event will add the extra data to the output tank just before it declares itself done
// that'll force it to add this to the copied file
bigWriteStream.tank.on("before-stream-end", ({tank,context}) => tank.add (context), extraBytes)

// do the pipe operation - thats it
bigReadStream.tank.pipe(bigWriteStream.tank).throw()
append example

Unit testing

I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responses to expect from calls. This shows how to test the functions mentioned in this article.

const testPipeDrv = ({
force = false,
unit } = {}) => {

// control which tests to skip
const skipTest = {
pipe: false & !force
}

// get a testing instance
unit = unit || new bmUnitTester.Unit({
showErrorsOnly: true,
showValues: true
})

// import required modules
const { Plugins, SuperFetch } = bmSuperFetch
const { Drv } = Plugins

// use this basic drv setup
const superFetch = new SuperFetch({
fetcherApp: UrlFetchApp,
tokenService: ScriptApp.getOAuthToken,
cacheService: CacheService.getUserCache()
})


unit.section(() => {
const drv = new Drv({
superFetch,
showUrl: false,
noCache: true
})
const bigPath = drv.files.path({ id: '11qAslAJmLCLpjkQZPz3G45DmYCHtzl7J' })
const tankPath = drv.files.path({ path: 'test/tanked' })

// get the csv data in total
const { actual: csv } = unit.not(null, bigPath.download().throw(), {
description: 'get reference data'
})

// do a tanked upload
// do a stream down/upload
const { actual: bigReadStream } = unit.not(null, bigPath.readStream().create(), {
description: 'create bigread stream'
})

const extraBytes = Utilities.newBlob('some extra data to add to the end').getBytes()


const { actual: bigWriteStream } = unit.not(null, tankPath.writeStream({
contentType: bigReadStream.contentType,
name: `big-stream-upload-${bigReadStream.name}`,
size: bigReadStream.size extraBytes.length
}).create(), {
description: 'create bigwritestream'
})
// do an append just before finalizing
bigWriteStream.tank.on("before-stream-end", ({tank,context}) => tank.add (context), extraBytes)

bigReadStream.tank.pipe(bigWriteStream.tank).throw()
unit.not(
bigReadStream.init.data.md5Checksum,
drv.files.get({ id: bigWriteStream.upload.data.id }).throw().data.md5Checksum, {
description: 'md5 is different'
})

unit.is(
csv.blob.getBytes().concat(extraBytes),
drv.files.download({ id: bigWriteStream.upload.data.id }).throw().blob.getBytes(), {
description: 'copy with extra is good'
})

}, {
description: 'checking appending',
skip: skipTest.appending
})
tests

Next

Next I’ll be showing how to pipe between different services – eg gcs and drv, and how to use transform tanks. Over time, many SuperFetch plugins will support piping between them.

Links

bmSuperFetch: 1B2scq2fYEcfoGyt9aXxUdoUPuLy-qbUC2_8lboUEdnNlzpGGWldoVYg2

IDE

GitHub

bmUnitTester: 1zOlHMOpO89vqLPe5XpC-wzA9r5yaBkWt_qFjKqFNsIZtNJ-iUjBYDt-x

IDE

GitHub

Related

file conversion

Convert any file with Apps Script

The Drive API offers a whole range of conversions between mimeTypes, but it's a little fiddly to figure out exactly ...
Superfetch plugin

Caching, property stores and pre-caching

I've written many times about various Apps Script caching techniques such as how to deal with size limits and use ...
document AI add-on

State management across CardService, HtmlService and Server side Add-ons

Motivation I've been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things ...
Secret Manager

SuperFetch Plugin: Cloud Manager Secrets and Apps Script

Smg is a SuperFetch plugin to access the Google Cloud Secrets API. SuperFetch is a proxy for UrlFetchApp with additional ...
Superfetch plugin

SuperFetch caching: How does it work?

SuperFetch is a proxy for UrlFetchApp with additional features such as built-in caching – see SuperFetch – a proxy enhancement ...
superfetch tank drive

SuperFetch plugins: Tank events and appending

Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
superfetch tank drive

SuperFetch Plugins: Apps Script streaming with Tank and Drive

Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
superfetch tank apps script streaming

SuperFetch Tank Plugin: Streaming for Apps Script

Tank is a SuperFetch plugin to emulate streaming with Apps Script. SuperFetch is a proxy for UrlFetchApp with additional features ...
superfetch drive plugin logo

SuperFetch plugin – Google Drive client for Apps Script – Part 1

Drv is a SuperFetch plugin to access the Google Drive API. SuperFetch is a proxy for UrlFetchApp with additional features ...
Superfetch plugin twitter

SuperFetch – Twitter plugin for Apps Script – Get Follows, Mutes and blocks

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Superfetch plugin twitter

SuperFetch plugin – Twitter client for Apps Script – Counts

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
goa twitter oauth2 apps script

OAuth2 and Twitter API – App only flow for Apps Script

I covered how to handle the somewhat more complex OAUTH2 authorization flow for the Twitter v2 API (OAuth 2.0 Authorization ...
Superfetch plugin twitter

SuperFetch plugin – Twitter client for Apps Script – Search and Get

Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Goa Oauth2 for Apps Script

Apps Script Oauth2 library Goa: tips, tricks and hacks

Motivation Goa is a library to support OAuth2 for Apps Script connecting to a variety of services, using a variety ...
goa twitter oauth2 apps script

Apps Script Oauth2 – a Goa Library refresher

It's been a few years since I first created the Goa library. Initially it was mainly to provide OAuth2 authorization ...
SuperFetch

SuperFetch plugin – Firebase client for Apps Script

Frb is a SuperFetch plugin to easily access a Firebase Real time database. SuperFetch is a proxy for UrlFetchApp with ...
SuperFetch

SuperFetch plugin – iam – how to authenticate to Cloud Run from Apps Script

SuperFetch is a proxy for UrlFetchApp with additional features - see SuperFetch - a proxy enhancement to Apps Script UrlFetch ...
SuperFetch

SuperFetch – a proxy enhancement to Apps Script UrlFetch

I've written a few articles about JavaScript proxying on here, and I'm a big fan. I also use a lot ...
SuperFetch

Apps script caching with compression and enhanced size limitations

Motivation Caching is a great way to improve performance, avoid rate limit problems and even save money if you are ...

Simple but powerful Apps Script Unit Test library

Why unit testing? There are many test packages for Node (my favorite is ava) and there are also a few ...
uper