Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script.
SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.
This is another in my series on SuperFetch plugins.
Tank events
You can see more about Pseudo streaming for Apps Script (it doesn’t support real streaming natively), at SuperFetch Tank Plugin: Streaming for Apps Script and how to use it to copy very large files using the drv plugin at SuperFetch Plugins: Apps Script streaming with Tank and Drive
This article will cover how to use events to tweak data passing through the pipes between tanks on its streaming journey.
Events
You can set up event callbacks by the tank.on method (and remove them with with tank.off). You provide a function to be triggered when any of these events happen – this is not an exhaustive list as more will be added from time to time – so check the library code (details at the bottom) for which events are supported.
'data',
'filler-start',
'filler-end',
'emptier-start',
'emptier-end',
'transformer-start',
'transformer-end',
'done',
'empty',
'full',
'error',
'level',
'pipe-done',
'stream-end',
'before-stream-end'
tank events that can be monitored
Callback actions
You set up an event callback like this
tank.on (eventNames, action[ ,context])
where
- eventNames – a single or an array of eventNames from the list above that will trigger your action function when detected
- action – a function to handle the event when it triggers
- context – an optional argument that will be passed to your action function
action function
An action function looks like this
const action = ({readings , tank , context}) => { … do something }
where
Example
In previous articles I’ve shown how to log movement of data through tanks and visualize them using various event callbacks and the readings they return.
This time, we’re going to tweak the behavior of the tank, specifically to append new data while copying a file. Drive doesn’t have an append method, so you always have to copy the content, add new content, then make a completely new copy. Even an update/patch operation requires that you write the entire content again.
If your file is too big for apps script to handle in one go, you can use streaming with a call back on the ‘before-stream-end’ event to append the extra data.
Preparation
Enable the Drive API in your cloud project and set the required scopes. If you need help on how to do this – see SuperFetch plugin: Google Drive client for Apps Script – Part 1
Import the plugins
// import required modules
const { Plugins, SuperFetch } = bmSuperFetch
const { Drv } = Plugins
// use this basic drv setup - don't need caching for this
const superFetch = new SuperFetch({
fetcherApp: UrlFetchApp,
tokenService: ScriptApp.getOAuthToken
})
// create a drv instance
const drv = new Drv({
superFetch
})
import
The copy and append code
// the input file
const bigPath = drv.files.path({ id: '11qAslAJmLCLpjkQZPz3G45DmYCHtzl7J' })
// the folder the result will go to
const tankPath = drv.files.path({ path: 'test/tanked' })
// create a readstream to get the input file data in chunks
const bigReadStream = bigPath.readStream().create()
// this is some data we want to append - you need to convert it to a byte array
const extraBytes = Utilities.newBlob('some extra data to add to the end').getBytes()
// create a writestream
const bigWriteStream = tankPath.writeStream({
// inherit the content type from the input stream
contentType: bigReadStream.contentType,
// derive the ourput file name from the input files name
name: `big-stream-upload-${bigReadStream.name}`,
// the size will be the original size the extrabytes
size: bigReadStream.size extraBytes.length
}).create()
// this event will add the extra data to the output tank just before it declares itself done
// that'll force it to add this to the copied file
bigWriteStream.tank.on("before-stream-end", ({tank,context}) => tank.add (context), extraBytes)
// do the pipe operation - thats it
bigReadStream.tank.pipe(bigWriteStream.tank).throw()
append example
Unit testing
I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responses to expect from calls. This shows how to test the functions mentioned in this article.
const testPipeDrv = ({
force = false,
unit } = {}) => {
// control which tests to skip
const skipTest = {
pipe: false & !force
}
// get a testing instance
unit = unit || new bmUnitTester.Unit({
showErrorsOnly: true,
showValues: true
})
// import required modules
const { Plugins, SuperFetch } = bmSuperFetch
const { Drv } = Plugins
// use this basic drv setup
const superFetch = new SuperFetch({
fetcherApp: UrlFetchApp,
tokenService: ScriptApp.getOAuthToken,
cacheService: CacheService.getUserCache()
})
unit.section(() => {
const drv = new Drv({
superFetch,
showUrl: false,
noCache: true
})
const bigPath = drv.files.path({ id: '11qAslAJmLCLpjkQZPz3G45DmYCHtzl7J' })
const tankPath = drv.files.path({ path: 'test/tanked' })
// get the csv data in total
const { actual: csv } = unit.not(null, bigPath.download().throw(), {
description: 'get reference data'
})
// do a tanked upload
// do a stream down/upload
const { actual: bigReadStream } = unit.not(null, bigPath.readStream().create(), {
description: 'create bigread stream'
})
const extraBytes = Utilities.newBlob('some extra data to add to the end').getBytes()
const { actual: bigWriteStream } = unit.not(null, tankPath.writeStream({
contentType: bigReadStream.contentType,
name: `big-stream-upload-${bigReadStream.name}`,
size: bigReadStream.size extraBytes.length
}).create(), {
description: 'create bigwritestream'
})
// do an append just before finalizing
bigWriteStream.tank.on("before-stream-end", ({tank,context}) => tank.add (context), extraBytes)
bigReadStream.tank.pipe(bigWriteStream.tank).throw()
unit.not(
bigReadStream.init.data.md5Checksum,
drv.files.get({ id: bigWriteStream.upload.data.id }).throw().data.md5Checksum, {
description: 'md5 is different'
})
unit.is(
csv.blob.getBytes().concat(extraBytes),
drv.files.download({ id: bigWriteStream.upload.data.id }).throw().blob.getBytes(), {
description: 'copy with extra is good'
})
}, {
description: 'checking appending',
skip: skipTest.appending
})
tests
Next
Next I’ll be showing how to pipe between different services – eg gcs and drv, and how to use transform tanks. Over time, many SuperFetch plugins will support piping between them.
Links
bmSuperFetch: 1B2scq2fYEcfoGyt9aXxUdoUPuLy-qbUC2_8lboUEdnNlzpGGWldoVYg2
IDE
GitHub
bmUnitTester: 1zOlHMOpO89vqLPe5XpC-wzA9r5yaBkWt_qFjKqFNsIZtNJ-iUjBYDt-x
IDE
GitHub
Related
The Drive API offers a whole range of conversions between mimeTypes, but it's a little fiddly to figure out exactly ...
I've written many times about various Apps Script caching techniques such as how to deal with size limits and use ...
Motivation I've been working on a CardService Add-on lately which also uses HtmlService, and also runs quite a few things ...
Smg is a SuperFetch plugin to access the Google Cloud Secrets API. SuperFetch is a proxy for UrlFetchApp with additional ...
SuperFetch is a proxy for UrlFetchApp with additional features such as built-in caching – see SuperFetch – a proxy enhancement ...
Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
Tank and Drv are SuperFetch plugins to emulate streaming and use the Drive REST API with Apps Script. SuperFetch is ...
Tank is a SuperFetch plugin to emulate streaming with Apps Script. SuperFetch is a proxy for UrlFetchApp with additional features ...
Drv is a SuperFetch plugin to access the Google Drive API. SuperFetch is a proxy for UrlFetchApp with additional features ...
Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
I covered how to handle the somewhat more complex OAUTH2 authorization flow for the Twitter v2 API (OAuth 2.0 Authorization ...
Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with ...
Motivation Goa is a library to support OAuth2 for Apps Script connecting to a variety of services, using a variety ...
It's been a few years since I first created the Goa library. Initially it was mainly to provide OAuth2 authorization ...
Frb is a SuperFetch plugin to easily access a Firebase Real time database. SuperFetch is a proxy for UrlFetchApp with ...
SuperFetch is a proxy for UrlFetchApp with additional features - see SuperFetch - a proxy enhancement to Apps Script UrlFetch ...
I've written a few articles about JavaScript proxying on here, and I'm a big fan. I also use a lot ...
Motivation Caching is a great way to improve performance, avoid rate limit problems and even save money if you are ...
Why unit testing? There are many test packages for Node (my favorite is ava) and there are also a few ...
uper