Apps Script V8 implements ArrayBuffers and Typed arrays from ES6. If you’ve used Apps Script Crypto Utilities or done any work with Blobs, then this is what’s going on behind the scenes, now exposed as part of the language.

What are ArrayBuffers and Typed Arrays

C programmers will be familiar with the concept of allocating an arbitrary piece memory and mapping it as a structure to more easily get at the contents. ArrayBuffers are the memory, and Typed Arrays are the structures.

Why you might need them

If you are dealing with Binary files or streaming, ArrayBuffers and Typed arrays complements the Apps Script Blob utilities.

Creating a buffer

Here we’ll create a buffer, 64 bytes long. There’s nothing in it, but it shows on the console as a JavaScript object

  // an arbitrary buffer
  const buffer = new ArrayBuffer (64)
  // shows as {} an object
  console.log(buffer)
  // confirmed its an object
  console.log(typeof buffer)

but it’s not an Array

  // false
  console.log(Array.isArray(buffer))

Typed Array Uint8Array

We can declare a Typed Array as an overlay to the same piece of memory. In this example the entire buffer is being mapped as and examined as if it were a an array of 64 1 byte unsigned numbers

  // let's say that buffer would contain 64 1 byte numberi
  const numbers = new Uint8Array(buffer)
  // 64, { '0': 0,'1': 0,'2': 0,'3': 0, ...etc
  console.log(numbers.length, numbers)

Typed Array Uint32Array

Further, we can map a different kind of array over exactly the same memory space. In this case, instead of 64 1 byte numbers we can map 16 4 byte numbers

  // what about if the numbers were 4 byte numbers
  const bigNumbers = new Uint32Array(buffer)
  // 16, { '0': 0,'1': 0,'2': 0,'3': 0, ...etc
  console.log(bigNumbers.length, bigNumbers)

Typed Array Float64Array

We can even map floating point numbers over the same space. In this case 8 x 8 byte floats

  const bigFloatNumbers = new Float64Array(buffer)
  // 8 { '0': 0, '1': 0, '2': 0, '3': 0, '4': 0, '5': 0, '6': 0, '7': 0 }
  console.log(bigFloatNumbers.length, bigFloatNumbers)

Iteration

Although these are not Arrays, the have much the same methods as an array – for example

  // iterable - yes!
  bigFloatNumbers.forEach(f=>console.log(f))

Converting to a regular Array

And like every Array like object, you can easily convert them into  actual arrays

  // make into a regular array 
  // [ 0, 0, 0, 0, 0, 0, 0, 0 ]
  console.log(Array.from(bigFloatNumbers))

Assigning values

You can assign values just like a regular array

  // assigning values
  bigFloatNumbers.forEach((f,i)=> bigFloatNumbers[i] = Math.log(2+i))
  // 8 { '0': 0.6931471805599453,'1': 1.0986122886681096} etc..
  console.log(bigFloatNumbers.length, bigFloatNumbers)

Remapping

All this is happening in the ArrayBuffer originally created. Because the Typed Arrays are simply different views into the same data, we can look at the floats just created byte by byte (as they would be in a blob)

  // whats in the buffer, byte by byte
  // [ 239, 57,254,-2, 66, 46, etc...
  const bytes = Array.from(new Uint8Array(buffer))

Blobs

An ArrayBuffer provides an array of arbitrary memory bytes, and Typed Arrays provide a convenient way to access that memory depending on the type of data to be stored there. A Blob (Binary large object) is how we usually package up this arbitrary binary data (A blob contains meta data as well as the data itself) so that it can be conveniently written to and read from a binary file (such as an image file).

Apps Script has a number of Utilities which both convert to and from Blobs, but which also use blobs for exchanging data (for example, cryptographic Utilities).  The DriveApp service can also accept a blob as content to be written to a file, so let’s look at the relationship between ArrayBuffers and Blobs.

Uint8Array view to create a file

To create a Blob, we need to get the bytes that make up the ArrayBuffer. One way to do this is simply to map a Uint8Array (1 byte per element) to the buffer, then use the Apps Script newBlob() method to wrap those bytes up in metadata. DriveApp.createFile() accepts a blob as input. I won’t bother to specify a mimetype as its purely for my use.

  const bytes = Array.from(new Uint8Array(buffer))
  // can make a blob
  const blob = Utilities.newBlob(bytes,null,"blobby")
  // and write it to a file
  const file = DriveApp.createFile(blob)

It’s a binary file, so it’s normal that it’s unreadable.

Reading file content

Quite a few steps in converting from a file to an ArrayBuffer (there are some shortcuts which I’ll cover later), but in summary

  1. Read back the file, extract the content as a blob, and extract out the array of bytes contained within that blob
  2. Make a new ArrayBuffer the same length as the array of bytes
  3. Map a new Uint8Array over the buffer to access it byte by byte
  4. Transfer each of the bytes into the Uint8Array (and therefore the underlying buffer)
  5. Map a new Float64Array to the buffer (because that’s what we started with)
  6. Display the result
  7. Checks it’s what we started with
  // read it in again
  const blobBytes = DriveApp.getFileById(file.getId()).getBlob().getBytes()
  // make another buffer
  const newBuffer = new ArrayBuffer(blobBytes.length)
  // map some bytes over it
  const newBytes = new Uint8Array(newBuffer)
  // copy the bytes in from the file blob
  blobBytes.forEach((f,i)=>newBytes[i]=f)
  // map some floats over it
  const newFloats = new Float64Array(newBuffer)
  // same as we started with!
  // { '0': 0.6931471805599453,'1': 1.0986122886681096, '2': 1.3862943611198906, etc..
  console.log(newFloats)

Using classes with ArrayBuffers

Those methods looked at so far are fine when the data is simple and homogenous, but let’s say the data is more complex (and you don’t want to use JSON to simply write/read to files, but rather to write the actual binary representation of the data). In other words, how do you get an object like this to and from an ArrayBuffer.

  const player = {
    id: 100,
    firstName: 'john',
    lastName: 'doe',
    scores: [1,2,3]
  }

A nice way of representing such data in an ArrayBuffer is to use a class (if you don’t know about classes see Apps Script V8: Multiple script files, classes and namespaces ). In this example, BufPerson is actually an ArrayBuffer that knows how to access properties by their type and position.

  const person = new BufPerson()
  // add some data
  person.id = 100
  person.firstName = 'john'
  person.lastName = 'doe'
  person.scores = [1,9,3,5,6]

As a bonus, we also want to be able to create an instance of this class from an existing buffer (which may or may not contain data already), or even from bytes read from a binary file.

The constructor

It accepts 3 types of arguments

  • none – create a buffer and map the class properties to it
  • a buffer – use that and map the class properties to it
  • bytes – create a buffer, map the class properties to it and populate with the bytes
class BufPerson {
  static getSize () {
    return 56
  }
  // takes a buffer or creates one if not given
  // can also take bytes
  constructor (content) {
    let buffer = null
    // this object has a specific size
    this.size = this.constructor.getSize()
    if (Array.isArray(content)) {
      // it's bytes
      buffer = new ArrayBuffer(content.length)
      const newBytes = new Uint8Array(buffer)
      // copy the bytes in 
      content.forEach((f,i)=>newBytes[i]=f)
    } else {
      // we got an initial buffer, or create one
      if (content) {
        if (content.contructor && content.constructor.name !== 'ArrayBuffer') {
          throw 'expected an ArrayBuffer but got ' + content.contructor && content.constructor.name
        }
        buffer = content
      } else {
        buffer = new ArrayBuffer(this.size)
      }
    }
    // now populate
    this.buffer = buffer
    if (this._buffer.byteLength !== this.size) {
      throw 'buffer should be '+this.size+' not '+this._buffer.byteLength
    }
  }

Notes

  1.  size: since we know the structure, we also know the size of the ArrayBuffer. Exposing it as a static property in case any callers need to know. Also, check that any data supplied is the right size
  2. detecting the argument type. We’ve already seen that ArrayBuffers are not actually an array, so if we get an array – assume its bytes. If it’s not an array and the argument is present, check it’s actually an ArrayBuffer.
  3. populating. We’ll use a setter for this, so it can be reused.

Setting up the mapping

Previously, we looked at how Typed Arrays could map over a buffer to provide different views on the same data. Typed Array constructors take additional arguments (byte offset, number of elements) so as to map them to different parts of the parent ArrayBuffer. In the context of a class, these parts of the buffer can be abstracted as properties of the class. If there is already some data in the buffer (passed to the constructor), they’ll automatically be able to be viewed as the correct type through these abstractions.

  get buffer () {
    return this._buffer
  }
  set buffer (initial) {
    this._buffer = initial
    // different parts of the buffer can hold different types of data
    // args are byteoffset, number of items
    // a 4 byte value in bytes 0-3
    this._id = new Int32Array(this._buffer, 0 ,1)
    // 16 chars in bytes 4 - 19
    this._firstName = new Uint8Array(this._buffer, 4 ,16)
    // 16 chars in byes 20 - 35
    this._lastName = new Uint8Array(this._buffer, 20 ,16)
    // 5 Int16 in bytes 36-55
    this._scores = new Int16Array(this._buffer, 36, 5)
  }

Utilities

No point in repeating these common operations for each field, so they can be declared as reusable methods

  fillBuffer (buf, value = 0) {
    buf.forEach((f,i)=>buf[i]=value)
  }
  stringToBuffer (str, buf) {
    this.fillBuffer(buf)
    str.split('').forEach((f,i)=>buf[i]=f.charCodeAt(0))
  }
  bufferToString (buf) {
    // like in C, a string will be terminated with binary 0
    // so find that to see how much of the buffer is actually occupied
    const b = new Uint8Array(buf)
    const len = b.indexOf(0)
    return String.fromCharCode.apply(null, buf.slice(0,len < 0 ? b.length : len ) || b.length);
  }

Getters and setters

Using the utilities above, these are now quite unremarkable setters and getters

set id (id) {
    this._id[0] = id
  }
  get id () {
    return this._id[0]
  }
  set firstName (firstName) {
    this.stringToBuffer(firstName,this._firstName)
  }
  get firstName () {
    return this.bufferToString(this._firstName)
  }
  set lastName (lastName) {
    this.stringToBuffer(lastName,this._lastName)
  }
  get lastName () {
    return this.bufferToString(this._lastName)
  }
  set scores (scores) {
    this.fillBuffer(this._scores)
    scores.forEach((f,i)=>this._scores[i]=f)
  }
  get scores () {
    return Array.from(this._scores)
  }

bytes

Finally add a convenience getter for extracting the bytes from the buffer

  get bytes () {
    return Array.from(new Uint8Array(this._buffer))
  }

Using the class

Now this can be treated pretty much like a regular class

  const person = new BufPerson()
  // add some data
  person.id = 100
  person.firstName = 'john'
  person.lastName = 'doe'
  person.scores = [1,9,3,5,6]
  
  // what does that look like
  // 100 'john' 'doe' [ 1, 9, 3, 5, 6 ]
  console.log(person.id,person.firstName,person.lastName,person.scores)

but of course underlying it is an Array buffer

  // what does it look like in bytes
  // [ 100,0,0,0,106,111,
  console.log(person.bytes)

Make a copy

It’s as simple as taking a copy of the buffer with slice() and instantiating. Note that if you simply mapped the same buffer to a second instance of the class (without taking a copy), both instances of the class would point to the same buffer and changing one would change the other.

  // make another one from the old
  const p2 = new BufPerson(person.buffer.slice())
  // 100 'john' 'doe' [ 1, 9, 3, 5, 6 ]
  console.log(p2.id,p2.firstName,p2.lastName,p2.scores)
  // change the id of the second one
  p2.id = 200
  p2.firstName = 'jane'
  // 200 'jane' 'doe' [ 1, 9, 3, 5, 6 ]
  console.log(p2.id,p2.firstName,p2.lastName,p2.scores)

Make a blob and write to file

Now the buffer (or more specifically, the bytes from the buffer), can be wrapped in a blob and written to a file on Drive. In this case, both person instances are combined to create a single blob – so the file will contain data for both the people created above

  // make a blob of an array of them
  const blob = Utilities.newBlob(person.bytes.concat(p2.bytes),null,"personBlob")
  // write
  const file = DriveApp.createFile(blob)
  // read it in again
  const blobBytes = DriveApp.getFileById(file.getId()).getBlob().getBytes()

Read it back

Finally, we can read back from that file and construct new instances from the blob by chopping it up into the length of buffer needed for each instance.

// split up into chunks
  const items = []
  while (blobBytes.length) {
    items.push(blobBytes.splice(0,person.size))
  }
  // make new people  
  const people = items.map(item=>new BufPerson(item))

  // 100 'john' 'doe' [ 1, 9, 3, 5, 6 ]
  // 200 'jane' 'doe' [ 1, 9, 3, 5, 6 ]
  people.forEach(p=>console.log( p.id,p.firstName,p.lastName,p.scores))

The whole thing

The class is fairly verbose, but once you’ve done it for one it’s easy to tweak it for others you want to treat the same way. Here’s the whole class for convenience.

class BufPerson {
  static getSize () {
    return 56
  }
  // takes a buffer or creates one if not given
  // can also take bytes
  constructor (content) {
    let buffer = null
    // this object has a specific size
    this.size = this.constructor.getSize()
    if (Array.isArray(content)) {
      // it's bytes
      buffer = new ArrayBuffer(content.length)
      const newBytes = new Uint8Array(buffer)
      // copy the bytes in 
      content.forEach((f,i)=>newBytes[i]=f)
    } else {
      // we got an initial buffer, or create one
      if (content) {
        if (content.contructor && content.constructor.name !== 'ArrayBuffer') {
          throw 'expected an ArrayBuffer but got ' + content.contructor && content.constructor.name
        }
        buffer = content
      } else {
        buffer = new ArrayBuffer(this.size)
      }
    }
    // now populate
    this.buffer = buffer
    if (this._buffer.byteLength !== this.size) {
      throw 'buffer should be '+this.size+' not '+this._buffer.byteLength
    }
  }

  get bytes () {
    return Array.from(new Uint8Array(this._buffer))
  }
  get buffer () {
    return this._buffer
  }
  set buffer (initial) {
    this._buffer = initial
    // different parts of the buffer can hold different types of data
    // args are byteoffset, number of items
    // a 4 byte value in bytes 0-3
    this._id = new Int32Array(this._buffer, 0 ,1)
    // 16 chars in bytes 4 - 19
    this._firstName = new Uint8Array(this._buffer, 4 ,16)
    // 16 chars in byes 20 - 35
    this._lastName = new Uint8Array(this._buffer, 20 ,16)
    // 5 Int16 in bytes 36-55
    this._scores = new Int16Array(this._buffer, 36, 5)
  }
  fillBuffer (buf, value = 0) {
    buf.forEach((f,i)=>buf[i]=value)
  }
  stringToBuffer (str, buf) {
    this.fillBuffer(buf)
    str.split('').forEach((f,i)=>buf[i]=f.charCodeAt(0))
  }
  bufferToString (buf) {
    // like in C, a string will be terminated with binary 0
    // so find that to see how much of the buffer is actually occupied
    const b = new Uint8Array(buf)
    const len = b.indexOf(0)
    return String.fromCharCode.apply(null, buf.slice(0,len < 0 ? b.length : len ) || b.length);
  }
  set id (id) {
    this._id[0] = id
  }
  get id () {
    return this._id[0]
  }
  set firstName (firstName) {
    this.stringToBuffer(firstName,this._firstName)
  }
  get firstName () {
    return this.bufferToString(this._firstName)
  }
  set lastName (lastName) {
    this.stringToBuffer(lastName,this._lastName)
  }
  get lastName () {
    return this.bufferToString(this._lastName)
  }
  set scores (scores) {
    this.fillBuffer(this._scores)
    scores.forEach((f,i)=>this._scores[i]=f)
  }
  get scores () {
    return Array.from(this._scores)
  } 
}

 

Summary

ArrayBuffers are not something that every Apps Scripter will need to use daily, but if you are playing with blobs, the idea of mapping types over the data can be very handy. For example, you could dig into the details of an image file. Here’s a class to get at the beginning of the header file of a gif file.

class Giffer {
  constructor (bytes) {
    this._buffer = new ArrayBuffer(bytes.length)
    const newBytes = new Uint8Array(this._buffer)
    bytes.forEach((f,i)=>newBytes[i]=f)
    console.log(bytes.slice(0,10))
    this._version = new Uint8Array(this._buffer,0, 6 )
    this._width = new Uint16Array(this._buffer,6, 1 )
    this._height = new Uint16Array(this._buffer,8, 1 )
  }
  bufferToString (buf) {
    const b = new Uint8Array(buf)
    const len = b.indexOf(0)
    return String.fromCharCode.apply(null, buf.slice(0,len < 0 ? b.length : len ) || b.length);
  }
  get buffer () {
    return this._buffer
  }
  get version () {
    return this.bufferToString(this._version)
  }
  get width () {
    return this._width[0]
  }
  get height () {
    return this._height[0]
  }
}

and it can be used like this

const readGif = () => {
  const gif = new Giffer(DriveApp.getFileById('1qOvw2VKX7CjQy7PmerbPC55sZrCgv3s9')
                 .getBlob().getBytes())
  // GIF89a 168 160
  console.log(gif.version, gif.width, gif.height)
}

Endian and DataView

Finally a word on this complication. Endian refers to the order of bytes in a number (for example an Int2 spans two bytes and to be able to interpret it you need to know which is the high part and which is the low). Little endian means that the least significant part comes first, and big endian means the least significant comes last.

Sadly, there are other endian variants too. It’s all a hangover from a lack of standards between manufacturers in the early days, and disagreements about the efficiency of one method over the other (the life of an assembler programmer in those days was a nightmare).

You don’t normally need to care about it, but if you do, ArrayBuffers have another method of overlay viewing  – DataViews. These will allow you to specify the endianness of the underlying data. That is further complicated because a JavaScript integer isn’t really an integer (it’s a float, which means that Number.MAX_SAFE_INTEGER is 2**53 -1 rather than 2**64 -1 as you’d expect), so that means that 64-bit numbers with unusual endianness will be really messed up. However, that’s for another day.