Apps Script V8 implements ArrayBuffers and Typed arrays from ES6. If you’ve used Apps Script Crypto Utilities or done any work with Blobs, then this is what’s going on behind the scenes, now exposed as part of the language.

What are ArrayBuffers and Typed Arrays

C programmers will be familiar with the concept of allocating an arbitrary piece memory and mapping it as a structure to more easily get at the contents. ArrayBuffers are the memory, and Typed Arrays are the structures.

Why you might need them

If you are dealing with Binary files or streaming, ArrayBuffers and Typed arrays complements the Apps Script Blob utilities.

Creating a buffer

Here we’ll create a buffer, 64 bytes long. There’s nothing in it, but it shows on the console as a JavaScript object

but it’s not an Array

Typed Array Uint8Array

We can declare a Typed Array as an overlay to the same piece of memory. In this example the entire buffer is being mapped as and examined as if it were a an array of 64 1 byte unsigned numbers

Typed Array Uint32Array

Further, we can map a different kind of array over exactly the same memory space. In this case, instead of 64 1 byte numbers we can map 16 4 byte numbers

Typed Array Float64Array

We can even map floating point numbers over the same space. In this case 8 x 8 byte floats

Iteration

Although these are not Arrays, the have much the same methods as an array – for example

Converting to a regular Array

And like every Array like object, you can easily convert them into  actual arrays

Assigning values

You can assign values just like a regular array

Remapping

All this is happening in the ArrayBuffer originally created. Because the Typed Arrays are simply different views into the same data, we can look at the floats just created byte by byte (as they would be in a blob)

Blobs

An ArrayBuffer provides an array of arbitrary memory bytes, and Typed Arrays provide a convenient way to access that memory depending on the type of data to be stored there. A Blob (Binary large object) is how we usually package up this arbitrary binary data (A blob contains meta data as well as the data itself) so that it can be conveniently written to and read from a binary file (such as an image file).

Apps Script has a number of Utilities which both convert to and from Blobs, but which also use blobs for exchanging data (for example, cryptographic Utilities).  The DriveApp service can also accept a blob as content to be written to a file, so let’s look at the relationship between ArrayBuffers and Blobs.

Uint8Array view to create a file

To create a Blob, we need to get the bytes that make up the ArrayBuffer. One way to do this is simply to map a Uint8Array (1 byte per element) to the buffer, then use the Apps Script newBlob() method to wrap those bytes up in metadata. DriveApp.createFile() accepts a blob as input. I won’t bother to specify a mimetype as its purely for my use.

Here’s what the content of the file looks like when viewed in a text editor. It’s a binary file, so it’s normal that it’s unreadable.

Reading file content

Quite a few steps in converting from a file to an ArrayBuffer (there are some shortcuts which I’ll cover later), but in summary

  1. Read back the file, extract the content as a blob, and extract out the array of bytes contained within that blob
  2. Make a new ArrayBuffer the same length as the array of bytes
  3. Map a new Uint8Array over the buffer to access it byte by byte
  4. Transfer each of the bytes into the Uint8Array (and therefore the underlying buffer)
  5. Map a new Float64Array to the buffer (because that’s what we started with)
  6. Display the result
  7. Checks it’s what we started with

Using classes with ArrayBuffers

Those methods looked at so far are fine when the data is simple and homogenous, but let’s say the data is more complex (and you don’t want to use JSON to simply write/read to files, but rather to write the actual binary representation of the data). In other words, how do you get an object like this to and from an ArrayBuffer.

A nice way of representing such data in an ArrayBuffer is to use a class (if you don’t know about classes see Apps Script V8: Multiple script files, classes and namespaces ). In this example, BufPerson is actually an ArrayBuffer that knows how to access properties by their type and position.

As a bonus, we also want to be able to create an instance of this class from an existing buffer (which may or may not contain data already), or even from bytes read from a binary file.

The constructor

It accepts 3 types of arguments

  • none – create a buffer and map the class properties to it
  • a buffer – use that and map the class properties to it
  • bytes – create a buffer, map the class properties to it and populate with the bytes
Notes
  1.  size: since we know the structure, we also know the size of the ArrayBuffer. Exposing it as a static property in case any callers need to know. Also, check that any data supplied is the right size
  2. detecting the argument type. We’ve already seen that ArrayBuffers are not actually an array, so if we get an array – assume its bytes. If it’s not an array and the argument is present, check it’s actually an ArrayBuffer.
  3. populating. We’ll use a setter for this, so it can be reused.

Setting up the mapping

Previously, we looked at how Typed Arrays could map over a buffer to provide different views on the same data. Typed Array constructors take additional arguments (byte offset, number of elements) so as to map them to different parts of the parent ArrayBuffer. In the context of a class, these parts of the buffer can be abstracted as properties of the class. If there is already some data in the buffer (passed to the constructor), they’ll automatically be able to be viewed as the correct type through these abstractions.

Utilities

No point in repeating these common operations for each field, so they can be declared as reusable methods

Getters and setters

Using the utilities above, these are now quite unremarkable setters and getters

bytes

Finally add a convenience getter for extracting the bytes from the buffer

Using the class

Now this can be treated pretty much like a regular class

but of course underlying it is an Array buffer

Make a copy

It’s as simple as taking a copy of the buffer with slice() and instantiating. Note that if you simply mapped the same buffer to a second instance of the class (without taking a copy), both instances of the class would point to the same buffer and changing one would change the other.

Make a blob and write to file

Now the buffer (or more specifically, the bytes from the buffer), can be wrapped in a blob and written to a file on Drive. In this case, both person instances are combined to create a single blob – so the file will contain data for both the people created above

Read it back

Finally, we can read back from that file and construct new instances from the blob by chopping it up into the length of buffer needed for each instance.

The whole thing

The class is fairly verbose, but once you’ve done it for one it’s easy to tweak it for others you want to treat the same way. Here’s the whole class for convenience.

 

Summary

ArrayBuffers are not something that every Apps Scripter will need to use daily, but if you are playing with blobs, the idea of mapping types over the data can be very handy. For example, you could dig into the details of an image file. Here’s a class to get at the beginning of the header file of a gif file.

and it can be used like this

Endian and DataView

Finally a word on this complication. Endian refers to the order of bytes in a number (for example an Int2 spans two bytes and to be able to interpret it you need to know which is the high part and which is the low). Little endian means that the least significant part comes first, and big endian means the least significant comes last.

Sadly, there are other endian variants too. It’s all a hangover from a lack of standards between manufacturers in the early days, and disagreements about the efficiency of one method over the other (the life of an assembler programmer in those days was a nightmare).

You don’t normally need to care about it, but if you do, ArrayBuffers have another method of overlay viewing  – DataViews. These will allow you to specify the endianness of the underlying data. That is further complicated because a JavaScript integer isn’t really an integer (it’s a float, which means that Number.MAX_SAFE_INTEGER is 2**53 -1 rather than 2**64 -1 as you’d expect), so that means that 64-bit numbers with unusual endianness will be really messed up. However, that’s for another day.