It’s a pretty common requirement, especially when you’re posting to an API, to split an array of data into manageable chunks, and most often the solution is to make an array of arrays – with each one being up to a maximum size. That’s fine, but we can also make an iterator.

Let’s take a look at how these approaches differ, and take a look into iterators – a very interesting addition to Apps Script as a result of v8.

Make an array of arrays

This is the usual solution.

  const chunker = (inputArray, size) => {
    const chunks = []
    const items = inputArray.slice()
    while (items.length) chunks.push(items.splice(0, size))
    return chunks
  }
  
  const size = 5
  const chunkFixture = [1, 2, 3, 4, 5, 6, 7, 8, 9]

  const chunks = chunker(chunkFixture, size)
  console.log(chunks)
  
  // [ [ 1, 2, 3, 4, 5 ], [ 6, 7, 8, 9 ] ]
  
  // or
  for (let chunk of chunks) {
    console.log(chunk)
  }
  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]
  
chunk into an array of arrays

 

Using an iterator

This is slightly more complicated but has the advantage of only creating chunks when they are needed. In the first approach we’d have multiple versions of the input array – a chunked and an unchunked. It’s a more satisfying solution all round.

First we need a function that can make an iterator. We use the [Symbol.iterator] well known symbol to create the iterator method and signal that it is iterable.

  const chunkIt = (inputArray, size) => {
    // like slice
    const end = inputArray.length
    let start = 0
    return {
      *[Symbol.iterator]() {
        while (start < end) {
          const chunk = inputArray.slice(start, Math.min(end, size + start))
          start += chunk.length
          yield chunk
        }
      }
    }
  }
make iterator function

Now we can loop through the chunks creating them on demand.

  const it = chunkIt(chunkFixture, size)
  for (let chunk of it) {
    console.log(chunk)
  }

  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]
simple iterator

Adding start and end

It might be useful to start and end at a particular slice of the inputArray. Again, to avoid making a redundant slice of the input array, we can build an optional start and end into the iterator.

  const chunkItFancier = ({ inputArray, size, start = 0, end = inputArray.length }) => {
    // like slice
    return {
      *[Symbol.iterator]() {
        while (start < end) {
          const chunk = inputArray.slice(start, Math.min(end, size + start))
          start += chunk.length
          yield chunk
        }
      }
    }
  }
with start and end

The default is to use the entire input array as before

  const itFancier = chunkItFancier({
    inputArray: chunkFixture,
    size
  })
  for (let chunk of itFancier) {
    console.log(chunk)
  }
  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]
default to entire array

but we can also select just a section of the input array

  const itSlice = chunkItFancier({
    inputArray: chunkFixture,
    size,
    start: 4,
    end: 7
  })
  for (let chunk of itSlice) {
    console.log(chunk)
  }
  // [ 5, 6, 7 ]
a slice of the input array

Converting back to an array

Since we can make arrays out of iterators, duplicating the output of the first method (array of arrays) is simple

  console.log(Array.from(chunkItFancier({
    inputArray: chunkFixture,
    size
  })))
  
  // or
  
  console.log([...chunkItFancier({
    inputArray: chunkFixture,
    size
  })])
  
  // each give the same result
  // [[ 1, 2, 3, 4, 5 ],[ 6, 7, 8, 9 ]]

Related

Iterator magic – Splitting an array into chunks

It's a pretty common requirement, especially when you're posting to an API, to split an array of data into manageable ...
Read More
admin.google.com

The nodejs Drive API, service account impersonation and async iterators

This is part of the series on sharing data between Apps Script and Node on various backends, Apps script library with ...
Read More

Rate limit handler, helper and iterator: Apps Script use cases

Motivation for apps script example I originally create both qottle and rottler  for various node projects, but realized that I ...
Read More
apps script v8

ES6 Symbols – what on earth is all that about ?

javaWhen I first read about Symbols coming to JavaScript, I couldn't figure out what the point was or even what ...
Read More
apps script drive pile of files

Getting ‘a pile of files’ from Google Drive with Apps Script

The method for doing this is actually part of the bmFolderFun library documented in A handier way of accessing Google ...
Read More

Supercharging copying files between Drive and Cloud Storage from Apps Script with Cloud Run

In Blistering fast file streaming between Drive and Cloud Storage using Cloud Run I showed how you could use Cloud ...
Read More
apps script v8

A handier way of accessing Google Drive folders and files from Apps Script

The usual way to access files and folders on Drive from Apps Script is via their ID, yet on other ...
Read More
copt between drive and gcs

Blistering fast file streaming between Drive and Cloud Storage using Cloud Run

A friend of mine hit an Apps Script problem the other day when transferring large amounts of data between Drive ...
Read More
crusher files on drive

Sharing data between Apps Script and Node using Google Drive back end – 5 minute example

Another quick demo of data sharing Here's a challenge that shares the data in a spreadsheet with node, set up ...
Read More
scrviz - vizzy - libraries

Searching and cataloging Apps Script projects on Github

In Every Google Apps Script project on Github visualized  I demonstrated an app that could be used to explore what every ...
Read More