Iterator magic - Splitting an array into chunks

It’s a pretty common requirement, especially when you’re posting to an API, to split an array of data into manageable chunks, and most often the solution is to make an array of arrays – with each one being up to a maximum size. That’s fine, but we can also make an iterator.

Let’s take a look at how these approaches differ, and take a look into iterators – a very interesting addition to Apps Script as a result of v8.

Page Content hide

1 Make an array of arrays

2 Using an iterator

2.1 Adding start and end

2.2 Converting back to an array

3.2 The nodejs Drive API, service account impersonation and async iterators

3.3 Rate limit handler, helper and iterator: Apps Script use cases

3.4 ES6 Symbols – what on earth is all that about ?

3.5 Getting ‘a pile of files’ from Google Drive with Apps Script

3.6 Supercharging copying files between Drive and Cloud Storage from Apps Script with Cloud Run

3.7 A handier way of accessing Google Drive folders and files from Apps Script

3.8 Blistering fast file streaming between Drive and Cloud Storage using Cloud Run

3.9 Sharing data between Apps Script and Node using Google Drive back end – 5 minute example

3.10 Searching and cataloging Apps Script projects on Github

Make an array of arrays

This is the usual solution.

  const chunker = (inputArray, size) => {
    const chunks = []
    const items = inputArray.slice()
    while (items.length) chunks.push(items.splice(0, size))
    return chunks
  }
  
  const size = 5
  const chunkFixture = [1, 2, 3, 4, 5, 6, 7, 8, 9]

  const chunks = chunker(chunkFixture, size)
  console.log(chunks)
  
  // [ [ 1, 2, 3, 4, 5 ], [ 6, 7, 8, 9 ] ]
  
  // or
  for (let chunk of chunks) {
    console.log(chunk)
  }
  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]

chunk into an array of arrays

Using an iterator

This is slightly more complicated but has the advantage of only creating chunks when they are needed. In the first approach we’d have multiple versions of the input array – a chunked and an unchunked. It’s a more satisfying solution all round.

First we need a function that can make an iterator. We use the [Symbol.iterator] well known symbol to create the iterator method and signal that it is iterable.

  const chunkIt = (inputArray, size) => {
    // like slice
    const end = inputArray.length
    let start = 0
    return {
      *[Symbol.iterator]() {
        while (start < end) {
          const chunk = inputArray.slice(start, Math.min(end, size   start))
          start  = chunk.length
          yield chunk
        }
      }
    }
  }

make iterator function

Now we can loop through the chunks creating them on demand.

  const it = chunkIt(chunkFixture, size)
  for (let chunk of it) {
    console.log(chunk)
  }

  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]

simple iterator

Adding start and end

It might be useful to start and end at a particular slice of the inputArray. Again, to avoid making a redundant slice of the input array, we can build an optional start and end into the iterator.

  const chunkItFancier = ({ inputArray, size, start = 0, end = inputArray.length }) => {
    // like slice
    return {
      *[Symbol.iterator]() {
        while (start < end) {
          const chunk = inputArray.slice(start, Math.min(end, size   start))
          start  = chunk.length
          yield chunk
        }
      }
    }
  }

with start and end

The default is to use the entire input array as before

  const itFancier = chunkItFancier({
    inputArray: chunkFixture,
    size
  })
  for (let chunk of itFancier) {
    console.log(chunk)
  }
  // [ 1, 2, 3, 4, 5 ]
  // [ 6, 7, 8, 9 ]

default to entire array

but we can also select just a section of the input array

  const itSlice = chunkItFancier({
    inputArray: chunkFixture,
    size,
    start: 4,
    end: 7
  })
  for (let chunk of itSlice) {
    console.log(chunk)
  }
  // [ 5, 6, 7 ]

a slice of the input array

Converting back to an array

Since we can make arrays out of iterators, duplicating the output of the first method (array of arrays) is simple

  console.log(Array.from(chunkItFancier({
    inputArray: chunkFixture,
    size
  })))
  
  // or
  
  console.log([...chunkItFancier({
    inputArray: chunkFixture,
    size
  })])
  
  // each give the same result
  // [[ 1, 2, 3, 4, 5 ],[ 6, 7, 8, 9 ]]