Recursive async functions

Page Content hide

Background

I have a GraphQL server on which I’ve deliberately limited the amount of data that can be returned in a single query to avoid daft requests. That means that you need to do paging to get a set of results of more than that limit. There are various ways of paging in GraphQL, but this one uses a simple system of limits and offsets to get chunks of results. Something like this

query ($limit: Int , $offset: Int) {
   People ( limit: $limit , offset: $offset ) {
    id
   name
  }
}

That leaves the problem of how exactly to do queries like this when the calls themselves are asynchronous. If they were synchronous, then a simple recursive approach would be easy enough to implement, but asynchronous calls are a little more tricky. Using Promises improves things a little, but using ES6 “await” makes it no more complex than a synchronous solution. Let’s take a look.

Fetcher

Start with a simple fetcher (I’m using axios) that makes a request given a query an some optional variables.

/**
   * fetcher for graphql
   * @param {string} query the graphql query
   * @param {object} [variables] the optional graphql variables (the .data part of the graphql response)
   * @return {object} the result 
   */
  ns.gfetch = (query, variables) => {
    
    // ns.curl is already set to the endpoint for graphql server
    return axios
      .post(ns.cUrl, {
        query,
        variables
      })
      // strip off the axios wrapper
      .then(result => result.data.data)
      
      .catch(err => {
        // strip down the axios error to something meaningful
        const pack = {
          status: err.response.status,
          statusText: err.response.statusText,
          query,
          variables,
          errors: err.response.data.errors
        };
        // for quick tracking down
        console.log(JSON.stringify(pack));
        return Promise.reject(pack);
      });
  };

Pager

The pager is called to bunch up pages of graphql results into a single response. The inline comments describe what’s going on, but in summary

Pages are recursively fetched until either there are no more or the max requested is reached.
After each page, the offset and limit variables are tweaked to get the next page.
The use of await means it reads almost like a synchronous recursion

ns.paged = (options) => {

    let { vars, q, max, offset, limit, once } = options;
    // total result will be accumulated here
    const bunch = [];

    // set up default values and check params
    offset = offset || 0;
    limit = limit || 50;

    // max is the overall max number of rows to return
    // by default, it's all of them.
    if (max && limit > max) limit = max;

    // once is a single query with no paging required
    if (!once) {
      // if we are paging, we'll need some variables to control it.
      vars = vars || {};
      vars.limit = limit;
      vars.offset = offset;
      // and the query must be set up to allow limit and offet
      if (!q.match("limit:Int")) throw "must have a limit in the query for paging";
      if (!q.match("offset:Int")) throw "must have an offset in the query for paging";
    }

    // this is the recursive function
    // the object returned from graphql can be extracted using its key
    const doFetch = (v) => ns.gfetch(q, v).then(r => {
      // this is the property key of the first object
      const [stem] = Object.keys(r);
      if (Object.keys(r).length !== 1) throw 'paged only handles single top level queries ' + JSON.stringify(Object.keys(r));
      return r[stem];
    });

    // can detect a next page if we havent got enough recs or if the full limit was returned
    const isNextPage = (data) => (bunch.length < max || !max) && data.length === limit && !once;

    // get all the pages
    // its an async function
    const getPages = async function(v) {

      // so we can use await{
      const data = await doFetch(v);

      // accumulate what just got returned
      Array.prototype.push.apply(bunch, data);

      // if there's any more then carry on
      if (isNextPage(data)) {

        // modify the offset to skip what we have
        v.offset = bunch.length + offset;

        // modify the limit if it would return more than we need
        if (max && max - bunch.length < limit) v.limit = max - bunch.length;

        // go round again
        return await getPages(v);
      }
      else {
        return bunch;
      }
    };

    // start the whole thing off and return the accumulated result
    return getPages(vars);


  };

Using it

Get all the people named smith in a single result

paged({
    q: `query ($value:Value,$limit:Int,$offset:Int) {
          People (value: $value, limit: $limit, offset: $offset) {
            id
            name
          }
        }`,
    vars: {
       value:"Smith"
    }
  }).then (people =>{
    // do something with the result
  });

For more like this see Google Apps Scripts Snippets