Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.

This is another in my series on SuperFetch plugins. It also includes a script to analyze your twitter circle to a spreadsheet.

Page Content hide

1.1 Script and Twitter Preparation

1.2.3.1 superFetch (required)

1.2.3.2 noCache property (optional)

1.2.3.3 showUrl property (optional)

1.3 Calling structure

1.5 Using public metrics

1.6 Writing the lists to a sheet

1.7 Unit testing

1.8 Links

1.9 Related

1.9.1 Convert any file with Apps Script

1.9.2 Caching, property stores and pre-caching

1.9.3 State management across CardService, HtmlService and Server side Add-ons

1.9.4 SuperFetch Plugin: Cloud Manager Secrets and Apps Script

1.9.5 SuperFetch caching: How does it work?

1.9.6 SuperFetch plugins: Tank events and appending

1.9.7 SuperFetch Plugins: Apps Script streaming with Tank and Drive

1.9.8 SuperFetch Tank Plugin: Streaming for Apps Script

1.9.9 SuperFetch plugin – Google Drive client for Apps Script – Part 1

1.9.10 SuperFetch – Twitter plugin for Apps Script – Get Follows, Mutes and blocks

1.9.11 SuperFetch plugin – Twitter client for Apps Script – Counts

1.9.12 OAuth2 and Twitter API – App only flow for Apps Script

1.9.13 SuperFetch plugin – Twitter client for Apps Script – Search and Get

1.9.14 Apps Script Oauth2 library Goa: tips, tricks and hacks

1.9.15 Apps Script Oauth2 – a Goa Library refresher

1.9.16 SuperFetch plugin – Firebase client for Apps Script

1.9.17 SuperFetch plugin – iam – how to authenticate to Cloud Run from Apps Script

1.9.18 SuperFetch – a proxy enhancement to Apps Script UrlFetch

1.9.19 Apps script caching with compression and enhanced size limitations

1.9.20 Simple but powerful Apps Script Unit Test library

Motivation

This is a follow on from SuperFetch plugin – Twitter client for Apps Script – Search and Get but will cover the followers, following, muting and blocking methods. This returns user data for each of those endpoints

Script and Twitter Preparation

You’ll need the bmSuperFetch and cGoa libraries and if you want to run some tests, the bmUnitTest library – details at the end of the article.

Like SuperFetch plugin – Twitter client for Apps Script – Search and Get you can use either Twitter Oauth2 User flow or the less secure OAuth2 and Twitter API – App only flow for Apps Script

I recommend using the user Flow where you can.

Instantiation

There’s a few things you need to initialize before you can start running your script

Goa

First you’ll need a Goa to provide a token service – see Twitter Oauth2 User flow for how to set up. All the code you need is below. Just patch in your credentials, run the onceoff, deploy as a webapp and use test deployment to get the consent process kicked off. If you’ve done all this before for previous endpoints, note that there’s a couple of extra scopes you’ll need in the goaPackage for these methods.

  // we can use this for user 
  const goa = makeTwitterGoa()
  
  // set this up somewhere
  const SETTINGS = {
    twitter: {
      propertyService: PropertiesService.getUserProperties(),
      name: 'twitter'
    }
  }
	  
  // this will nake a Goa - you can ignore (e) for now 
  const makeTwitterGoa = (e) => {
    const { twitter } = SETTINGS
    return cGoa.make(
      twitter.name,
      twitter.propertyService,
      e
    )
  }
  

  // run this once 
  const oneoffTwitter = () => {
    const { twitter } = SETTINGS
	
	// you'll need this to initialize goa for the first time then you can delete them
    const goaPackage = {  
      // replace with your credentials
      clientId: "exxxQ",
      clientSecret: "dxxxa",
	  
      scopes: ["tweet.read", "users.read", "follows.read", "block.read", "mute.read"],
      service: 'twitter',
      packageName: 'twitter'
    }
    cGoa.GoaApp.setPackage(twitter.propertyService, goaPackage)
  }

 
  // deploy this once off to kick off the goa consent process
  
  function doGet(e) {
    return doGeTwitter(e)
  }

  function doGeTwitter(e) {

    const goa = makeTwitterGoa(e)

    // it's possible that we need consent - this will cause a consent dialog
    if (goa.needsConsent()) {
      return goa.getConsent();
    }

    // get a token
    const token = goa.getToken()

    // if we get here its time for your webapp to run and we should have a token, or thrown an error somewhere
    if (!goa.hasToken()) throw 'something went wrong with goa - did you check if consent was needed?';

    // now we can use the token in a query or just leave it there registered for future server side use
    return HtmlService.createHtmlOutput(`Got this access token ${token}`)

  }

get an user flow twitter

SuperFetch

Include the plugins you need, and an instaniated SuperFetch instance. If you don’t need caching, then just omit the cacheService property (but I highly recommend caching with this API).

  // import required modules
  // import required modules
  const { Plugins, SuperFetch } = bmSuperFetch
  const { Twt } = Plugins

  const goa = makeTwitterGoa()
  
  // if you want to change the cache expiry time from the default ( 1 hour)
  // add the expiry property with the number of seconds cache entries should live for
  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: goa.getToken,
    cacheService: CacheService.getUserCache()
  })

superfetch instance

Twt instance

This will be the handle for accessing Twitter.


  const t = new Twt({
    superFetch
  }).users

twt instance

There are various other standard SuperFetch Plugin parameters that I’ll deal with later, but the ones of most interest are:

superFetch (required)

The instance you created earlier

noCache property (optional)

In this instance, I’m turning caching off for now.

showUrl property (optional)

Normally this would be false, but if you want to see you how your calls to the twt client are translated to native Rest calls, you can add this and set it to true.

Calling structure

The twt plugin uses closures heavily. A closure is a function which carries the references to its state (the lexical environment) with it. This encapsulation is very handy for creating reusable short cuts to standard queries, as we’ll cover later.

Get my id

We’ll be using my id for theses test, so first here’s how to get your own record

const me = t.me().throw().data.items[0]

get my own user record

Get followers

Here’s how to get a list of users that follow you.


 const {items} = t.followers(me.id).throw().data

followers

Get following

Here’s how to get a list of users that you follow.


 const {items} = t.following(me.id).throw().data

following

Get blocking

Here’s how to get a list of users that you have blocked


 const {items} = t.blocking(me.id).throw().data

blocking

Get muting

Here’s how to get a list of users that you have muted


 const {items} = t.muting(me.id).throw().data

muting

Example

Let’s say we want to find out which users I follow who are also following me back, those that are following me but that I don’t follow back, and those that I follow but are not following me back, plus the number I’ve blocked and muted.

const articleFollows = () => {

  const { Plugins, SuperFetch } = bmSuperFetch
  const { Twt } = Plugins

  const goa = makeTwitterGoa()
  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: goa.getToken,
    cacheService: CacheService.getUserCache(),
  })
  const t = new Twt({
    superFetch
  })

  // get my profile
  const me = t.users.me().throw().data.items[0]

  // get them all
  const everybody = t.users.query({ id: me.id }).page({ max: Infinity })

  // those I'm following
  const { items: following } = everybody.following().throw().data

  // my followers
  const { items: followers } = everybody.followers().throw().data
  
  // those I'm blocking
  const { items: blocking } = everybody.blocking().throw().data
  
  // those I'm muting
  const { items: muting } = everybody.muting().throw().data
  

  // now look for connections
  const both = following.filter(f => followers.find(({ id }) => id === f.id))
  const followersOnly = followers.filter(f => !both.find(({ id }) => id === f.id))
  const followingOnly = following.filter(f => !both.find(({ id }) => id === f.id))
  


  console.log(
    'total followers',
    followers.length,
    '\ntotal following',
    following.length,
    '\nmutual follows',
    both.length,
    '\nfollowers only',
    followersOnly.length,
    '\nfollowing only',
    followingOnly.length,
	'\nblocking only',
    blocking.length,
	'\nblocking only',
    muting.length
  )
}
/*
total followers 3699 
total following 3172 
mutual follows 2943 
followers only 756 
following only 229
*/

profile

Rate limit

At this point, depending on how many followers etc you have, you’re likely to hit a rate limit problem.

Using Superfetch’s built in caching is a great way to minimize the number of API requests you make, but when cache has expired (you can set the expiry time as a SuperFetch parameter – by default it’s 1 hour) or it’s the first time you make a particular query, it’ll count against your rate limit. The full story on Twitter rate limits is here

There are a number of ways of dealing with rate limits, some of which I’ve already covered in other SuperFetch articles, but the twitter one gives some clues to help so we don’t have to rely on exponential backoff or throttling calls as described in SuperFetch – a proxy enhancement to Apps Script UrlFetch

In the case of these user endpoints, twitter only allows 15 requests in a 15 minute period. Unlike tweet searching (which only allows only 100 reponses per page), these allow 1000 responses per page, so the Twt plugin will automatically adjust the request pagesize to the maximum it can get away with for each endpoint.

The plugin also returns a special rateLimit property as a response to each query which gives info about how much you have left – but that’ll be the subject of another SuperFetch/Twt article.

Using public metrics

In the previous example, we’ve retrieved the data for each of the users following etc. to get the counts. That’s because we need to analyze the intersection between them. If you just want the raw counts, there’s a much easier way using the user’s public metrics

Here’s how we can request some additional fields about your own profile. Of course this only gives a couple of summaries, but perhaps it’s enough

const articleMetrics = () => {

  const { Plugins, SuperFetch } = bmSuperFetch
  const { Twt } = Plugins

  const goa = makeTwitterGoa()
  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: goa.getToken,
    cacheService: CacheService.getUserCache(),
  })
  const t = new Twt({
    superFetch
  })

  // these are the extra user fields we want
  const fields =  {
    "user.fields": "public_metrics,profile_image_url,url,name,description,created_at,username"
  }

  // get my profile
  const me =  t.users.query({fields}).me().throw().data.items[0]
  console.log(me)
}
/*
{ description: 'Unprofessional musician, author and ancient geek. #gde',
  id: '17517365',
  public_metrics: 
   { followers_count: 3700,
     following_count: 3172,
     tweet_count: 25994,
     listed_count: 49 },
  created_at: '2008-11-20T18:10:27.000Z',
  username: 'brucemcpherson',
  name: 'Bruce McPherson 🇪🇺🇫🇷🏴󠁧󠁢󠁳󠁣󠁴󠁿',
  url: 'https://t.co/yHMsJL26Vk',
  profile_image_url: 'https://pbs.twimg.com/profile_images/1309151914551136256/Cw4iPy32_normal.jpg' }
 */

public metrics

Writing the lists to a sheet

Since we’re in Apps Script here, we as may well get some more info about our followers and write it all to a sheet. I’m going to use the bmPreFiddler library for this, as it makes sheet manipulation a bit easier. ID details at end of post.



const articleSheets = () => {

  const { Plugins, SuperFetch } = bmSuperFetch
  const { Twt } = Plugins

  const goa = makeTwitterGoa()
  const superFetch = new SuperFetch({
    fetcherApp: UrlFetchApp,
    tokenService: goa.getToken,
    cacheService: CacheService.getUserCache(),
  })

  const t = new Twt({
    superFetch
  })


  // these are the extra user fields we want
  const fields = {
    "user.fields": "public_metrics,profile_image_url,url,name,description,created_at,username"
  }

  // get my profile
  const me = t.users.query({ fields }).me().throw().data.items[0]

  // closure to get them all
  const everybody = t.users.query({ fields, id: me.id }).page({ max: Infinity })

  // those I'm following
  const { items: following } = everybody.following().throw().data

  // my followers
  const { items: followers } = everybody.followers().throw().data

  // those I'm blocking
  const { items: blocking } = everybody.blocking().throw().data

  // those I'm muting
  const { items: muting } = everybody.muting().throw().data

  // merge into a single table and remove dups and decorate and flatten for sheet
  const data = Array.from(
    new Map([me].concat(following, followers, blocking, muting).map(f => [f.id, f])).values())
    .map(item => ({
      ...item,
      ...item.public_metrics,
      following: Boolean(following.find(f => f.id === item.id)),
      followers: Boolean(followers.find(f => f.id === item.id)),
      blocking: Boolean(blocking.find(f => f.id === item.id)),
      muting: Boolean(muting.find(f => f.id === item.id))
    }))

  // write it all to this sheet and remove the defunct public_metrics column
  bmPreFiddler.PreFiddler().getFiddler({
    sheetName: 'twitter-circle',
    id: '1UU6t01SRssYQhZGSSo62HD7u7rVw6NtZ3jCPadmY7uA',
    createIfMissing: true
  })
    .setData(data)
    .filterColumns(name => name !== "public_metrics")
    .dumpValues()

}

writing profile of twitter circle to a sheet

That gives us a sheet of everyone in your twitter universe along with their stats and relationship to you. It’ll make a a good crib sheet for a twitter clearout.

Unit testing

I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls. These tests demonstrate in detail each of the topics mentioned in this article, and a few others, and could serve as a useful crib sheet for the plugin



const testTwt = ({ force = false, unit } = {}) => {

  // manage skipping individual tests

  const skipTest = {
    userFollowers: false && !force
  }

  // get a testing instance (or use the one passed over)
  unit = unit || new bmUnitTester.Unit({
    showErrorsOnly: true,
    maxLog: 200,
    showValues: true
  })


  // import required modules
  const { Plugins, SuperFetch } = bmSuperFetch
  const { Twt } = Plugins

  unit.section(() => {

    const goa = makeTwitterGoa()
    const superFetch = new SuperFetch({
      fetcherApp: UrlFetchApp,
      tokenService: goa.getToken,
      cacheService: CacheService.getUserCache()
    })

    const t = new Twt({
      superFetch,
      max: 30
    }).users

    // get my profile to play around with
    const { actual: me } = unit.not(null, t.me().throw().data.items[0], {
      description: 'get my profile'
    })
    const { actual: followers } = unit.not(null, t.followers(me.id).throw(), {
      description: 'get some followers'
    })
    const { actual: following } = unit.not(null, t.following(me.id).throw(), {
      description: 'get some following'
    })

    // try a closure
    const fQuery = t.query({ id: me.id })
    unit.is(followers.data.items, fQuery.followers().throw().data.items, {
      description: 'following closure'
    })
    unit.is(following.data.items, fQuery.following().throw().data.items, {
      description: 'following closure'
    })

    // override max
    unit.is(followers.data.items.slice(0, 20), fQuery.page({ max: 20 }).followers().throw().data.items, {
      description: 'different max'
    })
    const overMax = 110
    const { actual: overSize } = unit.is(t.page({ max: overMax }).following(me.id).throw().data.items, fQuery.page({ max: overMax }).following().throw().data.items, {
      description: 'mixed closure and query'
    })
    unit.is(overMax, overSize.length, {
      description: 'mixed correct number of items'
    })
    const fields = {
      "user.fields": "name,profile_image_url,username"
    }
    const pQuery = t.query({ fields }).page({ max: 10 })
    unit.is(pQuery.followers(me.id).throw().data, t.page({ max: 10 }).followers(me.id, fields).throw().data, {
      description: 'expansion matches on closure'
    })

    unit.not(null, t.blocking(me.id).throw(), {
      description: 'got blocking'
    })

    unit.not(null, t.muting(me.id).throw(), {
      description: 'got muting'
    })


  }, {
    description: 'get users by followers',
    skip: skipTest.userFollwers
  })
  unit.report()
}

tests