Twt is a SuperFetch plugin to easily access to the Twitter v2 API. SuperFetch is a proxy for UrlFetchApp with additional features – see SuperFetch – a proxy enhancement to Apps Script UrlFetch for how it works and what it does.
This is another in my series on SuperFetch plugins.
I’m planning to create a SuperFetch plugin for all the APIS I use regularily. The Twitter V2 API is a great improvement on the original one, and now uses OAuth2 for authorization. See Apps Script Oauth2- a Goa Library refresher for how this works.
I wanted to keep the SuperFetch structure and approach so although this uses the Twitter REST API, the call structure is a bit different to, for example, the TypeScript Twitter client.
The usual Superfetch goodies like rate limiting, depaging and caching are all built in. This article will cover a subset of the methods available – there are many. I’ll post another few of articles in due course to cover the others.
Script and Twitter Preparation
You’ll need the bmSuperFetch and cGoa libraries and if you want to run some tests, the bmUnitTest library – details at the end of the article.
You’ll also need to set up a Twitter application in the Twitter developer console, and use the Goa library to handle OAuth2 authentication and authorization for you. This article lays out how to do that in detail. I suggest you go there when you are ready to start coding.
There’s a few things you need to initialize before you can start running your script
First you’ll need a Goa to provide a token service – The detail is in this article or for a simpler kind of ‘app only’ oauth see OAuth2 and Twitter API – App only flow for Apps Script
I’ll do a very quick recap on Goa for Twitter later in the article.
First you need to include the plugins you need, and an instaniated SuperFetch instance. If you don’t need caching, then just omit the cacheService property (but I highly recommend caching with this API).
This will be the handle for accessing Twitter.
There are various other standard SuperFetch Plugin parameters that I’ll deal with later, but the ones shown are of special interest to this article.
The instance you created earlier
noCache property (optional)
In this instance, I’m turning caching off for now.
max property (optional)
This is the default maximum number of items to return for a query. You can change this for individual queries of course. Note that the Twitter API insists on a minimum of 10, and that max should be a multiple of 10.
showUrl property (optional)
Normally this would be false, but if you want to see you how your calls to the twt client are translated to native Rest calls, you can add this and set it to true.
The twt plugin uses closures heavily. A closure is a function which carries the references to its state (the lexical environment) with it. This encapsulation is very handy for creating reusable short cuts to standard queries, as we’ll cover later.
The Plugin is divided up into ‘domains’. This article will cover searching and getting tweets and users. Here’s how you’d create a shortcut to each of the domains referenced in this article.
The tweets domain returns 1 item for each qualifying tweet, plus various expansions which I’ll cover as we go through the article
Here’s how you’d get the most recent set of tweets. The query is constructed using the same syntax as you’d use in the normal web client. The full details are here.
All responses have exactly the same format. You’ll get back a standard SuperFetch Response, which looks like this
Twitter API data
The response from the Twitter API will be in the data property of the SuperFetch response.
Actually the twitter response will have been massaged a bit so that every API call has exactly the same format. There are 2 properties of interest
- items – an array of standard data from the Twitter API
- expansions – some API methods return enhanced data – we’ll look at that later
The items are an array of basic data matching the search criteria
The Twitter API supports paging (the default page size is 10, and the maximum is 100). You’ll notice though that we got 200 items without bothering to handle paging at all. For a web app you’ll probably want paging to render results, but in Apps Script you’ll seldom want to bother with that – you just want all the results (up to ‘max’) in one shot.
SuperFetch ‘depages’ queries like this so that you get the ‘max’ number of results consolidated into a single list of items. It will automatically optimize the pageSizes it requests from the Twitter API to minimize the number of external API fetches it does.
If you do need to handle paging yourself, then there is a paging mechanism too, which we’ll cover later.
You can enhance the results from the plugin by specifying fields. The complete list of fields available as parameters is here.
The Plugin provides a number of ways of specifying fields – the most reusable one being to create a query closure. Let’s say we want to enhance the tweets data with the user data for the author and the tweet created time
You’ll notice that the expansions property is now populated with that extra data about the author, and we have an extra field, created_at in the tweet data.
The reason that expansions are separate from to the basic data, is to avoid repetition. For example, there may be 100 tweets by the same author. In that case we’d have 100 tweet data items, but only 1 expansions.include.users entry. These can be matched on author_id if you need to do that.
Field closure shortcuts
You’ll notice that we created a closure for the query in the previous example. If you are doing multiple queries and want to keep the same fields definition, you can just use the closure like this and you’ll get the same shaped results for each query
Query and Field closure shortcuts
You can also add the query into the closure like this
You can combine a query closure with a regular search querystring like this. Query strings from both places are combined so will give the same result.
Fields in the search method
Finally you can do all of it in the search method, which may seem the most straightforward, but misses out on some of the reusability of the other approaches
We haven’t handled errors in the examples up to now. If SuperFetch encounters an error it’ll be in the error property of the response
- you can handle this yourself by checking the response error property.
- The plugin can automatically throw an error on your behalf if you can add the .throw() method to any request. All SuperFetch plugins have the same error handling approach. Think of it as a built in try/catch.
SuperFetch plugins always have a ref method – this allows you to create a new instance of the plugin which inherits all the characteristics of the source instance. You can pass any constructor options changes via .ref(). Any constructor options can be passed, but the most usual one would be to create a noCache instance.
In the examples so far we’ve had caching enabled, but we may want to do some queries avoiding cache.
As mentioned previously, you usually don’t need to bother handling paging, but you may want to. Twt will return up to ‘max’ items. A pageToken in the response indicates that there are more results available. If handling paging yourself, it’s best to use a noCache version of the Plugin as old pageTokens would not be valid, so they are not cached.
Provide both the pageToken, and optionally a different max using the page method to get the next set of results.
Using closures, this can be better written as
You’ve seen how the .page method can be used with the pageToken property to start a search at a particular point. You can use the max property in its own to control how items many you want. By default, max is set to 100 – I decided on a low amount because the Twitter API has a bunch of rate limits and caps, so it’s best to try to be cautious. I’ll deal with how Superfetch can help to deal with these rate limits in another article.
Here’s an individual query with no limit on the number of items
You can also create a twt instance that has no limit on searches using it.
Caching is built into SuperFetch (for details see SuperFetch – a proxy enhancement to Apps Script UrlFetch) so you get caching out of the box.
However, there are a couple of general things to be aware of
- Caching is particularily important with this API as currency is probably less important than with more transactional APIS, especially given Twitter Caps and Rate limits. Caching is also faster than hitting the API
- You’ll have seen there are a number of ways of constructing the same query. However you construct it, the caching normalization mechanism will notice that two queries trying to access the same data are the same as each other
- Searches that return more than 100 items (the twitter API maximum page size), will automatically make multiple API calls and consolidate the results to the maximum number of items you have set. If such a query is found in cache, it only has to make 1 cache transaction to get all of it. However if you handle paging yourself (using pageToken), it will have to hit the API directly and bypass cache.
- An uncached query clears the cached version of that search. The logic here is that if you are deliberately making an uncached call, it follows that any cached version will probably be stale
It’s possible (unlikely, but possibly for repeatable testing), that you’ll want to specifically clear any cached entries for a given query. To do this just repeat the query exactly, and replace the search method with deCache
Changing cache parameters
Cache parameters are set when you create the superFetch instance – here’s a typical one
There are 3 cache parameters available when you create the superFetch instance.
- cacheService – Usually this would be the userProperties service, but if you want to cache queries across all users of your script you could user scriptProperties
- prefix – This can be used to partition cache into groups – if one set of queries should somehow be completely separate from another set. The prefix can be any string you want and the cache entries will be shared with any superFetch instance sharing the same prefix
- expiry – this is the number of seconds to keep cache entries in place. The default is 1 hour after which they expire
Here’s an example of initializing a superFetch with different values
We can then get a ref of the existing twt instance, but with this new superFetch
Notice now that the cache entries for the new and old twt instances are now unaware of each other’s cache
Recent and All
- recent – tweets from the last 7 days
- all – all the twitter archive
The plugin supports both, but – from the API docs regarding the ‘all’ endpoint.
“This endpoint is only available to those users who have been approved for Academic Research access.
The full-archive search endpoint returns the complete history of public Tweets matching a search query; since the first Tweet was created March 26, 2006.”
This is the default, and you never need to specify it. For completeness it looks like this
Only Academic researchers have access to this, and most of us will get this error
Tweets can also be retrieved by a single or list of ids. The response format and fields are exactly the same as for searching, but this time instead of providing a search query you provide an array of ids. There is no paging for getting by id, and the maximum number that can be retrieved in one shot is the twitter API maximum – 100.
Getting a single basic tweet
Get a list of tweets by id
Get can have all the same field and query closures as search – for example
Get also supports compound queries, where the lists of ids are concatented
Get supports a list of ids, so the max returned is the number in the list, so there’s no paging support. If you need to get a list of more than 100, then split it and make several fetches. The 100 is the Twitter API limit – actually it’s probably driven by the maximum length of a url – it takes the ids as url parameters.
Works exactly the same way as for searches.
There is no search available in users – instead we get by id or by username. I the response, there will be one item for each id, plus some expansion data if you’ve requested it.
get – Getting by user ids
The Users domains for getting by Ids is exactly the same as for the Tweets domain, so I won’t repeat it all. Everywhere you see twt.tweets, replace it with twt.users.
Of course the fields and expansions available from the users domains are different from the tweets domain. See the API documentation for which fields you can use.
Here’s a complete example getting the pinned tweets and profile image url of a few users who have recently tweeted about Google apps script
Getting by user names
I hesitated in naming this method, as getByUsernames is a bit of a mouthful. In the end I decided to stick with the name of the Twitter API endpoint which is oddly, but simply, “by”
by – Getting by usernames
This works pretty much the same as get by id, except of course we replace all references to ‘ids’ by ‘usernames’. The usernames can be either a single username or an array or usernames, and is limited to 100 just as ids fetches.
Let’s simply redo the Get by user example, except find the individuals by username.
In fact there’s only 2 lines that need changed, and since we are using closures, the changes are very trivial
Getting your own account
Allows you to get your own user object
me – Getting your own user account
Although this will return only 1 item, the response is exactly the same as all other methods. ie. an array of of items (with just 1 member) plus an expansions property.
Goa is covered in detail elsewhere, and you’ll need to set that up before you can access the API – see Apps Script Oauth2- a Goa Library refresher
However, in this article I’ve used a couple of shortcuts for the token service. This is all the code you’ll need to implement and use goa as a twitter token service.
Most of the searching in this article applies to public data, so it’s possible that ‘app only’ oauth will suit you better if you don’t plan to do any user specific operations – it’s still enabled by Goa, but doesn’t need user consent. The detail is in OAuth2 and Twitter API – App only flow for Apps Script
I’ll use Simple but powerful Apps Script Unit Test library to demonstrate calls and responses. It should be straightforward to see how this works and the responsese to expect from calls. These tests demonstrate in detail each of the topics mentioned in this article, and a few others, and could serve as a useful crib sheet for the plugin
Warning – It’s a big read and many tests.
cGoa library 1v_l4xN3ICa0lAW315NQEzAHPSoNiFdWHsMEwj2qA5t9cgZ5VWci2Qxv2
Twitter API docs https://developer.twitter.com/en/docs/twitter-api
Twitter Developer profile https://developer.twitter.com/en/portal