In Every Google Apps Script project on Github visualized  I demonstrated an app that could be used to explore what every Apps Script developer who has shared their code is working on. One of the features is to be able to see which libraries are being used, find out who owns them, examine the code and clone entire projects directly into apps script from Github.

Finding Apps Script projects

It’s easy enough to find Apps Script projects on github. They have a manifest file – appsscript.json that contains various settings, including their dependencies. From that you can visualize a project and everything related to it.

For example – here are my public Apps Script projects on Github

bruce scrviz projects

Picking up a little more on one of those projects, you can see it uses 3 libraries – bmPreFiddler, cGoa and bmVizzyCache.

vizzycache project

Looking at another view, by library, we can see where else one of those libraries is being used

That’s all fine, because the code for cGoa is also on github and scrviz has been able to relate the repos together. However the project manifest for bmVizzyCacheLibrarySheet lists cGoa as a dependency by its scriptId, but the cGoa repo doesn’t necessarily know its own scriptId

Why is scriptId unkown?

It’s a good question – it wouldn’t be difficult for the appsscript manifest to contain its own id but Google haven’t done that, so we have to use a few hacks to get some clue as to precisely which script project a given github project belongs.

The scrviz components

The scrviz ecosystem has these components

I will also release a DataStudio community connector shortly to directly access and report on the catalog. Watch this space, or contact me for more details.

Some challenges

Associating scriptIds with Githib repos is not an easy problem to solve. Here are some of the challenges

  • Some repos contain multiple projects, each of which has its own appsscript.json
  • Some libraries are not public and/or don’t have their code on github
  • The structure of repos varies – some developers add a ./src or ./dist folder, and others dont
  • Using the Script API to narrow down the owner doesn’t help as we’d need to add oauth to the scraping process, and the scripts may not be public anyway. Furthermore, not everyone shows their email address on their github profile, and when they do, it may not be the one they use for scripting.

Some approaches

All this would be solved if the appscript.json file contained the id of the script it belonged to – but it doesn’t, so we need some workarounds.

  • The .clasp.json file that’s pushed to github along with the clasp project contains the script id it created. Scrgit reads the .clasp file, if there is one, to find out the project id. However, only 20+% of Apps Script projects on Github are created with clasp.
  • If you use my gasgit (Getting your apps scripts to Github) project to push your code to github, it already manages all the dependencies for your project (it predates the appsscript.json manifest) and creates a file info.json which references them all, and also contains the project id. Scrgit reads the info.json file, if there is one, to find out the project id.
  • Even if you don’t use gasit, you can still advertise your script id by creating a file called info.json and adding it to your project directory containing {“id”: “your script id” }
  • The only remaining way to identify the scriptId is if the project happens to be a referenced library.  Because scrgit creates a list of all dependencies of all projects by their id, comparing the usersymbol assigned to libraries with repo (or folder when there are multiple projects in a repo) name gives a clue which scriptid to assign to a project that happens to be a library. There is of course plenty of scope for mismatch here because of duplication or variations of library names/user symbols etc but it’s an other approach if the other two fail.

In action

Here’s some messages we see when getting scriptId clues while running using these techniques to assign github repos (and folders within repos where multiple projects are in the same repo)

Finding id clues in .clasp.json

…found scriptId clue file geekyme adr-logger .clasp.json
…found scriptId clue file SocialEconomyDataLab spec templates/scripts/.clasp.json
…found scriptId clue file watchdogpolska wd-gdocs-addons addon/.clasp.json
…found scriptId clue file gregoryfryns similarweb-gds-connector src/.clasp.json
…found scriptId clue file linhnguyen-pnl linhnguyen-pnl clasp/.clasp.json
…found scriptId clue file sfwatanabe gas-clasp-example cats/.clasp.json
…found scriptId clue file ropeck calendar-color .clasp.json
…found scriptId clue file shuhankuang Track-Multiple-Package .clasp.json

Finding id clues in info.json

From Gasgit or manually created info.json repo files

…found scriptId clue file brucemcpherson bmAlasql info.json
…found scriptId clue file brucemcpherson bmMathjs info.json
…found scriptId clue file brucemcpherson execTester info.json
…found scriptId clue file brucemcpherson cjsDiff info.json
…found scriptId clue file brucemcpherson cEs6Shim info.json
…found scriptId clue file brucemcpherson cEffexApiClient info.json
…found scriptId clue file brucemcpherson cLucky info.json
…found scriptId clue file brucemcpherson cCoupon info.json
…found scriptId clue file brucemcpherson d3Gas info.json
…found scriptId clue file brucemcpherson GasWatcher info.json
…found scriptId clue file brucemcpherson cGoingGas info.json
…found scriptId clue file brucemcpherson cUAMeasure info.json
…found scriptId clue file brucemcpherson cSheetExec info.json
…found scriptId clue file brucemcpherson bmSwopCxProxy info.json
…found scriptId clue file brucemcpherson bmFixer info.json
…found scriptId clue file brucemcpherson bmSwopCx info.json
…found scriptId clue file brucemcpherson bmFolderFun info.json
…found scriptId clue file brucemcpherson bmTemporal info.json
…found scriptId clue file brucemcpherson bmRandomThings info.json
…found scriptId clue file brucemcpherson bmFaker info.json
…found scriptId clue file brucemcpherson bmLibraryTracking info.json
…found scriptId clue file brucemcpherson bmUpstash info.json

Deducing scriptId from library dependency references

Unambigous usersymbol references matched repo (or folder within repo) names

…found scriptId clue library cdriversheet cDriverSheet 18fvqHqSs2YwU2ZMUcx6-9GE30u6i663rZTz7K0xNsStHoiJcs487JvN- brucemcpherson generateLogs libraries/cDriverSheet/appsscript.json
…found scriptId clue library classroomdocs gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom-docs/appsscript.json
…found scriptId clue library appsscripttestjson gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom/appsscript.test.json
…found scriptId clue library appsscripttestjson gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom-docs/appsscript.test.json
…found scriptId clue library classroom gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom/appsscript.json
…found scriptId clue library DocsServiceApp 108j6x_ZX544wEhGkgddFYM6Ie09edDqXaFwnW3RVFQCLHw_mEueqUHTW tanaikech DocsServiceApp appsscript.json
…found scriptId clue library cdbabstraction cDbAbstraction 1Ddsb4Y-QDUqcw9Fa-rJKM3EhG2caosS9Nhch7vnQWXP7qkaMmb1wjmTl brucemcpherson gasThreader libraries/cDbAbstraction/appsscript.json
…found scriptId clue library cdbabstraction cDbAbstraction 1Ddsb4Y-QDUqcw9Fa-rJKM3EhG2caosS9Nhch7vnQWXP7qkaMmb1wjmTl brucemcpherson migrateParseToMongo libraries/cDbAbstraction/appsscript.json

Rejected if ambiguous references 

This would happen if multiple repos shared a usersymbol reference, but their dependencies pointed to different scripts

…rejected ambiguous scriptId clue library (6) matches [
‘BkperApp:1fSZnepYcDUjxCsrWYD3452UJ5nJiB4js0cD45WWOAjMcKJR_PKfLU60X’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’
] bkper bkper-app src/appsscript.json

Afterwards

Now many more repos know their own scriptId, so they can be opened in the IDE directly from scrviz. Here’s an example

 

How many did we find.

Prior to this hack, we could find 799 out of 3471 projects (23%) – essentially these were the clasp projects only. Now we have found 1348 out of 4000 (34%)

 

Summary and links

35% is still not great, but it does help a bit, and you can all help by creating an info.json with the id in it (or by using gasgit). A problem easily solved if only the manifest contained the id of the project it refers to. Maybe one day it will.

More about this topic

All about scrviz