In Every Google Apps Script project on Github visualized I demonstrated an app that could be used to explore what every Apps Script developer who has shared their code is working on. One of the features is to be able to see which libraries are being used, find out who owns them, examine the code and clone entire projects directly into apps script from Github.
Finding Apps Script projects
It’s easy enough to find Apps Script projects on github. They have a manifest file – appsscript.json that contains various settings, including their dependencies. From that you can visualize a project and everything related to it.
For example – here are my public Apps Script projects on Github
Picking up a little more on one of those projects, you can see it uses 3 libraries – bmPreFiddler, cGoa and bmVizzyCache.
Looking at another view, by library, we can see where else one of those libraries is being used
That’s all fine, because the code for cGoa is also on github and scrviz has been able to relate the repos together. However the project manifest for bmVizzyCacheLibrarySheet lists cGoa as a dependency by its scriptId, but the cGoa repo doesn’t necessarily know its own scriptId
Why is scriptId unkown?
It’s a good question – it wouldn’t be difficult for the appsscript manifest to contain its own id but Google haven’t done that, so we have to use a few hacks to get some clue as to precisely which script project a given github project belongs.
The scrviz components
The scrviz ecosystem has these components
- Scrgit – this is a node app that from time to time, uses the github API to search for and catalog all apps script projects on github. It compresses and writes a catalog to a public gist file for consumption by any clients
- Scrviz – is a web app that consumes this catalog and creates a visualization of relationships between projects, dependencies, repos and owners. It also allow you to create and clone apps script project straight into IDE and to enrich your own or your project’s profile so others can find them
- bmVizzyCache – is an Apps script library that you can use to directly access the compressed github catalog if you want to build something using the data. Here’s its direct scrviz link. , and a link to an example project that uses it.
I will also release a DataStudio community connector shortly to directly access and report on the catalog. Watch this space, or contact me for more details.
Some challenges
Associating scriptIds with Githib repos is not an easy problem to solve. Here are some of the challenges
- Some repos contain multiple projects, each of which has its own appsscript.json
- Some libraries are not public and/or don’t have their code on github
- The structure of repos varies – some developers add a ./src or ./dist folder, and others dont
- Using the Script API to narrow down the owner doesn’t help as we’d need to add oauth to the scraping process, and the scripts may not be public anyway. Furthermore, not everyone shows their email address on their github profile, and when they do, it may not be the one they use for scripting.
Some approaches
All this would be solved if the appscript.json file contained the id of the script it belonged to – but it doesn’t, so we need some workarounds.
- The .clasp.json file that’s pushed to github along with the clasp project contains the script id it created. Scrgit reads the .clasp file, if there is one, to find out the project id. However, only 20+% of Apps Script projects on Github are created with clasp.
- If you use my gasgit (Getting your apps scripts to Github) project to push your code to github, it already manages all the dependencies for your project (it predates the appsscript.json manifest) and creates a file info.json which references them all, and also contains the project id. Scrgit reads the info.json file, if there is one, to find out the project id.
- Even if you don’t use gasit, you can still advertise your script id by creating a file called info.json and adding it to your project directory containing {“id”: “your script id” }
- The only remaining way to identify the scriptId is if the project happens to be a referenced library. Because scrgit creates a list of all dependencies of all projects by their id, comparing the usersymbol assigned to libraries with repo (or folder when there are multiple projects in a repo) name gives a clue which scriptid to assign to a project that happens to be a library. There is of course plenty of scope for mismatch here because of duplication or variations of library names/user symbols etc but it’s an other approach if the other two fail.
In action
Here’s some messages we see when getting scriptId clues while running using these techniques to assign github repos (and folders within repos where multiple projects are in the same repo)
Finding id clues in .clasp.json
…found scriptId clue file geekyme adr-logger .clasp.json
…found scriptId clue file SocialEconomyDataLab spec templates/scripts/.clasp.json
…found scriptId clue file watchdogpolska wd-gdocs-addons addon/.clasp.json
…found scriptId clue file gregoryfryns similarweb-gds-connector src/.clasp.json
…found scriptId clue file linhnguyen-pnl linhnguyen-pnl clasp/.clasp.json
…found scriptId clue file sfwatanabe gas-clasp-example cats/.clasp.json
…found scriptId clue file ropeck calendar-color .clasp.json
…found scriptId clue file shuhankuang Track-Multiple-Package .clasp.json
Finding id clues in info.json
From Gasgit or manually created info.json repo files
…found scriptId clue file brucemcpherson bmAlasql info.json
…found scriptId clue file brucemcpherson bmMathjs info.json
…found scriptId clue file brucemcpherson execTester info.json
…found scriptId clue file brucemcpherson cjsDiff info.json
…found scriptId clue file brucemcpherson cEs6Shim info.json
…found scriptId clue file brucemcpherson cEffexApiClient info.json
…found scriptId clue file brucemcpherson cLucky info.json
…found scriptId clue file brucemcpherson cCoupon info.json
…found scriptId clue file brucemcpherson d3Gas info.json
…found scriptId clue file brucemcpherson GasWatcher info.json
…found scriptId clue file brucemcpherson cGoingGas info.json
…found scriptId clue file brucemcpherson cUAMeasure info.json
…found scriptId clue file brucemcpherson cSheetExec info.json
…found scriptId clue file brucemcpherson bmSwopCxProxy info.json
…found scriptId clue file brucemcpherson bmFixer info.json
…found scriptId clue file brucemcpherson bmSwopCx info.json
…found scriptId clue file brucemcpherson bmFolderFun info.json
…found scriptId clue file brucemcpherson bmTemporal info.json
…found scriptId clue file brucemcpherson bmRandomThings info.json
…found scriptId clue file brucemcpherson bmFaker info.json
…found scriptId clue file brucemcpherson bmLibraryTracking info.json
…found scriptId clue file brucemcpherson bmUpstash info.json
Deducing scriptId from library dependency references
Unambigous usersymbol references matched repo (or folder within repo) names
…found scriptId clue library cdriversheet cDriverSheet 18fvqHqSs2YwU2ZMUcx6-9GE30u6i663rZTz7K0xNsStHoiJcs487JvN- brucemcpherson generateLogs libraries/cDriverSheet/appsscript.json
…found scriptId clue library classroomdocs gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom-docs/appsscript.json
…found scriptId clue library appsscripttestjson gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom/appsscript.test.json
…found scriptId clue library appsscripttestjson gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom-docs/appsscript.test.json
…found scriptId clue library classroom gsetools 1SRl5DGs_tVG42joBNHBFhPvivWtX07XRmIo533wfxzXgkM4ahPT8E5JA teachduttonteach gse-tools src/classroom/appsscript.json
…found scriptId clue library DocsServiceApp 108j6x_ZX544wEhGkgddFYM6Ie09edDqXaFwnW3RVFQCLHw_mEueqUHTW tanaikech DocsServiceApp appsscript.json
…found scriptId clue library cdbabstraction cDbAbstraction 1Ddsb4Y-QDUqcw9Fa-rJKM3EhG2caosS9Nhch7vnQWXP7qkaMmb1wjmTl brucemcpherson gasThreader libraries/cDbAbstraction/appsscript.json
…found scriptId clue library cdbabstraction cDbAbstraction 1Ddsb4Y-QDUqcw9Fa-rJKM3EhG2caosS9Nhch7vnQWXP7qkaMmb1wjmTl brucemcpherson migrateParseToMongo libraries/cDbAbstraction/appsscript.json
Rejected if ambiguous references
This would happen if multiple repos shared a usersymbol reference, but their dependencies pointed to different scripts
…rejected ambiguous scriptId clue library (6) matches [
‘BkperApp:1fSZnepYcDUjxCsrWYD3452UJ5nJiB4js0cD45WWOAjMcKJR_PKfLU60X’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’,
‘BkperApp:1hMJszJGSUVZDB3vmsWrUZfRhY1UWbhS0SQ6Lzl06gm1zhBF3ioTM7mpJ’
] bkper bkper-app src/appsscript.json
Afterwards
Now many more repos know their own scriptId, so they can be opened in the IDE directly from scrviz. Here’s an example
How many did we find.
Prior to this hack, we could find 799 out of 3471 projects (23%) – essentially these were the clasp projects only. Now we have found 1348 out of 4000 (34%)
Summary and links
35% is still not great, but it does help a bit, and you can all help by creating an info.json with the id in it (or by using gasgit). A problem easily solved if only the manifest contained the id of the project it refers to. Maybe one day it will.
- This complete app on github
- The client visualizer on github
- Try out the visualization
- An article about the visualizer
More about this topic
- 4 ways to customize scrviz info card behavior
- Apps Script Vizzy – new views, searches and filtering examples
- Direct scrviz link to your Apps Script github repo, project or owner
- Enrich your developer profile on scrviz
- Enrich your scrviz repo profiles and drive traffic to your website or blog
- Find an Apps Script library id in 10 seconds with scrviz
- Find and clone Apps Script projects from github directly into the IDE with scrviz
- Google gapi, Vue and Vuex together
- More scrviz updates to help you enrich your profile and repos
- Searching and cataloging Apps Script projects on Github
- Showcase your Apps Script work and get hired via vizzy
- Vizzy github data open access – how to build something with Apps Script
All about scrviz