I am supporting CandidateX

CandidateX is a startup that focuses on creating inclusion-focused hiring solutions, designed to increase access to job opportunities for underestimated talent. Check them out if you have a few minutes to spare. They need visibility!

Here’s a bit of fun with Google Language API, which uses ML to analyze the sentiment of text it receives. The idea here is to pass a Google Document over for analysis and colorize the sentences in a copy of the document using my bmChroma library. (Content oriented color mixing with Apps Script)

Sentiment analysis example

Here’s a small colorized snippet from article from the New York times on Brexit.

colorized brexit docs sentiment new york times

Getting started

I use the REST API version of the Docs, Drive and Language APIs rather than any of the built in Apps Script services as there are few things that we can’t easily get at from the normal services.

Libraries

You’ll need a couple of my libraries.

bmApiCentral – 1L4pGblikbjQLQp8nQCdmCfyCxmF3MIShzsK8yy_mJ9_2YMdanXQA75vI github

bmChroma – 1zjBPTX8meADK6W2tbw-sNB0479OMN2hhT1O5MGna7v5liAj7paj-W8QE github

Cloud Project

You’ll also need to use a regular cloud project and replace the default project in your Apps Script settings. I won’t go into the details of all that here, as if you’re reading this, you probably already know how to do all that. That cloud project will need these APIS enabled

  • Google Drive
  • Google Docs
  • Cloud Natural Language API

Api keys

I also like to create and use API key and restrict it to these APIS, which you can do via the API credentials section of the Cloud Console. This is not manadatory, but it helps with traceability. Add this as a script property in your project settings.

Project manifest

Finally, update your manifest file with these scopes and whitelist

  "oauthScopes": [
"https://www.googleapis.com/auth/cloud-language",
"https://www.googleapis.com/auth/script.external_request",
"https://www.googleapis.com/auth/documents",
"https://www.googleapis.com/auth/drive"
],
"urlFetchWhitelist": [
"https://www.googleapis.com/drive/",
"https://language.googleapis.com/v1beta2/documents:analyzeSentiment",
"https://docs.googleapis.com/v1/documents"
]
manifest file changes

Exports

I recommend you use an Exports definition in all your projects to organize where all the code is coming from, and to spot call errors in your code. Here’s one that will work for this project. Just create a script called Exports and paste this in.

var Exports = {

get libExports() {
return bmApiCentral.Exports
},

get ColorWords() {
return this.guard(bmChroma.Exports.ColorWords)
},

get Deps() {
return this.libExports.Deps
},


newDocs(...args) {
return this.libExports.newDocs(...args)
},


/**
* Language instance with validation
* @param {...*} args
* @return {Language} a proxied instance of Language with property checking enabled
*/
newLanguage(...args) {
return this.libExports.newLanguage(...args)
},


/**
* Drv instance with validation
* @param {...*} args
* @return {Drv} a proxied instance of Drv with property checking enabled
*/
newDrv(...args) {
return this.libExports.newDrv(...args)
},

/**
* Utils namespace
* @return {Utils}
*/
get Utils() {
return this.libExports.libExports.Utils
},

// used to trap access to unknown properties
guard(target) {
return new Proxy(target, this.validateProperties)
},

/**
* for validating attempts to access non existent properties
*/
get validateProperties() {
return {
get(target, prop, receiver) {
// typeof and console use the inspect prop
if (
typeof prop !== 'symbol' &&
prop !== 'inspect' &&
!Reflect.has(target, prop)
) throw `guard detected attempt to get non-existent property ${prop}`

return Reflect.get(target, prop, receiver)
},

set(target, prop, value, receiver) {
if (!Reflect.has(target, prop)) throw `guard attempt to set non-existent property ${prop}`
return Reflect.set(target, prop, value, receiver)
}
}
}


}
Exports

The code

First step is to initialize all the APIs we’ll be using. The libraries are dependency free, so Deps.init will give them your local UrlFetch and token service to use. The apiOptions are common across all library classes.

  // initialize library with apps script depdendencies
Exports.Deps.init({
tokenService: ScriptApp.getOAuthToken,
fetch: UrlFetchApp.fetch
})

// get the apiKey
const apiKey = PropertiesService.getScriptProperties().getProperty("apiKey")

// initialize the language, drv and docs api
const language = Exports.newLanguage({
apiKey
})
const drv = Exports.newDrv({
apiKey
})
const docs = Exports.newDocs({
apiKey
})

// default api options for all apis for this run
const apiOptions = {
noisy: true,
throwOnError: true,
noCache: false
}
initialize APIS

Copy the source document

Next we’ll copy the source document, since we’ll be making formatting changes to the copy rather than the original. Subsitute the doc id with your own, or use this one – it’s public.

  // file to play with
const originalDocId = "1lCg7AMwM5fE-LfQi2ZBlYnIzkQrsl_R8K2ivuFGWo1w"

// make a copy of the original file, and get it back via the docs api
const { data: copy } = drv.copy({ ...apiOptions, id: originalDocId })
const { data: doc } = docs.get({ ...apiOptions, id: copy.id })
const { body, title, documentId } = doc
console.log('working on copy ', title, documentId, 'with', body.content.length, 'items')

Do the sentiment analysis

Get the content from the document and call the language API to do the sentiment analysis.

  // we only want to make 1 call to the language api - we can combine it all later
// lets start by just sending the whole thing exactly as is
const elements = body.content.reduce((p, c) => {
if (Reflect.has(c, "paragraph")) {
c.paragraph.elements.forEach(e => p.push(e))
}
return p
}, [])

// this is what we'll send to be analyzed
const joinedContent = elements.map(e => e.textRun.content).join("")

// analyze that
const result = language.analyzeSentiment({
...apiOptions,
content: joinedContent
})
do the analysis

Reconstruct the response

The data we get back won’t be exactly the same as the text we sent, partly because the language API does a bit of trimming, but also because the idea of a ‘sentence’ doesn’t map to any document element and may span many elements, or be just part of an element. We have to rely on progressively matching the text fragments to establish its original position in the document, using this function. This will create a series of ranges that describe the positions of the analyzed sentences.

  // rebuild the document elements with styling based on the sentence sentiments
const reconstruct = ({ sectionOffset, joinedContent, sentences }) => {
// how far along the original content we are
let pointer = 0
// look at each sentence returned - we should find it in the original
return sentences.map(sentence => {
const { text } = sentence
const { content } = text

// only consider that which we havent already consumed
const nextContent = joinedContent.substring(pointer)

// find the sentence content
// this'll be where the text value returned matches what was sent
const nextIndex = nextContent.indexOf(content)

// it should always exist
if (nextIndex === -1) {
throw `Couldnt find ${content} in text sent to api`
}
// adjust for where we started how far in the text was found
const startIndex = nextIndex pointer

// the end will be the size of the content returned
const endIndex = startIndex content.length

// format a section
sentence.element = {
range: {
startIndex: startIndex sectionOffset,
endIndex: endIndex sectionOffset
}
}

// move to next chunk
pointer = endIndex

// make sure all the maths worked
const check = joinedContent.substring(startIndex, endIndex)
if (check !== content) {
throw `${check.length} - ${content.length}`
}
return sentence
})

}
Reconstruct

Create a text style

We’ll use the Content oriented color mixing with Apps Script to manipulate a color and contrasting text color for each analyzed sentence.

  // generate a textstyle basedon the sentiment
const getTextStyle = (color) => {
const { getContrast, getChroma } = Exports.ColorWords
const contrast = getChroma()(getContrast(color))
const conRgb = contrast.rgb(false)
const p = {
backgroundColor: {
color: docRgb(color.rgb(false))
},
foregroundColor: {
color: docRgb(conRgb)
}
}
return p
}
get text style

Make a Docs colors

Colors in the Docs API are specified using rgb base with values between 0 and 1 as opposed to the normal 0-255, so we need a function to make those.

  /**
* the docs api has a wierd way to represent color
* @param {number[]} base the rgb array
* @return {RgbColor} the docs representatino of an rgb color
*/
const docRgb = (rgb) => {
const [red, green, blue] = rgb
return {
rgbColor: {
red: red / 255,
green: green / 255,
blue: blue / 255
}
}
}
docRgb

A color scale

I’m going to use one of the ColorBrewer scales for this. ChromaJs knows all about them – you can of course use any scale that suits. The values the sentiment API will return are between -1 and 1. The returned value, scale, will be a function that will accept values between -1 and 1 and return a suitable color from the selected scale.

  //the sentiment scale

const scale = Exports.ColorWords.getChroma().scale('RdYlGn').domain([-1, 1]);
get a scale function

Do the reconstruction

Now we’re ready to reconstruct the api response, which will attach a document range to each sentence

  const sentences = reconstruct({
// only handling 1 section in this test, but it may consumes some index points
sectionOffset: elements[0] && elements[0].startIndex,
joinedContent,
sentences: result.sentences
})
reconstruct

Make the textStyles requests.

Now we can use the scale to calculate an appropriate background color and highlight the places in the document that generated them. The requests will be batched up and sent to the Docs API in one post.

  // generate the textSryle requests
const textStyleRequests = sentences.map(sentence => {
const { element } = sentence
const { magnitude, score } = sentence.sentiment
const { range } = element
// skip any ranges that are no length
return range.endIndex - range.startIndex > 1 && Math.abs(score) > 0.1 ? {
updateTextStyle: {
textStyle: getTextStyle(scale(score)),
fields: "backgroundColor,foregroundColor",
range
}
} : null
}).filter(f => f)
make text style requests

Update the document

Apply the formatting requests to the document.

  // update the document
const batchResult = docs.batch({
...apiOptions,
requests: textStyleRequests,
id: documentId
})
do the batchUpdate

The overall document score

This is the overall sentiment for the entire document. There’s a good explanation on how to interpret these (and the inidivual sentence scores) here.

console.log ("overall document sentiment", result.documentSentiment)
doc score

Links

bmApiCentral – 1L4pGblikbjQLQp8nQCdmCfyCxmF3MIShzsK8yy_mJ9_2YMdanXQA75vI

bmChroma – 1zjBPTX8meADK6W2tbw-sNB0479OMN2hhT1O5MGna7v5liAj7paj-W8QE

Next

This is just some reference code you might use to build on. It might make an interesting Add-on, or could be extended to apply to Mail Drafts before sending them and so on.