Colorize sentiment in a Google document

Here’s a bit of fun with Google Language API, which uses ML to analyze the sentiment of text it receives. The idea here is to pass a Google Document over for analysis and colorize the sentences in a copy of the document using my bmChroma library. (Content oriented color mixing with Apps Script)

Page Content hide

1 Sentiment analysis example

3.1 Copy the source document

3.2 Do the sentiment analysis

3.3 Reconstruct the response

3.4 Create a text style

3.5 Make a Docs colors

3.6 A color scale

3.7 Do the reconstruction

3.8 Make the textStyles requests.

3.9 Update the document

3.10 The overall document score

4 Links

5 Next

Sentiment analysis example

Here’s a small colorized snippet from article from the New York times on Brexit.

colorized brexit docs sentiment new york times

Getting started

I use the REST API version of the Docs, Drive and Language APIs rather than any of the built in Apps Script services as there are few things that we can’t easily get at from the normal services.

Libraries

You’ll need a couple of my libraries.

bmApiCentral – 1L4pGblikbjQLQp8nQCdmCfyCxmF3MIShzsK8yy_mJ9_2YMdanXQA75vI github

bmChroma – 1zjBPTX8meADK6W2tbw-sNB0479OMN2hhT1O5MGna7v5liAj7paj-W8QE github

Cloud Project

You’ll also need to use a regular cloud project and replace the default project in your Apps Script settings. I won’t go into the details of all that here, as if you’re reading this, you probably already know how to do all that. That cloud project will need these APIS enabled

Google Drive
Google Docs
Cloud Natural Language API

Api keys

I also like to create and use API key and restrict it to these APIS, which you can do via the API credentials section of the Cloud Console. This is not manadatory, but it helps with traceability. Add this as a script property in your project settings.

Project manifest

Finally, update your manifest file with these scopes and whitelist

  "oauthScopes": [
    "https://www.googleapis.com/auth/cloud-language",
    "https://www.googleapis.com/auth/script.external_request",
    "https://www.googleapis.com/auth/documents",
    "https://www.googleapis.com/auth/drive"
  ],
  "urlFetchWhitelist": [
    "https://www.googleapis.com/drive/",
    "https://language.googleapis.com/v1beta2/documents:analyzeSentiment",
    "https://docs.googleapis.com/v1/documents"
  ]

manifest file changes

Exports

I recommend you use an Exports definition in all your projects to organize where all the code is coming from, and to spot call errors in your code. Here’s one that will work for this project. Just create a script called Exports and paste this in.

var Exports = {

  get libExports() {
    return bmApiCentral.Exports
  },

  get ColorWords() {
    return this.guard(bmChroma.Exports.ColorWords)
  },

  get Deps() {
    return this.libExports.Deps
  },


  newDocs(...args) {
    return this.libExports.newDocs(...args)
  },


  /**
   * Language instance with validation
   * @param {...*} args
   * @return {Language} a proxied instance of Language with property checking enabled
   */
  newLanguage(...args) {
    return this.libExports.newLanguage(...args)
  },


  /**
   * Drv instance with validation
   * @param {...*} args
   * @return {Drv} a proxied instance of Drv with property checking enabled
   */
  newDrv(...args) {
    return this.libExports.newDrv(...args)
  },

  /**
   * Utils namespace
   * @return {Utils} 
   */
  get Utils() {
    return this.libExports.libExports.Utils
  },

// used to trap access to unknown properties
  guard(target) {
    return new Proxy(target, this.validateProperties)
  },

  /**
   * for validating attempts to access non existent properties
   */
  get validateProperties() {
    return {
      get(target, prop, receiver) {
        // typeof and console use the inspect prop
        if (
          typeof prop !== 'symbol' &&
          prop !== 'inspect' &&
          !Reflect.has(target, prop)
        ) throw `guard detected attempt to get non-existent property ${prop}`

        return Reflect.get(target, prop, receiver)
      },

      set(target, prop, value, receiver) {
        if (!Reflect.has(target, prop)) throw `guard attempt to set non-existent property ${prop}`
        return Reflect.set(target, prop, value, receiver)
      }
    }
  }


}

Exports

The code

First step is to initialize all the APIs we’ll be using. The libraries are dependency free, so Deps.init will give them your local UrlFetch and token service to use. The apiOptions are common across all library classes.

  // initialize library with apps script depdendencies
  Exports.Deps.init({
    tokenService: ScriptApp.getOAuthToken,
    fetch: UrlFetchApp.fetch
  })

  // get the apiKey
  const apiKey = PropertiesService.getScriptProperties().getProperty("apiKey")

  // initialize the language, drv and docs api
  const language = Exports.newLanguage({
    apiKey
  })
  const drv = Exports.newDrv({
    apiKey
  })
  const docs = Exports.newDocs({
    apiKey
  })

  // default api options for all apis for this run
  const apiOptions = {
    noisy: true,
    throwOnError: true,
    noCache: false
  }

initialize APIS

Copy the source document

Next we’ll copy the source document, since we’ll be making formatting changes to the copy rather than the original. Subsitute the doc id with your own, or use this one – it’s public.

  // file to play with
  const originalDocId = "1lCg7AMwM5fE-LfQi2ZBlYnIzkQrsl_R8K2ivuFGWo1w"

  // make a copy of the original file, and get it back via the docs api
  const { data: copy } = drv.copy({ ...apiOptions, id: originalDocId })
  const { data: doc } = docs.get({ ...apiOptions, id: copy.id })
  const { body, title, documentId } = doc
  console.log('working on copy ', title, documentId, 'with', body.content.length, 'items')

Do the sentiment analysis

Get the content from the document and call the language API to do the sentiment analysis.

  // we only want to make 1 call to the language api - we can combine it all later
  // lets start by just sending the whole thing exactly as is 
  const elements = body.content.reduce((p, c) => {
    if (Reflect.has(c, "paragraph")) {
      c.paragraph.elements.forEach(e => p.push(e))
    }
    return p
  }, [])

  // this is what we'll send to be analyzed
  const joinedContent = elements.map(e => e.textRun.content).join("")

  // analyze that
  const result = language.analyzeSentiment({
    ...apiOptions,
    content: joinedContent
  })

do the analysis

Reconstruct the response

The data we get back won’t be exactly the same as the text we sent, partly because the language API does a bit of trimming, but also because the idea of a ‘sentence’ doesn’t map to any document element and may span many elements, or be just part of an element. We have to rely on progressively matching the text fragments to establish its original position in the document, using this function. This will create a series of ranges that describe the positions of the analyzed sentences.

  // rebuild the document elements with styling based on the sentence sentiments
  const reconstruct = ({ sectionOffset, joinedContent, sentences }) => {
    // how far along the original content we are
    let pointer = 0
    // look at each sentence returned - we should find it in the original 
    return sentences.map(sentence => {
      const { text } = sentence
      const { content } = text

      // only consider that which we havent already consumed
      const nextContent = joinedContent.substring(pointer)

      // find the sentence content
      // this'll be where the text value returned matches what was sent
      const nextIndex = nextContent.indexOf(content)

      // it should always exist
      if (nextIndex === -1) {
        throw `Couldnt find ${content} in text sent to api`
      }
      // adjust for where we started   how far in the text was found
      const startIndex = nextIndex   pointer

      // the end will be the  size of the content returned
      const endIndex = startIndex   content.length

      // format a section
      sentence.element = {
        range: {
          startIndex: startIndex   sectionOffset,
          endIndex: endIndex   sectionOffset
        }
      }

      // move to next chunk
      pointer = endIndex

      // make sure all the maths worked
      const check = joinedContent.substring(startIndex, endIndex)
      if (check !== content) {
        throw `${check.length} - ${content.length}`
      }
      return sentence
    })

  }

Reconstruct

Create a text style

We’ll use the Content oriented color mixing with Apps Script to manipulate a color and contrasting text color for each analyzed sentence.

  // generate a textstyle basedon the sentiment
  const getTextStyle = (color) => {
    const { getContrast, getChroma } = Exports.ColorWords
    const contrast = getChroma()(getContrast(color))
    const conRgb = contrast.rgb(false)
    const p = {
      backgroundColor: {
        color: docRgb(color.rgb(false))
      },
      foregroundColor: {
        color: docRgb(conRgb)
      }
    }
    return p
  }

get text style

Make a Docs colors

Colors in the Docs API are specified using rgb base with values between 0 and 1 as opposed to the normal 0-255, so we need a function to make those.

  /**
   * the docs api has a wierd way to represent color
   * @param {number[]} base the rgb array
   * @return {RgbColor} the docs representatino of an rgb color
  */
  const docRgb = (rgb) => {
    const [red, green, blue] = rgb
    return {
      rgbColor: {
        red: red / 255,
        green: green / 255,
        blue: blue / 255
      }
    }
  }

docRgb

A color scale

I’m going to use one of the ColorBrewer scales for this. ChromaJs knows all about them – you can of course use any scale that suits. The values the sentiment API will return are between -1 and 1. The returned value, scale, will be a function that will accept values between -1 and 1 and return a suitable color from the selected scale.

  //the sentiment scale

  const scale = Exports.ColorWords.getChroma().scale('RdYlGn').domain([-1, 1]);

get a scale function

Do the reconstruction

Now we’re ready to reconstruct the api response, which will attach a document range to each sentence

  const sentences = reconstruct({
    // only handling 1 section in this test, but it may consumes some index points
    sectionOffset: elements[0] && elements[0].startIndex,
    joinedContent,
    sentences: result.sentences
  })

reconstruct

Make the textStyles requests.

Now we can use the scale to calculate an appropriate background color and highlight the places in the document that generated them. The requests will be batched up and sent to the Docs API in one post.

  // generate the textSryle requests
  const textStyleRequests = sentences.map(sentence => {
    const { element } = sentence
    const { magnitude, score } = sentence.sentiment
    const { range } = element
    // skip any ranges that are no length 
    return range.endIndex - range.startIndex > 1 && Math.abs(score) > 0.1 ? {
      updateTextStyle: {
        textStyle: getTextStyle(scale(score)),
        fields: "backgroundColor,foregroundColor",
        range
      }
    } : null
  }).filter(f => f)

make text style requests

Update the document

Apply the formatting requests to the document.

  // update the document
  const batchResult = docs.batch({
    ...apiOptions,
    requests: textStyleRequests,
    id: documentId
  })

do the batchUpdate

The overall document score

This is the overall sentiment for the entire document. There’s a good explanation on how to interpret these (and the inidivual sentence scores) here.

console.log ("overall document sentiment", result.documentSentiment)

doc score

Links

bmApiCentral – 1L4pGblikbjQLQp8nQCdmCfyCxmF3MIShzsK8yy_mJ9_2YMdanXQA75vI

bmChroma – 1zjBPTX8meADK6W2tbw-sNB0479OMN2hhT1O5MGna7v5liAj7paj-W8QE

This is just some reference code you might use to build on. It might make an interesting Add-on, or could be extended to apply to Mail Drafts before sending them and so on.