Datastore driver


This is the the driver for DB.DATASTORE described in Database abstraction with google apps script 

The library reference is MPZF_EC6nOZFAjMRqCxEaUyz3TLx7pV4j
 
This tutorial references the the cDataHandler and cEzyOauth2, both of which have been superseded by cDbAbstraction and cGoa which are easier to implement. If you'd rather work with those, the slides below show an example.

do something in 5 with gas-9-copy between databases with oauth2


 

Background to implementation.

This driver has been tricky to implement. The Cloud Datastore calls itself  "Google Cloud Datastore: A Fully Managed NoSQL Data Storage Service" and is normally used with Google App Engine using Python, Java, PHP or Go. There is no API for Google Apps Script, it is not normally used directly from a JavaScript client, and neither is it a database as 'we would know it'.

There is a JSON API, and along with the new concepts of entities , paths, ancestors and tricky indexing it's quite difficult to get started with. It does have a GQL query language which is SQL-ish, but you can only use it for querying (not updating). It is also only really able to handle objects that are flat, and complicated queries need to have indexes pre-setup - in other words you need to know what you'l ask before you ask. 

Since this is a database abstraction driver, I don't need the full capabilities of DataStore since I only need those that are present in other databases in the sub-universe. I also need to use  Flattening an object with dot syntax to allow for multi level object storage and querying, and because of the limitation around pre-indexing, I need to do hybrid querying with some of it happening in datastore, and more complex constraints happening in the DataHandler. 

Here's a quick primer.

Quotas

Before you choose to use DataStore, you'll need to think about quotas. It's very easy to blow the free daily allowance - for any meaningful application you'll probably end up paying something. There is a quota dashboard on the Google Developers Cloud console.

The data abstraction library has caching enabled from the start, so any data taken from cache doesn't eat into your quota.

Authentication

If you've been avoiding oAuth2, you can't. DataStore demands authentication. The DataHandler can accept and access token, and you can easily use EzyOauth2 - taking some pain out of Apps Script API authentication to get one for both web app and regular applications. More about that later.

Locking

Although Datastore has the concept of 'Transactions', you can't protect sections of code like you can when you use it with App Engine. I'm Using named locks with Google Apps Scripts to protect sensitive transactions.


Setup

There's a little bit of setting up to do
  • Create an application in the Google Cloud Console
  • Enable datastore
  • Get some oAuth2 credentials
  • Create your app from EzyOauth2 patterns
  • Run a web app to allow access and store your access/refresh token environment.


Here's some libraries you'll need or are used internally

library

key

comments

cDataHandler

Mj61W-201_t_zC9fJg1IzYiz3TLx7pV4j

Abstracted interface to back end databases, and all known drivers

cCacheHandler

M3reA5eBxtwxSqCEgPywb9ai_d-phDA33

Manages caching of query results

cNamedLock

Mpv7vUR0126U53sfSMXsAPai_d-phDA33

Cross script locking of abstract resources

cEzyOauth2

MSaYlTXSVk7FAqpHNCcqBv6i_d-phDA33

Manages oAuth2 credentials and offline refreshing of access tokens

cFlatten

MqxKdBrlw18FDd-X5zQLd7yz3TLx7pV4j

Flattens complex objects to 1 level dot syntax objects  so they can be stored/queries in a 2 dimensional space

ezyOauth2  templates

https://script.google.com/d/1ll5nvMOZL5YODaj71l0-XSaD0BBciG_zIV2I0Neu3Nz1LaKY6-4WiJAt/edit?usp=sharing

Patterns for web and nonweb apps for easy oath2 authentication. Take a copy of this

How it all fits together


Oauth2 pattern

As described in EzyOauth2 patterns, apps that require oAuth2 can be complicated. However, once you've copied the template, the pattern is simple and repeatable. Here are the steps
  • First time in - store your credentials and scope. You do this only once. You can delete this after running it once, so you don't have to carry credentials in your code.
  setAuthenticationPackage_ ({ 
    clientId : "xxx.apps.googleusercontent.com",
    clientSecret : "xxx",
    scopes : ['https://www.googleapis.com/auth/datastore','https://www.googleapis.com/auth/userinfo.email'],
    service: 'google',
    packageName: 'googleDatastore'
  });
  • First time in, provoke a doGet() dialog by publishing as a web app to create an access and refresh token. This will grab an offline authorization which will refresh automatically when needed. These will be used for both web apps and non-webapps. However you do need to publish and run once to create the dialog. If this is a non-web app, then unpublish it again when done. Your doGet() function looks like this. If this is a web actually a webapp, then the doSomething() function is where you create your app.
/** 
 * this is your web app
 * @param {object} webapp param object
 * return {HtmlOutput} 
 */

function doGet (e) {
  return doGetPattern(e, constructConsentScreen, doSomething, 'googleDatastore') ;
}

/**
 * tailor your consent screen with an html template
 * @param {string} consentUrl the url to click to provide user consent
 * @return {string} the html for the consent screen
 */
function constructConsentScreen (consentUrl) {
  return '<a href = "' + consentUrl + '">Authenticate to datastore</a> ';
}

/**
 * this is your main processing - will be called with your access token
 * @param {string} accessToken - the accessToken
 */
function doSomething (accessToken) {
 
   var options = {
     method: "POST",
     headers: {
       authorization: "Bearer " + accessToken
     }
   };

  var result = UrlFetchApp.fetch("https://www.googleapis.com/datastore/v1beta2/datasets/xliberationdatastore/lookup", options);
  return HtmlService.createHtmlOutput (' it worked' + result.getContentText());

}

/**
 * gets the property key against which you want the authentication package stored
 * @param {string} optPackageName
 * @return {string}
 */
function getPropertyKey_ (optPackageName) {
  return "EzyOauth2Datastore" + (optPackageName ? '_' + optPackageName : '');
}

If this is not a webapp, then you need something that looks like this too, where dataStoreTest is your application that is going to d the work.

function dataStoreExample() {
  // this will get an access token and pass it to doTheWork()
  return doGetPattern({} , constructConsentScreen, dataStoreTest,'googleDatastore');
}

More info

The datastore code

If you are interested in the implementation, you'll find the code here

The datastore test.

If you've already used Database abstraction with google apps script then this will be familiar. The database access is the same for datastore as it is for any other backend. The only specific coding is setting up the handler. Here are 3 different setups for 3  different backends - datastore, parse.com and Drive. You'll notice they are very similar. In this i've opted out of analytics tracking, and turned off caching- since I'm going to run the same tests on each platform to make sure I get the same result. You'll find these tests in the Patterns template and you may find them useful to check out your set up.

function dataStoreTest (accessToken) {
   
   // get a datastore handler
   var handler = new cDataHandler.DataHandler (
     'flattener',                              // Kind
     cDataHandler.dhConstants.DB.DATASTORE,    // Datastore
     undefined, 
     'xliberationdatastore',                   // project id
     undefined, 
     undefined,
     true,                                     // analytics opt out
     'bruce',                                  // analytics debugging tracking 
     accessToken,                              // the access token
     true);                                    // disable caching for testing  
     

  assert(handler.isHappy(), 'unable to get datastore handler','handler');
  return testFlattener (handler, accessToken)
}

function driveTest (accessToken) {
   
   // get a datastore handler
   var handler = new cDataHandler.DataHandler (
     'flattener.json',                              // Kind
     cDataHandler.dhConstants.DB.DRIVE,    // Datastore
     undefined, 
     '/datahander/driverdrive',                   // project id
     undefined, 
     undefined,
     true,                                     // analytics opt out
     'bruce',                                  // analytics debugging tracking 
     accessToken,                              // the access token
     true);                                    // disable caching for testing  
     

  assert(handler.isHappy(), 'unable to get drive handler','handler');
  return testFlattener (handler, accessToken)
}

function parseTest () {
   
   // get a parse handler
  var userStore = PropertiesService.getScriptProperties();
  var handler = new cDataHandler.DataHandler (
      'flattener',                            // class
      cDataHandler.dhConstants.DB.PARSE,      // parse
      undefined,
      'mp',
      JSON.parse(userStore.getProperty("parseKeys")), // parse credentials
      undefined,                       
      true,             
      'bruce',
      undefined,
      true);
     

  assert(handler.isHappy(), 'unable to get parse handler','handler');
  return testFlattener (handler,null)
}

Here's a set of tests - it doesn't matter which back end you are using. The code is the same.
function testFlattener (handler, accessToken) {

  var testData = [
      {
        name:'john',
        stuff:{age:25,sex:'male'},
        networks:{
          google:{name:'john',usage:'frequent'},
          facebook:{name:'jsmith',usage:'occassionally'}
        }
      },
      { 
        name:'mary',
        stuff:{age:26,sex:'female'},
        networks:{
          google:{name:'mjones',usage:'frequent'},
          facebook:{name:'mary',usage:'occassionally'}
        }
      },
      { 
        name:'ethel',
        stuff:{age:50,sex:'female'},
        networks:{
          google:{name:'ethel',usage:'seldom'},
          facebook:{name:'ejones',usage:'seldom'}
        }
      },
      { 
        name:'fred',
        stuff:{age:60,sex:'male'},
        networks:{
          google:{name:'fred',usage:'occassionaly'},
          facebook:{name:'fred',usage:'seldom'}
        }
      },
      { 
        name:'freda',
        stuff:{age:61,sex:'female'},
        networks:{
          google:{name:'freda',usage:'often'},
          facebook:{name:'freda',usage:'seldom'}
        }
      },
      { 
        name:'wilma',
        stuff:{age:20,sex:'female'},
        networks:{
          google:{name:'wilma',usage:'occassionaly'},
          facebook:{name:'wflintstone',usage:'seldom'}
        }
      },
      { 
        name:'betty',
        stuff:{age:21,sex:'female'},
        networks:{
          google:{name:'betty',usage:'occassionaly'},
          facebook:{name:'brubble',usage:'often'}
        }
      },
      { 
        name:'barney',
        stuff:{age:21,sex:'male'},
        networks:{
          google:{name:'barney',usage:'often'},
          facebook:{name:'rubble',usage:'never'}
        }
      },
      { 
        name:'mr slate',
        stuff:{age:91,sex:'male'},
        networks:{
          google:{name:'slate',usage:'seldom'},
          facebook:{name:'slate',usage:'never'}
        }
      },
      { 
        name:'dino',
        stuff:{age:10,sex:'male'},
        networks:{
          google:{name:'dino',usage:'seldom'},
          facebook:{name:'dino',usage:'often'}
        }
      }
    ];
    

  // delete everything
  var result = handler.remove();
  assert (result.handleCode >= 0 ,
    result,'remove');

  //query everything
  var result = handler.query();
  assert (result.handleCode >= 0 &&
    result.data.length === 0,
    result, 'querypostdelete');


  // add some data that needs flattened
  var result = handler.save (testData);
  assert (result.handleCode >= 0 ,
    result,'save');  

  //query everything
  var result = handler.query();
  assert (result.handleCode >= 0 &&
    result.data.length === testData.length,
    result, 'query');

  // query everything with limit
  var result = handler.query(undefined,{limit:1});
  assert (result.handleCode >= 0 && 
    result.data.length === 1, 
    result,'querylimit');

  // sort reverse
  var result = handler.query(undefined,{sort:'-name'});
  assert (result.handleCode >= 0 &&
    result.data.length === testData.length,
    result, 'querysort');

  // sort reverse, skip 1
  var result = handler.query(undefined,{skip:9,sort:'name'});
    var result = handler.query(undefined,{skip:9});
  Logger.log(result.data.length);
  Logger.log(testData.length);
  assert (result.handleCode >= 0 &&
    result.data.length === testData.length -1,
    result,'querysort+skip');
  
  // flattten query everything
  var result = handler.query({"name":'ethel'});
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { return d.name ==='ethel'; }).length === result.data.length ,
    result,'filterdot0'); 

  // flattten query everything
  var result = handler.query({stuff:{sex:'female'}});
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { return d.stuff.sex ==='female'; }).length === result.data.length ,
    result,'filter'); 
    
  // try constraints
  var c = cDataHandler.dhConstants.CONSTRAINTS;

  // check a single constraint works
  var result = handler.query({"stuff.age":handler.constraints([[c.GT,25]])});
  assert (result.handleCode >= 0 && 
    testData.filter(function(d) { 
      return   d.stuff.age > 25; })
      .length === result.data.length , 
      result,'filterdotc1'); 

  // two constraints on same property
  var result = handler.query ( { 
      'stuff.age': handler.constraints([[c.LT,60],[c.GT,25]])
    });
  assert (result.handleCode >= 0 && testData.filter(function(d) { 
    return  d.stuff.age > 25 && d.stuff.age < 60; }
    ).length === result.data.length , result,'filterdotc2'); 
    
 
  // check two constraints plus a different property
  var result = handler.query ( { 
    'stuff.age': handler.constraints([[c.LT,60],[c.GTE,25]]),
    'stuff.sex': 'male'},
    undefined,1,true);
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { 
      return  d.stuff.age >= 25 && d.stuff.age < 60 && d.stuff.sex === 'male'; }
    ).length === result.data.length ,
    result,'filterdotc3'); 
  
  var result = handler.query ( { name:handler.constraints([[c.IN,['ethel','fred']]])});
  assert (result.handleCode >= 0 && testData.filter(function(d) { 
    return   ['ethel','fred'].indexOf(d.name) != -1; }
    ).length === result.data.length , result,'filterdotc4'); 
  
  // check text constraints  
  var result = handler.query ( { 
    'stuff.age': handler.constraints([[c.GT,25]]), 
    'stuff.sex': 'male',
    'name':handler.constraints([[c.IN,['john','mary']]])
    });
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { 
      return  d.stuff.age > 25 && d.stuff.sex === 'male' && ['john','mary'].indexOf(d.name) != -1; }
    ).length === result.data.length 
    , result,'filterdotc5'); 

  // make sure we're getting the right id with complex constaints
  var result = handler.query ( { 
      'stuff.age': handler.constraints([[c.GT,25]])
    },
    {limit:1}, 
    1,
    true );
  assert (result.handleCode >= 0 &&
    result.data.length ===1 && 
    result.handleKeys.length ===1, 
    result, "limitkeycheck1");

    
  var r2 = handler.get(result.handleKeys);
  assert (
    r2.handleCode >= 0 && 
    r2.data.length === result.data.length &&
    r2.data.every (function(d) { return d.stuff.age > 25 ; }), r2, "limitgetcheck1");
    
   // lets try updating
  var result = handler.query ( { 
    'stuff.sex': 'male'},
    undefined,1,true);
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { 
      return d.stuff.sex === 'male'; }
    ).length === result.data.length ,
    result,'update1'); 
   
  var r2 = handler.update ( result.handleKeys, result.data.map (function(d) { d.stuff.man = d.stuff.sex === 'male'; return d; } ));
  assert (r2.handleCode >= 0, r2, 'update 2');
  
  var r2 = handler.query ( { 
    'stuff.man': true} );
  assert (r2.handleCode >= 0 &&
    testData.filter(function(d) { 
      return d.stuff.sex === 'male'; }
    ).length === result.data.length ,
    result,'update 3'); 
 
  // lets try counting
  var result = handler.count ();
  assert (result.handleCode >= 0 &&
    testData.length === result.data[0].count ,
    result,'count 1'); 
    
  // check count
  var result = handler.count ( { 
    'stuff.age': handler.constraints([[c.GT,25]]), 
    'stuff.sex': 'male',
    'name':handler.constraints([[c.CONTAINS,'el']])
    });
  assert (result.handleCode >= 0 &&
    testData.filter(function(d) { 
      return  d.stuff.age > 25 && d.stuff.sex === 'male' && d.name.indexOf('el') != -1; }
    ).length === result.data[0].count
    , result,'count 2'); 
}



For help and more information join our forum,follow the blog or follow me on twitter .