In Node focus explorer without excel I showed a d3.js app that could navigate a google site using tags and nodes. This was actually pretty generalized, so it was a minor modification to add blogger post navigation capability also.
Again we’ll use Google Apps Script and adapt Analyzing site content with GAS to take data from blogger posts rather than Google Site content (actually .. as well as .. but more of that later).
Getting the blogger API key
Unlike Google Sites, you do need to get an API key to access the blogger API. You can apply for one here. A day or two later, someone sends you a mail with details of your key.
The API data
This is a straightforward REST api, but it does have a limit of 20 posts, so you have to call it repeatedly until you have them all as follows
var cb = new mcpher.cBrowser();
var p, items=[];
while (!p || p.nextPageToken) {
p = JSON.parse(cb.get(initBlogger(eArgs,source) + ( p ? '&pageToken='+p.nextPageToken : '')));
if (p && p.items) items = items.concat (p.items);
if (!p) break;
}
Storing your API key securely
One of the challenges about publishing code where there is private information is how to hide it. I use Database abstraction and Google Apps Script and store stuff like this on my private MongoDb store, but you may find it easier to use something l like your property store. Here’s how I do it.
// my private store
var myKey = myStuff.getMyStuff(api, myStuff.myStuffDb());
if (myKey.status.code == "good") {
return myKey.result;
}
else {
throw ("unable to find your key for " + api);
return null;
}
}
Creating the dataset
Retrieving the data takes the same structure as for Google Sites, except its a little simpler. There is no need for recursion since the blog posts are flat. In fact – you can see from the blogger and sites version below there is a very minor difference to get the data from different properties, and to recurse or not.
function blogChildren(tags,parent,pages,a) {
var resultsArray = a || [];
for (var i =0;i<pages.length;i++) {
// process this page
var page = pages[i];
resultsArray.push( {
parent: parent,
name: page.title,
key: page.id,
url: page.url,
title: page.title,
tags: addCounts(cloneObject(tags),getTextFromHtml(page.content))
} );
// and its children
}
return resultsArray;
}
function logChildren(tags,parent,pages,a) {
var resultsArray = a || [];
for (var i =0;i<pages.length;i++) {
// process this page
var page = pages[i];
resultsArray.push( {
parent: parent.getName (),
name: page.getName (),
key: page.getName () + "_" + resultsArray.length,
url: page.getUrl(),
title: page.getTitle(),
tags: addCounts(cloneObject(tags),page.getTextContent())
} );
// and its children
logChildren (tags,pages[i], pages[i].getChildren(), resultsArray);
}
return resultsArray;
}
The only other wrinkle is that the blog content comes as html, so it needs to be rendered as plain text to avoid the html markup being considered as linkable data. I found this neat solution for that on stackOverflow by Cory Goldfeder who continually comes up with great stuff like this.
Function getTextFromHtml(html) {
return getTextFromNode(Xml.parse(html, true).getElement());
}
function getTextFromNode(x) {
switch(x.toString()) {
case 'XmlText': return x.toXmlString();
case 'XmlElement': return x.getNodes().map(getTextFromNode).join('');
default: return '';
}
}
Scheduling data updates
Now I have a Google site version and a blogger version. Scheduling GAS showed how to add a nightly update to write the data to Google Drive. I’ll just update that to write the blogger version to Google Cloud store (for more on that see Cloud Storage and Apps Script) and here is the result for the blogger. You can navigate in the same way as in the Google site version, except this time you get to both. You can write it to Drive if you prefer by changing the parameter from gcs to drive, but then you’ll need to find somewhere to host it. Using cloud storage I can host it in place from where I write it. tagoutput :'gcs', // change this to drive if you want to write to drive instead The navigator takes an argument that is the name of the input data (the default is the data for this site). So you can use it as is, once you have created a data set as described here and in the Google site version. Finally, I’ve built in a proxy to allow the data to come from some other place if you don’t have a site to host your data on. For details see Apps Script as a proxy. If you don’t want to use a proxy, you’ll need to change a couple of lines to get the data directly instead of via a proxy. It is embedded below or you can try it out here.
Combining data sets
This is where it gets real interesting – now you can combine multiple sites or blogs into a single navigation app. I have scheduled a nightly trigger that creates a dataset combing the sites and blogger data, giving us this (it’s a bit much but once you start focusing in it can be useful). It will take a while to settle down as there’s a huge amount of content between the two sources. You can run it here
The code
Here is the Google Apps Script code to generate the data
Here is the web app code
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>d3.js from Excel - ramblings.mcpher.com</title>
<base href="https://storage.googleapis.com/xliberation.com/">
<!-- Always force latest IE rendering engine (even in intranet) & Chrome Frame
Remove this if you use the .htaccess -->
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
<style>
body {
position: relative;
font-family: "Helvetica Neue", Helvetica, sans-serif;
margin: 1em auto 4em auto;
width: 960px;
}
h1 {
font-size: 64px;
font-weight: 300;
letter-spacing: -2px;
margin: .3em 0 .1em 0;
}
h2 {
margin-top: 2em;
}
h1, h2 {
text-rendering: optimizeLegibility;
}
h2 a {
color: #ccc;
left: -20px;
position: absolute;
width: 740px;
}
footer {
font-size: small;
margin-top: 8em;
}
header aside {
margin-top: 20px;
}
header aside, footer aside {
color: #636363;
text-align: right;
}
aside {
float:right;
display:inline;
font-size: small;
}
.attribution {
font-size: small;
margin-bottom: 2em;
}
body > p, li > p {
line-height: 1.5em;
}
body > p {
width: 720px;
}
body > blockquote {
width: 640px;
}
li {
width: 680px;
}
pre, code, textarea {
font-family: "Menlo", monospace;
}
code {
line-height: 1em;
}
textarea {
font-size: 100%;
}
body > pre {
border-left: solid 2px #ccc;
padding-left: 18px;
margin: 2em 0 2em -20px;
}
.html .value, .javascript .string, .javascript .regexp {
color: #756bb1;
}
.html .tag, .css .tag, .javascript .keyword {
color: #3182bd;
}
.comment {
color: #636363;
}
.html .doctype, .javascript .number {
color: #31a354;
}
.html .attribute, .css .attribute, .javascript .class, .javascript .special {
color: #e6550d;
}
svg {
font: 10px sans-serif;
}
.axis path, .axis line {
fill: none;
stroke: #000;
shape-rendering: crispEdges;
}
sup, sub {
line-height: 0;
}
q:before, blockquote:before {
content: "?";
}
q:after, blockquote:after {
content: "?";
}
blockquote:before {
position: absolute;
left: 2em;
}
blockquote:after {
position: absolute;
}
</style>
<style>
#chart {
height: 600px;
}
.node rect {
cursor: move;
fill-opacity: .9;
shape-rendering: crispEdges;
}
.node text {
pointer-events: none;
text-shadow: 0 1px 0 #fff;
}
.link {
fill: none;
stroke: #000;
stroke-opacity: .2;
}
.link:hover {
stroke-opacity: .5;
}
circle.node-dot {
fill: DarkSlateGray;
stroke: SlateGray;
stroke-width: 1px;
}
line.link {
fill: none;
stroke: SlateGray;
stroke-width: 1.5px;
}
marker#defaultMarker {
fill: SlateGray;
}
line.link.defaultMarker {
stroke: SlateGray;
}
circle {
fill: #ccc;
stroke: #333;
stroke-width: 1.5px;
}
text {
pointer-events: none;
}
text.shadow {
stroke: #fff;
stroke-width: 3px;
stroke-opacity: .8;
}
</style> </head><body>d3.js force directed node focus:<strong>Site Tag Explorer</strong> <small>click a node to focus, double click to visit page</small><aside><small>
<a href='http://ramblings.mcpher.com'>ramblings.mcpher.com</a><br>
works best on Chrome<br> ackowledgements:<a href='http://bost.ocks.org/mike/'>Mike Bostok for d3.js</a></small>
</aside>
</body>
</html>
For more on this , see GAS and sites and d3.js and this excelramblings blogpost.