d3 concept browser

In Node focus on Blogger posts and Node focus explorer without excel I showed how you might browse sites by connected topic, rather than hierarchically. This used a D3.js force diagram as the navigation tool.

 
I got to thinking about ‘interactive concept browsing’, and wondered if the d3 work could be tweaked to be a kind of hybrid list of pages, with connections through topics. Here’s what it looks like, using the posts from the Excel liberation blog.
 

And the live version is here

As in the previous explorer a number of the same techniques are used for retrieving the data, you can also focus in on particular topics or pages by clicking or visit the blog page by double clicking. In addition hovering will show connections between the concepts.
 
Page Content hide
1 Hovering
2 Selecting a topic and related pages with single click
3 Selecting related topics by a single click on a page
4 Finding related pages for multiple topics
5 Fixed positioning
6 Moving fixed items
7 Navigating
8 The Data

Hovering

Selecting a topic and related pages with single click

Selecting related topics by a single click on a page

Finding related pages for multiple topics

Fixed positioning

 

This kind of approach is possible because you can mix fixed positioning and dynamic positioning in D3. simply setting d.fixed to true in your data will keep that item fixed, whilst the others are calculated relative to each other and the fixed items. So in this way you can build up a list of pages that stay where they are, whilst the topic nodes float around as needed. 
 

Moving fixed items

 

But of course you may want to move these items, perhaps to group them in some way. Well you can. When you drag a page to a new position just stays there. In this example I wanted to focus on the page ‘Publishing Google Apps Script snippets’, so i pulled it to the center of the diagram. Anything else I open up will now be centered around that item I moved.

 

Navigating

You can navigate to the underlying pages by double clicking on any page node. I think this provides an interesting alternative to normal site navigation, and of course the data does not have to be a single site – it can be from many related sources.
 

The Data

 

Just like in Node focus on Blogger posts, the navigation data is being created each night by a Google Apps Script crawling the Excel Liberation blog and this site. It’s very straightforward – an array of topics, each one looking like this, which contain an array of pages that mention that topic. The concept explorer makes connections between topics by finding pages that are common between each topics pages array. In the case of the blogger data, the name and the title are the same. When we get to data generated from Google Sites, the name is different. Name and Title are both used in the concept explorer in different contexts. The key is a unique value that will be used to match pages and topics, and the count is used to size the node. In this case the count is the number of times the ‘cdataset’ topic is mentioned in all the following pages (potentially more than once per page)
"name": "cdataset",
"count": 87,
"key": "cdataset",
"pages": [
{
"name": "Sankey diagrams direct from Excel - update",
"key": "7899286130266944386",
"title": "Sankey diagrams direct from Excel - update",
"url": "https://ramblings.mcpher.com/sankeyexcel2/"
},
{
"name": "Connections in electoral data - D3 and VBA follow on from oUseful post",
"key": "861843895324588546",
"title": "Connections in electoral data - D3 and VBA follow on from oUseful post",
"url": "https://ramblings.mcpher.com/connections-in-electoral-data-d3-and/"
},
{
"name": "Mashing up electoral data - follow on from oUseful post",
"key": "2877216136096344149",
"title": "Mashing up electoral data - follow on from oUseful post",
"url": "https://ramblings.mcpher.com/mashing-up-electoral-data-follow-on"
},
You can see it would be straightforward to generate this kind of data from other sources, including crawling other websites and combining the results.  Here are the combined results for this site, or for this site combined with the blogger.
 
If you can massage your data into the same shape, you can just pass its url as follows to visualize it
 

https://storage.googleapis.com/xliberation.com/googlecharts/d3concept.html?data=https%3A%2F%2Fstorage.googleapis.com%2Fxliberation.com%2Fdump%2Fsite

The GAS code is here.

The Code to write the HTML app is below. 

<!DOCTYPE html>
<html lang="en">
 <head>
  <meta charset="utf-8" />
  <title>d3.js concept browser - ramblings.mcpher.com</title>
<base href="https://storage.googleapis.com/xliberation.com/">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
  <link rel="stylesheet" type="text/css" href="cdn/css/d3direct.css">
  <script type="text/javascript" src="https://d3js.org/d3.v3.min.js"></script>
  <script type="text/javascript" src="https://www.google.com/jsapi"></script>


  <script type="text/javascript">
   google.load("jquery", "1");
   google.setOnLoadCallback(function() {
        initialize().then (
            function (control) {
                doTheTreeViz(control);
            }
        );
   });

  function doTheTreeViz(control) {

    var svg = control.svg;

    var force = control.force;
    force.nodes(control.nodes)
        .links(control.links)
        .start();

    // Update the links
    var link = svg.selectAll("line.link")
        .data(control.links, function(d) { 
            return d.key; 
        } );
 
   // Enter any new links
    var linkEnter = link.enter().insert("svg:line", ".node")
        .attr("class", "link")
        .attr("x1", function(d) { return d.source.x; })
        .attr("y1", function(d) { return d.source.y; })
        .attr("x2", function(d) { return d.target.x; })
        .attr("y2", function(d) { return d.target.y; })
      .append("svg:title")
        .text(function(d) { return d.target.name + ":" + d.source.name ; });
    
    // Exit any old links.
    link.exit().remove();


  // Update the nodes
    var node = svg.selectAll("g.node")
        .data(control.nodes, function(d) { return d.key; });

    node.select("circle")
        .style("fill", function(d) {
            return getColor(d);
        })
        .attr("r", function(d) {
            return getRadius(d);
        })

  // Enter any new nodes.
    var nodeEnter = node.enter()
      .append("svg:g")
        .attr("class", "node")
        .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; })
        .on("dblclick", function(d){
            control.nodeClickInProgress=false;
            if (d.url)window.open(d.url);
        })
        .on("click", function(d){
            // this is a hack so that click doesnt fire on the1st click of a dblclick
            if (!control.nodeClickInProgress ) {
                control.nodeClickInProgress = true;
                setTimeout(function(){
                    if (control.nodeClickInProgress) { 
                        control.nodeClickInProgress = false;
                        if (control.options.nodeFocus) {
                            d.isCurrentlyFocused = !d.isCurrentlyFocused;
                            doTheTreeViz(makeFilteredData(control));
                        }
                    }
                },control.clickHack); 
            }
        })
        .call(force.drag);

    nodeEnter
      .append("svg:circle")
        .attr("r", function(d) {
            return getRadius(d);
        })
        .style("fill", function(d) {
            return getColor(d);
        })
        .on("mouseover", function(d){
            // enhance all the links that end here
            enhanceNode (d);
        })
        
        .on("mouseout", function(d){
            resetNode(d);
            
        })
      .append("svg:title")
        .text(function(d) { return d[control.options.nodeLabel]; })
        
    function enhanceNode(selectedNode) {
        link.filter ( function (d) {
            return d.source.key == selectedNode.key || d.target.key == selectedNode.key;
        } )
        .style("stroke", control.options.routeFocusStroke)
        .style("stroke-width", control.options.routeFocusStrokeWidth);
        
        if (text) {
            text.filter ( function (d) {
                return areWeConnected (selectedNode,d);
            } )
            .style("fill", control.options.routeFocusStroke);
        }
    }
    
    function areWeConnected (node1,node2) {
        for (var i=0; i < control.data.links.length ; i++) {
            var lnk = control.data.links&#91;i&#93;;
            if ( (lnk.source.key === node1.key && lnk.target.key === node2.key) ||
                 (lnk.source.key === node2.key && lnk.target.key === node1.key) ) return lnk;
        }
        return null;
    }
    function resetNode(selectedNode) {
        link.style("stroke", control.options.routeStroke)
            .style("stroke-width", control.options.routeStrokeWidth);
        if (text) { 
            text.style("fill", control.options.routeStroke);
        }
    }

   if (control.options.nodeLabel) {       
       // text is done once for shadow as well as for text
        var textShadow = nodeEnter.append("svg:text")
            .attr("x", function(d) {
                var x = (d.right || !d.fixed) ? 
                    control.options.labelOffset : 
                    (-d.dim.width - control.options.labelOffset)  ;
                return x;
            })
            .attr("dy", ".31em")
            .attr("class", "shadow")
            .attr("text-anchor", function(d) { 
                return !d.right ? 'start' : 'start' ;
            })
            .style("font-size",control.options.labelFontSize + "px")
            .text(function(d) {
                return d.shortName ? d.shortName : d.name;
            });

        var text = nodeEnter.append("svg:text")
            .attr("x", function(d) {
                var x = (d.right || !d.fixed) ? 
                    control.options.labelOffset : 
                    (-d.dim.width - control.options.labelOffset)  ;
                return x;
            })
            .attr("dy", ".35em")
            .attr("class", "text")
            .attr("text-anchor", function(d) { 
                return !d.right ? 'start' : 'start' ;})
            .style("font-size",control.options.labelFontSize + "px")
            .text(function(d) {
                return d.shortName ? d.shortName : d.name;
            })
            
            .on("mouseover", function(d){
            // enhance all the links that end here
                enhanceNode (d);
                d3.select(this)
                    .style('fill',control.options.routeFocusStroke);
            })
        
            .on("mouseout", function(d){
                resetNode(d);
            });
    }

    // Exit any old nodes.
    node.exit().remove();
    control.link = svg.selectAll("line.link");
    control.node = svg.selectAll("g.node");
    force.on("tick", tick);



    if (control.options.linkName) {
        link.append("title")
            .text(function(d) {
                return d&#91;control.options.linkName&#93;;
        });
    }


    function tick() {
        link.attr("x1", function(d) { return d.source.x; })
            .attr("y1", function(d) { return d.source.y; })
            .attr("x2", function(d) { return d.target.x; })
            .attr("y2", function(d) { return d.target.y; });
        node.attr("transform", function(d) {
            return "translate(" + d.x + "," + d.y + ")";
        });

    }
 
    function getRadius(d) {
        return makeRadius(control,d);
    }
    function getColor(d) {
        return control.options.nodeFocus && d.isCurrentlyFocused ? control.options.nodeFocusColor  : control.color(d.group) ;
    }

   }
   
function makeRadius(control,d) {
     var r = control.options.radius * (control.options.nodeResize ? Math.sqrt(d&#91;control.options.nodeResize&#93;) / Math.PI : 1);
     return control.options.nodeFocus && d.isCurrentlyFocused ? control.options.nodeFocusRadius  : r;
}

function makeFilteredData(control,selectedNode){
    // we'll keep only the data where filterned nodes are the source or target
    var newNodes = &#91;&#93;;
    var newLinks = &#91;&#93;;

    for (var i = 0; i < control.data.links.length ; i++) {
        var link = control.data.links&#91;i&#93;;
        if (link.target.isCurrentlyFocused || link.source.isCurrentlyFocused) {
            newLinks.push(link);
            addNodeIfNotThere(link.source,newNodes);
            addNodeIfNotThere(link.target,newNodes);
        }
    }
    // if none are selected reinstate the whole dataset
    if (newNodes.length > 0) {
        control.links = newLinks;
        control.nodes = newNodes;
    }
    else {
        control.nodes = control.data.nodes;
        control.links = control.data.links;
    }
    return control;
    
    function addNodeIfNotThere( node, nodes) {
        for ( var i=0; i < nodes.length; i++) {
            if (nodes&#91;i&#93;.key == node.key) return i;
        }
        return nodes.push(node) -1;
    }
}

function getPixelDims(scratch,t) {
    // scratch is an elemen with the correct styling, t is the text to be counted in pixels
    scratch.empty();
    scratch.append(document.createTextNode(t));
    return { width: scratch.outerWidth(), height: scratch.outerHeight() } ;
}
function initialize () {
   
    var initPromise = $.Deferred();
    var control = {};
    control.divName = "#chart";
    
    //some basic options
    var newoptions = {  nodeLabel:"label", 
                    nodeResize:"count", height:900,
                    nodeFocus:true, radius:3, charge:-500};
    // defaults
    control.options = $.extend({
            stackHeight : 12,
            radius : 5,
            fontSize : 14,
            labelFontSize : 8,
            labelLineSpacing: 2.5,
            nodeLabel : null,
            markerWidth : 0,
            markerHeight : 0,
            width : $(control.divName).outerWidth(),
            gap : 1.5,
            nodeResize : "",
            linkDistance : 80,
            charge : -120,
            styleColumn : null,
            styles : null,
            linkName : null,
            nodeFocus: true,
            nodeFocusRadius: 25,
            nodeFocusColor: "FireBrick",
            labelOffset: 5,
            gravity: .05,
            routeFocusStroke: "FireBrick",
            routeFocusStrokeWidth: 3,
            circleFill: "Black",
            routeStroke: "Black",
            routeStrokeWidth: 1,
            height : $(control.divName).outerHeight()
           
        }, newoptions);
        
    var options = control.options;
    options.gap = options.gap * options.radius;
    control.width = options.width;
    control.height = options.height;
    // this is an element that can be used to determine the width of a text label
    
    control.scratch = $(document.createElement('span'))
        .addClass('shadow')
        .css('display','none')
        .css("font-size",control.options.labelFontSize + "px");   
    $('body').append(control.scratch);
    
    getTheData(control).then( function (data) {  
        
        control.data = data;
        control.nodes = data.nodes;
        control.links = data.links;
        control.color = d3.scale.category20();
        control.clickHack = 200;
    
        control.svg = d3.select(control.divName)
            .append("svg:svg")
            .attr("width", control.width)
            .attr("height", control.height);
        
        control.force = d3.layout.force().
            size(&#91;control.width, control.height&#93;)
            .linkDistance(control.options.linkDistance)
            .charge(control.options.charge)
            .gravity(control.options.gravity);
    
       initPromise.resolve(control);
    });
    return initPromise.promise();
}

function getTheData(control) {
    var dataPromise = getTheRawData();
    var massage = $.Deferred();
    dataPromise.done ( function (data) {
        // need to massage it
        massage.resolve ( dataMassage (control,data));    
    })
    .fail (function (error) {
        console.log (error);
        massage.reject(error);
    });
    return massage.promise();
}

function dataMassage(control,data) {

var ind = data, nodes = &#91;&#93;,links =&#91;&#93;;
   // the tags are to be circles
   for (var i=0;i<ind.length;i++) {
        ind&#91;i&#93;.isCurrentlyFocused = false;
        nodes.push(ind&#91;i&#93;);
       // add links to pages
       for ( var j=0; j < ind&#91;i&#93;.pages.length; j++) {
           //push this page as a node
           var node = findOrAddPage(control,ind&#91;i&#93;.pages&#91;j&#93;,nodes);
           node.isCurrentlyFocused = false;
           // create a link
           var link = { source:node , target:ind&#91;i&#93;, key : node.key + "_" + ind&#91;i&#93;.key };
           links.push(link);
       }
   }
   // sort nodes alpha
   nodes.sort ( function (a,b) { return a.name < b.name  ? -1 : (a.name == b.name ? 0 : 1 ) ; });
   control.pageCount = 0;
   control.pageRectSize = {width:0,height:0,radius:0};   
   for ( var i = 0; i < nodes.length ; i++) {
       page= nodes&#91;i&#93;;
       page.group =0;
       page.dim = getPixelDims(control.scratch, page.name);
       if (page.fixed) { 
           control.pageCount++;
          // this will calculate the width/height in pixels of the largest label
           control.pageRectSize.width = Math.max(control.pageRectSize.width,page.dim.width);
           control.pageRectSize.height = Math.max(control.pageRectSize.height,page.dim.height);
           control.pageRectSize.radius = Math.max(control.pageRectSize.radius,makeRadius(control,page));
           page.group =1;    
       }
       
   }
   var options= control.options;

   // we're going to fix the nodes that are pages into two columns
    for ( var i = 0, c=0; i < nodes.length ; i++) {
        var page = nodes&#91;i&#93;;
        if (page.fixed) {
            page.right= (c > control.pageCount/2);
            // y dimension calc same for each column
            page.y = ((c % (control.pageCount/2)) + .5) * (control.pageRectSize.height)  ;
            
            // x based on right or left column
            page.x = page.right ? 
                        control.width - control.pageRectSize.width - options.labelOffset  :
                        page.dim.width + options.labelOffset ;
            c++;
        }
        
   }

   return {  nodes: nodes, links: links };

}

function findOrAddPage(control,page,nodes) {
    for ( var i=0;i<nodes.length;i++) {
        if ( nodes&#91;i&#93;.key === page.key ) { 
            nodes&#91;i&#93;.count++;
            return nodes&#91;i&#93;;
        }
    }
    page.fixed = true;
    page.count = 0;
    return nodes&#91;nodes.push(page) - 1&#93; ;
}


// modify with your proxy and dataurl
// take the raw data and prepare it for d3
function getParameterByName(name) {
    name = name.replace(/&#91;\&#91;&#93;/, "\\\&#91;").replace(/&#91;\&#93;&#93;/, "\\\&#93;");
    var regex = new RegExp("&#91;\\?&&#93;" + name + "=(&#91;^&#&#93;*)"),
        results = regex.exec(location.search);
    return results == null ? "" : decodeURIComponent(results&#91;1&#93;.replace(/\+/g, " "));
}
// modify with your proxy and dataurl
// take the raw data and prepare it for d3
function getTheRawData() {
    // here's a php proxy to make jsonP
    var proxyUrl="https://script.google.com/macros/s/AKfycbzMhwJy-OAR28YTBxO1AbVdSQQFI101X3UDzY-1yc9lUgBfpSc/exec";;


    // blogwheel.json - blog only
    // sitewheel.json - both
    // allwheel.json - site

    var dataUrl = getParameterByName('data') || "https://storage.googleapis.com/xliberation.com/dump/blogwheel.json";
      
    // promise will be resolved when done
    return getPromiseData(dataUrl,proxyUrl);
}

// no need to touch this
// general deferred handler for getting json data through proxy and creating promise
function getPromiseData(url,proxyUrl){
  var deferred = $.Deferred();
  var u = proxyUrl+"?url="+encodeURIComponent(url);

  $.getJSON(u, null, 
        function (data) {
            if (Math.floor(data.responseCode/100 ) !== 2) {
              throw 'error ' + data.responseCode + ' getting data ' + data.result;
            }
            // data returned by apps script proxy is encoded json.
            deferred.resolve(JSON.parse(data.result));
    })
    .error(function(res, status, err) {
        deferred.reject("error " + err + " for " + url);
    });
  
  return deferred.promise();
}

</script></head><body><div>d3.js force directed node focus:<strong>Site Concept Browser</strong><span style="color:darslateblue;"> <small>click a node to focus, double click to visit page</small></span><aside><small>
   <a href='http://ramblings.mcpher.com'>ramblings.mcpher.com</a> ackowledgements:<a href='http://bost.ocks.org/mike/'>Mike Bostok(d3.js)</a></small>
   </aside>
</div>
  <div id="chart">

  </div>

 </body>
</html>

Return to Gas and Sites to read more topics