Abstract: IPLD is the underlying data structure for IPFS. It is a huge hash-linked directed acyclic graph, with all files, git repos, blockchains, etc., within it. IPLD is the heart of IPFS. The tooling for manipulating IPLD directly has recently landed into both go-ipfs and js-ipfs under the API/commands ipfs dag. What we need next is to be able to make it nicer for humans to interact with IPLD directly. For this purpose, we have a few different "graph explorers" in mind. You can think of them similar to a visual explorer for a git repo (eg rendering the commit graph), or file explorers, or blockchain explorers. We aim to create several graph explorers, (1) one based on a traditional programming REPL, (2) another based on JSON tree explorers, (3) another based on a column-based tree viewer, (4) and a graphical version using d3 graph plotting, and (5) one in 3D which allows VR exploration.
To learn more about IPLD, please check out:
- https://github.com/ipld/ipld
- https://github.com/ipld/cid
- IPLD: Enter the Merkle Forest - conceptual talk from @jbenet
- IPFS, IPLD, Blockchain Fun - demo talk from @whyrusleeping
- http://ipld.io/
- website is outdated and will change soon to reflect important API changes, but explains the ideas ok. Talks may be better.
The "Relevant API" section in this document gives examples on ipfs node calls to make.
Examples:
node- https://repl.it/languages/javascript
- https://binary-studio.com/wp-content/uploads/2015/05/node-repl.png
The first idea is to part from a very simple REPL. The REPL would be primarily about hashes and content. It would have a few "builtin" functions:
resolve(<path>)- walk down a path and return the object found therels(<path>)- list the entries in a pathtree(<path>, <depth>)- list all entries starting from path, and up to<depth>graph depth. (may be useful)get(<path>)- walks down a path and returns that path
Entering <path> is the equivalent of get <path>, and every non-path value is added to ipfs, hashed, and its CID is returned.
A more advanced version would allow:
eval(<path>), which would load the value at<path>, interpret it as javascript, and eval it, adding it to the current context \o/.
Examples:
- https://shrimpworks.za.net/assets/projects/json-explorer/jsonex.png
- http://www.webandsay.com/images/json-diff-released/jsondiff.png
- http://visualizer.json2html.com/
- http://www.jsoneditoronline.org/
Structured explorers:
- https://blockexplorer.com/block/00000000000000000156c68693b097733b83084f327a10a591948b9dec54032e
- https://explorer.zcha.in/blocks/000000006263c2e8a35c0e950f5d139be1b6060554895f4327c81db4b1f00813
The tree explorer builds a very similar editor to traditional JSON tree explorers. This may be overkill for now.
File browsers / explorers:
- https://en.wikipedia.org/wiki/File_manager
- https://tawus.files.wordpress.com/2011/08/filebrowser.png
- https://i.stack.imgur.com/j1v4A.jpg <-- columns
The column-based file system exploration fits our graphs very well, as well. This should perhaps be the easiest to get working well. There's many examples and libraries out there.
D3 was already used in some of our prior art and probably will end up using it directly or indirectly.
Open technical questions that probably do not belong to design discussion: what will be the data format? should D3 be used in raw form, or via a library? are there any alternatives to D3 ecosystem? We will answer them during the implementation, raising them here just for the record.
D3→Shapes→Links: https://github.com/d3/d3-shape#links
Live demo with annotated sources: https://bl.ocks.org/mbostock/4339184
This version visualizes the IPLD graph using d3:
ipfs dataviz is currently working, and showing files. It should also show the raw data. Now that IPLD is out, the project could be updated. Though because having access to the file graph is useful, we may want to copy the visualization and remix it to navigate on nodes, instead of files.
Dead Demo (trying to load big flat directory with XKCD archive): https://ipfs.io/ipfs/QmX5smVTZfF8p1VC8Y3VtjGqjvDVPWvyBk24JgvnMwHtjC/viz#Qmb8wsGZNXt5VXZh1pEmYynjB6Euqpq3HYyeAdw2vScTkQ
Given enough time it will eventually render this mess.
(Does not really support dynamic transformations in web browser, but mentioning it for the record)
Why mentioning GraphViz and Dot format? People can easily reuse it in papers, publications etc.
Prior art exists in IPFS ecosystem, see notes on Graphing Objects:
Web Version implemented in D3 exists: https://github.com/mstefaniuk/graph-viz-d3-js
Examples:
Blockchain explorers:
- in 3d: https://www.youtube.com/watch?v=3ujUIz9hQ7c
- http://www.bitcoinlinks.net/files/styles/card_image__320x240_/public/img/bitbonkers_0.png?itok=2vKUm481
This explorer is a native way to look at the bitcoin blockchain; it's just done on a 3D environment! This is possibly super cool, but we'd love to let users navigate and explore, jumping from block to block.
All of these explorers do one thing: traverse the graph. Therefore, we only need a couple of api calls to do this: ipfs.dag.ls, and ipfs.dag.get. See more in the Relevant API section below.
All explorers have to do with a given path. It is important to keep this path in the hash/anchor of the html page, as we want to be able to link to exact locations in the graph.
Another thing all explorers track is navigation path. While this may be multiple nodes open in some explorers, it is enough to keep track of a single string path for the others. It would be useful if this navigation path is also given through the URL bar, so that people can link each other to the exact same location and "path used to get there".
All of the explorers here will use the following api calls:
ipfs.dag.lsipfs.dag.get
See the DAG API, which works in both js-ipfs and js-ipfs-api.
var IPFSAPI = require('ipfs-api')
var ipfs = IPFSApi('/ip4/127.0.0.1/tcp/5001')
// given an object such as:
//
// var obj = {
// "a": 1,
// "b": [1, 2, 3],
// "c": {
// "ca": [5, 6, 7],
// "cb": "foo"
// }
// }
//
// ipfs.dag.put(obj, function(err, cid2) {
// assert(cid == zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y)
// })
ipfs.dag.ls('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y', function(err, result) {
for (var i in result.entries) {
console.log(result.entries[i])
}
})
// Returns:
// a
// b
// c
ipfs.dag.ls('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/c', function(err, result) {
for (var i in result.entries) {
console.log(result.entries[i])
}
})
// Returns:
// ca
// cb
ipfs.dag.ls('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/c/ca', function(err, result) {
for (var i in result.entries) {
console.log(result.entries[i])
}
})
// Returns:
// 0
// 1
// 2var IPFSAPI = require('ipfs-api')
var ipfs = IPFSApi('/ip4/127.0.0.1/tcp/5001')
// given an object such as:
//
// var obj = {
// "a": 1,
// "b": [1, 2, 3],
// "c": {
// "ca": [5, 6, 7],
// "cb": "foo"
// }
// }
//
// ipfs.dag.put(obj, function(err, cid2) {
// assert(cid == zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y)
// })
ipfs.dag.tree('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y', function(err, result) {
for (var i in result) {
console.log(result[i])
}
})
// Returns:
// a
// b
// b/0
// b/1
// b/2
// c
// c/ca
// c/ca/0
// c/ca/1
// c/ca/2
// c/cb
ipfs.dag.tree('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/c', function(err, result) {
for (var i in result) {
console.log(result[i])
}
})
// Returns:
// ca
// ca/0
// ca/1
// ca/2
// cbvar IPFSAPI = require('ipfs-api')
var ipfs = IPFSApi('/ip4/127.0.0.1/tcp/5001')
// given an object such as:
//
// var obj = {
// "a": 1,
// "b": [1, 2, 3],
// "c": {
// "ca": [5, 6, 7],
// "cb": "foo"
// }
// }
//
// ipfs.dag.put(obj, function(err, cid2) {
// assert(cid == zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y)
// })
function errOrLog(err, result) {
if (err) console.error('error: ' + err)
else console.log(result)
}
ipfs.dag.get('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/a', errOrLog)
// Returns:
// 1
ipfs.dag.get('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/b', errOrLog)
// Returns:
// [1, 2, 3]
ipfs.dag.get('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/c', errOrLog)
// Returns:
// {
// "ca": [5, 6, 7],
// "cb": "foo"
// }
ipfs.dag.get('zdpuAkxd9KzGwJFGhymCZRkPCXtBmBW7mB2tTuEH11HLbES9Y/c/ca/1', errOrLog)
// Returns:
// 6