/
the-final-hack-of-the-gephi-streaming-plugin.html
76 lines (68 loc) · 17.6 KB
/
the-final-hack-of-the-gephi-streaming-plugin.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><title>The Final hack of the Gephi Streaming Plugin - Toto do stuff</title><meta name="description" content="We saw that we can use Gephi Streaming Plugin to push data into Gephi. It works well , but it brings a lot of constrains…"><meta name="generator" content="Publii Open-Source CMS for Static Site"><link rel="canonical" href="https://totetmatt.github.io/the-final-hack-of-the-gephi-streaming-plugin.html"><link rel="alternate" type="application/atom+xml" href="https://totetmatt.github.io/feed.xml"><link rel="alternate" type="application/json" href="https://totetmatt.github.io/feed.json"><meta property="og:title" content="The Final hack of the Gephi Streaming Plugin"><meta property="og:site_name" content="Toto do stuff"><meta property="og:description" content="We saw that we can use Gephi Streaming Plugin to push data into Gephi. It works well , but it brings a lot of constrains…"><meta property="og:url" content="https://totetmatt.github.io/the-final-hack-of-the-gephi-streaming-plugin.html"><meta property="og:type" content="article"><style>:root{--body-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--heading-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--logo-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--menu-font:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen,Ubuntu,Cantarell,"Fira Sans","Droid Sans","Helvetica Neue",Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol"}</style><link rel="stylesheet" href="https://totetmatt.github.io/assets/css/style.css?v=825c89ac06c7215b642eda05e8a14751"><script type="application/ld+json">{"@context":"http://schema.org","@type":"Article","mainEntityOfPage":{"@type":"WebPage","@id":"https://totetmatt.github.io/the-final-hack-of-the-gephi-streaming-plugin.html"},"headline":"The Final hack of the Gephi Streaming Plugin","datePublished":"2015-11-05T08:55","dateModified":"2020-06-20T00:35","description":"We saw that we can use Gephi Streaming Plugin to push data into Gephi. It works well , but it brings a lot of constrains…","author":{"@type":"Person","name":"Totetmatt","url":"https://totetmatt.github.io/authors/totetmatt/"},"publisher":{"@type":"Organization","name":"Totetmatt"}}</script></head><body><div class="site-container"><header class="top" id="js-header"><a class="logo" href="https://totetmatt.github.io/">Toto do stuff</a></header><main><article class="post"><div class="hero"><figure class="hero__image hero__image--overlay"><img src="https://totetmatt.github.io/media/website/bg.jpg" srcset="https://totetmatt.github.io/media/website/responsive/bg-xs.jpg 300w, https://totetmatt.github.io/media/website/responsive/bg-sm.jpg 480w, https://totetmatt.github.io/media/website/responsive/bg-md.jpg 768w, https://totetmatt.github.io/media/website/responsive/bg-lg.jpg 1024w, https://totetmatt.github.io/media/website/responsive/bg-xl.jpg 1360w, https://totetmatt.github.io/media/website/responsive/bg-2xl.jpg 1600w" sizes="(max-width: 1600px) 100vw, 1600px" loading="eager" alt=""></figure><header class="hero__content"><div class="wrapper"><div class="post__meta"><time datetime="2015-11-05T08:55">15/11/05</time></div><h1>The Final hack of the Gephi Streaming Plugin</h1><div class="post__meta post__meta--author"><a href="https://totetmatt.github.io/authors/totetmatt/" class="feed__author invert">Totetmatt</a></div></div></header></div><div class="wrapper post__entry"><p>We saw that we can use Gephi Streaming Plugin to push data into Gephi. It works well , but it brings a lot of constrains such as :</p><ul><li>You need to know Gephi Instance Address, which makes it tricky when you want to do over internet communication.</li><li>It's HTTP Push based, which technicaly isn't optimum, especially for high throughput stream.</li></ul><p>It could be nice to have reverse working : Gephi connecting to an entrypoint and generates graph from it.</p><h1>Gephi as consumer of an entry point</h1><p>I manage to find how to do that : HTTP Streaming. I won't go to the details, because spoiler alert : it's nice but not that much.</p><p>The trick is to create a HTTP Server that streams the data from an entrypoint that you access with the client part of the Gephi Stream plugin. That works, and that's interesting because you can then provide an entrypoint on your webserver in internet and share it to other gephi users to have the same stream.</p><p>It might be interesting for some use cases so I share you the code of a simple web server here : <a href="https://github.com/totetmatt/graphStreamServer">https://github.com/totetmatt/graphStreamServer</a></p><p>But with this method, Gephi behaves weird when you use filters and I found it more unstable than when you're using it as a "Master".</p><p>So here is the deal, we would like to share across the internet a Graph Stream, but with Gephi set as Master. It's theorically feasable, but it involves networking operation in your home. Not acceptable, we are targeting a solution that (almost) everybody can do.</p><h1>Websocket strikes back</h1><p><a href="https://en.wikipedia.org/wiki/WebSocket">Websocket</a> is bi-directonnal communication, which means you can receive and send data within the same "connection" and asyncronously (you still have a Client / Server relation, but the Client can be as talkative as needed). If you follow this blog, I've already talked about <a href="http://matthieu-totet.fr/Koumin/2014/06/15/lets-play-gephi-streaming-api-the-hidden-websocket/">Websocket and the Gephi Streaming plugin</a> , where I explain that Gephi was able to push data of it's own graph to client over websocket protocol.</p><p>Now, we want the opposite. It's possible...... and it works smooth !</p><p>What we need to test is a simple http web page, that will connect to Gephi and send data to generate a Graph.</p><p>Gist file : https://gist.github.com/totetmatt/49d2c5dd243918068320</p><pre><html>
<head>
<script type="text/javascript">
window.onload = function () {
// Gephi Streaming Master should be launch in your computer
var websocket = new WebSocket("ws://localhost:8080/workspace0?action=updateGraph")
websocket.onopen = function (event) {
// Sending event following the API and message structure defined
websocket.send('{"an":{"a":{"label":"a"}}}')
websocket.send('{"an":{"b":{"label":"b"}}}')
websocket.send('{"ae":{"ab":{"source":"a","target":"b"}}}')
randomGenerate()
};
// A Quick & Dirty example to see the "real-time" graph
function randomGenerate(){
setTimeout(function(){
var test = (Math.floor( Math.random() * 50 ) + 1)%2 ;
for (i = 0; i < 10; i++) {
var id = Math.floor( Math.random() * 50 ) + 1 ;
websocket.send('{"an":{"'+id+'":{"label":"'+id+'"}}}')
}
for (i = 0; i < 10; i++) {
var source = Math.floor( Math.random() * 50 ) + 1 ;
var target = Math.floor( Math.random() * 50 ) + 1 ;
if(test===0) {
websocket.send('{"ae":{"'+source+'-'+target+'":{"source":"'+source+'","target":"'+target+'"}}}')
} else {
websocket.send('{"de":{"'+source+'-'+target+'":{"source":"'+source+'","target":"'+target+'"}}}')
}
}
randomGenerate()
}, 2000)
}
// Triggered when we received a message
// Here Gephi is actually propagating any changes to all the client (including yourself)
// It can be usefull but can also be ignored
/*
websocket.onmessage = function(message){
console.log(message)
}
*/
}
</script>
<title>Websocket Gephi</title>
</head>
<body>
Run Gephi and Gephi Streaming Master, then refresh this page.
</body>
</html></pre><p>(Pro Tip, Websocket isn't bounded to javascript within Html page, if you try with other languages with a good websocket library, it will works the same way)</p><p>The graph is quite trivial, but you just need to extand that with a backend or another JavaScript process within your page to generate complex graph data.</p><p>We still need to connect to the master, why is it an improvment then ?</p><p>It's not out of the box, but actually your html page can be a "proxy" page that do the bridge between a server and a gephi instance.</p><figure class="alignnone wp-image-516 size-full">Which means you can create application with this architecture :<a href="https://totetmatt.github.io/media/posts/37/streamWebSocket.png"><img loading="lazy" src="https://totetmatt.github.io/media/posts/37/streamWebSocket.png" sizes="(max-width: 48em) 100vw, 768px" srcset="https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-xs.png 300w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-sm.png 480w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-md.png 768w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-lg.png 1024w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-xl.png 1360w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket-2xl.png 1600w" alt="streamWebSocket" width="769" height="113"></a></figure><p></p><p>With this, you can generate a graph within your remote server, send it to all clients and then the client decide where to send the data. No too complex operation for both part, and no big issue with Gephi, as it runs as a master.</p><p>We can even think about having data processing within the html page, with the advantage that you could let the web application doing a custom computation before generating the graph.</p><figure class="wp-image-518 size-full"><a href="https://totetmatt.github.io/media/posts/37/streamWebSocket21.png"><img loading="lazy" src="https://totetmatt.github.io/media/posts/37/streamWebSocket21.png" sizes="(max-width: 48em) 100vw, 768px" srcset="https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-xs.png 300w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-sm.png 480w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-md.png 768w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-lg.png 1024w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-xl.png 1360w, https://totetmatt.github.io/media/posts/37/responsive/streamWebSocket21-2xl.png 1600w" alt="streamWebSocket2" width="918" height="585"></a></figure>This is madness ? THIS IS LIVE GRAPH !<p></p><h1>Websocket vs REST call</h1><p>What is the advantage of this technics over the old one that consist to generate one call per actions (used in Naoyun for example) ?</p><p>On the main advantage is technical : It creates a unique connection in Gephi that stays until it stops. I didn't perform proper test, but I might expect it helps a lot the JVM in terms of memory an garbage collector.</p><p>It doesn't really solves the fact that you need to have a Gephi Master that listen to event rather than a consumer, but the solution proposed here is quite convenient without touching the source of the plugin.</p><h2>Next hack</h2><p>With the differents post about Gephi Streaming Plugin, I think we are reaching the limit of hacking around the plugin. Most of the improvment need to be coded in the plugin. I've tried several time to contribute to the code, but I never manage to get used with the Gephi Java API and the way the plugin is coded.<br>With the release of Gephi 0.9 approaching, and the various change it will bring, I think spending time on it won't be a good bet. I prefer to wait the release of Gephi 0.9, check the compatibility, and try to release a complete new Streaming Plugin. Mainly could be nice to have a default client / server websocket + other new broker system like Kafka or Redis pubsub. The internal data managment need to be review also to be more flexible and stable than today.</p><p>It doesn't mean no more fancy live graph ! There is multiple technologies evolution today that fit exactly into subject of live graph analysis and that will be interesting to deep dive, like Spark with the Streaming and GraphX module for live graph analysis.</p><p>One of the next challenge is to "get rid" of Gephi by going full web, which sounds promising with libraries like Linkurious and Vivagraph.js .</p><p>Stay tuned :-)</p></div><footer class="wrapper post__footer"><p class="post__last-updated">This article was updated on 20/06/20</p><ul class="post__tag"><li><a href="https://totetmatt.github.io/gephi/">Gephi</a></li><li><a href="https://totetmatt.github.io/stream/">Stream</a></li></ul><div class="post__share"></div><div class="post__bio bio"><div class="bio__info"><h3 class="bio__name"><a href="https://totetmatt.github.io/authors/totetmatt/" class="invert" rel="author">Totetmatt</a></h3></div></div></footer></article><nav class="post__nav"><div class="post__nav-inner"><div class="post__nav-prev"><svg width="1.041em" height="0.416em" aria-hidden="true"><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#arrow-prev"/></svg> <a href="https://totetmatt.github.io/lets-play-gephi-dbpedia-rdf-sparql-and-your-favorite-actors.html" class="invert post__nav-link" rel="prev"><span>Previous</span> Let's Play Gephi : Dbpedia, RDF, Sparql and your favorite Actors</a></div><div class="post__nav-next"><a href="https://totetmatt.github.io/request-for-comments-reference-map.html" class="invert post__nav-link" rel="next"><span>Next</span> Request for Comments Reference Map </a><svg width="1.041em" height="0.416em" aria-hidden="true"><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#arrow-next"/></svg></div></div></nav><div class="post__related related"><div class="wrapper"><h2 class="h5 related__title">You should also read:</h2><article class="related__item"><div class="feed__meta"><time datetime="2016-04-25T20:33" class="feed__date">16/04/25</time></div><h3 class="h1"><a href="https://totetmatt.github.io/twitter-streaming-importer-naoyun-as-a-gephi-plugin.html" class="invert">Twitter Streaming Importer : Naoyun as a Gephi Plugin</a></h3></article><article class="related__item"><div class="feed__meta"><time datetime="2015-09-06T22:47" class="feed__date">15/09/06</time></div><h3 class="h1"><a href="https://totetmatt.github.io/lets-play-gephi-dbpedia-rdf-sparql-and-your-favorite-actors.html" class="invert">Let's Play Gephi : Dbpedia, RDF, Sparql and your favorite Actors</a></h3></article><article class="related__item"><div class="feed__meta"><time datetime="2014-06-15T09:55" class="feed__date">14/06/15</time></div><h3 class="h1"><a href="https://totetmatt.github.io/lets-play-gephi-streaming-api-the-hidden-websocket.html" class="invert">Let's Play Gephi : Streaming API - The 'hidden' Websocket</a></h3></article></div></div></main><footer class="footer"><div class="footer__copyright"><p>Powered by <a href="https://getpublii.com" target="_blank" rel="nofollow noopener">Publii Static CMS</a></p></div><button class="footer__bttop js-footer__bttop" aria-label="Back to top"><svg><title>Back to top</title><use xlink:href="https://totetmatt.github.io/assets/svg/svg-map.svg#toparrow"/></svg></button></footer></div><script>window.publiiThemeMenuConfig = {
mobileMenuMode: 'sidebar',
animationSpeed: 300,
submenuWidth: 'auto',
doubleClickTime: 500,
mobileMenuExpandableSubmenus: true,
relatedContainerForOverlayMenuSelector: '.top',
};</script><script defer="defer" src="https://totetmatt.github.io/assets/js/scripts.min.js?v=f4c4d35432d0e17d212f2fae4e0f8247"></script><script>var images = document.querySelectorAll('img[loading]');
for (var i = 0; i < images.length; i++) {
if (images[i].complete) {
images[i].classList.add('is-loaded');
} else {
images[i].addEventListener('load', function () {
this.classList.add('is-loaded');
}, false);
}
}</script></body></html>