Real time plotting. Any recommendations?

[zibrov-zlobin]

I'm sure you've thought of all the possible solutions to how to best organize real-time plotting, so I was wondering if anyone could give me some tips.
I need to plot time series and possibly 2d heat plots in real time.

As I understand, the way to do it would be:

The Client retrieves data from the devices and sends it off to the Data Vault server. This creates/ updates a csv file. The data vault on the other hand at the event of new data received sends a signal to all other listening servers/clients and particularly some plotting client. The plotter then reads the csv file and append the last row to the graph.

If the dataset becomes large, wouldn't this pose speed issues, since the whole csv file needs to be read in order to update the graph?

Am I missing something?

Or the scheme to do would be to have a data_plotting server that receives data from the client without ever querying the data_Vault?
Would it be beneficial to use some sort of TSDB for this? Say influxdb + graphite/graphana? Maybe something working already exists?

Thanks,
Sasha

[DanielSank]

The Client retrieves data from the devices and sends it off to the Data Vault server. This creates/ updates a csv file. The data vault on the other hand at the event of new data received sends a signal to all other listening servers/clients and particularly some plotting client. The plotter then reads the csv file and append the last row to the graph.

That's exactly how we do it.

If the dataset becomes large, wouldn't this pose speed issues, since the whole csv file needs to be read in order to update the graph?

Well, yes, large data sets take a few moments to load. We have client-side wrappers representing a data set which manage fetching data in reasonable chunks, etc.

Am I missing something?

Doesn't sound like it.

Or the scheme to do would be to have a data_plotting server that receives data from the client without ever querying the data_Vault?

I suppose that's possible, but it hasn't been necessary for us.

We use a combination of tools for plotting. Our most time honored one is a legacy program written in Delphi. We are phasing that out in favor of a web-based system (which functions almost exactly as you've proposed). We have also used PyQT with the associated matplotlib plug-ins for logging cryostat temperature, pressure, etc.

[jayich]

For fast plotting pyqtgraph is decent. I use this for displaying camera images.

[zibrov-zlobin]

might be of interest: http://vispy.org/index.html

[DanielSank]

@zibrov-zlobin if you happen to get that working with pyqt please let everyone know.

[zibrov-zlobin]

@DanielSank there is this answer
http://stackoverflow.com/questions/27932721/how-to-embed-vispy-graph-in-pyqt
having tried it yet though.
The package looks like still in pretty early development.