Graphing app..

[MB6]

I've made a commonscontrol api graphing app, you can find it at mb6/commonsgraph. Please check it out.

[wrongu]

Very cool! Takes forever to install all of the dependencies, but graphtk.py ran fine once I got everything. (frz.py and commonsgraph.py both break when run from the command line, FYI)

I have to ask, though - what's the plan for using this app? Seems very similar to what the website's analyze page already does.

[MB6]

Try to run the webapp for over one month of data...

-Max B.

[MB6]

frz is broken because it is me trying to distribute on windows without installing anything... bbfreeze is unmaintained and doesn't work.

[MB6]

I could not use ciso8601, but it is the fastest iso 8601 timestamp parser.. i need to make a requirements.txt to speed that up (you just do pip install -r requirements.txt)

[wrongu]

Good point. The long-term solution should be to fix the site.

In terms of downloading big CSV files, using generators to stream rows back would be ideal. The reason that hasn't been done yet is because the split_on_indexes function is a batch operation. The first step would be to streamline that function (for example, we could have it generate the indexes quickly based on only the most recent data points analogous to how we made /data/api/systems faster)

In terms of plotting tons of points with highcharts, this appears relevant.

I'll be working on these things over winter break (when school lets up)

[wrongu]

if you use virtualenv, requirements are easy. pip freeze > requirements.txt

[wrongu]

requirements:

altgraph==0.9
bbfreeze==1.1.3
bbfreeze-loader==1.1.0
ciso8601==1.0.1
matplotlib==1.4.2
mock==1.0.1
nose==1.3.4
numpy==1.9.1
pyparsing==2.0.3
python-dateutil==2.2
pytz==2014.10
requests==2.4.3
six==1.8.0
wsgiref==0.1.2

[MB6]

BTW, try using that app and make a request for something with indexes for 10 or so months, all that waiting time is the server. I checked.

[wrongu]

Right... Big queries are definitely a bottleneck. Same problem with too much batch processing. There are 2 cards on the ol' Trello that would address this: caching queries and streaming them