Frame discontinuities in mainbrain files

[sdvillal]

Continuation of #24, to update on the issue of trials in mainbrain warping in time, possible fixes at analysis level for old files and possible fixes when writing.

This code is a mixture of ad-hoc script and reusable module wannabe to check for "continuity" problems and generate reports to assess the extent of the problem in our files and suggest test cases. Some of it parses outputs from flydra's check continuity script. Most important is this function, which we could adapt to check stuff at least in flydra, if not also during the auto-pipeline, combine and others. We also generate a couple of handy dataframes.

Current logs live here:
/mnt/strawscience/santi/mainbrain-holy-logs
There there are some h5 files with data aggregated in pandas dataframes.

I think we have at least two kinds of discontinuities:

• Old (fixed around September 2014): many trials with many duplicated frames (but different data), "jumps" of between -3 and 0 frames (0 = frame repeated). A complete summary for 20131205_174729, one of Lisa's DCN experiments, is in /mnt/strawscience/santi/mainbrain-holy-logs/mainbrain-full-logs. If we need a fix, we could just drop repeated frames when resmoothing (if we agree that won't hurt much the estimates). These are some summary stats:

Max number of jumps in an obj_id:  196
Max number of missing frames in an obj_id:  0
Max number of exceeding obs in an obj_id:  413
Seen jump sizes:  [-3.0, -2.0, -1.0, 0.0]

• New: a few trials at the beginning; these seem to just have two blocks of observations (like 3,4,5,0,1,2), so we could simply merge them in the proper order or just drop them.

I haven't really made sure that there are not other types of errors.