visualization of optical flow & lead field vector on cortical surfaces

[tmedani]

Hi all,

Following the discussion that we had with @sbaillet @martcous and Leo (joining soon), regarding the visualization of the vectors flow on the cortical surface.
We decided to open this thread to discuss and to adapt the 'view_leadfield' for the visualization of the optical vector flow, improve it and add functionalities.

Leo will start testing and share its results and feedback soon.

Thanks

[LeoNouvelle]

Hi all,

I went through @tmedani's code for lead_field vizualisation, it is actually very similar to what was coded for optical flow. The differences I observed are:

• For lead field, the flow is displayed on each vertice of the cortex // For optical flow the vectors are filtered so that only those of highest norm are visualized.
• For lead field, the vector's display scale is readapted at each image // For optical flow the vectors are scaled across each image of the time interval to get a sense of when optical flow is most important. This induces visualization problems, the data can be unreadable.

The great advantage of view_leadfields over optical flow vizualisation is the user interface through keyboard shortcuts. A generalization of this interface to the display of vector fields in Brainstorm could be a good improvement of the software.

Do you agree ?

Thanks,

[sbaillet]

Thanks Leo: I agree. Sylvain. -

…

________________________________ From: LeoNouvelle <notifications@github.com> Sent: Monday, April 6, 2020 11:44 AM To: brainstorm-tools/brainstorm3 <brainstorm3@noreply.github.com> Cc: Sylvain Baillet, Dr <sylvain.baillet@mcgill.ca>; Mention <mention@noreply.github.com> Subject: Re: [brainstorm-tools/brainstorm3] visualization of optical flow & lead field vector on cortical surfaces (#277) Hi all, I went through @tmedani<https://github.com/tmedani>'s code for lead_field vizualisation, it is actually very similar to what was coded for optical flow. The differences I observed are: * For lead field, the flow is displayed on each vertice of the cortex // For optical flow the vectors are filtered so that only those of highest norm are visualized. * For lead field, the vector's display scale is readapted at each image // For optical flow the vectors are scaled across each image of the time interval to get a sense of when optical flow is most important. This induces visualization problems, the data can be unreadable. The great advantage of view_leadfields over optical flow vizualisation is the user interface through keyboard shortcuts. A generalization of this interface to the display of vector fields in Brainstorm could be a good improvement of the software. Do you agree ? Thanks, [view_panel_optical_flow(initial_view)]<https://user-images.githubusercontent.com/61158704/78577320-e58de600-77fb-11ea-80b3-41259ef610a3.jpg> [view_leadfields]<https://user-images.githubusercontent.com/61158704/78577322-e6267c80-77fb-11ea-8dea-9beea12b2895.jpg> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#277 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEOWKPJ63DSKTNBYN2WPFTDRLH2FRANCNFSM4L3BSRYA>.

[LeoNouvelle]

Hi everyone,

It could be intersting to precise the features we would like to include in the brainstorm gui for brainstorm vizualisation.

Here is a list of a few features which would eb interesting from the optical flow perspective:

• Vector length (Scaling parameter over the whole field)
• Vector width
• Vector field coloring (Colormap sensible to the norm of vectors)
• Vector field filtering (Excluding vectors of insignificant norm and those of disproportionately important norm for better vizualisation)
• Vector field smooting

Some of the features above such as vector vield filtering or vector field smoothing could havec several implementations.

• Vector field inflating
  (Implemented for optical flow, partially within Brainstorm, projects the vector flow on an inflated sphere outside the brain. Helps vizualising vectors which disapear when displayed on the brain's surface because it's non-convex irregularities.) Please find an example attached.

Do you have any other ideas of useful features to include ? In particular, @tmedani what would be useful to add for the display of lead fields ?

Thanks,

Léo

[tmedani]

Hi @LeoNouvelle
Sorry, I'm less reactive these days, I have some troubles with my internet.

I went through @tmedani's code for lead_field vizualisation, it is actually very similar to what was coded for optical flow. The differences I observed are:

Are your codes available somewhere?
or it's in your local PC?

The great advantage of view_leadfields over optical flow vizualisation is the user interface through keyboard shortcuts. A generalization of this interface to the display of vector fields in Brainstorm could be a good improvement of the software.

indeed, we can use it to add more functions, but it can have some limitations.

Here is a list of a few features which would eb interesting from the optical flow perspective:

• Vector length (Scaling parameter over the whole field) ---> 1
• Vector width ----> 2
• Vector field coloring (Colormap sensible to the norm of vectors) --->3
• Vector field filtering (Excluding vectors of insignificant norm and those of disproportionately important norm for better visualisation) --->4
• Vector field smoothing --->5
• Vector field inflating ---> 6

For the 1, 2 and 4, I think it's the easiest to integrate.
However, for 3, 5 and 6, it needs more investigations.

Also, what do you mean by smoothing (5)?
and what is the difference between Vector field inflating (6) and Vector width (2) ?

What are the different inputs data that you need in order to display this figure?

Could you share your code?
and an example of the data then I could reproduce your figure?

Thank you.
Takfarinas

[LeoNouvelle]

Hi @tmedani ,

Are your codes available somewhere?
or it's in your local PC?

I only have the codes on my local computer. I'm not sure I can/should upload code on the Brainstorm github... Perhaps it's best if you have an email adress I can send you the code + data + explanations on how to use it with brainstorm ?

What are the different inputs data that you need in order to display this figure?

For all the figures the only input I need is the brain tessalation (Surface File in Brainstorm) and the optical flow vector field (or the cortical activity which I can process in optical flow).

For the 1, 2 and 4, I think it's the easiest to integrate.
However, for 3, 5 and 6, it needs more investigations.

I totally agree.

Also, what do you mean by smoothing (5)?

The the optical flow data is noisy even when processed on smoothed sources so I am looking into applying a smoothing filter on the vectors to improve results and vizualisation. It would be nice to be able to adapt the smoothing of the vector field from the user interface while looking at it.
However I am not sure what kind of filter would work the best. A convolution mask works quite ok but I am still exploring other options.

and what is the difference between Vector field inflating (6) and Vector width (2) ?

The vector field inflating would projet the vectors from the brain's surface on to a smooth, convex sphere outside the brain (it's the case in the last image I imported). The purpose is to visualize all the vectors without any being hidden by the brain's irregularities (some vectors can be pointing towards the inside of the brain and hidden). it is less useful if we only project on fully smoothed brain surfaces.

About the user interface, would it be complicated to adapt your work with the keyboard shortcut to a visual command pannel ? Because I was thinking it could be more adapted for actions such as vector field filtering. We could integrate the pannel to the Brainstorm pannel as @martcous suggested during the call ?

Thank you,

Léo

[martcous]

About the user interface, would it be complicated to adapt your work with the keyboard shortcut to a visual command pannel ? Because I was thinking it could be more adapted for actions such as vector field filtering. We could integrate the pannel to the Brainstorm pannel as @martcous suggested during the call ?

Hi guys,

I don't think this is too complicated, so long as the parameters can be represented with one of the basic Java Swing GUI components (such as buttons, sliders, text fields, etc). I can definitely help with this once the parameters are set in stone and callback functions have been written. Depending on how many there are, we could add it to an existing panel (Surface?) or create a new one. You can have a look at the panel_surface.m function to see how it works if you're curious.

Thanks!
Martin

[ftadel]

Indeed, together with Martin, we can help you with the interface.

Please avoid adding buttons or other controls in the figures.
Prefer adding options either in the popup menu of the figure (function DisplayFigurePopup of the corresponding figure function), or in a separate option panel if you think it is really relevant (= if it is going to be used intensively by other users outside of your own local projects).

Examples for separate panels:

• panel_coordinates.m, which gets displayed when you right-click on a 3D figure > Get coordinates
  
• panel_dipoles.m, which gets added to the Brainstorm window when loading some dipoles: https://neuroimage.usc.edu/brainstorm/Tutorials/TutDipScan#Dipole_scanning

There is a panel_opticalflow.m and "Plug-in > Optical-flow" popup menu when you right-click on a "full matrix" source file... Is this what you are using?
I have never reviewed this code or used its interface. It has never been documented and I don't think it has been used much.
If you want to use this in a new project, we should plan some dusting of the code and tutorial writing.

[tmedani]

Hello all,

Thank you @ftadel & @martcous for your help

I only have the codes on my local computer. I'm not sure I can/should upload code on the Brainstorm github... Perhaps it's best if you have an email adress I can send you the code + data + explanations on how to use it with brainstorm ?

@LeoNouvelle could you either send it by email or share a Gdrive/dropbox link here?
then I can have a look and try to reproduce your results.

There is a panel_opticalflow.m and "Plug-in > Optical-flow" popup menu when you right-click on a "full matrix" source file... Is this what you are using?

@ftadel : I can't reproduce this process, what do you mean by source file?

I don't know how @LeoNouvelle do it for now,
but I think it's outside from brainstorm.

[ftadel]

I can't reproduce this process, what do you mean by source file?

Right-click on some recordings > Compute sources [2018] > Show details : select "Full results".
Right-click on the source file created > Plug-ins > [Experimental] Optical flow

@sbaillet Can you tell us what this was about?

[tmedani]

Right-click on some recordings > Compute sources [2018] > Show details : select "Full results".
Right-click on the source file created > Plug-ins > [Experimental] Optical flow

Thank you @ftadel,
as it's now this process is not working.

[LeoNouvelle]

Hello everyone,

Thank you @martcous @ftadel for your help !

There is a panel_opticalflow.m and "Plug-in > Optical-flow" popup menu when you right-click on a` "full matrix" source file... Is this what you are using?
I have never reviewed this code or used its interface. It has never been documented and I don't think it has been used much.
If you want to use this in a new project, we should plan some dusting of the code and tutorial writing.

Indeed, it's the function I've been using for visualizing vector fields.
It does need some dusting (I think it was written in 2010). Actually, if you touch any of the settings other than the time interval in the function's panel it won't work. I have spent some time understanding and correcting the problems. I now use my corrected version of the function which is only stored locally on my PC. I shared it in the google drive link below.

@tmedani in fact you are also right because I mainly work outside of Brainstorm on re-processing the flow field I want to visualize. I then reimport the data in Brainstorm and visualize it using the optical flow plugin @ftadel mentionned. I keep switching in/out of Braisntorm which is quite time consuming.

@LeoNouvelle could you either send it by email or share a Gdrive/dropbox link here?
then I can have a look and try to reproduce your results.

Of course, here is a link towards useful functions and data : https://drive.google.com/open?id=1zV9YgUcP8WV4zcv6j-ET-LvJEWXjPJE6
The different files are:

• Subject1Avg and DefaultCortex_15002V, data exported from Brainstorm, respectively a full dource matrix and the default brain surface.
• leo_hhdronbraintest is a function outside of Brainstorm to visualize the vector fields and their HHD decomposition. You will have to import the data as well as hhdr and hhdr_recursive functions to run the complete code.
• The corrected panel_opticalflow_ function I use for the visualizing part. It is a function intern to Brainstorm. Normally you shouldn't need to replace the version of the function stored in your Brainstorm to visualize a vector field. Following @ftadel's explanations should be enough to visualize an optical flow fiels (without touching the settings in the optical flow panel), it works on my PC.
  If you want to visualize the inflated sphere results we talked about above you need to replace the 'panel_opticalflow_' function in your Brainstorm with the one in my drive and select the option in the optical flow panel.
  We can arrange some time when I could show you if it still doesn't work on your PC @tmedani ?

Depending on how many there are, we could add it to an existing panel (Surface?) or create a new one. You can have a look at the panel_surface.m function to see how it works if you're curious.

Thanks @martcous, I will start studying the function !

[ftadel]

@LeoNouvelle Are these developments you make with the goal of distributing them with Brainstorm later, or is this just for one single specific project?
In the first case, it would be good if we could discuss the general ideas for the implementation before you go too far in the coding, so that the final integration is smooth and requires as little recoding as possible.

Have you already presented this at one of our biweekly Brainstorm meetings? (I'm sorry if this is one I missed, in that case maybe we could catch up with a short skype meeting)

[sbaillet]

Hi Francois: This is an effort to revive and update the optical flow developments we’ve done over the years, with now full integration in Brainstorm. Sylvain.

[tmedani]

Hi @LeoNouvelle,

I have tried with your files but it does not work correctly.

I tried to change the code but it's not efficient ...

Maybe we need to set a discussion together and show me how you use it.

Thanks

[LeoNouvelle]

Hi @tmedani,

Sorry to hear that it didn't work. Perhaps I made changes in other functions related to panel_opticalflow_ on my computer I don't remember.

I would be happy to show you. When are you available ? I am pretty much availbale at any time these days ...

See you soon,

Léo

[tmedani]

Hi Leo,

ok, we can have a discussion tomorrow, I will send you a Zoom link and the time by email.

best

[tmedani]

Hi @LeoNouvelle

it seems that the vectors are not displayed on the cortex vertices but moved on the normal direction of the vertices.

• Here is the view after removing the shift (on line 753)
  

Regarding the second function "quiver3" at line 656 it seems to be used to export the results (need to check).

So, for now, the main modifications would be around the sub-function "PlotOpticalFlow"

[LeoNouvelle]

Hi Takfarinas,

Glad you were able to reproduce the process ! Thanks for the figure, it looks good. So for the vector representation there isn't a problem of mismoatching as we were afraid of but a slight shift along the vertices Normals. (By the way, I just checked and the vertices normals do change and adapt when we apply a smoothing on the surface).

Also, I have attached the files I said I would send you to the google drive (24/04 file):
https://drive.google.com/open?id=1zV9YgUcP8WV4zcv6j-ET-LvJEWXjPJE6. Inside are the leodisplayfields function, the smoothed surface and a ResultsFile to use it. (Just so it isn't confusing, the colormap under the vectors is no longer the current. It is a potential field, the vector displayed are equal to the gradient of the field. This is why I don't display absolute values.)

I agree, the interesting part of the code is the PlotOpticalFlow sub-function for now. The last quiver3 you talked about is for a sub-function ("Print Stable/Transition States" on the panel) which isn't useful yet.
Once we have coded a vector display panel in Brainstorm it willl take over most of the problems !

I went (brieffly) through the panel_surface file you were talking about @martcous. I think I got a sense of the code. I also saw there are panel_record, panel_filter and panel_scout functions. I guess the final goal is to add a panel_vectors to the software. I will try to go over it again and come back with a few questions if that's all right ?

Thanks,

Léo

[ftadel]

I guess the final goal is to add a panel_vectors to the software. I will try to go over it again and come back with a few questions if that's all right ?

Not necessarily, it depends how many things you have to put in there. It could also be floating panel (not docked in the Brainstorm main window).
We should discuss before you make any decision of implementation for the GUI, so that we don't have to redo the same thing multiple times.

Would you be available for presenting this at our next Brainstorm meeting (May 6th)?
If you need to make decisions and move on with the developments sooner than this, we could have a skype meeting next week.

[LeoNouvelle]

Would you be available for presenting this at our next Brainstorm meeting (May 6th)?
If you need to make decisions and move on with the developments sooner than this, we could have a skype meeting next week

Yes I am available on May 6th, @sbaillet already asked me to present.
No it's fine I can work on other things until then. I'm just anticipating but there is no hurry.

We should discuss before you make any decision of implementation for the GUI, so that we don't have to redo the same thing multiple times

Yes, I agree of course.

[LeoNouvelle]

Hi everyone,

I just wanted to reactivate the conversation after the conversation we had at the brainstorm meeting wednesday.
For the visualization of optical flow, we had discussed the different features useful to be modified in Brainstorm earlier on in the conversation with @tmedani :

Here is a list of a few features which would eb interesting from the optical flow perspective:

Vector length (Scaling parameter over the whole field) ---> 1
Vector width ----> 2
Vector field coloring (Colormap sensible to the norm of vectors) --->3
Vector field filtering (Excluding vectors of insignificant norm and those of disproportionately important norm for better visualisation) --->4
(Vector field smoothing --->5)
(Vector field inflating ---> 6)

(5 and 6 were more ideas I proposed but didn't really seem essential. 5 wasn't actually well defined.)

@martcous @ftadel, do those features seem ok to add to Brainstorm without to much trouble ? What would be the best user interface to implement ?

Thank you

Léo

[martcous]

Hi Léo,

Thank you for the summary. Is it possible to detail the type of object you would like each option
to be and their possible values? For example, the vector length could be a slider (then what are its min and max values?), vector width either a slider or a text field, etc.

Also, did we decide that we would create an entirely new panel for optical flow, or add it to one dynamically like the Display panel?

Thanks!
Martin

[ftadel]

I've been working recently on very similar things with the display of the FEM conductivity tensors, so I have a good idea of what are the representation capabilities of the various graphic objects capable of drawing lines in Matlab.

We will need a clear test case in order to discuss what we want to implement.
@LeoNouvelle Can you please describe one specific example dataset? The easiest would be to start from the introduction tutorial dataset (auditory oddball), but if it doesn't work for you, please package an example dataset (maybe fully imported in Brainstorm then exported as .zip with File > Export subject - but please keep it small)
=> Describe precisely how to get to the file you want to display from the tutorial protocol (list of clicks, options to select shown in screen captures) to make it easier to reproduce exactly the same files as you.

Second step: we need to separate the display options from the computation options. For example, smoothing or inflating a vector field do not seem like simple display options to me...

Vector length (Scaling parameter over the whole field)

A fixed size maximum vector with all the other vectors scaled wrt the max norm is not a good option for you? You'd need to set the actual length of the maximal norm on which all the vectors are scaled?
If so, I don't think this is an option that would be manipulated very intensely by the few users who would use this feature. It's probably not worth the investment in an interactive slider in a dedicated panel (which represents a lot of code volume, and maintenance over the years). A right-click on the figure > "Vector field options > Set maximum vector length" could be enough.

Similarly to the 2DLayout display options:
https://github.com/brainstorm-tools/brainstorm3/blob/master/toolbox/gui/figure_3d.m#L1544
As long as this is documented in the tutorial explaining this tool, it's OK.

Vector width

Same thing: This is a display option that would be called once or twice, not continuously (like the "smooth surface" or "transparency" sliders).
Right-click on figure > "Vector field options > Set vector width" ?

Vector field coloring (Colormap sensible to the norm of vectors)

This is achievable with the patch objects. This is the way I implemented the colored lines for representing the FEM tensors:
https://github.com/brainstorm-tools/brainstorm3/blob/master/toolbox/gui/figure_3d.m#L4597

Vector field filtering (Excluding vectors of insignificant norm and those of disproportionately important norm for better visualisation)

The use case and implementation of the filtering needs to be better described.
A real life example will help.

---

General design considerations

Are you planning to write a tutorial for these tools?
Allowing people to do everything that they could imagine doing is impossible. A good way to design an interface is to define a few use cases for which you unfold full scenarios of actions to obtain the desired result. The display options that are really needed can be derived from such scenarios. Adding options and buttons that are not strictly necessary brings more confusion and cognitive overload than anything else.
Writing a step-by-step tutorial on a real case is a good way to design these GUI storyboards. Having a clear goal always help design the tools properly. It's best to define what are all the targets in terms of display (screen captures or figures from articles) before starting the implementation.

[ftadel]

Is this moving forward?

[sbaillet]

yes.

[ftadel]

Any update on this front?

[tmedani]

the todo list is getting bigger, and pushing this one for later,
there is also a PR about this but it needs more work.
@sbaillet do you have any students candidate for this task?

[sbaillet]

We’re actively working on the research end of this dossier. Software deployment will be next. Cheers, Sylvain.

[ftadel]

@sbaillet Any updates?