General Thoughts and Forward Looking

[CSSFrancis]

This comes up in offline discussions, so I figured I would post about this to let people know where I am and my future plans related to pyxem.

I'd also love to hear from other developers. What level of involvement do they plan to have in the future, what jobs have they accepted, and how life in general is going?

---

I'm finishing my PhD in the next month (or 2?), which should hopefully align with a pyxem 1.0.0 release around the same time and a publication to follow. This should leave pyxem in a nice state where it can exist as a stable, well-tested package for general 4-D STEM operations. The field as a whole has been missing that and in my opinion this is a huge barrier to continued research as packages like pyxem thrive on public or private workflows in the form of jupyter notebooks. It's part of the reason that a 2.0.0 release in hyperspy upstream was such a significant deal and part of the reason that a 1.0.0 release means a significant ammount.

Looking beyound the 1.0.0 release pyxem has quite a bit of flexibility. I recently accepted a job with Direct Electron, specifically I am working on developing accelerated methods for doing 4D STEM with a focus on GPU acceleration for doing live or simply faster data processing. That topic is quite broad/ general and I have a fair bit of flexibility in how I implement things.

4D STEM is new enough/ the analysis are fairly complex/the market is fairly small that there really are not general comercial softwares for doing analysis. In contrast the open source space there are (potentially too many) different options. From the prospective of detector manufacturers, open source provides a valuable resource as they provide cutting edge analysis for their hardware without the added cost to detector manufactures of maintaining a dedecated software team focused on tools for anaylsis. From the user standpoint, however, I can't help but feel that there is something left to be desired. I've taught quite a few people how to do 4D STEM over the last couple of years and find there are many barriers. I've tried to help fix these issues over the years but having a solution and teaching/disseminating that solution are vastly different things.

These generally boil down to:

• How do I load my data --> (Hopefully Rosettasciio fixes this problem!)
• Installing python packages, getting jupyter set up doesn't work! --> (Hyperspy-Bundle?)
• DoI have to find 100GB of RAM to view my data? --> (Lazy Processing)
• How do I visualize my data?--> (Lazy Processing + setting a navigator in hyperspy + better file saving from detector manufactures)
• I took a dataset and now 3 days later when I finally get around to looking at it doesn't look right!--> (Live processing? LiberTEM?)
• Things take too long --> (Try Zarr :) )

Just ball parking but my assuption is that ~70% of 4D STEM data is never analyzed. This leads to a great question which is should someone have to download pyxem to do something simple like a VDF or strain mapping? Does doing 4D STEM have to require some python programming experience? Should we be offering open source GUI solutions for 4D STEM?

My approach to this problem is to simplify things using disk detection and vectors. This will probably evolve with time but:

1. Make a truely fast/ robust disk detection algorithm. I want to start with a generic method that exploits the real space redundancy in peak positions and then compare that with a neural network approach optimized for speed. The hope is to be able to run on a high performance GPU at 100,000+ frames per second.
2. Expand the vector based operations in 4D STEM. This includes visualization, orientation mapping, clustering, strain mapping, DPC etc.
3. Improve GUI implementations. Support LiberTEMLive? Improve HyperspyUI for data visualization on the mircoscope using pyxem?

---

More on the nitty gritty side. What about liscenses, how to handle open-source vs corperate interests. GPL is infectious which is a blessing and a curse. On one hand open science is always the goal. On the other it tends to cause problems in many cases from an intelectual property side and many coorperations are hesisitent to touch it.

• Switching from GPL 3 to BSD 3 License orix#389
• Permissive license or LGPL? hyperspy/rosettasciio#51

I've previously thought about this and feel something like #933 would be a good step in the right direction as we could maintain a more permissive library for generic operations and then those could be seperately implemented in pyxem (or even in liberTEM/py4dstem).

I'd love to hear peoples thoughts and suggestions. I can be fairly flexible on what I work on so if you have suggestions I'd love to hear them.

@ericpre @magnunor @pc494

[pc494]

I probably won't get to a proper read of this until the weekend, but I just want to drop two notes here:

1. I think this is the correct time for this conversation
2. Congratulations @CSSFrancis on the new role :D

[CSSFrancis]

@pc494 Thanks! I'm hopeful that I will be able to support pyxem/hyperspy at least at the same level I am currently if not more but that remains to be seen. At the very least I can probably scheudle some portion of my time to review and maintain the package which is more than I can currently say :).

[hakonanes]

Congratulations on your new job, @CSSFrancis! I'm glad to hear you'll continue within the realm of microscopy and diffraction. I started working for Xnovotech on X-ray imaging and crystallography in December after handing in my PhD. Moving from academia to industry poses both challenges and opportunities; I enjoy it very much so far and still find time for OSS (maintenance, not so much development, unfortunately).

Just ball parking but my assuption is that ~70% of 4D STEM data is never analyzed.

If that's the case: wow. Seems like live processing can help mitigate that, to have at least some preliminary results right after a microscopy session. Your point 3. regarding improving existing GUI implementations is a good one. I would say yes to both questions. However, resources are limited, so supporting existing solutions (like LiberTEMLive) is a reasonable start.

Does doing 4D STEM have to require some python programming experience? Should we be offering open source GUI solutions for 4D STEM?

I'd say no, and yes... With the caveats that (1) GUI solutions should focus on simple operations to keep maintenance costs manageable, and that (2) a user finding the GUI too restrictive has to learn programming.

But, if a user wants to build their own workflows, the API should be easy to use! Sadly, this is the most overlooked aspect of software development within the pyxem packages, in my opinion. Yes, new functionality is added by academics with limited time and experience with software design (often). Even so, we must think twice about how a user can incorporate new functionality into their own workflows before deciding on the final API.

Structuring modules, classes, attributes, and methods in a way that is natural, easy to use, and maintainable is challenging and something that should be discussed when reviewing PRs to the pyxem packages.

What about liscenses, how to handle open-source vs corperate interests

I've been in favor of more permissive licensing (BSD, MIT, etc.) for a couple of years. I haven't found the time, but an approach I want to suggest for orix is to move the bits we can to separate files with permissive licensing. The LiberTEM maintainers have done this: Their software has a GPLv3 license, but their IO parts have an MIT license. OK, the software as a whole cannot be used by packages not wanting the GPLv3 license, but the code can be reused. With time, more things should have a more permissive license, allowing for use and contributions from industry people.

---

Lastly, I wish you good luck in finishing your thesis! 🚀

[CSSFrancis]

Congratulations on your new job, @CSSFrancis! I'm glad to hear you'll continue within the realm of microscopy and diffraction. I started working for Xnovotech on X-ray imaging and crystallography in December after handing in my PhD. Moving from academia to industry poses both challenges and opportunities; I enjoy it very much so far and still find time for OSS (maintenance, not so much development, unfortunately).

@hakonanes congratulations on your job! I'm glad that you are finding it rewarding! To your other point maintenance of OSS is one of the most important and helpful things. I really appricate your help!

If that's the case: wow. Seems like live processing can help mitigate that, to have at least some preliminary results right after a microscopy session. Your point 3. regarding improving existing GUI implementations is a good one. I would say yes to both questions. However, resources are limited, so supporting existing solutions (like LiberTEMLive) is a reasonable start.

Yea it's unfrotuantly even the case for my group which I consider to have relatively complete workflows. Even I am guilty of having TB of unanalyzed data sitting around. It gets worse with new users as lots of non microscopists end up taking the datasets which is actually fairly easy but then get stuck somwhere in the next couple of steps. It doesn't help that there aren't intermediary visualization steps which can be super discouraging if you don't have some reward after taking the dataset.

Does doing 4D STEM have to require some python programming experience? Should we be offering open source GUI solutions for 4D STEM?

I'd say no, and yes... With the caveats that (1) GUI solutions should focus on simple operations to keep maintenance costs manageable, and that (2) a user finding the GUI too restrictive has to learn programming.

I think thats very true. I really like the HyperspyUI even if it (maybe) falls into the box of too complex of operations with limited maintenance. But I do like the fact that you can record functions and there is a pretty natural progression from GUI to programming with the ability to record the eqivilent functions/ workflow in hyperspy. That being said people don't seem to really use it.

But, if a user wants to build their own workflows, the API should be easy to use! Sadly, this is the most overlooked aspect of software development within the pyxem packages, in my opinion. Yes, new functionality is added by academics with limited time and experience with software design (often). Even so, we must think twice about how a user can incorporate new functionality into their own workflows before deciding on the final API.

Yea I'd agree there. We benefit from consistancy across the hyperspy/pyxem/ kikchipy etc. which actually makes transition between packages quite nice from a developer standpoint. I at least feel semi competent reviewing code releated to signals in kikchipy even though I'm not very involved in development there. Orix being familar to people using mtex is nice, but maybe we are reaching a matplotlib junction where we have outgrown our inspiration in certain area. diffsims I would like to make more similar to hyperspy/pyxem. I think the idea of 1/2D simulations with navigation axes --> combinations of Phases/Rotations makes sense. I would even like to at least think about adjusting hyperspy to support visualizaiton of simulations in diffsims. There also is abtem which I think does a great job in creating modular simulations but might be in need of additional developer time/ support. I would love to think about extending abtem for kinemetic simulations and better integrating abtem into pyxem.

Structuring modules, classes, attributes, and methods in a way that is natural, easy to use, and maintainable is challenging and something that should be discussed when reviewing PRs to the pyxem packages.

Yea I think that comes with maturity continued development, and continued developers with long term plans/goals. The one thing I don't want is to make the barrier to the first code addition too high :) but that can be mitigated by more maintainers and adding fixes to peoples code.

What about liscenses, how to handle open-source vs corperate interests

I've been in favor of more permissive licensing (BSD, MIT, etc.) for a couple of years. I haven't found the time, but an approach I want to suggest for orix is to move the bits we can to separate files with permissive licensing. The LiberTEM maintainers have done this: Their software has a GPLv3 license, but their IO parts have an MIT license. OK, the software as a whole cannot be used by packages not wanting the GPLv3 license, but the code can be reused. With time, more things should have a more permissive license, allowing for use and contributions from industry people.

That is my opinion as well and why I wanted to make a feedstock for some different methods. I guess this leads me to the next question: If parts oforix were more permissive could you/ Xnovotech start to use it? Are there other parts of pyxem, kikcuipy etc. that might be relevant/ useful? I'm always more in support of permissive liscneses if there is a open dialouge (I want to know if people are using my code!) and am definately more in support of this if we can help to support contributors in making industry connections etc.

Lastly, I wish you good luck in finishing your thesis! 🚀

Thanks! It's going fairly well, just need to revise some final papers and get things out the door :).

[magnunor]

I agree on the point that most 4D-STEM data isn't really analyzed due to the data processing software being difficult to use. At least in Norway, more and more of the master and PhD students learn Python programming, and using Jupyter Notebooks during their studies. So the issue with people being intimidated by a Python command line, for example in a Jupyter Notebook, will decrease over time. I'm assuming this is the case other places as well.

Improve GUI implementations. Support LiberTEMLive? Improve HyperspyUI for data visualization on the mircoscope using pyxem?

One thing to keep in mind is that developing and maintaining GUIs is much more developer demanding compared to developing command line tools. Thus, one should be very careful before committing to something like that.

I do know the LiberTEM project have been looking into using the GUI which CEOS is developing: Panta Rhei. It is Python based, and they're planning on open sourcing it. I haven't used it much itself, but it is very extendable, and can handle live processing pretty well. @uellue, @sivborg and @sk1p would know more about this. @gguzzina is one of the developers working on this in CEOS.

Creating a 4-D STEM Feedstock Package, #933

I think this is a very good idea. The main issue is getting time to implement and maintain this.

[CSSFrancis]

I agree on the point that most 4D-STEM data isn't really analyzed due to the data processing software being difficult to use. At least in Norway, more and more of the master and PhD students learn Python programming, and using Jupyter Notebooks during their studies. So the issue with people being intimidated by a Python command line, for example in a Jupyter Notebook, will decrease over time. I'm assuming this is the case other places as well.

Improve GUI implementations. Support LiberTEMLive? Improve HyperspyUI for data visualization on the mircoscope using pyxem?

One thing to keep in mind is that developing and maintaining GUIs is much more developer demanding compared to developing command line tools. Thus, one should be very careful before committing to something like that.

I do know the LiberTEM project have been looking into using the GUI which CEOS is developing: Panta Rhei. It is Python based, and they're planning on open sourcing it. I haven't used it much itself, but it is very extendable, and can handle live processing pretty well. @uellue, @sivborg and @sk1p would know more about this. @gguzzina is one of the developers working on this in CEOS.

That's very good to hear! It might be worth thinking about how we can converge, and build out a community of support.

Creating a 4-D STEM Feedstock Package, #933

I think this is a very good idea. The main issue is getting time to implement and maintain this.

I will have to talk more with DE but I think this is something that we could help to support. I'd love to see if there is any interest from LiberTEM or CEOS. It seems like there is lots of interest in making an open source solution and it would be great to have some more open discussions on what that might look like.

[gguzzina]

I will add a few words on the CEOS angle on this.

To start: if anyone wants an offline copy of Panta Rhei to play around, just ask.

Panta Rhei was born, originally, to let our customers work effectively with the CEFID, our energy filter/spectrometer.

This brings the following requirements:

1. Hardware support: for the CEFID as well as tge different TEMs, cameras, and scan generators that are used together with the CEFID. It's a lost that keeps slowly expanding
2. Responsive and interactive GUI: the users need to be able to quickly inspect and assess their data. Is this data good enough? Can I go home and spend the next week analysing this in Hyperspy, or should I keep trying harder?
3. Basic analysis tools: the tools need to be good enough for data assessment, or for a quick report, nothing fancy. But they need to be quick, the user needs the answer within seconds, while at the microscope, or even in real time during acquisition. Examples are thickness mapping, ZLP realignment, linearised EELS quantification, centre-of-mass, virtual detector imaging.
4. Extensibile/Interoperable/Open: You can never account for all use cases, because the job of scientists is to come up with new and better ways to do things, and with new and interesting things to do. So we want to enable them to do this as flexibly as possible. Often these tools exist elsewere, (e.g. the HyperSpy family) and there's not point even trying to replicate it all, we just try to ensure easy interoperability (e.g. prz support inside hspy, hspy support inside PR). Making the whole thing open source just makes sense, we used a lot of open source libraries to develop this quickly, it only makes sense to give back something.

The GUI, as you see from this screenshot, ends up being reminiscent of some other software you may have seen before.

Point 1 may be of no interest to this community, but points 2, 3, and 4 are what motivated the interest from the LiberTEM crowd (I believe that @sk1p and @uellue can offer a bit more). They want to use Panta Rhei as a visualisation engine and GUI. Suppose you have computed a train map, and now want to extract a line profile over a certain feature... In a notebook this is a bit of work, in Panta Rhei it's two clicks. Making a plug-in dialogue with a bunch of buttons to help streamline certain operations is not too hard.

We would be very happy to see wider adoption of Panta Rhei. On the one hand, this may help us with the development, gives us wider testing in more scenarios and use-cases, and will hopefully lessen the dependency of the field on proprietary solutions. Also, selfishly, I think that if more people come in contact with Panta Rhei they will be less "scared" by our filter.

We even have some users that use Panta Rhei without having a filter, because they have a compatible camera, and they prefer Panta Rhei to the software provided by the camera vendors. This of course makes us very happy.

About opening our sources... We have made a clear commitment to publish the sources under a permissive license, but it's taking a little longer than we hoped, as there are a few things that need to be done before the complete source is ready for publication. While we believe that this is important and we are happy to support other open source projects within the community as much as possible, in the short term we need to support the experimental needs of our customers first and foremost, and for the rest we can't offer more than our "best effort" and hope it's enough.

I'd love to see if there is any interest from LiberTEM or CEOS. It seems like there is lots of interest in making an open source solution and it would be great to have some more open discussions on what that might look like.

I honestly don't think we should strive for an open source solution, but for an open source ecosystem.

From our side, it's great that we can have several developers working on Panta Rhei full time, but again our priority is with the need of our customers, and everyone is pretty busy as it is.

So we're not going to take responsibility for other projects, but we will be happy to collaborate and offer the occasional contribution (as we are already doing).

BTW we already collaborate with Direct Electron, and DE cameras can already be used within Panta Rhei.

Edit: Actually include the screenshot.

[uellue]

With LiberTEM we are working on blending live and offline processing, exactly to get quick feedback on the instrument and to remove the barrier between acquisition and analysis.

With notebooks we have hit limits on how fast one can plot (ASI CheeTah T3 does 4D STEM with 1 us dwell time), so @sk1p, @matbryan52 and I evaluated different options to build interactive GUIs for fast live processing. Panta Rhei came out on top because the shared memory interface allows fast inter-process communication, the plotting is fairly efficient, and as @gguzzina writes it already implements a lot of visualization and subsequent analysis out of the box.

Regarding interfaces between different projects, supporting interactive live processing at the current detector speeds requires the right architecture and programming model, and efficient implementations for the analysis methods. Dask likes to decide which chunk is processed when and where, and only provides a final result without intermediate updates. Live processing is driven by the data source and requires frequent updates of intermediate results for live visualization. That's why we created LiberTEM. Unfortunately it is quite a bit of work to refactor code written for offline processing to be compatible with high-throughput interactive live processing. The opposite direction is, in contrast, fairly easy. That applies both to analysis routines and file readers/data sources.

It would be great if more code would be structured for live processing so that one can create better interactive tools and reuse code across projects in the future! We've implemented full Dask support in LiberTEM to facilitate re-using LiberTEM code in HyperSpy-related code (the easy direction), but unfortunately that didn't materialize yet.