Discussion: Hyperspy metadata SEM vs TEM #2095

sem-geologist · 2018-11-19T12:02:51Z

It is probably not an issue to most of the users, but I find the current hyperspy metadata structure to be a bit clumsy. My main issue is with SEM and TEM leafs. These are like singletons. It makes the tree unnecessary more complicated and adds additional level of depth. I have not seen even a single case, where acquisition instrument would have both SEM and TEM (and EELS and EDS under both of these). I think simplifying the tree structure would simplify parsing of files (particularly Bruker files where are no information about what kind of instrument the EDX detector is mounted on). SEM or TEM leaf could be refactored into simple attribute. i.e. Acquisition_instrument.microscope_type. The children of SEM/TEM could be moved one level up in the tree. Would this be possible in next_major version?

The text was updated successfully, but these errors were encountered:

jlaehne · 2020-08-31T12:33:50Z

Good point. SEM could simply become another possible value of the acquisition_mode parameter that distinguishes between TEM/STEM.

jlaehne · 2021-03-19T13:52:25Z

We're discussing metadata for luminescence spectroscopy in LumiSpy/lumispy#53 and come to a similar conclusion as here. The metadata structure seems to date from a time where EELS and EDX in TEM were the only applications. The overall structure is not so adapted to what HyperSpy wants to be nowadays, open to different instrument types and methods. So maybe, we should think of a reorganisation here @ericpre @francisco-dlp @jordiferrero

sem-geologist · 2021-04-05T06:17:47Z

Again, additional problem with this clumsy metadata structure can be experienced when loading msa files produced with DTSA-II, which contains data_type='EDS'. Is there any EDS system which knows if it is installed in TEM or SEM at all?

jlaehne · 2022-09-14T12:57:41Z

I think the metadata structure should become something like we did for LumiSpy:
https://lumispy.readthedocs.io/en/latest/user_guide/metadata_structure.html

The basic idea is to split detection and excitation, as it would more versatile when different instruments can be used for excitation, but the types of detectors that can be used are the same.

The simplest step would be to just move Detector one level up in https://hyperspy.org/hyperspy-doc/current/user_guide/metadata_structure.html
However, as there is also a lot of common metadata between SEM and TEM, we should probably have a more general EM node that joins the two, where the acquisition_mode does the dinstinction between SEM/TEM/STEM.
Alternatively, a much more general approach would be to counter the Detector node with a Source node. The distinction between SEM/TEM (X-ray source for XRF, laser for optical spectroscopy, ...) would come in an source_type (or just type) field. Then, also EDS/EELS should not be subnodes of Detector, but the value of the detector_type field with the other metadata on the same level.

jlaehne · 2022-09-14T13:06:57Z

Ad 1.:

├── Acquisition_instrument
│   ├── SEM
│   │   ├── beam_current (nA)
│   │   ├── ...
│   │   ├── Stage
│   │   │   ├── rotation (º)
│   │   │   ├── ...
│   │   └── working_distance (mm)
│   └── TEM
│   │   ├── Biprism
│   │   │   ├── azimuth_angle (º)
│   │   │   ├── position
│   │   │   └── voltage (V)
│   │   ├── acquisition_mode
│   │   ├── beam_current (nA)
│   │   ├── ...
│   │   └── Stage
│   │       ├── rotation (º)
│   │       ├── ...
│   ├── Detector
│   │   ├── detector_type
│   │   ├── EDS
│   │   │   ├── azimuth_angle (º)
│   │   │   ├── ...
│   │   └── EELS
│   │       ├── aperture (mm)
│   │       ├── ...

jlaehne · 2022-09-14T13:07:15Z

Ad 2.:

├── Acquisition_instrument
│   ├── EM
│   │   ├── acquisition_mode
│   │   ├── beam_current (nA)
│   │   ├── beam_energy (keV)
│   │   ├── ...
│   │   ├── Stage
│   │   │   ├── rotation (º)
│   │   │   ├── ...
│   │   └── working_distance (mm)
│   ├── Detector
│   │   ├── detector_type
│   │   ├── EDS
│   │   │   ├── azimuth_angle (º)
│   │   │   ├── ...
│   │   └── EELS
│   │       ├── aperture (mm)
│   │       ├── ...

jlaehne · 2022-09-14T13:07:39Z

Ad 3.:

├── Acquisition_instrument
│   ├── Source
│   │   ├── source_type
│   │   ├── beam_current (nA)
│   │   ├── beam_energy (keV)
│   │   ├── ...
│   │   ├── Stage
│   │   │   ├── rotation (º)
│   │   │   ├── ...
│   │   └── working_distance (mm)
│   ├── Detector
│   │   ├── detector_type
│   │   ├── azimuth_angle (º)/aperture (mm)
│   │   ├── ...

jlaehne · 2022-09-14T13:12:49Z

I would favor 3, as it is the most versatile and concise solution -- leaving out a lot of unneccesary nodes, while all the information will be contained. source_type and detector_type would determine what other leafs/nodes follow.

jlaehne · 2022-09-14T13:23:13Z

Alternative to source/detector could be excitation/detection.
source_type/detector_type could just become type to make it even simpler.

jordiferrero · 2022-09-18T11:11:07Z

I would favor 3, as it is the most versatile and concise solution -- leaving out a lot of unneccesary nodes, while all the information will be contained. source_type and detector_type would determine what other leafs/nodes follow.

Is this something worth discussing/implementing within the first version of RosettaSciIO #1978?

jlaehne · 2022-09-18T19:02:51Z

Is this something worth discussing/implementing within the first version of RosettaSciIO #1978?

Yes, that is why I added it to the HyperSpy v2.0 discussion: #2996

The convention is HyperSpy, though parsing is done in rosettasciio, but if we want to change anything the split would be the right moment to do so!

jordiferrero · 2022-09-19T09:24:58Z

So what are the action points for this @jlaehne?

Agree on one new metadata standard ((1, 2 or 3 above)
Update hyperspy 2.0 documentation on metadata structure
Implement new parsing in rosettasciio
Check for any uses of metadata items in HyperSpy and relevant extensions to adapt these

Anything else?

jlaehne · 2022-09-19T11:16:12Z

Added one more check. For the new metadata standard, we would first need an oppinion from more people, in particular @ericpre and @francisco-dlp. If we break it once, we should do it right ;-)

CSSFrancis · 2022-09-19T11:33:14Z

I don't have anything to add except for to ping @jat255 as well. I know he works a lot with metadata so he might have some experience/good ideas for standardizing how we deal with metadata.

jlaehne · 2022-09-19T11:58:49Z

As a sidenote, number 3 is somewhat similar as to how it is in handled in the NeXus applications definitions: https://manual.nexusformat.org/classes/applications/index.html#application-definitions
(also, there currently is a larger effort from the German FAIRmat consortium to work on application definitions for a wider range of experimental methods: https://github.com/FAIRmat-Experimental)

sem-geologist · 2022-09-19T12:16:38Z

Source and detection sounds better than current situation. I however would go one step further and divide the metadastructure into 3 not 2 parts. That would be Source, Sample and Detection. Why? Because such structure would mimic directly most of experiments (source of radiation -> sample -> detection). Also there could be same sample, and same detector, but different (simultaneous) sources. Same Source, same detection, different samples at different angle and position.
The Stage section in current 3rd proposal being under source looks IMHO currently out of place, it could move under sample.

jlaehne · 2022-09-19T12:54:09Z

Yes indeed, putting the Stage as part of the Source is a very specific electron microscopists perspective and not very generic.

However, currently we already have Sample one level higher than Source and Detector and it kind of doesn't fall into Acquisition instrument, while Stage does.

jlaehne · 2022-11-07T07:39:07Z

@ericpre @francisco-dlp it would be valuable to have your opinion on this proposal before starting to implement it. On the HyperSpy side, this should not be too much effort, but the refactoring of rsciio plugins to correctly handle the new metadata structure would be a little more effort due to the number of formats we already support. However, in view of us aiming at a more general audience, I think a more generic and less EM-specific metadata structure there would be really valuable and worth the effort.

ericpre · 2023-09-25T07:22:26Z

Moving back to the Discussion milestone, as this is not realistic for the 2.0 release.

tjof2 added this to the Discussion milestone Jul 31, 2020

jlaehne mentioned this issue Nov 23, 2020

Add handling for CL metadata in DM reader #2590

Merged

5 tasks

jlaehne mentioned this issue Mar 19, 2021

Luminescence specific metadata LumiSpy/lumispy#53

Closed

sem-geologist mentioned this issue Mar 24, 2022

Bruker M4 Tornado X-ray fluorescence BCF file import works incorrectly #2898

Closed

jlaehne mentioned this issue Sep 14, 2022

Release 1.7.x and 2.0.0 #2996

Closed

57 tasks

jlaehne modified the milestones: Discussion, v2.0 Split Jan 21, 2023

jlaehne linked a pull request Jan 28, 2023 that will close this issue

Update metadata convention #3093

Open

7 tasks

jlaehne linked a pull request Feb 17, 2023 that will close this issue

Update metadata convention #3093

Open

7 tasks

ericpre modified the milestones: v2.0 Split, Discussion Sep 25, 2023

jlaehne mentioned this issue Dec 21, 2023

Connection with and contribution to the MaRDA metadata extractors registry hyperspy/rosettasciio#207

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discussion: Hyperspy metadata SEM vs TEM #2095

Discussion: Hyperspy metadata SEM vs TEM #2095

sem-geologist commented Nov 19, 2018 •

edited

jlaehne commented Aug 31, 2020

jlaehne commented Mar 19, 2021

sem-geologist commented Apr 5, 2021

jlaehne commented Sep 14, 2022 •

edited

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022 •

edited

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022

jordiferrero commented Sep 18, 2022

jlaehne commented Sep 18, 2022

jordiferrero commented Sep 19, 2022 •

edited by jlaehne

jlaehne commented Sep 19, 2022

CSSFrancis commented Sep 19, 2022

jlaehne commented Sep 19, 2022

sem-geologist commented Sep 19, 2022 •

edited

jlaehne commented Sep 19, 2022

jlaehne commented Nov 7, 2022

ericpre commented Sep 25, 2023

Discussion: Hyperspy metadata SEM vs TEM #2095

Discussion: Hyperspy metadata SEM vs TEM #2095

Comments

sem-geologist commented Nov 19, 2018 • edited

jlaehne commented Aug 31, 2020

jlaehne commented Mar 19, 2021

sem-geologist commented Apr 5, 2021

jlaehne commented Sep 14, 2022 • edited

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022 • edited

jlaehne commented Sep 14, 2022

jlaehne commented Sep 14, 2022

jordiferrero commented Sep 18, 2022

jlaehne commented Sep 18, 2022

jordiferrero commented Sep 19, 2022 • edited by jlaehne

jlaehne commented Sep 19, 2022

CSSFrancis commented Sep 19, 2022

jlaehne commented Sep 19, 2022

sem-geologist commented Sep 19, 2022 • edited

jlaehne commented Sep 19, 2022

jlaehne commented Nov 7, 2022

ericpre commented Sep 25, 2023

sem-geologist commented Nov 19, 2018 •

edited

jlaehne commented Sep 14, 2022 •

edited

jlaehne commented Sep 14, 2022 •

edited

jordiferrero commented Sep 19, 2022 •

edited by jlaehne

sem-geologist commented Sep 19, 2022 •

edited