Unable to parse BASE error on TTree (TBaskets in TTree) #235
Labels
bug
Something isn't working
Comments
|
If needed we can provide an example file to reproduce this error |
|
example file would be great |
|
Thanks for looking into this. |
|
@tamasgal because he recently refactored those code. From what I can tell, it's not a new issue, but the error message is now obscure, before julia> LazyTree("/tmp/gears.root", "t")
ERROR: UndefVarError: `TAttMarker_1` not definedwhich we've seen many times. But in |
|
idk why Geant4 would write out files like that, I think they're including legacy metadata not used, a simple re-procesisng with In [18]: import uproot as up
In [19]: with up.recreate("/tmp/gears2.root") as f_out:
...: f_in = up.open("/tmp/gears.root")["t"]
...: f_out["t"] = {k: f_in[k].array() for k in f_in.keys()}julia> LazyTree("/tmp/gears2.root", "t")
Row │ xx npdg np k m y et nxx p nst ⋯
│ SubArray{Float6 Int32 Int32 SubArray{Float6 Int32 SubArray{Float6 SubArray{Float6 Int32 SubArray{Float6 Int ⋯
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ [-39.4, -39.1, - 4 4 [2600.0, 2600.0, -1 [9.24, 14.0, 72. [] 4 [2600.0, 2600.0, 4 ⋯
2 │ [-36.8, -35.3, - 5 5 [2600.0, 2600.0, -1 [6.12, 8.95, 48. [] 5 [2600.0, 2600.0, 5 ⋯
3 │ [-37.3, -36.1, - 9 9 [2600.0, 2600.0, 1 [6.71, 9.85, 25. [1860.0, 1860.0] 9 [2600.0, 2600.0, 9 ⋯
4 │ [-32.8, -29.4, - 10 10 [2600.0, 2600.0, 1 [2.28, 3.36, 11. [1910.0, 1910.0] 10 [2600.0, 2600.0, 10 ⋯
5 │ [-33.6, -30.6, 4 5 5 [2600.0, 2600.0, -1 [2.93, 4.29, 20. [] 5 [2600.0, 2600.0, 5 ⋯
6 │ [-36.3, -34.6, - 15 15 [2600.0, 2600.0, -1 [5.61, 8.18, 30. [] 15 [2600.0, 2600.0, 15 ⋯ |
|
Gotcha, this is very helpful for now, thanks! |
|
this is the change from 1->2. Content-wise I don't see any difference tbh, so idk... |
|
Yes, the current error is suboptimal. The main issue here is that we have very little test coverage for 32bit ROOT files and this is one of those: I need to dig deeper, but it's apparently using a TTree v5 and we only have support for more recent versions. I added the bootstrap logic for TTree v5 but it's not even hitting the TTree parsing, so it already chokes before that. Here is the full debug log of the file opening, as we can see, a bunch of stuff is parsed somehow but it's unclear yet what's wrong. I'm having a look... |
we can error on this if we can parse 0% of them |
|
Currently it's not even reaching the TTree parsing, so the problem is before that. Something is different in the 32bit version |
are you referring to the distinction in file header, or is it something else |
|
Yes and there might be some other differences downstreams. I am still checking what's going on. The TTree v5 logic is in-place, but I did not manage to reach that part yet. I will work on it this night. |
|
ok it sounds pretty bad that we categorically cannot process "small" root file, I'm surprised we don't run into this more often.
|
|
Yes indeed but we will figure it out ;) |
|
Actually I was wrong. Almost every file in our test-suite is a 32bit ROOT file 🙈 What I've just found is that Now the funny part is that a number below 1000 means "32bit" ROOTDirectoryHeader, and above it's 64bit. Somehow this "gears.root" file is a 32 bit ROOT file with a 64bit ROOTDirectoryHeader, because it reads I am very confused... let's dig deeper ;) |
|
Here is an example showing the here are also some differences (notice TKey64 vs. TKey32), but as you can see, the entry is (most probably) correctly read. At least the TTree shows the correct name ;) |
|
I got a bit further but I don't fully understand what's happening. I find random I added TTree v5, TBranch v8 and AttMarker v1. With that, the metadata is correctly parsed but then it chokes on some baskets, as seen below, with a proper error now. The basket parsing is not done via the streamer logic in UnROOT, so I need to figure that out. I don't want to ping Jim yet, but that will be my last resort. First I'll need to dig through all my notes and conversations with him |
|
I'll continue here: #236 |
|
OK found the quote: scikit-hep/uproot3#401 (comment) I'll check out the TBasket implementation in Rust now. |
|
Jerry pointed me to another comment of Jim with more information: #14 (comment) I forgot all these stuff, I need to work on my memory 🙈 Anyways, long story short, #236 already set the pavement to read out the baskets, now they need to be used when the number of entries is not what's written in the metadata. Meaning that the data in these ttree-tbaskets needs to be concatenated (they are uncompressed) to the actual basket data. That should be it. I will try to finish it this week. |
|
Thanks so much! |
tamasgal
changed the title
|
A little update: when opening the (KM3NeT) file, which also has TBaskets in the TTree, The file attached to this issue gives one warning (nothing else): |
|
The basket recovery mechanism is finished: #238 We are now able to fully read the KM3NeT files with baskets inside the trees. However the file attached to this issue still causes issues. I have a similar problem with another file I found in the SciKit-HEP test data, which is certainly helpful 😉 I'll keep you updated. |
|
At the moment I am stuck at the I am also pretty sure that I found the place where things go wrong and I have the feeling I am pretty close to the solution. I printed the The |
|
Hi @tamasgal thanks a lot for all the efforts! It's highly appreciated! I just tried your recent changes with the (Geant4 generated) file I mentioned here. It now gives the following error message: |
|
Thanks @Cornelius-G that's helpful! I will continue this nicht. I was fiddling with the bits and bytes all day long, it really annoys me and I want to solve it 😅 |
|
Sorry for being silent so long, I still have no news. I got a little bit further yesterday but not working patch yet... :( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With a fairly simple TTree generated by gears we (@fhagemann and me) get this error
Is this a known thing?
The text was updated successfully, but these errors were encountered: