requirements: use pandas >= 2 #128
Conversation
|
Looks like only 1 test failure, in fairly recent code and er... >2000 warnings. Probably from hitting the same code over and over. |
|
At present, navis works with both pandas 1 and 2. Locking to 2 would enable some optimisations (e.g. using the arrow backend has some speed and memory advantages), but could possibly break some downstream users' workflows. Of course, they are welcome to use an older navis version (the last pandas1-compatible version should be clearly signposted). Implementing the optimisations conditionally based on which pandas version was installed would be a headache. |
|
I don't have any strong opinions in this case. I've been using pandas >=2 for a while and haven't had any major issues - neither with Did you have anything specific in mind re speed and memory advantages? |
|
I suppose there's no rush on this while everything is still working, but I think that if an incompatibility were to arise, pandas 1 should probably be put on the chopping block before too much effort is expended accounting for both. Pandas' integration with arrow is a work in progress but is already faster for a number of operations. I think at the moment, pandas would just use arrow instead of numpy arrays for columns rather than directly wrapping an arrow table; I also think it doesn't directly support arrow map, struct, and list types for now (it just turns them into python dicts, dicts, and lists). The advantage is probably more theoretical for now. |
If pandas 2 works without much effort, we should use it as it can be much more efficient and sane with the arrow backend. If it doesn't work, we should constrain the install requirements.