ENH: Explore SIMDJSON for JSON I/O

[WillAyd]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

We have used a vendored port of ultrajson for quite some time, but I think it is starting to show its age and could use a fresh look. The SIMDJSON project looks promising for our needs and claims really faster performance:

https://github.com/simdjson/simdjson?tab=readme-ov-file

Feature Description

Leverage SIMDJSON. There is a Python package that is worth exploring, although I think to get the most out of it we are looking at a C++ extension with nanoarrow

Luckily SIMDJSON is supported in the Meson Wrap DB, and nanoarrow should be with 0.5.0. So vendoring would be less of a thing - just would integrate with Meson

Alternative Solutions

status quo

Additional Context

No response