(I'm putting a comment here, for lack of a better place.)

The implementation of _get_from_value() and _get_to_value() currently depends on the name of the file.

This is too brittle, and will be hard to maintain.

Better: auto-detect the values by reading the file itself. It's not that expensive, and will be far better to work with. This is what we've done in the past; and we can re-use that past code. Details:

• A baseline version: simply load the whole file, and read the first entry and last entry.
• Even faster: open the file but don't load it; then read either the second line (for "from" value); or rapidly iterate to the last line and read it (for "to" value).
• The latter has already been built for you! It's how we used to do things, before porting to polars.
  • It's in the file pdutil.py, circa Nov 23, 2023
  • Function oldest_ut() finds the "from" value
  • Function newest_ut() finds the "to" value
  • They both rely on lower-level helper functions. Nice and clean:)
  • And it's all unit-tested too. See test_pdutil.py from that time
  • The function load_csv() may be useful too

We create a folder for each dataset and populate them with files named according to the timestamp values of their data. Each file contains 1000 rows of data.

test_data_folder/x_from_0000000001_to_00000212112.csv
test_data_folder/x_from_00000212112_to_00004912112.csv
test_data_folder/x_from_00004912112_to_00408912312.csv

A baseline version: simply load the whole file, and read the first entry and last entry.

so actually _get_from_value and _get_to_value methods do not read the same file, they detect the last file and return the "from" or "to" value