Improve pit performance

[PaleNeutron]

Description

see #1671

Consider pit data, assume we have T trade days and N report_period record:

	date	report_period	value
0	2011-10-18 00:00:00	201103	0.318919
1	2012-03-23 00:00:00	201104	0.4039
2	2012-04-11 00:00:00	201004	0.403925
3	2012-04-11 00:00:00	200904	0.403925

We access PIT table in 3 Ways:

1. observe latest data each trade day

Just loop through table and keep only latest report_date value. consume O(N)

2. observe latest several report_period data for expression like P(Mean($$roewa_q, 2))

Read data file once.

• Loop through trade day, slice data[:tradeday],
  • groupby report_period, get the last item.
  • return last X item

Algorithm could be improved by loop back from the end until find X different period. But groupby use C level loop which should be faster.

3. observe specific period from each trade day

Get all data belong to given period

How Has This Been Tested?

• Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
• If you are adding a new feature, test on your own test scripts.

Screenshots of Test Results (if appropriate):

Types of changes

• Fix bugs
• Add new feature
• Update documentation

[PaleNeutron]

Anyone can fix main branch? CI fails due to main branch problem.

[Fivele-Li]

It seems that the index file mentioned here and the _next column in the data file will not be used in this PR. Are you going to delete them together?

qlib/scripts/dump_pit.py

Lines 198 to 204 in 98f569e

	if not overwrite and index_file.exists():
	with open(index_file, "rb") as fi:
	(first_year,) = struct.unpack(self.PERIOD_DTYPE, fi.read(self.PERIOD_DTYPE_SIZE))
	n_years = len(fi.read()) // self.INDEX_DTYPE_SIZE
	if interval == self.INTERVAL_quarterly:
	n_years //= 4
	start_year = first_year + n_years

[PaleNeutron]

It seems that the index file mentioned here and the _next column in the data file will not be used in this PR. Are you going to delete them together?

qlib/scripts/dump_pit.py

Lines 198 to 204 in 98f569e

	if not overwrite and index_file.exists():
	with open(index_file, "rb") as fi:
	(first_year,) = struct.unpack(self.PERIOD_DTYPE, fi.read(self.PERIOD_DTYPE_SIZE))
	n_years = len(fi.read()) // self.INDEX_DTYPE_SIZE
	if interval == self.INTERVAL_quarterly:
	n_years //= 4
	start_year = first_year + n_years

The whole dump_pit.py should be rewrited since we implement FilePitStorage. So current dump file should look like

s = FilePitStorage("000001.SZ", "ROE")
s.write(np_data)

[PaleNeutron]

@Fivele-Li, I think rewrite dump scripts could be done in another PR, since normal feature dump script should also be rewrited using LocalFeatureStorage and LocalCalendarStorage.