You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When we are reading a Lance file the scheduler will usually submit requests in the order that they are needed. The exception to this is any field that needs indirect I/O (for example, a list field). When the scheduler reaches that field it will immediately schedule the I/O requests to fetch the data offsets. However, we cannot schedule the data at this time because we do not know what I/O to issue. Instead, the data is requested as soon as the offsets are fetched. By this time the scheduler may have scheduled many more requests.
We want these indirect IOPS to take priority over IOPS scheduled for later pages. We can easily assign a priority for all IOPS based on the top-level row number of the first row the request is fetching (e.g. if we have List<List> and we ask for rows 10..20 then three different requests will be generated and they will all have priority 10.
One potential wrinkle is that these priorities only apply with the context of a single file. If two files are being read simultaneously then we the normal first-come-first-serve priority should apply. So the priority order is roughly...
This also renames store scheduler to scan scheduler. I'm thinking I
don't want to get into the thorny issue of how to prioritize I/O
requests across different scans. So, with this approach, if multiple
scans are running at the same time, then they will overschedule (and let
the OS deal with it). This can be revisited in the future.
Closes#1958
This also renames store scheduler to scan scheduler. I'm thinking I
don't want to get into the thorny issue of how to prioritize I/O
requests across different scans. So, with this approach, if multiple
scans are running at the same time, then they will overschedule (and let
the OS deal with it). This can be revisited in the future.
Closes#1958
When we are reading a Lance file the scheduler will usually submit requests in the order that they are needed. The exception to this is any field that needs indirect I/O (for example, a list field). When the scheduler reaches that field it will immediately schedule the I/O requests to fetch the data offsets. However, we cannot schedule the data at this time because we do not know what I/O to issue. Instead, the data is requested as soon as the offsets are fetched. By this time the scheduler may have scheduled many more requests.
We want these indirect IOPS to take priority over IOPS scheduled for later pages. We can easily assign a priority for all IOPS based on the top-level row number of the first row the request is fetching (e.g. if we have List<List> and we ask for rows 10..20 then three different requests will be generated and they will all have priority 10.
One potential wrinkle is that these priorities only apply with the context of a single file. If two files are being read simultaneously then we the normal first-come-first-serve priority should apply. So the priority order is roughly...
However, the above function is not transitive and so it wouldn't work as a sort function.
The text was updated successfully, but these errors were encountered: