Dataset Support in EVA

[gaurav274]

Ongoing

Add dataset support to EVA. Allow loading multiple video to a single dataset.

LOAD Syntax

LOAD INFILE ‘1.mp4’ AS mondayMorning INTO UADETRAC;
LOAD INFILE ‘2.mp4’ AS mondayNight INTO UADETRAC;

SELECT Syntax

1. select from dataset

SELECT * FROM UADETRAC; 
or explicitly (haven't decided) SELECT * FROM DATASET UADETRAC;

Output schema = table_name + the schema for tables in the dataset. For videos:

table_name	id	data
mondayMorning	1	nd.array(...)
mondayMorning	2	nd.array(...)
mondayNight	1	nd.array(...)
mondayNight	2	nd.array(...)

2. select from table

SELECT * FROM mondayMorning; 

id	data
1	nd.array(...)
2	nd.array(...)

TODO

• Fix the dataframe and dataset support in catalog
• Get operator for dataset
• Testcases

[gaurav274]

Pending Questions:

1. How to scan over all the frames in a dataset?
   SELECT * FROM UADETRAC; or
   SELECT * FROM DATASET UADETRAC;
   Comments: The second option will be easier to implement as we won't need to infer if the table is a video table or a dataset table.
2. How do we support listing all the videos in a dataset?
   SELECT * FROM UADETRAC.VIDEOS;

[jarulraj]

1. We could complicate the internal implementation a bit to infer that it is a regular table or a video table. Otherwise, the user must be aware of two types of tables. For instance, when a LOAD INFILE is used, then infer that it is a video table.
2. We could rely on just DISTINCT(video_name) to infer that. The less metadata we separately maintain (e.g., list of videos in a video table) for speeding up a particular query, the more maintainable and easier it is.

[xzdandy]

We can have a SHOW DATASET UADETRAC; to list all videos in a dataset.

[pchunduri6]

A few questions:

1. Are all tables (videos) within a dataset assumed to have the same resolution?
2. Would we be able to run an action recognition UDF over a DATASET? If we support this, we might have to explicitly handle the video boundaries when constructing segments.

[xzdandy]

1. Resolution is not enforced in EVA now (i.e., We do not store the resolution of the video in catalog now). So the answer now is no.
2. What is the current plan to support action recognition UDF? The window key word? There is nothing preventing us to run UDF over a DATASET. But you are right, if it is not a per-frame UDF, we would need some support to handle video boundaries.