Reading CSV file with no index column

[ANeeK181]

Hi

I am writing for a project in which i am trying to mimic pandas and pytorch in cpp. First step is reading csv files, and i have found way to manipulate header so that it is in the format required by dataframe.read method. However, these csv files have no index column, so can someone guide what to do in that case? i have turned columns_only flag to true, but that gives error "Data size is larger than index size of 0". Any suggestions?

[ANeeK181]

i mean is there any option with reading csv files into dataframe with default index?

[hosseinmoein]

Data columns in a DataFrame can be at most as long as the index column. That means if you want to read a csv file without an index, you must load the index into the DataFrame first and then read the csv file. For example, you can use DataFrame::gen_sequence_index() and load_index() first.

See docs https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/DataFrame.html

[ANeeK181]

Thanks. Got it. Following code correctly loads data with predefined integer index.

StdDataFrame<int> csv_file_df;

csv_file_df.load_index(csv_file_df.gen_sequence_index(0, 1001, 1));

// Load the data
csv_file_df.read("./test_data/06-24-2022.csv", io_format::csv2, true);

One question, why do we have to specify column types and values types in functions like get_column_info() and get_row() etc? once dataframe is setup, shouldn't it automatically infer column types as they are known now?

[hosseinmoein]

C++ is a statically typed language. It means all types must be known at compile time (i.e. when you write the code).
In contrast Python is a dynamically typed language. It means it figures out the types at run time. That's why it is so slow

[ANeeK181]

right, yeah i get that. Two more questions if you might help:

1. i have csv data of all numbers (float), so i know the data type beforehand, and get no of observations by getting no. of lines in file - one header row. Now my question is, given that i already know the header (all column names, their types and their number), can i somehow skip the header row, and read only data? in other words, can i read csv file without the header format dataframe library is expecting?

I know it expects the header row be like "INDEX::OBSERVATIONS,COL_NAME_1::OBSERVATIONS,...". But is there a way to circumvent that if i already know all that information beforehand?

I have thought of a hack that i would insert the first line in csv files, but that would require modifying the original files, which others would also be using, so i have to make a copy and that would be wasteful. Any guidance on how to approach this issue?

2. Secondly, i would have dataframe of all numbers, and i would be loading them in torch through torch::from_blob(). Now, that is ok when doing with Eigen matrix, or cv2 image, but for dataframe, would that work? I haven't tried it yet, but from_blob expects void* pointer towards some memory location, and then dimesions of to be tensor.

[hosseinmoein]

1. The input file must have the header. DataFrame cannot read the file without it
2. Read the docs for the read() function. read() function doesn't support pointers. It just supports the types in the doc. But once you load the file into the DataFrame, you can add any columns of any type to the DataFrame

In general it doesn't make sense to read pointers to objects from a file. Pointers are assigned at runtime by the OS

[ANeeK181]

looking at the documentation, DataFrame can also read from stream. So if i read file in memory, modify it a bit, and then write it to buffer and stringstream, then dataframe can read from stringstream, right?

[hosseinmoein]

yeah, you can play around with it that way

[ANeeK181]

yeah, got it to work that way. Read file in buffer, modify first line to make it compatible with DataFrame.read() function expectations, and then read it by stringstream. works perfectly.

Can you help a bit with torch::from_blob()? it wont read dataframe per se

[hosseinmoein]

Sorry, cannot help with that

[ANeeK181]

thanks, no issues. I will post solution here once i figure out the best way.

[ANeeK181]

just for information of others, we can use torch::from_blob on each column to read data column wise, and then use torch::cat to concatenate all tensors. This serves two purposes, first obviously it reads all data into torch tensor. secondly, from_blob does not allocate any new memory, and is only reading directly from passed column, so if any thing changes, it becomes invalid. torch::cat makes new memory, so now we have a completely new tensor.

If someone thinks of a better way, do let me know please.