Row counts

[unkinected]

I love this tool, thank you for all the work!

I would love it if somehow it could spit out the row counts in a odbc2parquet query command. The number of rows that are written to the file would be immensely helpful. Thanks!

[pacman82]

Hello @unkinected ,

thanks for the kind words. Please help me understand how that would help. Logging the number of rows to standard error would not be any problem. Writing a report to standard out is trickier, since standard out can also be used to stream the parquet, rather than writing to a file. I wonder what you want to do with that number, so I know how to report it best.

On another note: The number of rows is also written in the parquet file itself. Does your use-case prevent you from using the schema information of the generated parquet file?

Best, Markus

[unkinected]

Hi there. My use case is to record the number in my logging (whether in a text file or in a SQL table). I'd like to be able to go back into logs and say, "at time XX:XX:XX I extracted 1.56 million records." I too was wondering how exactly this would be done since standard error doesn't feel right, and the standard out is already used. I don't know if that could be output to a separate variable or "results" file?

To your note... yes, my next thought was to just read the parquet file, but I was hoping to involve as few external tools here as possible. I'm not very familiar with reading parquet files outside of something like Databricks, so I'd have to do some research on that. If odbc2parquet automatically spit out the row count, that would save effort on my part (selfish, I know!).

On the other hand, I thought about using odbc2parquet to read the parquet file.. The "odbc2parquet insert" command requires dumping the parquet data to a SQL table. If there was perhaps an "odbc2parquet readparquetmetadata" command, that might be a good option too.

[pacman82]

I released odbc2parquet which emits a log message on info level to standard error. Standard error is not wrong for log messages. Even if they are not errors. It is intended for non-essential output. More like a "how did we get to this result" rather than the result itself.

To see an info level log message you have to specify at least one level of verbosity though (using -v on the command line).

If you want to keep a separate method you would still need to parse it. I will think about your suggestions with the sub-command. Adding the extra log message though is quick and at least does not hurt.

Best, Markus

[unkinected]

This is the command I'm using:

odbc2parquet.exe --verbose query --connection-string "Driver={ODBC Driver 17 for SQL Server};Server=REDACTED" --no-empty-file --column-length-limit 80000 "c:\temp\myfile.parquet" "MY SQL QUERY"

I'm not including any file splitting parameters by default, but the the "_01.par" still gets appended. If I specify "suffix-length 0" then the file is appended like this: "_1.par." I've tried on very small tables too that should not be split with the default 65k batch limit. Is my usage correct? Prior to version 5.1 this worked correctly, with no appended suffix.

[pacman82]

thanks. odbc2parquet 5.1.2 is released. This works now.

[unkinected]

Confirmed, all working for me! Thank you again so much for the help. This tool was a lifesaver for me.

[pacman82]

You are welcome. Lifesaver is likely a bit dramatic, but may I know how it helped?

[unkinected]

Mostly for logging. We do a lot of automated bulk extractions from SQL, so having the row count logged somewhere for analysis later is very useful.