Can't run SQL on log files.

[jubinmytheen]

Describe the bug and expected behavior

Getting error while running SQL query on log files like auth.log. Also tried with the file testdata/logfmt/log.logfmt (https://github.com/multiprocessio/dsq/blob/main/testdata/logfmt/log.logfmt). Is it possible to run SQL quries on the log files?

Reproduction steps

# cat log.logfmt | dsq -s text/logfmt 'SELECT level FROM {}'
Input is not an array of objects: /tmp/dsq-stdin3752862510.

# cat /var/log/auth.log | dsq -s text/logfmt 'SELECT * FROM {}'
Input is not an array of objects: /tmp/dsq-stdin1533956686.

Versions

• OS: Ubuntu 22.04 LTS (Jammy Jellyfish)
• Shell: bash
• dsq version: dsq 0.20.2

[eatonphil]

Hey! I notice that the README incorrectly says it must only be piped in.

There are tests in place that test it not being piped in: https://github.com/multiprocessio/dsq/blob/main/scripts/test.py#L193.

Does that work for you?

[jubinmytheen]

Hi,
Thanks for the info.
Unfortunately, it didn't work for me. Getting mimetype error while trying without piping.

# ./dsq ./testdata/logfmt/log.logfmt 'SELECT level FROM {}'
Unknown mimetype for file: ./testdata/logfmt/log.logfmt.

[eatonphil]

Support for logfmt was merged but not released. I just released it now https://github.com/multiprocessio/dsq/releases/tag/v0.22.0.

Tested and working for me.

[jubinmytheen]

It's working now.
Also, is it possible to run SQL on log files like /var/log/auth.log?

[eatonphil]

Good question! That used to be supported and it was accidentally dropped.

I've added an issue to support reading syslog: multiprocessio/datastation#292.