Inconsistent handling of LF and CRLF with the shebang trick.

[chaoticryptidz]

Describe the bug
The current released version of jq cannot handle either LF or CRLF with the shebang trick.
Currently version c95b34f is the only one able to handle this particular trick and only with LF endings.

To Reproduce
The script is a bit weird as it needs to import itself as a module, and is also running under NixOS where /bin/sh is discouraged.

#!/usr/bin/env -S bash -x --
# \
exec jq -L "$(dirname $0)" -ne 'include "a"; main' -- "$@"

def main: true;

lappy-t495 :: ~/Temp » cat a.jq           
#!/usr/bin/env -S bash -x --
# \
exec jq -L "$(dirname $0)" -ne 'include "a"; main' -- "$@"

def main: true;%                                                                                                                                                                                                                                lappy-t495 :: ~/Temp » jqtest() { (dos2unix ~/Temp/a.jq && ~/Temp/a.jq testarg; echo; unix2dos ~/Temp/a.jq && ~/Temp/a.jq testarg) }
lappy-t495 :: ~/Temp » jqtest                                                                                                       
dos2unix: converting file /home/chaos/Temp/a.jq to Unix format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -ne 'include "a"; main' -- testarg
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
exec jq -L "$(dirname $0)" -ne 'include "a"; main' -- "$@"     
jq: 1 compile error

unix2dos: converting file /home/chaos/Temp/a.jq to DOS format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -ne 'include "a"; main' -- $'testarg\r'
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
     jq -L "$(dirname $0)" -ne 'include "a"; main' -- "$@"
jq: 1 compile error
lappy-t495 :: ~/Temp » PATH=~/Projects/jq:$PATH jq --version                                                                                                                                                                                3 ↵
jq-1.7.1-29-gc95b34f
lappy-t495 :: ~/Temp » PATH=~/Projects/jq:$PATH jqtest      
dos2unix: converting file /home/chaos/Temp/a.jq to Unix format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -ne 'include "a"; main' -- testarg
true

unix2dos: converting file /home/chaos/Temp/a.jq to DOS format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -ne 'include "a"; main' -- $'testarg\r'
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
     jq -L "$(dirname $0)" -ne 'include "a"; main' -- "$@"
jq: 1 compile error

Another Example:
This time without -n, which appears to be even harder to get working.

[chaos@lappy-t495:~]$ jqtest() { export DATA='{"key": "value"}'; jq --version; unix2dos -f ~/Temp/a.jq; (~/Temp/a.jq <<< "$DATA"); echo; dos2unix -f ~/Temp/a.jq; (~/Temp/a.jq <<< "$DATA") }

[chaos@lappy-t495:~]$ PATH=/home/chaos/Projects/jq:$PATH jqtest
jq-1.7.1-29-gc95b34f
unix2dos: converting file /home/chaos/Temp/a.jq to DOS format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -r 'include "a"; . | main'
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
     jq -L "$(dirname $0)" -r 'include "a"; . | main' # nya
jq: 1 compile error

dos2unix: converting file /home/chaos/Temp/a.jq to Unix format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -r 'include "a"; . | main'
value

[chaos@lappy-t495:~]$ jqtest
jq-1.7.1
unix2dos: converting file /home/chaos/Temp/a.jq to DOS format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -r 'include "a"; . | main'
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
     jq -L "$(dirname $0)" -r 'include "a"; . | main' # nya
jq: 1 compile error

dos2unix: converting file /home/chaos/Temp/a.jq to Unix format...
++ dirname /home/chaos/Temp/a.jq
+ exec jq -L /home/chaos/Temp -r 'include "a"; . | main'
jq: error: syntax error, unexpected IDENT, expecting end of file (Unix shell quoting issues?) at /home/chaos/Temp/a.jq, line 3:
exec jq -L "$(dirname $0)" -r 'include "a"; . | main' # nya     
jq: 1 compile error

Expected behavior
The script should return true regardless of line ending. Or in the second example, value.

Environment (please complete the following information):

• OS and Version: NixOS nix run github:NixOS/nixpkgs/9099616b93301d5cf84274b184a3a5ec69e94e08#jq
• jq version: jq-1.7.1 ( nix run github:NixOS/nixpkgs/9099616b93301d5cf84274b184a3a5ec69e94e08#jq) / jq-1.7.1-29-gc95b34f (freshly built)

Additional context
Behaviour seems to have changed around here 71e7bcdfc154ddbd27

[emanuele6]

This report does not make sense to me; jq has never supported files with CR LF line endings

$ printf '"hi"\r\n+"hello"' > f.crlf.jq
$ jq -nf f.crlf.jq
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting end of file (Unix shell quoting issues?) at <top-level>, line 1:

jq: 1 compile error
$ ./jq-master -nf f.crlf.jq
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting end of file (Unix shell quoting issues?) at <top-level>, line 1:

jq: 1 compile error
$ ./jq-1.6 -nf f.crlf.jq
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:

jq: 1 compile error
$ od -t c f.crlf.jq
0000000   "   h   i   "  \r  \n   +   "   h   e   l   l   o   "
0000016
$ dos2unix f.crlf.jq
dos2unix: converting file f.crlf.jq to Unix format...
$ od -t c f.crlf.jq
0000000   "   h   i   "  \n   +   "   h   e   l   l   o   "
0000015
$ jq -nf f.crlf.jq
"hihello"
$ ./jq-master -nf f.crlf.jq
"hihello"
$ ./jq-1.6 -nf f.crlf.jq
"hihello"

And, before 1.7.1, CR characters were not even allowed in comments:

$ printf '# foo\r\n"hi"\n' > f.crlf.jq
$ od -An -tc f.crlf.jq
   #       f   o   o  \r  \n   "   h   i   "  \n
$ jq -nf f.crlf.jq
"hi"
$ ./jq-master -nf f.crlf.jq
"hi"
$ ./jq-1.6 -nf f.crlf.jq
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:

jq: 1 compile error

You make it sound like something has changed with regards to line endings, but that really does not seem to be the case...

I have also tried checking if CR after # \ was allowed in jq 1.6, since that is what was changed by the commit you have mentioned, but that does not seem to be the case; as you can also see from the code that was removed, only \\, \\, \n triggered a line continuation.

At most, jq 1.6 allowed CR at the end of (or anywhere in) the line after # \ (and in the shebang line; while as mentioned, before it rejects CR anywhere else).
But jq 1.7.1 allows CR in the same places and more since it allows CR in any comments.

Can you clarify what you think the problem is?

If it's just "jq does not accept scripts with CRLF line endings", that has nothing to do with # \, and "it has never done that".