\i command, bufio.Scanner: token too long

[katmfoo]

When attempting to import a large sql file (mine was over 2 GB), I would eventually get the following error:

bufio.Scanner: token too long

[nineinchnick]

We might try to use bufio.Reader.ReadLine() instead of a Scanner in

usql/handler/handler.go

Line 1139 in b376ee5

s := bufio.NewScanner(f)

but huge lines might cause issues in other places.

Until we address this, I'd recommend using tools dedicated to a particular DB for loading a large amount of data, as usql might not have the best performance for this.

[kenshaw]

I'd like to echo @nineinchnick -- perhaps one day usql will be ready to deal with bulk-loading data into any database, unfortunately it isn't ready for that yet. I'm quite familiar with this specific hack I wrote. I'l try to push out a fix, but as above, for bulk-loading data, I'd suggest relying on the database's native client.

[kenshaw]

@pricheal + @nineinchnick I just pushed a commit changing to bufio.Reader.ReadLine. I don't have a multi-gigabyte file to test with, but I'll try to generate one later. My guess is if the original SQL script involved was trying to import a multi-gigabyte entry into a single row, this is going to cause the same problem. Either way, the new implementation is likely better. All of my superficial tests/checks are passing using this changed implementation, but requires more scrutiny.

[kenshaw]

@pricheal were you able to test this change?

[katmfoo]

@pricheal were you able to test this change?

If I remember correctly I tested it and experienced the same issue