About the compression mechanism

[chainsawriot]

As there are many bugs (#395 #396 #399) after really testing this bunch of code (#354)

Other than the issues of compression detection, this

rio/R/compression.R

Lines 25 to 44 in 2fb1373

	filename <- normalizePath(filename)
	tmp <- tempfile()
	dir.create(tmp)
	on.exit(unlink(tmp, recursive = TRUE), add = TRUE)
	file.copy(from = filename, to = file.path(tmp, basename(filename)), overwrite = TRUE)
	wd <- getwd()
	on.exit(setwd(wd), add = TRUE)
	setwd(tmp)
	if (type == "zip") {
	o <- utils::zip(cfile2, files = basename(filename))
	} else {
	if (type == "tar") {
	type <- "none"
	}
	o <- utils::tar(cfile2, files = basename(filename), compression = type)
	}
	setwd(wd)
	if (o != 0) {
	stop(sprintf("File compression failed for %s!", cfile))
	}

is super not robust (e.g. it won't return a file and o is still 0). Instead of using this, the plan is to use the battle-tested R.utils compression. Well, even data.table is using it and that's the reason for introducing R.utils in the first place #362 .

https://github.com/Rdatatable/data.table/blob/2487c61656335764980e478c323f7e6ce4e6d4ca/R/fread.R#L123

But the problem is that R.utils doesn't support xz. (But I think it should be okay to cut xz. No one complains about xz not functioning, although it is not functioning for so many years.)

[chainsawriot]

Despite the code has been added since 2016 e980077 , the (current) doc never says that .{format}.xz, .{format}.bzip, and .{format}.gz (except ".csv.gz") are supported.

rio/R/export.R

Line 11 in 144ebca

#' The output file can be to a compressed directory, simply by adding an appropriate additional extensiont to the `file` argument, such as: \dQuote{mtcars.csv.tar}, \dQuote{mtcars.csv.zip}, or \dQuote{mtcars.csv.gz}.

("compressed directory" is weird, probably means "compressed file.")

I think another solution is to clarify only ".{format}.zip", ".{formart}.tar", ".{format}.tar.gz", ".csv.gz" (via data.table::fwrite()) and ".csv.bz" (via data.table::fwrite()) are supported, because they are tested. Probably this is not a breaking change, because gzip, bzip2 and xz never work.

@schochastics Thoughts?

[schochastics]

If we properly document what is supported and what not we can take the "not-support-xz" road and use R.utils to fix the remaining compression

[chainsawriot]

Cool. Then let's do it.

• Correct DOC import() export() doc is outdated #402
• DOC Mention csv.gz (and maybe csv.bz) are exceptions Except csv.gz, no compression for .gz extensions #399
• Use R.utils to enable gzip and bzip support bzip2 and xz don't work #396
• Disable xz bzip2 and xz don't work #396