things we should deprecate, 0.6 edition

[StefanKarpinski]

I'd like to collect a list of things that should be deprecated before 0.6 is released, even if we don't implement the corresponding feature until 1.0. Some examples I can think of:

• Strings: "hello".data access – we need to change the representation of strings to make them more efficient, which is almost certainly not going to allow accessing their underlying bytes like this. So we need to deprecate this and provide an alternative API for this sort of thing. faster String allocation #19449.

• Backtick syntax: deprecate using shell-special characters inside of backtick command syntax. At this point we can write most shell constructs in Julia, but it's gotten a bit more verbose and inconvenient than I'd like. So, in 1.0 I'd like to make, e.g. `echo hello & echo world | sort >> file.txt` construct a pipeline that runs two echo commands in parallel and connects them to the stdin of a sort command and then connects the output of sort to the file file.txt opened in append mode. deprecate unescaped shell special chars in commands #19786

• the elementwise meaning of | and & given we're about to have .| and .& Deprecate manually vectorized methods in favor of dot syntax, monolithic edition #19791

• depreaction of sumabs, sumabs2, maxabs and minabs which are not necessary with fast function arguments. Deprecate sumabs, sumabs2, minabs, maxabs #19616

• Array(T, dims...) (RFC: Deprecate Array(T, dims...) #19989)

• partial linear indexing, done in RFC: Deprecate partial linear indexing #20079

• produce/consume (issue Deprecate produce-consume #17699)

• vectorized isinteger, et al. (deprecate array-reducing isinteger #19925)

• vectorized frexp (Deprecate frexp{T<:AbstractFloat}(A::Array{T}) in favor of broadcast ? #19817)

• promote_eltype_op (Remove broadcast_elwise_op and deprecate promote_eltype_op #19814)

• deprecate vectorized ifelse in favor of dot syntax (Deprecate vectorized ifelse in favor of dot syntax #19931)

• deprecate _ as a rvalue (Dummy variable _ #9343, deprecate using _ as an rvalue #20328)

• ~ macro syntax the overall plan from things we should deprecate, 0.6 edition #19598 (comment):

  • change the one user of this as a macro – i.e. StatsModels (WIP: Replace infix ~ for formulas with a model macro JuliaStats/StatsModels.jl#9)

  • update DataFrames to depend on the new version of StatsModels (issue/PR link?)

  • change parsing of ~ in Base from macro to plain infix operator

• Open for read and write: we need to get rid of readandwrite so whatever deprecation we need to make there should be made. This may be harder since we may need to implement the change. Change open(cmd) to return Process instead of Tuple #12807

• Inner constructor syntax, see things we should deprecate, 0.6 edition #19598 (comment)

• Static parameter syntax, see things we should deprecate, 0.6 edition #19598 (comment)

• Deprecate reducer form of @parallel: RFC: "for-loop" compliant @parallel for.... take 2 #20259, waiting for a review

Probably in 1.0 rather than 0.6:

• Various special functions, pending Special functions to Base #4301/move special functions out of Base? #8536

• full – @Sacha0, how are we on this?

[tkelman]

• the elementwise meaning of | and & given we're about to have .| and .&
  edit: done in Deprecate manually vectorized methods in favor of dot syntax, monolithic edition #19791 – @StefanKarpinski
• macro parsing of ~ if the stats packages can come up with a @formula replacement in time

[nalimilan]

macro parsing of ~ if the stats package can come up with a @formula replacement in time

Should probably go into StatsModels now.

[ararslan]

Regarding formulas, see JuliaStats/StatsModels.jl#3.

[Sacha0]

Alongside @tkelman's mention of vectorized & and |, remaining vectorized methods for abs, abs2, imag, trunc, floor, ceil, mod, div, rem, clamp, and a number of hyperbolic trig functions should be deprecated. Likely vectorized big methods as well, and possibly (discussion ongoing) vectorized float, real, and (two-argument) complex methods. (I have stale pull requests open for most of those deprecations, blocked on whatever mark we set for completion of the sparse broadcast overhaul.) (edit: These are done, less those struck through.)

With devectorization complete, deprecation of expm, logm, sqrtm, and friends (i.e. matrix functions specially named due to former conflict with vectorized scalar functions) in favor of non-m equivalents might be worth considering. Best!

Edit: Also full. (I hope to see deprecating full through after completing the sparse map/broadcast overhaul (post 0.6).)

[andreasnoack]

I'd also propose depreaction of sumabs, sumabs2, maxabs and minabs which are not necessary with fast function arguments.

Edit: done in #19616 – @StefanKarpinski

[simonbyrne]

Various special functions, pending #4301/#8536

[simonbyrne]

For strings, the decision was to define a codeunit(s::String, offset::Int) function for people who need access to bytes.

[StefanKarpinski]

If Jeff's recent String experiment pans out, we'll do that; otherwise we rename the .data field.

[tkelman]

• partial indexing deprecate (then remove) generalized linear indexing #14770, but maybe too big of a job to fit before 0.6 feature freeze

• vectorized isinteger (and maybe others) on arrays, replace with all(isinteger, x) (edit: isinteger done - @Sacha0)

[vtjnash]

If Jeff's recent String experiment pans out, we'll do that; otherwise we rename the .data field.

I think we said that Jeff's recent String experiment is expected to merge in the next few months, otherwise we don't spend any other effort on changing the representation. Probably by 1.0, we will need to have the infrastructure in place to handle Arrays more efficiently via an underlying Buffer type. At that time, we can delete the special case code for String and go back to having a .data field, by showing that it gives the same (or better) performance.

[JeffBezanson]

Another candidate: produce and consume, in favor of channels.

[amitmurthy]

Another candidate: produce and consume, in favor of channels.

#17699

[blakejohnson]

With devectorization complete, deprecation of expm, logm, sqrtm, and friends (i.e. matrix functions specially named due to former conflict with vectorized scalar functions) in favor of non-m equivalents might be worth considering.

I agree, but these need to wait one release so that we can suggest the alternate spelling without the m.

[ararslan]

The manual has claimed the Array(T, dims) syntax to be "deprecated" for a while now, but no deprecation warning is emitted. If we really want that syntax to be deprecated, we should formally deprecate it.

[goszlanyi]

What was wrong with Array(T,dims)?
It was concise, easy to remember and similar to zeros(T,dims) and ones(T,dims).
Could you point to the issue where it happened, please.

[ararslan]

What was wrong with Array(T,dims)?

It's not a proper type constructor; Array is a parametric type, so it should be constructed with a type parameter.

Could you point to the issue where it happened, please.

Jeff added the text in this commit: 23a6995

[StefanKarpinski]

Ok, as far as I can tell the remaining items to deprecate are:

[List moved to top comment]

Everything else is either already deprecated or has a fresh PR pending.

[mbauman]

partial linear indexing – any chance of this, @mbauman?

I can definitely deprecate it. I'm not sure I'll be able to resolve all the deprecations in a timely manner, though. Right now it's blocked on #18457. Once that is merged I can push a deprecation branch and we can crowd-source the deprecation fixes.

[ararslan]

~ macro syntax – how do we do this?

This is what @tkelman suggested for that:

maybe Base could keep ~ parsing as a macro but always throwing a depwarn? Looks like this may not change in Base for 0.6, but if packages can be made ready then maybe we hard change the parsing during 1.0-dev and packages might not notice if they've transitioned to this?

That would be the least immediately-breaking course of action, I would think. Once we do stop parsing ~ as a macro, we just have to make sure it still parses as an infix call with the same precedence.

[simonbyrne]

• vectorized frexp (Deprecate frexp{T<:AbstractFloat}(A::Array{T}) in favor of broadcast ? #19817)

[JeffBezanson]

Also:

• "Inner constructor" syntax. To be specific, the following constructor:

type Foo{T}
    x
    Foo(x) = new(x)
end

currently gets rewritten to

    (::Type{Foo{T}}){T}(x) = new{T}(x)

which is bad, because the definition looks like it defines Foo(x) but it doesn't.

• Static parameter syntax. The above constructor should instead be written as

    Foo{T}(x) where T<:Any = new{T}(x)   # or possibly just `new(x)`

These require #18457.

[simonbyrne]

There are also some changes to LibGit2 we should make before 1.0 (#19839), but I'm not sure if I'll have time to get to them before then.

[pabloferz]

I'd add

• promote_eltype_op (Remove broadcast_elwise_op and deprecate promote_eltype_op #19814)

[quinnj]

@nalimilan ^?

[StefanKarpinski]

Can I have a link to a PR for the DataFrames ~ change so we can track it from here more easily?

[StefanKarpinski]

I've reorganized this issue so that all the todos are in the top comment. A few comments:

• @nalimilan (or anyone who knows): if you could fill in the relevant issue/PR links for the ~-related issues in the top comment, that would be helpful.

• anyone: a PR in this repo changing ~ into an infix operator can be done in parallel and makes a pretty solid up-for-grabs project for someone wanting to hack on the parser a bit.

• @timholy: differentiating the two meanings of one doesn't involve any deprecation afaict – we can just introduce a new function and people can start using it.

• @JeffBezanson: are we realistically still going to deprecate inner constructor syntax and static parameter syntax in this release? If so cool, but obviously should happen soon.

• @musm, @simonbyrne: what's the status of moving "various special functions" out of base and deprecating them? Is that realistically still going to happen in this release?

• @Sacha0: is the full thing still plausible in this release?

If the answer to any of the above is "no", that's ok, just want to update the list so we know what we're still waiting on.

[nalimilan]

@ararslan is the one to ask regarding the ~ changes, but at least the needed changes have been merged in StatsModels (JuliaStats/StatsModels.jl#9, I've added the link to the description). We still need to make DataFrames use that, though.

[StefanKarpinski]

Now that the StatsModels change has happened (thanks, @ararslan!), I don't think the other two changes need to happen in order since they're about tagging new versions and getting the dependencies right. So the next action item there seems to be a PR to change ~ to an operator.

[timholy]

Yes, I agree that the changes for one are easy (docstring and adding a new function). Fixing base can come later.

[stevengj]

I actually started on a patch for one. Making Base dimensionally correct requires a big effort

[stevengj]

See #20268.

[mbauman]

Is it possible to deprecate using _ as an rvalue?

[StefanKarpinski]

Good call, @mbauman – need to do that now in order to use it for discarding values in 1.0.

[amitmurthy]

Also

• Deprecate reducer form of @parallel - RFC: "for-loop" compliant @parallel for.... take 2 #20259, waiting for a review.

[ararslan]

A few FYIs from me:

Special functions: I've been working on moving special functions out of Base. From a package standpoint it's nearly there, but from a modification to Base standpoint there's still a fair bit of work to be done. In particular, to remove the dependency on openspecfun, we'll need to reimplement rem2pi in pure Julia, which requires double-double arithmetic. Unless we can extend and use Base.TwicePrecision for this, we need a solid double-double arithmetic implementation in Base. I think it's unlikely that this will happen for 0.6.

FFTW: As another data point, I've also been working on moving FFTW into a package. That too is going well, though there is still quite a bit of work to be done in moving from makefiles to BinDeps. Similarly unlikely to make it into 0.6.

[StefanKarpinski]

As a temporary solution, could we just copy-pasta the one function you need from openspecfun into libjulia for now, with the longer term plan of reimplementing it in pure Julia?

[ararslan]

I'm cool with that.

[JeffBezanson]

How about moving SharedArrays to a package?

[ararslan]

Hm, SharedArrays seem pretty convenient for the parallel facilities we provide out of the box. If anything I would suggest that sparse arrays go first.

[JeffBezanson]

I think SharedArrays are pretty limited and difficult to use. They are not used anywhere in Base itself and SharedArrays is very self-contained. Of course sparse should be moved too, but I have a sense it will be more difficult. It has binary dependencies for example.

[ararslan]

Yeah, makes sense. Perhaps JuliaParallel could adopt a SharedArrays.jl package. @andreasnoack?

[ararslan]

Special functions are on their way out! See #20427

[PallHaraldsson]

"the elementwise meaning of | and & given we're about to have .| and .& #19791"

[EDIT: Made question clearer, I think..:] Does #19791 mean we already disallow overloading | and & ? I think that should be done; added to the list of deprecations if not already done.

[StefanKarpinski]

Does this that we can disallow overloading | and & ? I think it should be added to the list of deprecation if not already done.

Apropos of what?

[JeffBezanson]

I propose closing this. We can still entertain deprecation PRs as they are posted, but I think we're done actively seeking deprecations for 0.6.

[StefanKarpinski]

Sure, I think this has served its purpose now.