Roadmap for 0.3

[staticfloat]

This is meant to be a general roadmap issue for things that we'd like to shoot for inclusion into 0.3. Feel free to edit this post to get the list in one place, and try to link to specific issues as much as possible. For those of you without edit access, I suppose you can post your additions as comments and I (or someone else who happens to have edit access) will incorporate your changes into this top post.

Please note that this isn't a wishlist, new features that are being proposed or asking to be added into 0.3 should be opened as their separate issues, this list is for issues that we already know we want to be in 0.3.

Usability Improvements

• Julia-based REPL work (Merging this repo in base somehow, I assume)
• P.k.g. phone home (P.k.g. phone home #4609, I'm so glad that this issue is the top result for the google search "pkg phone home")

Base Julia Improvements

• Faster startup (Improving package load times  #4373, suggestions for faster startup #260)
  • This basically includes the whole idea of saving out compiled Julia code
• Switching to MCJIT (switching to MCJIT #3922)
• Background bitcoin miner thread (I kid, glad you're paying attention)
• Shared-memory parallelism (support shared-memory parallelism (multithreading) #1790)
• Overloading of a.b (allow overloading of a.b field access syntax #1974, note this is marked as 1.0 right now)
• Automatic recompiling of dependent functions (automatic recompilation of dependent functions #265)
• Tuples rewrite (Tuples made me sad (so I fixed them) #4042)
• C-style struct support
  • Types as c-style structs (types as C-structs #2818)
  • Passing Julia structs to C (RFC: Make struct passing work properly #3466)
• more inlining (more inlining #3796)
• advanced cfunction (https://gist.github.com/vtjnash/7296634)

Standard Library Improvements

• Better error reporting (Improve macro-related error reporting #685, improve error types and messages #4744)
• Linear Algebra improvements (Proposal: Better linear algebra benchmarks #4668)
• builtin Enum type (enum bit masks #2988)
• SubArray improvements (Re-working SubArray #3496, RFC: Improved implementation of SubArray  #3503, RFC: merging range indexes for faster subarray linear indexing #3778, range indexing should produce a subarray, not a copy #3701 )
• Faster sparse matrix vector products (slow array allocation #4707).
• Datetime support (consider merging into Base quinnj/Datetime.jl#12)
• Pkg.update() hooks (Pkg: provide hooks for Pkg.update() to packages #4201)

Distribution

• Remove dependence on Rmath (Rename libRmath to libRmath-julia #5524)

[blakejohnson]

Add C struct passing from #3466?

[davidssmith]

Seamless access to help from unloaded packages.

[simonster]

Is this meant to be a wishlist or a list of things that people have committed to implementing? In either case, improved SubArray performance (#3496, #3503, #3778, maybe others) should probably be on here.

[quinnj]

Furthering @davidssmith's comment, I think there have been several issues discussed/opened about the help system in general; specifically addressing the "we need more" discussions (i.e. examples + links + etc. in addition to simple text descriptions), and probably more urgently, the need for packages to be able to integrate into the help system.

I would also add the discussion at quinnj/Datetime.jl#12 about datetime support in Base. I've got a wonderfully long holiday break coming up and this is near the top of my to do list.

[ViralBShah]

Faster sparse matrix vector products (#4707).

[milktrader]

+1 datetime support in Base

• datetime support in Base

[JeffBezanson]

This is what the 0.3 milestone is for. We don't need an issue listing other issues.

[StefanKarpinski]

It is kind of nice to see it all in one place, honestly. And the combination of linking with checkmarks is effective.

[JeffBezanson]

The one place is the 0.3 milestone, and the checking is closing issues.

[ViralBShah]

This is useful because it can eventually become the release notes.

[JeffBezanson]

Isn't this just an extra thing to keep in sync? IIUC, changes for 0.3 will now need to update the relevant issue, the NEWS file, and this issue. Are all new 0.3 issues also supposed to be added to this one? And then when we close an issue we also check it off here? Why bother?

[vtjnash]

@JeffBezanson I think the idea was that we would be used to generate a time-table / prioritization listing

[JeffBezanson]

I'm ok if the work item here is to decide on the scope of 0.3 --- in other words, make sure issues are open for everything we want in 0.3, and then close this when we're satisfied with the set of issues. I just prefer issues to have clear criteria for what it takes to close them. And I prefer for issues not to merely ask that certain other issues be fixed. That is just redundant.

If people somehow prefer the UX of a checklist inside this issue to milestones, then I suppose we could switch over. I just don't want two copies of the issue list. Which one is authoritative?

[porterjamesj]

Two things I would love to see are slices returning references instead of copies (#3701) and type specialization of anonymous functions (#1864).

[johnmyleswhite]

From my perspective, the sooner we get to #3701, the better. I'm not aware of any comparably breaking change we have planned.

[staticfloat]

@JeffBezanson I agree. I had intended to better define this issue a bit more, but RL intervened as it is wont to do.

My intention here was to initiate discussion of what should make it into 0.3. I do not intend for this to be a wishlist or forum for feature suggestions, rather it should be a place for issues that have already been proposed (by that I mean a separate issue was opened discussing the idea) to be formally itemized as "things that we're actively working on for 0.3". I suppose this makes it easier to generate a NEWS.md for 0.3, but I just don't really find the 0.3 milestone marker as helpful as a checklist. It IS still helpful, and I think we should continue to tag things for milestones, but it's less organized, issues often have titles that aren't quite what is being discussed within them and I like being able to hierarchically order the issues in the rough outline I have above.

[StefanKarpinski]

I really don't see any problem with having both a milestone and an umbrella issue.

[jakebolewski]

I saw in a previous issue that a metadata field for function expression objects was being considered. Is this still planned? I really like this clojure feature as it is nice for macro writing / metaprogramming as well as tooling support.

[porterjamesj]

Oh, one more thing: the syntax sugar from #1255 (import Module as m) would be really awesome

[gitfoxi]

How about, install with Anaconda?

Surprisingly -- or maybe not -- the hardest platform to install Julia on is non-Ubuntu (or non-root-access, or slightly-out-of-date) Linux.

Julia includes a lot of hard-to-build dependencies -- much as Scipy once did. There's a lot of similar dependancies between Julia and Anaconda. Would be nice to type "conda install julia".

[davidssmith]

+1 to synergy with Anaconda.

[jiahao]

I would very much like to see #552 happen.

[lindahua]

The thing that I have been looking forward to for a long time is to allow specializing/inlining function arguments.

Currently, I have to avoid using higher-level functions (such as map, reduce, mapreduce, etc) in performance-critical paths, and write hand-crafted loops instead. They are often overly slow, as finding and invoking functions in each iteration of an inner loop simply kills performance.

@JeffBezanson listed this in #3440. I do hope this to be implemented before 0.3 is release. This would greatly simplify the implementation of many functions in Base, e.g. you can write

exp(x::AbstractArray) = map(exp, x)
sum(x::AbstractArray) = reduce(+, 0, x) 
...

This also makes functional programing a truly feasible solution even in the context where performance is important.

[johnmyleswhite]

Dahua's request would be really great if it's not too ambitious.

[StefanKarpinski]

I do think this should be a really high priority at this point. It could be a pure performance optimization, which is nice.

[Keno]

Background bitcoin miner thread

Wait a minute...

[JeffBezanson]

We were discussing the release cycle off-line a bit, and decided we'd like a faster cycle for 0.3. There are already some nice changes like the faster startup time and a couple big performance improvements, and we don't want to hold those up.

The list above has a couple big-ticket items that we're not going to get to. Ideally we'd like to declare a feature freeze at the end of January, let everything stabilize over February, and release 0.3 in early March.

[vtjnash]

Sounds reasonable to me. Especially if we are planning on keeping up the pace.
Stuff that's ready to merge (inlining, MCJIT) can go in. Other stuff will wait.

There's a few issues that need to be addressed too: #5142 and making make the sys.so image redistributable

[ViralBShah]

I guess that MCJIT also means that we should be updating to LLVM 3.4? It's best to do this as soon as we can, to give adequate time for wide testing. IIRC, the AVX2 also works in 3.4, and perhaps the autovectorization stuff can also make an appearance in 0.3.

[JeffBezanson]

We can upgrade to LLVM 3.4 in any case. I don't think we should switch to MCJIT for 0.3 unless it fixes something important. We can switch to it at the beginning of 0.4.

[ViralBShah]

I thought that 3.4 dropped support for the old JIT, or does that happen in 3.5? If we can upgrade to 3.4, let's do it.

[JeffBezanson]

I'm pretty sure 3.4 still has the legacy JIT, but it will be gone for sure by 3.6.

[vtjnash]

I don't mean that we have to enable the MCJIT, just merge the code so it is available.

[timholy]

@vtjnash, probably this is more long-term than 0.3, but what are your thoughts on integration of Gtk? Do you see it migrating to base, or will package pre-compilation (=fast loading of graphics) make that unnecessary?

[tknopp]

@timholy, do you have some experience how good Gtk works on OSX and Windows? GTK seems to be the perfect fit to me as it is a C library which makes integration with Julia a lot easier than for the other C++ libraries out there. Furthermore GTK is quite modern as it allows to define the GUI declarative using xml and glade.

[tknopp]

@vtjnash, @timholy I am currently trying out Gtk.jl for the first time (on Windows) =>

Gtk.jl == "awesome"

Seriously, having a high-quality GUI toolkit available for Julia is a very big deal.
I don't really think that this should be in base but I would vote for putting Gtk.jl to some sort of "standard library" (#5155) or "default package" (#1906)

[vtjnash]

@tknopp definitely hoping for some more speed improvements in the package loading area. not sure what will get into 0.3, but it's a known issue

[timholy]

@tknopp, no, I have no personal experience with the cross-platform issues. I've previously lobbied for Qt, in large measure because they were so astonishingly slow to port GTK+3 to other platforms; fortunately, it appears that very recently the situation has gotten a lot better. Now (1) Gtk.jl is awesome 😄 and keeps getting better, (2) C++ is indeed challenging, and (3) it would be great to pick one toolkit to rule them all. So I vote enthusiastically for Gtk. If going into base helps standardize on it, I could imagine it being a good thing. On the other hand, you probably can't "force" these things no matter how hard you try, so there may be no point. Certainly, fast package-loading would take away the pain.

[tknopp]

@timholy: After playing a little bit my impression is that Gtk is quite stable on Windows (on linux anyway). OSX seems worse. I still did not get Gtk3 compiled using homebrew.

I don't think privileging a package is really a problem. Currently, Julia is outstanding for doing command line stuff. But looking at Gtk.jl one can imagine that Julia could also get an excellent GUI building language.

And while I really like Qt, one should begin the Qt vs Gtk discussion when someone has a good idea how to interface with Qt from Julia.

[ViralBShah]

Why was this closed?

[JeffBezanson]

At this point we've decided the scope of 0.3. We did a quick triage session at MIT and you'll notice a bunch of issues were closed or moved out of the 0.3 milestone. Everybody should feel free to keep discussing, but the milestone will serve as the official to-do list for the release. If something doesn't seem right, relabel and discuss in the relevant issues.

[ViralBShah]

It is nice to have that note for those of us not present.

[StefanKarpinski]

In general, it's good to give a brief reason when closing an issue unless it's obvious.

[mseri]

You shall should seriously consider update to LLVM 3.4. The 3.3 does not accept patches anymore (see e.g.[https://github.com/Homebrew/homebrew-versions/issues/453] that would prevent Julia compilation on OS X 10.10 unless you patch it yourself)

[johnmyleswhite]

Maybe should is a better word choice than shall?

[Keno]

3.4 is in a weird in-between state that is unfortunately not usable for us. We can easily patch llvm to make this work and upgrade to 3.5 once that's released.

[mseri]

@johnmyleswhite yes, sorry about it.

I wanted to say simply that I;ve found worrisome that LLVM3.3 is no more updated, and maybe one should start to think about moving to a different version.

@Keno I see, I was not aware of it, thanks for the clarification.

[StefanKarpinski]

We are, I'm afraid, acutely aware of the status of various LLVM versions. As @Keno said, we cannot use 3.4 since both of LLVM's JITs are in an unusable state for Julia in that version. LLVM 3.5 has not yet been released. So we're stuck on 3.3 with patches. The notion that LLVM is just a system library that you can just link against some arbitrary version of, although widespread, is completely misguided.

[ViralBShah]

The julia source tree can download and build its own LLVM. Upstream is no longer maintaining 3.3 which is why distros are moving away, not wanting to maintain 3 toolchain releases as 3.5 is around the corner. We should be OK with 3.5, hopefully.