allow overloading of a.b field access syntax

[StefanKarpinski]

Brought up here: #1263.

[stevengj]

The ability to use dots as syntactic sugar for mutator/accessor methods would be nice for lots of things. I've always appreciated this in languages that provide it, so that you can turn structure fields into more complicated abstractions without breaking the API.

[toivoh]

+1

[JeffBezanson]

I have an absolutely awesome way to implement this.

[johnmyleswhite]

Interested in talking about it? I know that Tom Short is really interested in having this for DataFrames, although I've come to be increasingly skeptical about the wisdom of using this feature.

[stevengj]

This would make calling Python code (via PyCall) significantly nicer, since currently I'm forced to do a[:b] instead of a.b.

[stevengj]

@JeffBezanson, any chance of having this for 0.3? Would be great for inter-language interop, both for PyCall and for JavaCall (cc @aviks).

[ihnorton]

@JeffBezanson if not, is there any chance you could give some direction on how you want this implemented? (I have an absolutely awesome way to implement this.)

[StefanKarpinski]

In my experience, there is no faster nor surer way to get Jeff to implement something than to implement a version of it that he doesn't like ;-)

[JeffBezanson]

The basic idea is that you implement

getfield(x::MyType, ::Field{:name}) = ...

so that you can overload it per-field. That allows access to "real" fields to keep working transparently. With suitable fallbacks getfield(::MyType, ::Symbol) also works.

The biggest issue is that modules have special behavior with respect to .. In theory, this would just be another method of getfield, but the problem is that we need to resolve module references earlier since they basically behave like global variables. I think we will have to keep this a special case in the behavior of .. There is also a bit of a compiler efficiency concern, due to analyzing (# types) * (# fields) extra function definitions. But for that we will just see what happens.

[bfredl]

@JeffBezanson Do you also refer to const behavior in modules? It would be useful to have a user type emulating a module and be able to tell the compiler when the result of a dynamic field lookup is infact constant. (another approach would be to start with an actual module and be able to "trap" a failed jl_get_global and inject new bindings on demand)

I would find that to be very useful in combination with #5395. Then one be able to intercept a call to a undefined function or method MyMod.newfunction(new signature) and generate bindings to a (possibly large) API on demand. This would then be cached as usual const bindings I guess.

[cdsousa]

Let me, a simple Julia newbie, present a little concern: I think the possibility to overload the dot operator might imply that field access "purity" is somehow lost.

The user would generally lose the knowledge if doing a.b is just an access to a reference/value or if there can be a huge function machinery being called behind. I'm not sure how that could be bad though, it is just a feeling...

On the other hand, I see that indeed this is a big wish for syntax sugar for many cases (PyCall, Dataframes...), which is perfectly understandable.
Maybe it is time for .. #2614?

[johnmyleswhite]

I support doing this.

But the purity does have something to say for it, even if one can use names(Foo) to figure out what the real components of Foo are.

The purity argument is closely related to the main practical concern I have, which is how one handles name conflicts when the fields of the type interfere with names you might hope to use. In DataFrames, I think we'd resolve this by banning the use of columns and colindex as column names, but wanted to know what people's plan was for this.

[cdsousa]

I guess getfield(x::MyType, ::Field{:foo}) = ... would have to be forbidden when MyType has a field foo, otherwise the access to the real field would be lost (or a way to force access to the field would have to be available).
But then getfield could only be defined for concrete types, since abstract ones know nothing about fields.

(Meanwhile, I stumbled upon this about C++.)

[JeffBezanson]

It's not a major problem. We can provide something like Core.getfield(x, :f) to force access to the real fields.

[cdsousa]

Ok, maybe I'm sold. But then defining a shortcut to Core.getfield(x, :f) (e.g., x..f) will be nice, otherwise internal code of types overloading the . for all symbols (dataframes, probably dictionaries) have to be crowded with Core.getfields.

[toivoh]

I'm not worried about the purity aspect - until we have this, the only code
that should be using field access at all is code that belongs to the
implementation of a given type. When field access is part of an api, you
have to document it, as with any api. I agree that it might be handy with
some shortcut syntax for core.getfield though, when writing those
implementations.

[cdsousa]

It had already been pointed out in #4935, but let's pull it to here: dot overloading can overlap a little with classical Julian multiple dispatch if not properly used, since we can start doing

getfield(x::MyType, ::Field{:size}) = .........
for i=1:y.size .....

instead of

size(x::MyType) = ..........
for i=1:size(y) ....

While the dot would be great to access items in collections (Dataframes, Dicts, PyObjects), it can somehow change the way object properties (not fields) are accessed.

[nalimilan]

I think one thing to consider is that if you can overload accessing field, you should also be able to overload setting a field. Else this will be inconsistent and frustrating. Are you OK to go that far?

[stevengj]

@nalimilan, one absolutely needs a setfield! in addition to getfield. (Similar to setindex! vs. getindex for []). I don't think this is controversial.

[johnmyleswhite]

Agree with @stevengj: DataFrames will definitely be implementing setfield! for columns.

[lindahua]

I support this.

Experience with other languages (e.g. C# and Python) does show that the dot syntax does have a lot of practical value. The way that it is implemented through specialized methods largely addresses the concern of performance regression.

It is, however, important to ensure that the inlineability of a method won't be seriously affected by this change. For example, something like f(x) = g(x.a) + h(x.b) won't become suddenly un-inlineable after this lands.

If we decide to make this happen, it is useful to also provide macros to make the definition of property easier, which might look like:

# let A be a type, and foo a property name
@property (a::A).foo = begin
    # compute the return the property value
end

# for simpler cases, this can be simplified to
@property (a::A).foo2 = (2 * a.foo)

# set property 
@setproperty (a::A).foo v::V begin
    # codes for setting value v to a property a.foo
end

Behind the scene, all these can be translated to the method definitions.

[Liso77]

Pls change milestone to 1.0! :)

[nalimilan]

@Liso77 What do you mean? This is already implemented on git master, as noted in the status when closing.

[Liso77]

@nalimilan sorry if I understand it wrong! But I thought that as 1.x are labeled postponed things which are going to be solved after 1.0. And this is solved now...

[ChrisRackauckas]

Open source is a decentralized community. The milestone sets what should be complete by 1.0, but contributors and work on whatever they want. In this case someone wanted this in 1.0, so they contributed the code to get it there.

[kdheepak]

@Liso77 As I understand it, this will not be in v1.0. The Julia v1.0 feature freeze date was set to Dec 15th, but this issue was closed on Dec 17th so I think we can expect it in a 1.x release. Core devs can correct me if my interpretation is incorrect.

[tomaklutfu]

No, it is merged into master and will be in the next release.

[Liso77]

:) Well! I just thought that it is good to label 1.0 what is going out in 1.0. If it is not wanted then sorry for disturbing! :)

[KristofferC]

I think the NEWS file is a better way to see what is going into 1.0.

[nalimilan]

The milestone was added to mean "can be implemented without breaking compatibility in the 1.x release series", but then since the code was ready it's been merged anyway before the 1.0 feature freeze. I've removed the milestone for clarity, but at this point everything which is merged in master will be in 1.0.

[kdheepak]

Thanks for the clarification! This is exciting! The NEWS file was also particularly enlightening.

[Liso77]

Thanks for remove! I am also very glad it will come in 1.0! :)

[oschulz]

I wonder if there's a way to support these new "dynamically-defined fields" in auto-complete, e.g. by allowing to overload fieldnames?. This could be very powerful, for interactive use, e.g. when dealing with DataFrames (assuming they'll will support df.column_name in the future) with many columns and/or long column names.

I guess at the moment the REPL (tab-expansion), IJulia, etc. look at the type definition, not the instance? But maybe that could be changed, for interactive use. It's probably impossible to support in an IDE like Juno, though, as instances are not available during coding.

[yurivish]

@oschulz, you can already overload fieldnames:

julia> struct Foo; foo; end

julia> fieldnames(Foo)
1-element Array{Symbol,1}:
 :foo

julia> Base.fieldnames(::Type{Foo}) = [:bar, :baz]

julia> fieldnames(Foo)
2-element Array{Symbol,1}:
 :bar
 :baz

And it looks like fieldnames is what the REPL looks at:

julia> x = Foo(3)
Foo(3)

julia> x.ba<tab>
bar baz

[oschulz]

@yurivish right - but is it "safe" to do so, currently? I'm not sure what else relies on fieldnames.

[KristofferC]

If not safe, it should be possible to define a complete_fieldnames(x) = fieldnames(x), use complete_fieldnames for the REPL completions, and overload that for custom completions.

[oschulz]

Yes, this is why I was bringing this up - in case something needs to be changed/added in Base, so that REPL and friends can take advantage of it later. In view of the 0.7 feature freeze ...

[JeffBezanson]

This is why I'm glad we called the function getproperty; it helps clarify that it doesn't refer to the same thing as fieldnames. Interfering with fieldnames could definitely cause serious problems.

[StefanKarpinski]

We might need a corresponding propertynames to give the valid property values. Of course, that may not be a well-defined concept, so perhaps we want something more specific to completion.

[oschulz]

I had a feeling that this may require a bit of deeper analysis and input from the experts (thanks!) - should we open a new issue for this?