Integrate functionality from the atom-elixir alchemist-server version

[tonini]

@msaraiva it would be great if we could integrate the code/parser and the code/metabuilder (and the rest which is needed too) into the alchemist-server. Then I could start to refactor a lot of the server itself and also simplify the API a lot for example the definition and completion lookup for example. 🎉

[msaraiva]

@tonini I can start working on the integration in a couple of days. It would be great if we could eventually refactor the completion lookup or even reimplement it from scratch. During the whole process of creating atom-elixir, autocompletion was the hardest part to extend (and still is). I guess the reason is that it was originally designed only for iex. Right?

[tonini]

Sounds good to me :)

About the completion functionality. We used the autocompletion functionality served by elixir IEx.Helpers itself which we also improved with the help of José.

It would be great if you could write down some of the issues which you had, I'm really curious about these.

[msaraiva]

@tonini I'm currently adding a couple of features to the parser and also refactoring some parts. I expect to finish this by the end of the week. As soon I get everything working, I'll merge into my alchemist-server fork and create a PR.

Regarding the completion implementation, don't worry. As I wrote "we could eventually refactor the completion lookup". It's not a priority right now. I'll write down some of the issues I had when I have a chance ;)

Cheers.

[tonini]

@msaraiva These are great news :) Thanks a lot for all your effort.

[msaraiva]

@tonini Quick update:

There are two features I'm currently working on:

1. Introspecting behaviours information (callbacks, docs, ...), including those injected by use clauses)
2. Storing variables and attributes definition lines in the metadata

I'm resuming my work on these features today, so I hope I'll get something stable in the next few days. March was a really hard month for me to do any OSS work. Sorry about that.

Cheers.

[tonini]

Hey @msaraiva

This sounds great! 👍

Don't worry, just take your time.

Enjoy the weekend. 😄

[msaraiva]

@tonini Ok, let's get started ;)

msaraiva@9dd2331

[tonini]

@msaraiva this is amazing :D ❤️ 💛 💙

Btw, currently the alchemist-server also uses the complete autocompleter module because of the aliases usage. But since Elixir 1.2 we don't need that anymore, instead we could simple add the aliases to the application env when needed. -> https://github.com/elixir-lang/elixir/blob/master/lib/iex/lib/iex/autocomplete.ex#L175

[msaraiva]

@tonini I just took a look at the new autocomplete code and I afraid I'll still have to copy most of the code and then make my own changes. Here is the main issue:

• The only information the expand function returns is the hint and a list of suggestions as strings. As you can see in this screenshot, I introspect much more information, namely specs, doc summary, type of suggestion and the module where the function was originally defined.

The current implementation doesn't have any extension point where I can add what I need and, since I don't see iex ever using those extra info, I'll probably never be able to use this function directly.

Thoughts?

[tonini]

Hi @msaraiva

Sorry for my late response.

I think you're right about this. What do you think of implementing your autocomplete module for the alchemist-server? I guess people would really like it, I saw some people already talking about it in the slack and irc channel. :-) They love how atom-elixir does it. ;-)

[msaraiva]

Hi @tonini
/cc @mat-mcloughli

Months ago I started to remove all logic from atom-elixir and moved to a new pure Elixir lib (temporally called ElixirSense) that could be used not only by alchemist and atom but also by vscode or any other editor/tool. The main goal was to create a unified way to introspect information from code/beans and provide that information through a well defined API. All functions available are editor agnostic and exchange not only strings but also compound data types like lists and maps. Any editor specific implementation, e.g. formatting, can be done by the plugin/package's author easily. Here's an example of how Alchemist.API.Comp would look like:

defmodule Alchemist.API.Comp do

  @moduledoc false

  @spec request(String.t) :: no_return
  def request(args) do
    {{hint, buffer_file, line}, _} =  Code.eval_string(args)
    buffer = File.read!(buffer_file)

    ElixirSense.suggestions(hint, buffer, line)
    |> Enum.map(&format_suggestion/1)
    |> Enum.each(&IO.puts/1)
    IO.puts "END-OF-COMP"
  end

  defp format_suggestion(%{type: :variable, name: name}) do
    "#{name};var"
  end
  defp format_suggestion(%{type: :attribute, name: name}) do
    "#{name};attribute"
  end
  defp format_suggestion(%{type: :hint, value: value}) do
    "#{value};hint"
  end
  defp format_suggestion(%{type: :module, name: name, subtype: subtype, summary: summary}) do
    "#{name};module;#{subtype};#{summary}"
  end
  defp format_suggestion(%{type: :callback, name: name, arity: arity, args: args, origin: mod_name, summary: desc, spec: spec}) do
    "#{name}/#{arity};callback;#{args};#{mod_name};#{desc};#{spec}"
  end
  defp format_suggestion(%{type: :return, description: description, spec: spec, snippet: snippet}) do
    "#{description};return;#{spec};#{snippet}"
  end
  defp format_suggestion(%{type: type, name: func, arity: arity, args: args, origin: mod_name, summary: summary, spec: spec}) do
    "#{func}/#{arity};#{type};#{args};#{mod_name};#{summary};#{spec}"
  end
end

Note: Vscode and Atom will use a different formatter (JSON).

The code above actually works and I was using it in atom-elixir for my tests. As you can see, there's no need to do any parsing in the client and you don't need to pass any use/alias/import/behaviour information since the library will handle all that using the new parser. Here's the implementation of ElixirSense.suggestions/3:

  @spec suggestions(String.t, String.t, non_neg_integer) :: [Suggestion.suggestion]
  def suggestions(hint, code, line) do
    buffer_file_metadata = Parser.parse_string(code, true, true, line)
    %State.Env{
      imports: imports,
      aliases: aliases,
      vars: vars,
      attributes: attributes,
      behaviours: behaviours,
      module: module,
      scope: scope
    } = Metadata.get_env(buffer_file_metadata, line)

    Suggestion.find(hint, [module|imports], aliases, vars, attributes, behaviours, scope)
  end

So, basically the only thing we have to do is formatting the result of the available functions. Currently they are: docs/3, suggestions/3, definition/3 and a new function called signature/3 that can provide live information about the functions/params as you write (just like vscode's cmd+shift+space):

iex> code = ~S'''
...> defmodule MyModule do
...>   alias List, as: MyList
...>
...> end
...> '''
iex> ElixirSense.signature("MyList.flatten(par0, ", code, 3) 
%{active_param: 1,
  signatures: [
    %{name: "flatten", params: ["list"]},
    %{name: "flatten", params: ["list", "tail"]}]}

Other existing functions in atom-elixir like expand, quote and match can be easily moved to the new API in no time.

There are also other things halfway done like function find/refactor using xref and a new custom parser that will enable us to retrieve line+column information - last time I've checked, the default one could only retrieve the line, which is terrible and makes some of the functionality not as precise as it should be.

The bad news is that my last commit was about 3 months ago. Since then, I've had practically no time for OSS projects. The good news is that I will try to take a couple of days next week to finish the most important part so people can start using/testing and, depending on the feedback, we can finally release the first version.

So, let's see how it goes ;)

Cheers.

[mat-mcloughlin]

you missed an n its @mat-mcloughlin

[konstantinzolotarev]

Any news/progress on this ?

[msaraiva]

Hi all.

I've just pushed the first version of ElixirSense. As already mentioned, the final goal is provide a standard, easy to use API for retrieving context-aware information about Elixir code. The project is basically divided in two parts:

• The API itself, which is exposed by the ElixirSense module and can be used by any editor/tool.
• The server - It was initially intended to work with Atom and VS Code. However, it may also be used by any editor/tool as long as it can encode/decode Erlang's External Term Format.

This first version has all the existing functionality of atom-elixir plus some new features/enhancements:

• API
  • Simplified standard API. All functions available that depend on context information can now be called passing only the editor's buffer and cursor position (i.e. line and column). I believe this is the best way to provide a consistent, editor agnostic API that can be easily used by any tool, avoiding a lot of parsing/regex on the client side.
  • Signature Info. This is a new feature that works very similar to VS Code's Signature Help. Here's an example (Atom):
    
• Server
  • Binary Protocol. All messages are encoded/decoded into/from Erlang's External Term Format
  • TCP/IP and Unix Sockets. Communication between client and server through UNIX domain socket (OSX/Linux) or TCP/IP socket (Windows).
• Clients
  • JavaScript. I've already implemented a JS client for Atom that can be easily used by VS Code without any changes. This should probably become a npm package among with all required .ex files. This way we could avoid copying those files to Atom/VScode's plugins project manually. The client also automatically encodes/decodes Erlang terms into/from JS values, including nested list/maps into/from array/objects. This makes adding new features that exchange complex data much easier.
  • Java. I believe @JakeBecker has been working on a Java client for IntelliJ.

Other information/notes:

• @tonini, I guess you'll be using only the API, not the server, so you'll probably only need to access the ElixirSense module directly.
• Check each provider if you want to see the specification/structure of all returned types. For instance, all additional suggestions' types for auto complete can be found here.
• The RequestHandler module is responsible for mapping client requests into ElixirSense calls. This is how client calls should look like when using elixir-sense-client.js. Here is an example from atom-elixir.

The next release of atom-elixir will be the first release using the new API and Server. After this release, I'll try to find someone else to maintain it. My intention is to focus on improving/adding features to ElixirSense's API instead of dealing with specific client code. Atom still demands a lot of CoffeeScript/JS code to make simple stuff to work. VS Code has already a standard API for most of the features I have in mind. So since I've been already using VS Code for TypeScript projects in the last few months, I might just switch to VS Code for good and start using it with Elixir projects as well.

I apologize in case this first release took much longer than expected. Unfortunately, I can only work on this project on my spare time.

In case you have any questions, feel free to DM me.

/cc @mat-mcloughlin, @fr1zle, @slashmili

[tonini]

Just awesome! :) I try to get some free time to dive into ElixirSense!

[timmhirsens]

This looks awesome.
I will definately look into adding this to the vscode extension.
I would also happily accept your contribution @msaraiva if you feel like implementing this yourself 😉

[narendraj9]

This looks great! I would be amazing to add this to Emacs. :) I heavily depend upon eldoc-mode.