JavaScript: more async tagging

[jayceekay]

The name of the parser: javascript

The command line you used to run ctags:

$ ctags --options=NONE Units/parser-javascript.r/js-async2.d/input-more-async.js

The content of input file:

$ cat Units/parser-javascript.r/js-async2.d/input-more-async.js
class ES6Class {
  anotherAsyncMethod = async attrs => {}
}

The tags output you are not satisfied with:

$ cat tags
!_TAG_FILE_FORMAT       2       /extended format; --format=1 will not append ;" to lines/
!_TAG_FILE_SORTED       1       /0=unsorted, 1=sorted, 2=foldcase/
!_TAG_OUTPUT_MODE       u-ctags /u-ctags or e-ctags/
!_TAG_PROGRAM_AUTHOR    Universal Ctags Team    //
!_TAG_PROGRAM_NAME      Universal Ctags /Derived from Exuberant Ctags/
!_TAG_PROGRAM_URL       https://ctags.io/       /official site/
!_TAG_PROGRAM_VERSION   0.0.0   /00be957/
ES6Class        Units/parser-javascript.r/js-async2.d/input-more-async.js       /^class ES6Clas
s {$/;" c

The tags output you expect:

not sure? but basically the above, but with a tag for the async class method

The version of ctags:

$ ctags --version
Universal Ctags 0.0.0(00be957), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jul  8 2018, 17:27:03
  URL: https://ctags.io/
  Optional compiled features: +wildcards, +regex, +iconv, +option-directory, +xpath, +case-inse
nsitive-filenames

How do you get ctags binary: homebrew

i tried tackling this myself but there's not enough for me to get started by myself. as an aside, i went looking for a community chat and realized you guys are communicating using issues (hence this issue).

what i tried:

$ git diff --cached
diff --git a/Units/parser-javascript.r/js-async2.d/expected.tags b/Units/parser-javascript.r/js
-async2.d/expected.tags
index 18b00b08..8f058db6 100644
--- a/Units/parser-javascript.r/js-async2.d/expected.tags
+++ b/Units/parser-javascript.r/js-async2.d/expected.tags
@@ -7,3 +7,4 @@ method1 input.js        /^  method1 : function() {$/;"  m       class:Class
 method2        input.js        /^  method2 : async function() {$/;"    m       class:Class
 method3        input.js        /^  method3() {$/;"     m       class:Class
 method4        input.js        /^  async method4() {$/;"       m       class:Class
+anotherAsyncMethod     input.js        /^anotherAsyncMethod = async attrs => {}$/;"    m
       class:ES6Class
diff --git a/Units/parser-javascript.r/js-async2.d/input-more-async.js b/Units/parser-javascrip
t.r/js-async2.d/input-more-async.js
new file mode 100644
index 00000000..42fcfd36
--- /dev/null
+++ b/Units/parser-javascript.r/js-async2.d/input-more-async.js
@@ -0,0 +1,3 @@
+class ES6Class {
+  anotherAsyncMethod = async attrs => {}
+}

as an aside, are there any articles on how this code works? i'm not a c developer but when i inspected other PRs that added async tags to the js parser i didn't notice much change to logic outside of Units/. is it using black magic to derive tag definitions from the expected.tags file?! help on how i can help would be cool!

[b4n]

The reason you don't get a tag for attrs is not async, but that the JS parser currently doesn't understand the method syntax you used. I'm not familiar with it either, but I'll try and study it to add support for it.
You can check that removing async doesn't help getting the method tagged.

as an aside, are there any articles on how this code works? i'm not a c developer but when i inspected other PRs that added async tags to the js parser i didn't notice much change to logic outside of Units/. is it using black magic to derive tag definitions from the expected.tags file?! help on how i can help would be cool!

Unfortunately there's not much more documentation that the code itself, and hopefully a few comments in it. But no, it's not using black magic so much; the commit that added async "support" (41c2c77) did add some code to the javaScript parser: it added async as a keyword, plus handling of it specially in 4 places.

[maxnordlund]

To be fair, that is non-standard syntax. The static/instance properties is not set in stone yet. But I guess it would be useful to add none then less.

I noticed a sap keyword in the parser that I don't recognize, what is it for @b4n? I'm so curious 😄

[masatake]

To be fair, that is non-standard syntax. The static/instance properties is not set in stone yet. But I guess it would be useful to add none then less.

Where can I get more information about the non-standard syntax?

LANG=JavaScript; We, u-ctags developers, not experts of $LANG. Related language references, standard/non-standard information, the status of standardization, small example inputs, concrete expected tags output for the inputs,
the large codebase using the feature, and impact of users, etc. are needed to work on the issue.
This information drives ME.
(I should put above sentences to somewhere .github/ISSUE_TEMPLATE.md.)

I noticed a sap keyword in the parser that I don't recognize, what is it for @b4n? I'm so curious smile

We inherit the keyword from Exuberant-ctags, from where Universal-ctags was forked.

[b4n]

I noticed a sap keyword in the parser that I don't recognize, what is it for @b4n? I'm so curious 😄

As @masatake said, it comes from before my time :) But anyway, it's a sad hack for improving support for the SAPUI5 framework IIUC (I just found out it seems to be OpenUI5 now).

[masatake]

I would like to introduce OpenUI5 or SAPUI5 sub parser on JavaScript parser.
All kinds defined in JavaScript parser for SAPUI5 should be moved to the subparser.
@jayceekay, I would like to know the URL or something for the documentation explaining async.

If it is not in a popular standard, I will think about introducing subparser for it.

[maple-leaf]

async will break if Class has get or set before it. check example code below:

class Test {
    async before() { // ctags will recognize this
    }

    get xx() {
        return 1;
    }

    async after() { // ctags will NOT recognize this
    }

    set yy(val) {
        return 2;
    }

    async after2() { // ctags will NOT recognize this
    }
};

And one thing not related to async, xx and yy also ignored.

[masatake]

Not reproduced.

[jet@living]/tmp% u-ctags --version
u-ctags --version
Universal Ctags 0.0.0(0730e66d), Copyright (C) 2015 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jul 25 2019, 03:06:32
  URL: https://ctags.io/
  Optional compiled features: +wildcards, +regex, +iconv, +debug, +option-directory, +xpath, +json, +interactive, +sandbox, +yaml, +aspell, +packcc
[jet@living]/tmp% cat /tmp/foo.js 
cat /tmp/foo.js 
class Test {
    async before() { // ctags will recognize this
    }

    get xx() {
        return 1;
    }

    async after() { // ctags will NOT recognize this
    }

    set yy(val) {
        return 2;
    }

    async after2() { // ctags will NOT recognize this
    }
};
[jet@living]/tmp% u-ctags -o - /tmp/foo.js 
u-ctags -o - /tmp/foo.js 
Test	/tmp/foo.js	/^class Test {$/;"	c
after	/tmp/foo.js	/^    async after() { \/\/ ctags will NOT recognize this$/;"	m	class:Test
after2	/tmp/foo.js	/^    async after2() { \/\/ ctags will NOT recognize this$/;"	m	class:Test
before	/tmp/foo.js	/^    async before() { \/\/ ctags will recognize this$/;"	m	class:Test
xx	/tmp/foo.js	/^    get xx() {$/;"	G	class:Test
yy	/tmp/foo.js	/^    set yy(val) {$/;"	S	class:Test
[jet@living]/tmp% u-ctags --options=NONE  -o - /tmp/foo.js 
u-ctags --options=NONE  -o - /tmp/foo.js 
u-ctags: Notice: No options will be read from files or environment
Test	/tmp/foo.js	/^class Test {$/;"	c
after	/tmp/foo.js	/^    async after() { \/\/ ctags will NOT recognize this$/;"	m	class:Test
after2	/tmp/foo.js	/^    async after2() { \/\/ ctags will NOT recognize this$/;"	m	class:Test
before	/tmp/foo.js	/^    async before() { \/\/ ctags will recognize this$/;"	m	class:Test
xx	/tmp/foo.js	/^    get xx() {$/;"	G	class:Test
yy	/tmp/foo.js	/^    set yy(val) {$/;"	S	class:Test
[jet@living]/tmp% 

[maple-leaf]

Homebrew universal-ctags version is outdated, and can't upgrade so far.

[masatake]

@KazuakiM, could you look at the issue about Homebrew reported by @maple-leaf?

[KazuakiM]

@maple-leaf
I have uninstalled universal-ctags and then reinstalled it again now.
Or try brew reinstall universal-ctags .

ope log)
https://gist.github.com/KazuakiM/69f58a05f07e3f6dd6ea3d76bc977bea

Can you please say that again?

[maple-leaf]

@masatake @KazuakiM I try brew reinstall universal-ctags, and it works. Seems like homebrew can't upgrade HEAD version, issue: Homebrew/legacy-homebrew#13197

[KazuakiM]

@maple-leaf
Thanks, Do you try uninstall & install --HEAD ?

$ brew uninstall universal-ctags
$ brew install --HEAD universal-ctags

[maple-leaf]

@KazuakiM I just tried, and works too.

[jafl]

Since @jayceekay's original code is not standard (cfr. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes) and the current version of ctags correctly handle's @maple-leaf's code, I recommend closing this issue.

[jafl]

I tested both JavaScript snippets in this issue, and they both work correctly with the latest ctags on master.