Support run TestNG suites toward XML file

[jdneo]

The experience could be adding a command action into the file explorer context menu, like this:

[Mohannad91]

Thanks,
It would be a great feature that will help in debugging for automation testing frameworks written in Java.
Are we getting this soon or not ?

[jdneo]

Hi @Mohannad91,

Do you mean in the context menu, you want another item called Debug Test?

[Mohannad91]

Hi @jdneo .. yeah exactly what i meant..
it's a popular and important feature for automation engineers using java and TestNG to build their framework.. it'll make life way easier

[dg-ceiland]

@jdneo: While the "Run Test" context menu command would be handy, users can get equivalent functionality by right clicking a Maven project and choosing the test goal. To expand on the comments above, one thing that the VSCode extension can't do right now is run or debug at the class and method level using info specified in testng.xml.

Example: With TestNG, I can use the @Parameters decorator to specify that I'd like TestMethod1() to use a parameter called "username". Then, I can specify a value for "username" in a testng.xml file specified in the Maven project's pom.xml. I can also specify in the testng.xml file that I'd like to only run the TestMethod1() method next build. From there, I could run mvn test from the Terminal, only TestMethod1() would fire, TestNG would notice that I have a parameter called "username" specified for it in testng.xml, and that parameter would be plugged into my test method.

However, using the CodeLens "Run Test" or "Debug Test" action on TestMethod1() will result in a Skip by TestNG - by clicking on those, I'm running the class, not the whole project, so CodeLens doesn't know what "username" is. It does know that it's a @Parameter that's not specified in any file it knows about, so TestNG returns an error. Now, imagine this project has dozens or hundreds of test methods; running the whole project just to check one method is cumbersome, and editing my testng.xml each time I want to just run or debug one method is tedious.

A nice feature would be to automagically give Run Test and Debug Test visibility to the stuff specified in testng.xml. @Parameters would be one such example that's fairly straightforward. Getting more stuff out of testng.xml at runtime might be nontrivial and something you might consider splitting into subtasks.

[jdneo]

Hi @dg-ceiland,

Sorry for my late reply and thank you for the detailed explanation. Just as you have said, the extension currently has limited support on TestNG. And your suggestion is very inspiring.

I think I can first support running/debugging towards the testng.xml and then make the codelens(Run Test/Debug Test) more intelligent.

[jdneo]

Hi @ALL,

Now have one problem with this issue. The VS Code when API can only check the file name but has no ability to check the content. This limitation may introduce some problems here:

• Add Run/Debug Test menu item for all *.xml file: This implementation can make sure that we won't miss any TestNG XML files in the project. But it will be confusing when the user right clicks on other XML files, like pom.xml.
• Add Run/Debug Test menu item for all the XML files whose file name contains testng. By doing this, we limited the entry points but we may also miss some of the configuration files.

I prefer the first one, but I'm worrying about the confusion that may be introduced here. Do you guys have any thoughts on this?

[dg-ceiland]

@jdneo Just to confirm we're talking about the same thing (I don't believe we are), I've included a screenshot of the Run / Debug Test UI clickables that I'm thinking of.

Perhaps my suggestion is outside of the scope of this issue. In any case, it's tricky to say which of your options is better - option 1 would probably be my choice as well.

[jdneo]

@dg-ceiland I think you are talking about bringing the context defined in testng.xml to the Run Test & Debug Test actions. Am I right?

[dg-ceiland]

@jdneo Yes. Is this within scope, or even doable? If so, is this something that should be put into a different issue?

[jdneo]

@dg-ceiland Yes, I think it would be better to file this into another new issue.

Would you mind to file a new issue for that?

[oliversundell]

@jdneo
Any updates on this? Would really want the ability to run testng tests from xmls, just by right clicking and "Run Test"/"Debug Test". Often have many tests in the xmls that run parallel. As it is done in Eclipse today.

[jdneo]

@oliversundell,

There is one concern that so far I cannot find a way to figure out testng specific xml. If I simply add the test/debug menu item to all xml files, this would be a little bit annoying.

[oliversundell]

@jdneo Yeah, understand. Is there any way to customize the context menu my self to achieve this (I can accept that I get the options on all xmls)?

[jdneo]

@oliversundell Your comment inspired me! So actually there is some ways to filter the unexpected run/debug menu items, for example:

• When only the extension detects there is TestNG in class path, we show them
• Or we can provide an extension setting to let the users specify whether they want to have them show in the context menu for the .xml files.