-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect listener's onStart(ISuite suite) execution order in case of inner suites #1656
Comments
Currently, |
@juherr actually it's called twice (for each suite file). Has the behaviour changed since 6.11? |
Ok, two calls is an issue. @krmahadevan What is the expected behavior of |
I am able to reproduce the order issue but, in fact, TestNG was always running child suites before parent suite (https://github.com/cbeust/testng/blame/master/src/main/java/org/testng/TestNG.java#L1191-L1199) and listeners were not correctly added before #1285. |
6.13 should have fixed the duplicate issue with #1533 |
@avarabyeu Could you confirm that 6.13 is fixing the event duplication issue? |
@juherr Can confirm that 6.13 does not have issue with duplication.
Yeah it would be easier for us to handle events in this case. Thank you! |
@cbeust @krmahadevan WDYT if we change the order execution of suite/child? |
@avarabyeu - Just trying to understand, how does the order of execution matter ? Why is it problematic if the child suites execute first and then the parent suite ? @juherr - do we really need to change this ? We can easily have this sorted out at the test level itself by moving the contents of the master suite into a suite of its own, and then having the master suite just work as a suite of suites no ? Just trying to understand what we are going to get as benefits ? |
@krmahadevan You will find more details on reportportal/reportportal#273 |
@krmahadevan ReportPortal is a tool for real-time reporting of results from different test engines. So basically, its client handles events fired by testng listener and sends them to server. Order of that events matter because based on them ReportPortal fills internal data structures which are tree-based - for instance, it's impossible to create child before the parent. |
No. That is not what I meant. I am basically saying that the |
@avarabyeu - Thanks for sharing additional context. I am still not able to grasp the idea behind the order of execution being important, because TestNG does fire events before and after suite execution. So irrespective of whether child suites execute or the parent, the corresponding events get triggered no ? |
Ok! True, it is something we can check and warn against. |
Yeah, but we build the same structure in our internal database - parent suite -> child suite. So if child suite starts earlier then parent, we simply cannot handle it on server side. |
@avarabyeu - So what happens if you don't have any I am basically trying to see what can be done to help you get past the problem without any code change. |
@krmahadevan actually, this is good question. Issue came from our clients and i'm not really sure if they are able to adjust their testng.xml files in order to solve this integration issue. Let me check what happens if we have no in master suite. |
Closing this issue for now. @avarabyeu Please comment back if you have an update on the query that I raised so that we can further triage this issue if necessary. |
@krmahadevan @avarabyeu |
There were some changes made to the latest codebase in TestNG with respect
to ordering of suites etc.,
So can you please try once again using TestNG 7.0.0-SNAPSHOT and post back
your results ?
If its still a problem, request you to please help share a sample that can
be used to reproduce the problem.
|
The same problem happened on version 7.0.0-SNAPSHOT |
@ulmanA - The sample you shared doesn't say a lot of things (at-least it wasn't explicit for me). You basically have a suite file called <suite name="All Suites" verbose="2">
<suite-files>
<suite-file path="suite1.xml"/>
<suite-file path="suite2.xml"/>
</suite-files>
</suite> And when I run this, I notice that
So what is your expectation here ? Its not clear.
|
@avarabyeu |
@krmahadevan I am not the bug reporter but I would expect the following order of execution:
So yes, |
TestNG Version
6.11
Expected behavior
The listener is called for each suite, if the parent suite contains child suites then the parent suites should run first before running the child suite.
Actual behavior
The child suites are first run before running the parent suite.
Is the issue reproducible on runner?
Test case sample
The text was updated successfully, but these errors were encountered: