IntelliJ outputs error messages twice

[Skynet0]

I am working on IntelliJ 2019.3, with Truth 1.0 and Java 11.0.6, and running tests using Junit 5.6.0.

I'm trying to do test some basic assertions, such as checking the value of an int. However, IntelliJ is printing the messages twice:

@Test
@Tag("D")
public void testDuplicateDemo() {
    int x = 5;
    assertThat(x).isEqualTo(10);
}

Outputs:

This is rather annoying and unexpected behavior. If I test equality of Strings, the same behavior is shown, unless the string has multiple lines, in which case, only expected / but was appears.

Why is this happening? Is there a workaround / fix? Thanks!

[Skynet0]

Some more tests show me that this is something internal to how IntelliJ reports tests errors (considering that AssertJ behaves the same way). Going to close this and just use JUnit for now, which doesn't double up.

[cpovirk]

Thanks. I suspect that the cause is that we throw a ComparisonFailure. So IntelliJ outputs the Truth-generated failure message (the first 2 lines) but then also generates its own message by calling the getters on the ComparisonFailure.

I wouldn't be surprised if IntelliJ does the same for JUnit -- but only when the inputs are strings, since that's the only case in which JUnit creates a ComparisonFailure.

I'm going to reopen this for us to consider doing something better. That might mean talking to the IntelliJ people to ask them for a way to suppress the display of their "Expected" and "Actual" lines, or it might mean something else.

[cpovirk]

Quick grep shows at least a few files that deal with ComparisonFailure extraction:

• https://github.com/JetBrains/intellij-community/blob/master/java/java-runtime/src/com/intellij/rt/execution/junit/ComparisonDetailsExtractor.java
• https://github.com/JetBrains/intellij-community/blob/master/java/java-runtime/src/com/intellij/rt/execution/junit/ComparisonFailureData.java
• https://github.com/JetBrains/intellij-community/blob/master/plugins/junit_rt/src/com/intellij/junit3/JUnit3IdeaTestRunner.java

[Skynet0]

For completeness, here's the behavior of Junit5, which does not raise ComparisonFailure:

@Test
@Tag("demo")
public void testDuplicateDemoIntJUnit() {
    int x = 5;
    org.junit.jupiter.api.Assertions.assertEquals(10, x);
}

org.opentest4j.AssertionFailedError: 
Expected :10
Actual   :5
<Click to see difference>

@Test
@Tag("demo")
public void testDuplicateDemoStringJUnit() {
    String x = "foo";
    org.junit.jupiter.api.Assertions.assertEquals("bar", x);
}

org.opentest4j.AssertionFailedError: 
Expected :bar
Actual   :foo
<Click to see difference>

It does not have the double output issue, even on Strings. Hamcrest behaves similarly to JUnit.

[cpovirk]

Thanks! The different behavior might depend not only on the type of exception thrown (ComparisonFailure vs. AssertionError vs. AssertionFailedError) but also on the version of JUnit used to write/run the tests.

It's possible that a straight migration to the opentest4j exceptions would make things worse in some environments. Maybe we should first check with IntelliJ developers (and other tool owners) to ensure that they handle the opentest4j exceptions even under older versions of JUnit.

[Skynet0]

JUnit 4.12 behavior: no double output, even when a ComparisonFailure is raised:

@Test
public void testJunit4InJUnit4String() {
    System.out.println("JUnit version: " + junit.runner.Version.id());
    String x = "foo";
    assertEquals("bar", x);
}

JUnit version: 4.12

org.junit.ComparisonFailure: 
Expected :bar
Actual   :foo
<Click to see difference>

Truth has the same double output issue in JUnit 4.12.

[cpovirk]

Huh, thanks. I think I also saw something in IntelliJ that looked for messages that match a specific text format. Perhaps it's detecting the JUnit format and dropping the output in that case.

[Skynet0]

Oh, here's something interesting - AssertJ does not have duplicating behavior in JUnit4, and it raises junit ComparisonFailure for both int and String:

@Test
public void testAssertJInJUnit4Int() {
    int x = 5;
    org.assertj.core.api.Assertions.assertThat(x).isEqualTo(10);
}

@Test
public void testAssertJInJunit4String() {
    String x = "foo";
    org.assertj.core.api.Assertions.assertThat(x).isEqualTo("bar");
}

org.junit.ComparisonFailure: 
Expected :10
Actual   :5
<Click to see difference>

org.junit.ComparisonFailure: 
Expected :"bar"
Actual   :"foo"
<Click to see difference>

(Mentioned above, but AssertJ has duplicate output behavior in JUnit5.)

[cpovirk]

Thanks. I haven't organized the details of this in my head, but it might be that:

• Under JUnit 5, IntelliJ will duplicate the message unless you use the opentest4j AssertionFailedError.
• Under JUnit 4, IntelliJ will duplicate the message unless your message matches the default message style of JUnit's ComparisonFailure. (That's my best guess based on this and this.)

[Skynet0]

Don't think it's quite the first - in the AssertJ issue I linked above, it raises the opentest4j AssertionFailedError, but the output is still duplicated.

[cpovirk]

Oh, thanks. So it does.

The cause there may again be a custom exception message, [edit: which presumably IntelliJ detects under JUnit 5 as it might under JUnit 4]. AssertJ provides a custom message to the AssertionFailedError:
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L118
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L152
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L131
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L191

Contrast to JUnit, which uses the "expected:<...> but was:<...>" format:
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertEquals.java#L177
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertEquals.java#L182
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L62
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L112
https://github.com/junit-team/junit5/blob/d430c63460073b16049d506b05c82751f041d84e/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertionUtils.java#L119

The duplication wouldn't appear under JUnit 4 because AssertJ throws ComparisonFailure without a custom message if the opentest4j types aren't on the classpath:
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L127
https://github.com/joel-costigliola/assertj-core/blob/a63994016ff84e6847858c708597e1e4d388e72e/src/main/java/org/assertj/core/error/ShouldBeEqual.java#L205

[cpovirk]

There may be drawbacks to this that I'm not thinking of, but....

I would love to see this changed on the IntelliJ side. If the ComparisonFailure or AssertionFailedError has a message, I would like to see IntelliJ skip displaying its own "Expected" and "Actual" (though keep its "<Click to see difference>").

In the case of the "standard JUnit message format," I wouldn't mind if IntelliJ instead skipped displaying the message in favor of displaying only "Expected" + "Actual" + "<Click to see difference>." This doesn't matter to Truth, but it matters to AssertJ under JUnit 4 and to JUnit's own assertions (4 and 5 both, IIUC).

[Skynet0]

https://youtrack.jetbrains.com/issue/IDEA-238472