HTML report does not display any info

[textnow-andriizubenko]

Describe the bug

I am running tests using Flank(v 21.09) in Firebase, some test runs would generate an empty html report but Junit report does have a failed tests.

To Reproduce

java -jar flank.jar firebase test android run --config flank.yml

Expected behavior

Html report is generated properly for all failed tests

Details (please complete the following information):

Have you tested on the latest Flank snapshot?

No

Post the output of flank --version.

21.09

Additional context

HTML report is properly generating for some test runs.

flank.yml

    gcloud:
      record-video: true
      timeout: 15m
      async: false
      client-details:
      network-profile: null
      results-history-name: null
      # Android gcloud
      app: /tmp/myapp.apk
      test: /tmp/myapp_test.apk
      additional-apks:
      auto-google-login: true
      use-orchestrator: true
      directories-to-pull:
      grant-permissions: all
      type: null
      other-files:
      scenario-numbers:
      scenario-labels:
      obb-files:
      obb-names:
      performance-metrics: false
      num-uniform-shards: null
      test-runner-class: null
      test-targets:
        - annotation com.enflick.android.customAnnotations.TestRail
        - notAnnotation com.enflick.android.customAnnotations.IgnoreFirebaseTestLab
      robo-directives:
      robo-script: null
      device:
        - model: blueline
          version: 28
          locale: en
          orientation: portrait
      num-flaky-test-attempts: 0
      test-targets-for-shard:
      fail-fast: false
      parameterized-tests: default

    flank:
      max-test-shards: 30
      shard-time: -1
      num-test-runs: 1
      smart-flank-gcs-path: 
      smart-flank-disable-upload: false
      default-test-time: 120.0
      use-average-test-time-for-new-tests: false
      files-to-download:
      test-targets-always-run:
      disable-sharding: false
      project: api-project-xxxxxxxxxxxxxxxxxx
      local-result-dir: results
      full-junit-result: false
      # Android Flank Yml
      keep-file-path: false
      additional-app-test-apks:
      run-timeout: -1
      legacy-junit-result: false
      ignore-failed-tests: false
      output-style: multi
      disable-results-upload: false
      default-class-test-time: 240.0
      disable-usage-statistics: false
      output-report: none
      skip-config-validation: false
      custom-sharding-json:

[bootstraponline]

Thanks for reporting the bug!

[textnow-andriizubenko]

Thanks for reporting the bug!

I can send you junit xml and html report if needed

[bootstraponline]

I can send you junit xml and html report if needed

That'd be helpful. You can send it to matt@bootstraponline.com

[textnow-andriizubenko]

@bootstraponline After some investigation i found out that Html report gets broken when JSON that is being inserted into INJECT-DATA-HERE has some single backslash double quotes \" . This can be fixed by replacing \" with \\\"

[textnow-andriizubenko]

Json example that breaks HTML report

[
  {
    "label": "deviceName com.myApp.tests.SomeTest#exampleTest",
    "items": [
      {
        "label": "Error in 'single quotes \"Double quotes\"'",
        "url": "https://console.firebase.google.com/project/"
      }
    ]
  }
]

[textnow-andriizubenko]

Working JSON

[
  {
    "label": "deviceName com.myApp.tests.SomeTest#exampleTest",
    "items": [
      {
        "label": "Error in 'single quotes \\\"Double quotes\\\"'",
        "url": "https://console.firebase.google.com/project/"
      }
    ]
  }
]

[textnow-andriizubenko]

The problem seems to be happening with escaping double quotes in test results(\" or \\"). Example :

val someString = "Some error in Espresso test with stacktrace in 'single quotes \"Double quotes\"'"
// or
val someOtherString = "Some error in Espresso test with stacktrace in 'single quotes \\"Double quotes\\"'"
// or 
val someOtherOtherString = "Some error in Espresso test with stacktrace in 'single quotes \\\"Double quotes\\\"'"

So the workaround would be to replace \" with " and remove remaining \

val originalString = "Some error in Espresso test with stacktrace in 'single quotes \\\"Double quotes\\\"'"
val newString = originalString.replace("\"", """)?.replace("\\", "")

Temp workaround that worked for me :

flank/test_runner/src/main/kotlin/ftl/reports/HtmlErrorReport.kt

Line 69 in 3f509c2

label = testCase.stackTrace().split("\n").firstOrNull() ?: "",

private fun List<JUnitTest.Case>.createItems(): List<HtmlErrorReport.Item> = map { testCase ->
    HtmlErrorReport.Item(
        label = testCase.stackTrace().split("\n").firstOrNull()?.replace("\"", "&quot;")?.replace("\\", "") ?: "",
        url = testCase.webLink ?: ""
    )
}