XSLT code coverage incorrectly marks contents of xsl:choose as missed

[galtm]

In XSLT code coverage reports, I'm seeing cases where xsl:choose is correctly marked as hit, but the contents of xsl:choose are incorrectly marked as missed. The problem seems to be in the trace file, not the interpretation of the trace or the formatting of the coverage report.

I'm not sure if the problem in the trace file is due to a bug in Saxon or the XSpec trace listener. Does anyone know? The problem exists when I use Saxon-EE version 9.9.1.7 (which is the latest 9.9.x.x version right now) or 9.8.0.15.

I also tried using XSpec 1.2 to see if the issue is new for XSpec. I got an error message during report formatting (something not quite right about my environment, maybe) but the trace did get generated and it shows the same set of lines being hit.

Sample XSLT

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:f="myfunctions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0">
  <xsl:function name="f:fcn">
    <xsl:param name="node" as="element()"/>
      <xsl:choose>
         <xsl:when test="$node/self::child1">
            <xsl:sequence select="'first'"/>
         </xsl:when>
         <xsl:when test="$node/self::child2">
            <xsl:sequence select="'second'"/>
         </xsl:when>
         <xsl:otherwise>
            <xsl:sequence select="'third'"/>
         </xsl:otherwise>
      </xsl:choose>
  </xsl:function>
</xsl:stylesheet>

Sample XSpec

<?xml version="1.0" encoding="UTF-8"?>
<x:description xmlns:x="http://www.jenitennison.com/xslt/xspec"
  xmlns:f="myfunctions"
  stylesheet="coverage_choose.xsl">
   <!-- Why does all xsl:choose content get marked as missed? -->
  <x:scenario label="Tests for fcn">
    <x:scenario label="child2">
      <x:call function="f:fcn">
        <x:param name="node1"><child2/></x:param>
      </x:call>
      <x:expect label="2nd xsl:when block" select="'second'"/>
    </x:scenario>
    <x:scenario label="Neither child1 nor child2">
      <x:call function="f:fcn">
         <x:param name="node"><child3/></x:param>
      </x:call>
      <x:expect label="xsl:otherwise" select="'third'"/>
    </x:scenario>
  </x:scenario>
</x:description>

Coverage report shows hit on xsl:choose on line 5 (see <h l="5" m="0" c="142"/> below) but none of its descendants, not even xsl:otherwise. This is reflected in the trace file as follows:

<?xml version="1.0" encoding="UTF-8"?>
<trace xspec="...">
   <x u=".../coverage_choose-compiled.xsl"/>
   <u u=".../src/compiler/generate-tests-utils.xsl"/>
   <m id="0" u=".../coverage_choose.xsl"/>
   <c id="158" n="{http://www.w3.org/1999/XSL/Transform}function"/>
   <h l="3" m="0" c="158"/>
   <c id="142" n="{http://www.w3.org/1999/XSL/Transform}choose"/>
   <h l="5" m="0" c="142"/>
   <h l="3" m="0" c="158"/>
   <h l="5" m="0" c="142"/>
</trace>

[AirQuick]

I tested quickly on v1.0.0, v1.2.0 and v1.6.0 with your sample and Saxon-EE 9.9.1.7. The problem was reproduced on all of them. Setting SAXON_CUSTOM_OPTIONS=-opt:0 didn't help.
As for v1.2.0, no error message happened on my end.

[AirQuick]

With #941, you can see every enter() call on the trace listener.
It's called only on the line 3 and 5.

N:\xspec-939>set DEBUG_XSLT_COVERAGE_TRACE_LISTENER=1

N:\xspec-939>set SAXON_CUSTOM_OPTIONS=-opt:0

N:\xspec-939>set SAXON_CP=...\SaxonHE9-9-1-7J\saxon9he.jar

N:\xspec-939>...\bin\xspec.bat -c coverage_choose.xspec
...
Running Tests...
Collecting test coverage data...
****************************************
...
enter()          : file:/M:/Scripts/XSpec/issue/xspec-939/coverage_choose.xsl:3
...
enter()          : file:/M:/Scripts/XSpec/issue/xspec-939/coverage_choose.xsl:5
...

So, probably this is a very fundamental optimization by Saxon or its stylesheet interpretation we can't suppress.

I also took a look at an output from java -cp "%SAXON_CP%" net.sf.saxon.Transform -explain -it:{http://www.jenitennison.com/xslt/xspec}main -opt:0 -xsl:xspec\coverage_choose-compiled.xsl, but I couldn't figure out its implication.

[galtm]

Thanks for the due diligence.

probably this is a very fundamental optimization by Saxon or its stylesheet interpretation we can't suppress

I've asked Saxonica if they can take a look, and I'll report back here if I find out anything.

[galtm]

With #1833 and Saxon-HE 12.4, I still see this behavior, even with SAXON_CUSTOM_OPTIONS=-opt:0.

I didn't end up finding out anything from Saxonica back in 2020. Their usual policy is not to debug third-party applications, nothing obvious jumped out at Michael Kay, and I don't know how to reproduce the problem outside of XSpec.

[AirQuick]

I don't know how to reproduce the problem outside of XSpec.

-T (without [:classname]) prints tracing information (to stderr) using the built-in listener.

So, when I modified your stylesheet as follows

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:f="myfunctions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0">

  <xsl:template name="xsl:initial-template">
      <xsl:variable name="child1" as="element(child1)"><child1/></xsl:variable>
      <xsl:variable name="child2" as="element(child2)"><child2/></xsl:variable>
      <xsl:variable name="other" as="element(other)"><other/></xsl:variable>
      <xsl:sequence select="($child1, $child2, $other) ! f:fcn(.)" />
  </xsl:template>

  <xsl:function name="f:fcn">
    <xsl:param name="node" as="element()"/>
      <xsl:choose>
         <xsl:when test="$node/self::child1">
            <xsl:sequence select="'first'"/>
         </xsl:when>
         <xsl:when test="$node/self::child2">
            <xsl:sequence select="'second'"/>
         </xsl:when>
         <xsl:otherwise>
            <xsl:sequence select="'third'"/>
         </xsl:otherwise>
      </xsl:choose>
  </xsl:function>
</xsl:stylesheet>

and ran it with this command line

java -cp "%SAXON_CP%" net.sf.saxon.Transform -opt:0 -T -Tlevel:high -it -xsl:coverage_choose.xsl 2> trace.xml

trace.xml is generated without relying on third-party apps. The contents of trace.xml include tracing information similar to what XSpec Code Coverage collects. None of xsl:choose descendants are there. Perhaps, they have been optimized away.

[galtm]

Thanks for the information. I'll try to put together a question to ask Saxonica about this in the next week or so.

[galtm]

I created this issue report for Saxonica: https://saxonica.plan.io/issues/6295