Run-TestsInBcContainer sometimes does not seem to find any tests to run

[hemisphera]

Describe the issue
Run-TestsInBCContainer (and to some extent also Get-TestsFromBcContainer) sometimes fail to load the tests into the test suite even though test codeunits exist on the server.

This only seems to happen if apps containing tests have been installed just moments before the cmdlets are invoked. I therefore assume it is some kind of cache/async/state issue. When I run the same exact script a few seconds later the test codeunits are correcty loaded. Maybe the client session state is not yet done loading the tests before the cmdlet returns?

One more hint that pointed me in that direction is the fact that the action "Get Test Codeunits" on the normal AL test runner (not the command-line one) usually takes a very long time the first time it is run.

I have worked around this issue by repeatedly invoking Get-TestsFromBcContainer until it returns something and then continue (or until it returned nothing after 3 tries and then I fail the script). However, this is a bit messy as I never really know if the tests that I found are actually the ones I wanted or tests that were present already from before.

Unfortunately I cannot provide you a simple repro. This does seem to be related to the NST cache in some way. Once it happens, it's hard to reproduce. Even starting/stopping the service or the entire container not always yields this behaviour.

Scripts used to create container and cause the issue
Just install some apps (in whatever way) that contain test codeunits and then run the codeunits using

Run-TestsInBCContainer `
    -containerName <container> `
    -testCodeunit <codeunit-id-that-you-are-sure-exists-on-the-server> `
    -credential <credential> `
    -XUnitResultFileName <filename> `
    -AzureDevOps "warning" `
    -AppendToXUnitResultFile `
    -detailed

Full output of scripts

Connecting to http://localhost:80/BC/cs?tenant=default

Then nothing more as BCCH seems to find no codeunits to run. As I said: I am hardly able to reproduce on-demand.

Screenshots
n/a

Additional context

• does it happen all the time? no
• did it use to work? not really

[hemisphera]

I just happend to encounter it (somewhat). Here's what I did:

• stop the BC service in the container
• restore a fresh database backup to a container
• start the BC service in the container
• install all dependencies (via PowerShell)
• install the app with the tests (via PowerShell)

The I ran the following scripts, each within about 5-10sec from each other:

Get-TestsFromBcContainer -containerName BC23-RTM -credential (Get-ContainerCredentials)
Get-TestsFromBcContainer -containerName BC23-RTM -credential (Get-ContainerCredentials)
Get-TestsFromBcContainer -containerName BC23-RTM -credential (Get-ContainerCredentials)

Tests                                                                      Name                  Id
-----                                                                      ----                  --
{WarmupInvoicePosting}                                                     Sys. Warmup Scenarios 130411

As you can see only the third call actually returned any output.

[freddydk]

Is this a timing issue?
I mean, instead of calling the function three times - if you wait for a minute before the first call - does it then actually return the result the first time?

[hemisphera]

I was playing around a bit more and noticed the following:

I did everything as explained above (new database from .bak, restart service, install stuff ...). But then, instead of using BCCH, I opened up the command line test runner page (130455) in the BC web client and repeatedly kept inserting (and removing) the ID of my codeunit in the "Test Codeunit Range" field for about 5 minutes. The tests never ever showed up in the repeater below.

Then after the 5min I opened a second session and did the same, and the tests were instantly there on the first try.

Then I returned to my previous session and on the first try the tests also appeared.

Looks like AllObjWithCaption isn't properly loaded/updated or something like that.

[freddydk]

OK - and if you wait 5 minutes before creating the first session - does it then appear?
Is this because the cache gets filled with stale data?

[hemisphera]

Is this because the cache gets filled with stale data?

Don't know, how would I verify that?

OK - and if you wait 5 minutes before creating the first session - does it then appear?

No. The more things I try the more it gets confusing. Is it possible that an app is not really "loaded" until it is used? I am asking because until now I have (consciously) not provided an app-ID because I wanted to run all tests on the system and avoid to filter by app ID. And when not providing the app ID to Run-TestsInBcContainer it actually never works.

When I provide the app ID however, it works on the first try. Is this intended? I would've expected the parameter extensionID to be optional?

[freddydk]

If the AppId is not provided, I believe you must populate the test suite manually using an install codeunit or like.
I am not entirely sure.

[hemisphera]

Maybe it's just a missing function call?

navcontainerhelper/AppHandling/PsTestFunctions.ps1

Lines 345 to 351 in f578432

	if ($testPage -eq 130455) {
	Set-ExtensionId -ExtensionId $extensionId -Form $form -ClientContext $clientContext -debugMode:$debugMode
	Set-TestRunnerCodeunitId -TestRunnerCodeunitId $testRunnerCodeunitId -Form $form -ClientContext $clientContext -debugMode:$debugMode
	Set-RunFalseOnDisabledTests -DisabledTests $DisabledTests -Form $form -ClientContext $clientContext -debugMode:$debugMode
	$clientContext.InvokeAction($clientContext.GetActionByName($form, 'ClearTestResults'))
	}

You're not setting the value from $testCodeunit anywhere, so the trigger on the test page never runs. The page 130455 would have a field for TestCodeunitRangeFilter but it is not used.

Also Run-TestsInBcContainer does not seem to use that field, but rather you're using the value passed from $testCodeunit to apply some filters to the page "manually". Not sure why though?

[freddydk]

In the non-extensionId scenario it is enumerating the repeater in the page, finding lines with that codeunit id.
Feel free to create a PR utilizing the filter field (if the field exists - it might not exist in all versions since 15) - but it should still work if the codeunits are in the repeater.

[freddydk]

I have added a few comments to your PR

[hemisphera]

Seen that. But hold your horses, it's not even a PR yet. I am still working on it.

[freddydk]

:-) - you added a link to it - so I thought you wanted comments

[hemisphera]

:-) - you added a link to it - so I thought you wanted comments

Not explicitly, no. But hey :)

Anyway. I updated my code and pushed it again. I've opened PR #3492 so we can continue discussion there.