make install for tests

[vermaete]

Hi,

Am I correct that the make install in not installing the tests even if the tests where enabled in the configuration?
Is that as expected?
I was expecting it would be because when cross compiling it's kind of needed to have the packaged during the install phase.

br,

[iguessthislldo]

Yes, it is expected that it only builds them. Not everything built is installed, even in a --no-tests build. It's not clear what installing all the tests would look like. Doing testing for a cross-compiled build is a good question though. ACE's testing framework has support for running on a remote target system. I've used it to run ACE tests on Android before. It runs on a host where it copies needed files to the target system, runs them, copies the results back and sort of acts like it ran locally. However it might or might not support the system you're trying to target and I think this is the only place it's documented. Also I've also never got around to trying this on OpenDDS, so I'm not sure how well OpenDDS tests will run in this mode.

[vermaete]

@iguessthislldo
Some more background about the question:

Yocto has already ptest.
What's not that much more that a shell around the test of the packages/recipes themself to run them and collect the results.
There could be an opendds-test package that holds all the files to run the tests.

Running the tests is done with 'ptest-runner opendds' on the target.

Besides other things I'm missing right now the 'make install' part of tests.
It could be handles inside the bitbake recipe. But that's not really the task of the Yocto recipes.

Is there a solution with MPC by patching a file in Yocto to have the 'make install' target in the generted Makefiles?

If meta-opendds would have the ptests it would be great to test on the targets in a almost automatic way.

Br

[mitza-oci]

Tests are not installed, but they can be both compiled and run using an installed OpenDDS.
Let's assume for a moment that "make install" for tests is not feasible.
What have you tried in terms of getting the test programs to run on the target? What problems did you encounter? I suggest doing this independent from any other framework first.

[vermaete]

It is an attempt to use the ptest framework of Yocto on our devices. Not that it will discover big things, but it's also rather low hanging fruit to do the tests.
For now I think to use an rsync with some excludes (*.h, *.cpp, *.o, ...) when doing the do_install in the Bitbake recipe.
The Yocto ptest isn't doing much more that running the test suite of OpenDDS and collect and reformat the results.
The first target that can be shared in meta-opendds is to have them running on a RaspberryPi with meta-raspberry.

[vermaete]

@mitza-oci @iguessthislldo
Just a question more about this.

Is their a way to sort the tests that are useful to run on the target and those only useful to run at the native machine?
Being 'target' as the device that is cross compiled. And 'native' the machine where the cross compilation was running at.

Still WIP:
https://github.com/vermaete/meta-opendds/blob/vermaete/ptest/recipes-connectivity/opendds/opendds/run-ptest
But the test FAIL: tests/DCPS/Compiler/typeobject_hash_consistency/run_test.pl needs 'opendds_idl'. This seems to be not valid for the tests at the target.

[iguessthislldo]

You could add a new -Config value like CROSS_COMPILED.

For the tests that shouldn't run like typeobject_hash_consistency, you'd change:

OpenDDS/tests/dcps_tests.lst

Line 80 in 08a2763

tests/DCPS/Compiler/typeobject_hash_consistency/run_test.pl: !DCPS_MIN !OPENDDS_SAFETY_PROFILE

to:

tests/DCPS/Compiler/typeobject_hash_consistency/run_test.pl: !DCPS_MIN !OPENDDS_SAFETY_PROFILE !CROSS_COMPILED

Then you'd pass -Config CROSS_COMPILED to auto_run_tests.pl.

[vermaete]

Sound like a good solution. I will give it a try.
But is it something that I can upstream? Is it useful for OpenDDS?
And could the list of the filtered test be maintained by OpenDDS?

br

[iguessthislldo]

Yes, I wouldn't have a problem with a PR for it. We wouldn't have a way to see if we added a new test that requires the -Config though. Someone else might need it though.

[mitza-oci]

If they only difference is opendds_idl, we could have a CI job that removes it before running tests.

[vermaete]

@mitza-oci @iguessthislldo

I found some time to pick this up. Now the Yocto ptests stuff is working for meta-opendds.

I found a '!TARGET' in the dcps_tests.lst file.
If I scrolled thought the history of it, it seems to be to avoid running tests in the LynxOS target.
Although some of these tests do pass on my Raspberry Pi with an Yocto image on it.

Do we keep the proposed !CROSS_COMPILED?
Is reusing !TARGET ok?
Or do I take !TARGET_ONLY?

What's yours opinions?

Br,

[iguessthislldo]

I think we're okay with changing the name to CROSS_COMPILED. The current ones being excluded from TARGET would have to be evaluated to see if they actually cause an issue with cross compiled tests. They may have been specific to LynxOS.