-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test strategy #12
Comments
There's probably two levels of testing here. Unit and integration. Unit tests the APIs of the components, where integration would actually test against a running gdb and fake out a client. |
For the integration test using a fake client, this seems nice, part of the vscode-debugadapter repo: https://github.com/Microsoft/vscode-debugadapter-node/tree/master/testSupport I am planning to start writing some tests with it, any objections? What kind of test output does your Jenkins job want? It is possible to write new reporters for mocha, so I assume it's possible to output what you need. For the unit tests, mocha + chai is a combo that works well, that's what we use in Theia. |
That would be awesome! Go ahead.
Yeah, that's what we use for our web projects here at QNX. The biggest issue we've had is with finding good reporters that present the results nicely. I'll have to see what the current state of all that is. |
Actually we have a reporter we're using internally. I should be able to open it. |
Add an integration test that sets a breakpoint, runs, then expect to hit that breakpoint. If you forget a step (forget to build the package or the test examples), the errors are not very descriptive nor helpful, that's something I'd like to improve in the future. Refs eclipse-cdt-cloud#12 Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
Add an integration test that sets a breakpoint, runs, then expect to hit that breakpoint. If you forget a step (forget to build the package or the test examples), the errors are not very descriptive nor helpful, that's something I'd like to improve in the future. Refs eclipse-cdt-cloud#12 Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
Add an integration test that sets a breakpoint, runs, then expect to hit that breakpoint. If you forget a step (forget to build the package or the test examples), the errors are not very descriptive nor helpful, that's something I'd like to improve in the future. Refs eclipse-cdt-cloud#12 Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
Add an integration test that sets a breakpoint, runs, then expect to hit that breakpoint. If you forget a step (forget to build the package or the test examples), the errors are not very descriptive nor helpful, that's something I'd like to improve in the future. Refs #12 Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
We have the integration story in place. Thanks @simark! |
@thegecko Please comment here and I will try to figure it out. |
In #136 @thegecko mentions issues with MacOS. I didn't know that tests didn't work on MacOS. I am in the process of bringing up a machine to add to the CI to regtest on Windows, but I don't have any immediate plans on Mac - but I could do the same there at some point. @thegecko the best would be that tests "just worked" on Mac. What is the issue there? Is it the getting a GDB/GCC/toolchain that works easily, or something else? |
I see lots of gdb errors, so it's likely that needs to be configured properly. e.g.
I wonder if a docker container would make sense to encapsulate the test environment? |
There is already one based on cdt-infra, you can do something like this if you are in the root of the project. (Or just run bash in the container and do things manually)
For ease of use a "latest" tag would probably make sense - perhaps also one that does not start UI/VNC environment. |
The state of GDB on macOS isn't so great, so there is probably nothing else to do other than fix GDB itself. |
@jonahgraham I see you updated instructions in #144 On my machine this errors unfortunately:
|
I think you need to do the yarn install & yarn build first for the container, cross-env comes in there. |
Yup, that fixed the run. Unfortunately 8 of the 19 tests failed :( |
Does qemu work on Macos? Because so much of the focus is on embedded dev, I thought it would be good if we had a "true" cross platform / remote dev environment. That may be easier? |
I suspect that the 8 out of 19 failed is because the elf files were not rebuilt in the container? I can reproduce something similar. I have added to the documentation on running tests to say that you should be fully clean when changing platforms (the change is implicit when going from host to container run). |
See PR #146 |
We have fairly decent tests now - some more work to be done in #160 |
Time to put together a test strategy for the adapter. We want to be able to confidently make changes without breaking the clients.
We have experience with mocha but I hear that it doesn't have good Jenkins reporting which we need for the CI machine.
Anyone else have any ideas?
The text was updated successfully, but these errors were encountered: