integration tests

[drieseng]

We currently have a (non-public) IntegrationTests repository that itself uses SshNet.TestTools.OpenSSH project from the TestTools repository.

There were two reasons for having a separate repository for these integration tests:

• I am not 100% happy with neither the code nor the setup.
  This is food for another discussion.
• The integration tests project consumes SSH.NET using our public NuGet package(s) hereby allowing us to easily run the latest tests again an older version of SSH.NET.

Since we publish no NuGet package for SshNet.TestTools.OpenSSH, we rely on it being available locally. This was ok(ish) when I was the only one working on this, but this is something I want to fix.

I'd prefer not to publish SshNet.TestTools.OpenSSH on nuget.org as I made it to only cover our specific needs.

The way I see it, we have the following options:

1. Create a myget feed, and publish SshNet.TestTools.OpenSSH (with a better name?) to that feed (and consume it from that feed).
   We can eventually also publish continuous builds of SSH.NET to that feed.
2. Move both SshNet.TestTools.OpenSSH and the integration tests to the main SSH.NET repository.

Do you see other options?

In case of the second option, do we configure the integration tests project to use the SSH.NET assembly from the repository or will we continue to consume it as a NuGet package?

cc: @daviburg, @IgorMilavec, @zybexXL

[drieseng]

If we keep the integration tests in a separate repo, how do you see the workflow for running the integration tests again the branch of SSH.NET that you're working on?

[daviburg]

Is SshNet.TestTools.OpenSSH in SSH.NET repo because it is also consumed locally there?
If SshNet.TestTools.OpenSSH is consumed in two distinct repositories, it should be published from its own repository.
Would there be a circular dependency between SshNet.TestTools.OpenSSH and SSH.NET?

[drieseng]

Is SshNet.TestTools.OpenSSH in SSH.NET repo because it is also consumed locally there?

It is only used in the IntegrationTests repo.
If we decide to keep the integration tests outside the main SSH.NET repo, then we could of course still merge the TestTools repo (which contains SshNet.TestTools.OpenSSH) into the IntegrationTests repo.

If SshNet.TestTools.OpenSSH is consumed in two distinct repositories, it should be published from its own repository. Would there be a circular dependency between SshNet.TestTools.OpenSSH and SSH.NET?

It is only consumed in the IntegrationTests repo.

Do you prefer to keep the integration tests in its proper repo instead of integrating it into the main SSH.NET repo?

[daviburg]

You wrote initially that the reason to have a separate repo were:

• I am not 100% happy with neither the code nor the setup.
  This is food for another discussion.

This should be a short-term issue so probably it should not drive long term where the code lives.

• The integration tests project consumes SSH.NET using our public NuGet package(s) hereby allowing us to easily run the latest tests again an older version of SSH.NET.

Consuming SSH.NET from NuGet is possible regardless of where the integration test code lives. Even if it lives in the same repo, it can consume an older version of the published NuGet from its own repo.
It is a bit scary because having everything in the same repo has the appearance of a circular dependency. It also creates the temptation for developers to only test integration test changes against the current SSH.NET or to change the integration tests to fit a local change in SSH.NET but not working for backward compatibility.
On the other hand, it also allows running the integration tests again local SSH.NET code, therefore allowing to coordinate changing code in both SSH.NET and its integration tests for new capabilities.
Would it be possible for the integration tests to run twice, once against the current published NuGet, once against the local SSH.NET code, therefore driving the backward compatibility of changes?

@IgorMilavec, @zybexXL could you share your thoughts as well?

[WojciechNagorski]

I would like to improve integration tests in SSH.NET. My idea:

1. Move the integration tests repo to one SSH.NET repo.
2. Use the excellent library TestContainers for running containers. This library can build Dockerfile, run container and then clean up everything. Thanks to this library we can run integration tests from Visual Studio like normal unit tests, so it would be easier to prepare a PR for the community.
3. Refactor integration tests that currently are in SSH.NET and run them using docker.
4. Run integration tests inside CI/CD.
5. Optional: Migrate to GitHub Actions
6. Optional: Collect code coverage for unit tests and integration tests and merge reports and show the report in PR. Using RaportGenerator. There is an example report https://reportgenerator.io/resources/reports/Html/index.html

I would like to do this in stages. I've done similar tasks in the past so I know how to do it.

[drieseng]

@WojciechNagorski, can you prepare a PR where you do the setup and "move" a single integration test into the main repo?

[WojciechNagorski]

@drieseng yes. This is good idea. I will prepare a PR next week.