Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Light: Add smoke testcases for Example Msg Generator source #4216

Draft
wants to merge 34 commits into
base: master
Choose a base branch
from

Conversation

mitzkia
Copy link
Collaborator

@mitzkia mitzkia commented Nov 16, 2022

  • My goal with this PR series is to add a good test coverage (with the minimal effort) for the supported drivers in Light.
  • Of course these are only a smoke testcases, which did not checks the actual functionality, but checks if syslog-ng could configure/use that option and one message could arrive from source to destination

What are the benefits:

  • bigger (smoke) coverage for driver options
  • this will be always actual for the all options (if a PR introduces a new option for a driver it will be automatically tested) per driver
  • with the minimal testcase effort (with the help of pytest parametrize) we could write smoke testcases for drivers

What are the drawbacks:

  • smoke testcases needs to maintain only in special certain cases when an option introduces a custom (or new) option type.

How this works:

  • We have one API requirement from: contrib/config_option_database/utils/ConfigOptions. There is a function (get_driver_options()) which can provide all options with their option value types for a certain context (source or destination) and driver.
  • For all supported drivers in Light I would add one new testcase file which can iterate in all supported options (in this PR: source_drivers/generator_source/test_generator_source_smoke.py)
  • Smoke testcases needs a new option and value generator (generate_options_and_values_for_driver()). This will provide all possible options with an example value for tescases. With the help of pytest.mark.parametrize Pytest will generate a new testcase from each option and value.
  • In generate_options_and_values_for_driver() we construct a list of dictionaries where one dictionary contains a configured option and value. Example: {'keep-hostname': 'yes'} or {'hook-commands': {'teardown': "'almafa'"}} . These option elements will be the inputs for testcases.
  • In generate_options_and_values_for_driver() when get_driver_options() provide us an option type we will map it to a custom value. Here we need only a valid value, to check if syslog-ng could start with it.

You can check an example run here (under the Light target you can find like this: functional_tests/source_drivers/generator_source/test_generator_source_smoke.py::test_example_msg_generator_source_smoke[use-syslogng-pid_yes]):
https://github.com/mitzkia/syslog-ng/actions/runs/3476660034/jobs/5812084494
or
https://github.com/mitzkia/syslog-ng/actions/runs/3476660034/jobs/5812084655

@github-actions
Copy link
Contributor

No news file has been detected. Please write one, if applicable.

@mitzkia
Copy link
Collaborator Author

mitzkia commented Nov 16, 2022

This PR is depends on: #4181

@mitzkia
Copy link
Collaborator Author

mitzkia commented Nov 16, 2022

If it is needed we can 'mark' smoke testcases in Light, to enable-disable them from the actual run. But in case of GitHub Actions I would enable them to verify if a driver has not got a new regression.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
@mitzkia mitzkia force-pushed the micek_add_smoke_test_for_example_msg_gen branch from f501f3e to e733771 Compare November 21, 2022 13:55
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
We will be using pylint from our own build-time virtualenv.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
This change splits development and runtime related requirements so we
can deploy the two independently. Our root direction had
a dev-like requirements file, which is now renamed to dev-requirements.

Runtime dependencies for various new Python modules are put in
requirements.txt

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Rely on the build system to initialize a dev virtualenv automatically for this purpose.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
@@ -8,6 +8,7 @@ hyrule [devshell, kira] [
requests [kira] [python2, python3]

# pip packages for python functional tests
networkx [centos, fedora, debian, ubuntu, devshell] [python3]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we want to do this, I think this package should be moved out of contrib and probably into scripts.

If we did that, I would also try to run it from inside of the new build-time Python venv (e.g. #4193). This would mean that networkx should be added to dev-requirements.txt in the root and not here.

We might also consider dropping cleaning up/getting rid off pip_packages.manifest.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note.
I have rebased this PR on top of #4193 . From now this PR is depends on #4193 (how to install networkx package)
I have also replaced the commit where networkx has added. The new installation place is dev-requirements.txt file in the root.
After this new smoke testcases could run successfully for me.
If I am correct @alltilla has future plans with config_option_database. Could you please help a little for us?

bazsi and others added 17 commits November 21, 2022 16:47
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Remove the duplicated check for stat(db-file).

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
This allows us to get rid of the use of template->name in this instance.

Also it improves performance as the name-value pair handle is resolved in
init time, instead of at every match.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
…arrays

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
We originally planned feature flips to have an extremal value of zero,
but the G_STATIC_ASSERT() in cfg.h did not accept that.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
With v4.0, we enable all kind of new behaviours which was tested behind
code that enabled that feature flip explicitly.

Reverse this and use the new default by default, and test the legacy one
by flipping the version number to an old one.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
…flip

In preparation to 4.0 where the typing is finally enabled by default,
we don't need the testcase that was testing the experimental feature
transition.

Signed-off-by: Balazs Scheidler <bazsi77@gmail.com>
- this is a fixture which will called for all testcases separately
- this fixture will responsible for closing (teardown) callbacks
- users of this fixture could be other fixtures or testcases itself
- users of this fixture could register their closing (teardown)
mechanism which will called on teardown phase of testcases

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- the registered deinit with atexit will executed only at the end of each
testrun session
- with a new 'teardown' fixture there will be a possibility
to execute separate closing (deinit) mechanism per testcase function

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- this new fixture will allow the config drivers to register their
closing mechanisms, which will be executed on teardown phase for each
testcase

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
…rivers

- 3 drivers could have open file fds: file source, file destination and
example destination
- there was missing a function to close fds for these drivers separately,
this will be updated (added close_file() and related close_writeable_file()
 and close_readable_file())
- at the end we can register new close_file function for drivers

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
…lback

- the registered stop_listener with atexit will executed only at the end of each
testrun session
- with a new 'teardown' fixture there will be a possibility
to execute separate closing (deinit) mechanism per testcase function
- 3 drivers could have open socket fds: network, unix_dgram and unix_stream
destinations
- registering stop_listener function for drivers, this will be executed
on teardown phase for each testcase

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- this will allow for syslog_ng sub components to register their
closing mechanisms, which will be executed on teardown phase for each
testcase
- register syslog_ng.stop() in new teardown fixture

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- there could be an open fd for stderr after each
ConsoleLogReader:wait_for_messages_in_console_log() has been called
- this open fd will be closed with a registration of __stderr_file.close

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- Create write_content_and_close() function: this is simplified function
for a file open, write and close operations
- There were several places where this pattern was used, with this
the code will be more readable, and the internal file close will be forced

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
- this package will be needed for Light to be able to run smoke testcases
with using the exposed API of config_option_database

Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
Signed-off-by: Andras Mitzki <mitzkia@gmail.com>
@mitzkia mitzkia force-pushed the micek_add_smoke_test_for_example_msg_gen branch from e733771 to 3e38158 Compare November 22, 2022 06:24
@MrAnno MrAnno marked this pull request as draft March 7, 2023 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants