Skip to content

v0.72.0 (agent only)
Compare
Choose a tag to compare
@ndokos ndokos released this 06 Jun 08:13
· 11 commits to b0.72 since this release
v0.72.0
4391fbc
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

This is a minor release of the Pbench agent. It consists mostly of bug fixes and deletions of deprecated components that were announced previously.

The most visible parts of the changes are summarized below. The full change log can be found at Changelog, but note that most of the 408 commits are not for the agent: the list includes server and dashboard changes which are already incorporated into the current production Pbench server and dashboard.

Installation

The COPR repo names have changed: the pbench-agent RPM is now found in the pbench-0.72 repo. The reason for this change was that COPR does not allow us to keep different versions of the RPM in the same repo: it deleted the older one as soon as a newer one was built. We needed that capability however, so we chose to go with separate repos for each release.

There are some RPMs that are shared between versions (e.g. pbench-sysstat). We maintain those in the original COPR  pbench repo . The upshot is that the user now has to install two COPR repos (if doing it manually). The Ansible roles have been modified to do that, so if you are installing through Ansible, you don’t need to worry about that, except for adding the following line to the [servers:vars] section of the inventory file:

pbench_repo_name = pbench-0.72

The inventory file should look like this:

[servers]
<host1>
<host2>
...

[servers:vars]
pbench_repo_name = pbench-0.72

pbench_key_url =  <URL to directory containing the key>
pbench_config_url = <URL to directory containing the config file>

The version of the pbench-agent RPM for this release is 0.72.0-1g4391fbc01.

Ansible roles

New Ansible roles have been uploaded to Ansible Galaxy, so you will need to update your installation of those roles:

ansible-galaxy collection install pbench.agent -f

In addition, there is no default for pbench_repo_name any longer (see above for an explanation). You will need to set it in your inventory file like this:

...
[servers:vars]
pbench_repo_name = pbench-0.72

pbench_key_url =  <URL for ssh key>
pbench_config_url =  <URL for config file>

New user-visible utilities

The pbench-tools-kill script has been added. This is a new utility intended to provide complete cleanup if some Tool Meister component refuses to start. This is usually because there are old processes running and keeping network ports busy. The tool cleans up such errant processes so you can start afresh.

Bug fixes and enhancements

The Tool Meister subsystem has undergone a few fixes and some enhancements, primarily in logging and reporting of status; also, the state-signals work was integrated into the Tool Meister (thanks Mustafa Eyceoz!). pbench-linpack has had some fixes (primarily thanks to Lukas Doktor). In addition, pbench-specjbb, pbench-uperf and pbench-fio have had bug fixes.

The user-tool script was broken and that necessitated a few changes to the Tool Meister and also to pbench-postprocess-tools. Thanks to Keith Valin for finding the problem and to Michey Mehta for debugging it.

As usual, if you find problems, please open an issue on Github.

Support for latest RHEL and Fedora versions

V0.72.0 supports RHEL 8.8, RHEL9.2 and Fedora 37 and 38, in addition to the previously supported RHEL versions. Fedora 36 is not supported any longer (primarily because COPR has dropped it).

Deprecation notices and deletions of previously deprecated items

A default tool set was implicitly used by pbench-register-tool-set . It was deprecated in v0.71.0 and is still deprecated in v0.72.0 - it will finally go away in the next release and you will need to choose a tool set explicitly when registering tools. The name for what used to be the default tool set is legacy. In addition, there are light, medium and heavy tool sets. Not supplying an argument for the tool set is still a warning but it is going to become an error in the next release.

The pbench-generate-token command (see the "Futures" section below) is deprecated and will be deleted in the next release of the agent.

The following have been previously deprecated and have now been deleted: pbench-run-benchmark, pbench-cyclictest, pbench-dbench, pbench-iozone, pbench-migrate, pbench-netperf . In addition, two contributed bench-scripts, pbench-bzt and pbench-mpt, have been deleted.

The stockpile subproject has been removed, as well as the script pbench-avg-stddev which was unused.

Futures

The following describes some details about future directions. One component of that is containerization. The Pbench server on production is already running in a container. Here we describe the current, experimental version of the containerized agent. The second component is user authentication and ownership of datasets. That is work in progress and what we describe here is what is available in the v0.72.0 release of the Pbench agent and the current production Pbench server.

This section is meant as a foretaste of things to come. We expect most users to continue using RPMs for installing the agent and the pbench-move-results (or pbench-copy-results) utility to upload datasets, just as with previous versions of the Pbench agent.

If you'd like to kick the tires a bit, read on and feel free to experiment, although we recommend that you don't use the following for "real" data. Things should work but that's not guaranteed: if you do venture forth and encounter problems, we would really like to know about them. Thanks in advance!

Containerized Pbench agent

There is an experimental Pbench agent container, intended as a demonstration project, available in the contrib/containerized-pbench directory of the b0.72 branch (the branch that was used to cut this release). The directory contains a README file, a pbench command and a demo script, pbench-demo.

The demo script (which is to be thought of more as "executable documentation" than anything else at this point) uses the pbench command to execute a series of commands inside a container. The first time that the pbench command runs, it realizes that there is no container yet, so it downloads the pbench-agent-all-fedora-36:b0.72 image from quay.io and starts the container. It then executes the first command that it was given inside that container. Subsequent invocations of the pbench command execute their arguments inside that container, first registering tools, then listing the tools, then running a simple fio benchmark under pbench-user-benchmark and finally pushing the results to the configured Pbench server. Although this is a very simple set of commands, it indicates how things would go in a more complicated invocation.

There are a couple of significant caveats: this version of the demo script does NOT use pbench-move-results to send the results to the server (although it be could modified to do so). Instead it passes an authentication token to the pbench-results-move command (see below) to push the results. That token is generated by the pbench-generate-token script, which is invoked at the very beginning of the demo script: that script asks for a user ID and a password and then generates and stores that token in a file (the file is stored in a directory which is mapped into the container from the outside, so the token persists beyond the run of the demo script). That means you have to have a user ID and a password on the Pbench server before generating the token.

To create a user ID with a password, you have to visit the Pbench dashboard and click on Login in the upper right. That will pop up a login/sign up dialog through which you can create an account that will then allow you to generate a token (or login to the dashboard and look around). N.B. All data sent to the AWS Pbench Satellite or pbench.app.intlab.redhat.com “pass-through” server is owned by a legacy user: it’s all visible, but can’t be modified.

The trouble is that this is a very temporary arrangement: we expect that very soon, you will be able to use Red Hat SSO for logging in and generating the document. The accounts created as above will go away, as will the pbench-generate-token script which is already deprecated. Any datasets submitted through this mechanism will therefore be orphaned, hence the imprecation to use this to kick the tires, not for storing "real" results that you don't want to lose. We are NOT planning to migrate any such results.

New utilities to upload datasets as an authenticated user

The existing pbench-move-results command works in exactly the same way as before: it uses ssh/scp to copy the results to a pass-through server, but there is no notion of a user owning those
results. Although we expect that to continue to be the main mode of operation for users of v0.72.0, we are moving towards a future where users will authenticate using SSO and that authenticated identity will become the owner of the datasets that are submitted by that user. The new commands pbench-results-move and pbench-results-push use an HTTP PUT to send results to a v1.0 Pbench Server which now provides a RESTful API to its services (pbench-results-push is used by the pass-through server to send the results to a Pbench server using a legacy user ID). The new commands will eventually supplant the existing pbench-move-results. Note that an early Pbench Server 1.0 integration with Pbench Agent introduced the pbench-generate-token command described above (available only Pbench Agent 0.71.0, 0.72.0) which is not compatible with the Pbench Server OIDC model that will allow us to use Red Hat SSO for authentication; the command is now deprecated and will be removed in the next Pbench Agent release.

But we are not quite there yet: we may need to move to HTTPS in the Integration Lab and we need to finish setting up the authentication underpinnings in order to hook into the SSO infrastructure. Until that point, we encourage you to continue with pbench-move-results and the legacy interface to the server. If you have questions about any of this, ping us on GChat.

Changelog (abbreviated - only the agent-side commits are included here)

For the full Changelog, see the link near the top of these notes.

Agent changes between v0.71.0 and v0.72.0:

  • 4391fbc Add a constraints.txt file to limit python-daemon to versions < 3.0.0 (#3448)
  • c560bdf Clarify what tool-specific options are (#3444)
  • 5fe6a7b Fixes for user-tool (#3440)
  • d021d23 Fix the image tag (#3446)
  • 2915816 Disable redis protected mode (#3434)
  • bdc741a Correct the contents of the b0.72 jenkings/branch.name file
  • 3cf0f8c Open firewall ports immediately as well as permanently (backport of #3419 to b0.72) (#3421)
  • 89e45fb Ansible fixes b0.72 (#3407)
  • f6d8e10 Build pbench-tools-kill (#3408)
  • 3f84f33 Retire the previous containerized Agent example from contrib and update the README.md.
  • 9053af1 pbench-results-push: Backport of #3348 and #3378 to b0.72 (#3377)
  • 761445c Separate pbench-config and pbench-server-config
  • 6713d5f Add an explicit ENTRYPOINT to the Agent "all" containers (#3320)
  • d9c95e6 Update default Linux distros: drop Fedora 35 and add Fedora 37 (#3327)
  • a968551 Simple example of running pbench agent in a container
  • e496ad7 Add --output option to pbench-generate-token
  • 26f11d7 Drop controller from PUT agent side (#3220)
  • 0e92342 Add support for building Agent containers to the CI (#3109)
  • e07b6ba Stop using pbench logger in agent tests
  • 95c0c40 Tweak the infrastructure for Click-based commands (#3157)
  • 4292ae8 Add verify host names method used by Tool Meister
  • c2047b3 Send tarball with access param
  • b4a053b Prepare Agent container builds for use in the CI (#3086)
  • 9e5c67a Remove unused helper functions from agent/base
  • 26450d0 Add the new pbench-tools-kill command
  • 76d1aeb State Signals in Pbench
  • 8c3d1e0 Get the current Agent container build working again
  • b472d03 Remove the now unused pbench-avg-stddev (perl) script (#3004)
  • bacdd93 Tweak the help texts for pbench-tool-meister-start and pbench-fio (#3002)
  • 0f4f25b Add RPM builds to the CI pipeline (#2982)
  • b58845e Use major.minor version in COPR repo name
  • fc97eb7 Minor correction to yum invocation
  • 256c812 Correct utils.mk path reference
  • e2a11d8 Correct VERSION variable references
  • 7125060 Correct use of sha1 hash in rpm builds
  • 16de4c9 Remove pbench-run-benchmark
  • 9b9d8b5 Restore legacy tests for the agent
  • 70927a0 Remove default repo name
  • 84cce37 Consolidate ansible directory hierarchy
  • 5de49f2 Restore agent RPM builds
  • e5dc47e Disable legacy tests
  • b6e68ce Remove trafficgen vestiges
  • f8f00c7 PBENCH-697 Delete stockpile submodule (#2971)
  • 42f9508 Accessing results tarballs over http (#2967)
  • a61494e Fix port requirements in pbench-uperf doc (#2969)
  • cbf3b6a Framework for building RPMs in the CI (#2961)
  • e583efd Refactor support for distro/version pattern rules in makefiles
  • 8107277 Initial support for building replacements for COPR chroots (#2957)
  • 676ce68 Enhancements to the CI pipeline (Part 1, Take 2) (#2946)
  • 7105b25 Use the "epel" chroot for "Rocky Linux"
  • 0c8e182 Delete the epel role from pbench_agent_install.yml
  • 2bf939b Add example playbook to open ports for redis/TDS (#2943)
  • 8b602b6 Correct CentOS 7 container builds, update defaults
  • 1ff802e Avoid verify_common_bench_script_options in linpack pre-check (#2929)
  • 0f12276 Fix pbench-linpack options handling bug (#2926)
  • 52e8313 Remove wayward "shbangs" in Python 3 library files (#2917)
  • 32c8526 Correct dependencies in agent spec file
  • 9997558 Default to Python 3.9 for unit tests (#2913)
  • 657adef Ensure fio arguments are used with bash -c
  • f3eca50 Add more tests for pbench-register-tool-set
  • 6ba02ca Fix pbench-register-tool-set argument handling (#2904)
  • 2d97e96 Fix CentOS-Stream-8 repo
  • d8a7fdb CentOS-Stream-8/9 dependency fixes (#2893)
  • 0151038 Add unit tests for the ToolGroup class
  • 6de91b5 Restore default tool set and deprecate
  • beb61d1 Remove the unused check_install_rpm function
  • 4e6b9d4 Remove contibuted pbench-bzt and pbench-mpt
  • a957705 Initial changes to support Python 3.10
  • 2f93c6f Add numactl RPM requirement (#2879)
  • e160efa Tweak container builds for CentOS 9
  • c84c6ed Ensure use of localhost when stopping Redis
  • b51b116 Add local and remote pre-check for linpack
  • ba6a2b7 Stop resolving benchmark binary location
  • 2eff79b Revert "Provide sneaky way to list bench-scripts tests"
  • 32734f3 Revert "Add comments about the magic"
  • c474689 Revert "Remove use of benchmark_bin environment variable"
  • 56ff71d Revert "Add failing tests for uperf and fio behaviors"
  • 7859465 Revert "Correct fio and uperf existence checks"
  • add7416 Revert "Correct local assigment then return code check"
  • 04715fb Revert "Apply a round of code-review comments, still w which"
  • e79ad4d Revert "Replace which with command -v"
  • cd70a64 Revert "Stop resolving benchmark binary location"
  • 226e733 Stop resolving benchmark binary location
  • 42a8d12 Replace which with command -v
  • 877698f Apply a round of code-review comments, still w which
  • d439921 Correct local assigment then return code check
  • 19cf4ec Correct fio and uperf existence checks
  • 0146624 Add failing tests for uperf and fio behaviors
  • 50f0157 Remove use of benchmark_bin environment variable
  • ad1b263 Add comments about the magic
  • c38030b Provide sneaky way to list bench-scripts tests
  • fbcd7f6 Correct pbench-specjbb2005 use of -send-tools
  • 4227406 pbench-specjbb2005 now handles commented props
  • 1f64660 Add UUIDs to command line of running Tool Meisters
  • ea18664 Fix pbench-fio local fio-server execution
  • 32e07f5 Quick v0.71 fixes (#2821)
  • 70cce64 Isolate parameter transforms to ToolDataSink class
  • 28f95ca Fix Tool Meister instance exit code handling
  • 48dc95c Correct spelling in pidstat tool help text
  • 82b3ca6 Delete the spurious pbench3-devel RPM dependency (#2799)
  • e5e8855 Delete deprecated bench-scripts from agent Makefile (#2802)
  • 55ac486 Initial commit for merging sos and perf data
  • 509ffb3 Agent v0.72, Server v1.0
  • e4c184f Remove deprecated bench-scripts
Assets 2