host.package().is_installed is returning True incorrectly when the RPM database is corrupted.

[narmaku]

Summary

When calling host.package().is_installed in a system using RPMs (in our case, RHEL), and the actual RPM database is corrupted, the command returns True, but the package could not even be queried to know if it's installed or not.

We got a false positive in this test case:

[gw3] linux -- Python 3.9.16 /opt/app-root/bin/python3.9
[gw3] linux -- Python 3.9.16 /opt/app-root/bin/python3.9[gw3] linux -- Python 3.9.16 /opt/app-root/bin/python3.9

self = <test_aws.TestsAWS object at 0x7fba46f33c70>
host = <testinfra.host.Host paramiko://<masked>.us-gov-east-1.compute.amazonaws.com>

    @pytest.mark.run_on(['rhel'])
    def test_unwanted_packages_are_not_present(self, host):
        """
        Some pkgs are not required in EC2.
        BugZilla 1888695, 2075815
        """
        unwanted_pkgs = [
            'aic94xx-firmware', 'alsa-firmware', 'alsa-lib', 'alsa-tools-firmware',
            'ivtv-firmware', 'iwl1000-firmware', 'iwl100-firmware', 'iwl105-firmware',
            'iwl135-firmware', 'iwl2000-firmware', 'iwl2030-firmware', 'iwl3160-firmware',
            'iwl3945-firmware', 'iwl4965-firmware', 'iwl5000-firmware', 'iwl5150-firmware',
            'iwl6000-firmware', 'iwl6000g2a-firmware', 'iwl6000g2b-firmware', 'iwl6050-firmware',
            'iwl7260-firmware', 'libertas-sd8686-firmware', 'libertas-sd8787-firmware', 'libertas-usb8388-firmware',
            'firewalld', 'biosdevname', 'plymouth', 'iprutils', 'rng-tools', 'qemu-guest-agent'
        ]
    
        system_release = version.parse(host.system_info.release)
    
        # BugZilla 1888695
        if version.parse('8.3') > system_release >= version.parse('8.0'):
            unwanted_pkgs.remove('rng-tools')
    
        if system_release < version.parse('8.5'):
            unwanted_pkgs.remove('qemu-guest-agent')
    
        if test_lib.is_rhel_sap(host):
            # In RHEL SAP images, alsa-lib is allowed
            unwanted_pkgs.remove('alsa-lib')
    
        if test_lib.is_rhel_high_availability(host):
            unwanted_pkgs.append('rh-amazon-rhui-client')
    
        found_pkgs = [
            pkg for pkg in unwanted_pkgs if host.package(pkg).is_installed]
    
>       assert len(
            found_pkgs) == 0, f'Found unexpected packages installed: {", ".join(found_pkgs)}'
E       AssertionError: Found unexpected packages installed: alsa-lib
E       assert 1 == 0
E        +  where 1 = len(['alsa-lib'])

test_suite/cloud/test_aws.py:233: AssertionError[gw3] linux -- Python 3.9.16 /opt/app-root/bin/python3.9

Source: https://github.com/osbuild/cloud-image-val/blob/5e2d95961646dd66e5962c9f8c65133713b1ee40/test_suite/cloud/test_aws.py#L199

I found this issue debugging the tool and adding more information in some of our test results, as you can see below:

[gw11] linux -- Python 3.9.16 /opt/app-root/bin/python3.9
[gw11] linux -- Python 3.9.16 /opt/app-root/bin/python3.9[gw11] linux -- Python 3.9.16 /opt/app-root/bin/python3.9

self = <test_aws.TestsAWS object at 0x7fdcc584bd60>
host = <testinfra.host.Host paramiko://<masked>.af-south-1.compute.amazonaws.com>

    @pytest.mark.pub
    @pytest.mark.run_on(['rhel'])
    def test_rhui_pkg_is_installed(self, host):
        with host.sudo():
            # CLOUDX-590
            if host.run('rm -f /var/lib/rpm/__db.*').failed or host.run('rpm --rebuilddb').failed:
                change_permissions_cmd = 'chmod 755 /var/lock /var/lock/rpm ' \
                    '&& chown root.lock /var/lock ' \
                    '&& chown root.root /var/lock/rpm'
                assert host.run_test(change_permissions_cmd), 'Failed to change permissions'
                if host.file('/var/lock/rpm/transaction').exists:
                    assert host.run_test('rm -f /var/lock/rpm/transaction'), 'Failed to remove the transaction file'
    
        unwanted_rhui_pkgs = None
    
        if test_lib.is_rhel_high_availability(host):
            required_rhui_pkg = 'rh-amazon-rhui-client-ha'
        elif test_lib.is_rhel_sap(host):
            required_rhui_pkg = 'rh-amazon-rhui-client-sap-bundle'
        else:
            required_rhui_pkg = 'rh-amazon-rhui-client'
            unwanted_rhui_pkgs = [
                'rh-amazon-rhui-client-ha',
                'rh-amazon-rhui-client-sap',
            ]
    
        test_lib.print_host_command_output(host, 'rpm -qa | grep rhui')
    
        if unwanted_rhui_pkgs:
            for pkg in unwanted_rhui_pkgs:
                assert host.run(f'rpm -qa | grep {pkg}').failed, \
                    f'Unexpected rhui package installed: {pkg}'
    
>       assert host.run(f'rpm -qa | grep {required_rhui_pkg}').succeeded, \
            f'Package "{required_rhui_pkg}" should be present'
E       AssertionError: Package "rh-amazon-rhui-client" should be present
E       assert False
E        +  where False = CommandResult(command=b'rpm -qa | grep rh-amazon-rhui-client', exit_status=1, stdout=None, stderr=b'error: db5 error(-30986) from dbcursor->c_get: BDB0075 DB_PAGE_NOTFOUND: Requested page not found\n').succeeded
E        +    where CommandResult(command=b'rpm -qa | grep rh-amazon-rhui-client', exit_status=1, stdout=None, stderr=b'error: db5 error(-30986) from dbcursor->c_get: BDB0075 DB_PAGE_NOTFOUND: Requested page not found\n') = <bound method Host.run of <testinfra.host.Host paramiko://<masked>.af-south-1.compute.amazonaws.com>>('rpm -qa | grep rh-amazon-rhui-client')
E        +      where <bound method Host.run of <testinfra.host.Host paramiko://<masked>.af-south-1.compute.amazonaws.com>> = <testinfra.host.Host paramiko://<masked>.af-south-1.compute.amazonaws.com>.run

test_suite/cloud/test_aws.py:348: AssertionError[gw11] linux -- Python 3.9.16 /opt/app-root/bin/python3.9
 ------------------------------Captured stdout call------------------------------ 
---------------------------------------------------------------------------
                            rpm -qa | grep rhui
---------------------------------------------------------------------------
Exit code: 1

Stdout:


Stderr:
error: db5 error(-30986) from dbcursor->c_get: BDB0075 DB_PAGE_NOTFOUND: Requested page not found

Source: https://github.com/osbuild/cloud-image-val/blob/5e2d95961646dd66e5962c9f8c65133713b1ee40/test_suite/cloud/test_aws.py#L320

Expected behavior

is_installed fails since it cannot check for the package existence as rpm command crashed unexpectedly due to a DB error, or any other reason. And this reason is thrown as an a exception so the user know what happened.

Actual behavior

is_installed returns True when there is a rpm command failure and it is returning 1.

Suggestion / Proposal

Make RPM database integrity check, or at least improve the is_installed function to throw an exception when the rpm query could not be done (considering that other errors can actually also return 1 when running rpm command).

We can check the stdout to really know that the package is not installed:

package <pkg> is not installed (return code 1)

If not matching, we can print the actual stdout + stderr, or the CommandOutput object itself, such as it is being done in the rest of the project.