[config:read-releases] Fetch exact branch

[thegreyd]

We're using ghapi's list_files method which calls get_branch
which fetches the first matched branch with given prefix (!!)
So in my case when I pointed doozer to my fork and said fetch me openshift-4.7,
it returned "openshift-4.7-cachito" branch.

This is pretty alarming behavior, so let's not use methods that depend on get_branch()

Test

(I had to delete the openshift-4.7-cachito branch to unblock my failing job, but
the original behavior should be reproducible)

• doozer --data-path=https://github.com/thegreyd/ocp-build-data --group=openshift-4.14@disable_ptp config:read-releases --yaml

[ashwindasr]

Isn't it a good idea for us to add a new function like get_branch_exact like

# Define a new class that subclasses GhApi
class MyGhApi(GhApi):
    def get_branch_exact(self, branch=None):
        <match the exact branch>

[vfreex]

I think probably we need to avoid using those convenience methods. https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#get-repository-content should be preferred to get file content for it.

[thegreyd]

@vfreex You're right we shouldn't rely on convenience methods. I remember your errata_tool rant :) , we should put that in art-docs for guide to api packages use.

In this case if you see https://github.com/fastai/ghapi they have provided some convenience methods and others methods are generated from openapi spec. So I'd say let's use only the official methods (get_ref and get_tree are). I'll check out the contents endpoint, maybe it's better

[openshift-bot]

Build #5

py38: remove tox env folder /mnt/workspace/jenkins/working/art-tools_doozer_PR-803/.tox/py38
.pkg: remove tox env folder /mnt/workspace/jenkins/working/art-tools_doozer_PR-803/.tox/.pkg
py38: install_deps> python -I -m pip install -r requirements-dev.txt -r requirements.txt
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_wheel> python -I -m pip install wheel
.pkg: freeze> python -m pip freeze --all
.pkg: pip==23.2.1,setuptools==68.0.0,wheel==0.41.0
.pkg: prepare_metadata_for_build_wheel> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py38: install_package_deps> python -I -m pip install 'PyGitHub>=1.46' aiofiles aiohttp bashlex 'click>=8.1.3' defusedxml 'dockerfile-parse>=0.0.13' future ghapi jira==3.4.1 koji 'mysql-connector-python>=8.0.21' 'openshift-client>=1.0.12' 'pydantic~=1.10.7' 'python-dateutil>=2.8.1' 'pyyaml>=5.1' requests requests-kerberos ruamel.yaml semver 'setuptools>=65.5.1' tenacity wrapt
py38: install_package> python -I -m pip install --force-reinstall --no-deps /mnt/workspace/jenkins/working/art-tools_doozer_PR-803/.tox/.tmp/package/5/rh-doozer-0.0.0.tar.gz
py38: freeze> python -m pip freeze --all
py38: aiofiles==23.1.0,aiohttp==3.8.5,aiosignal==1.3.1,astroid==2.15.6,async-timeout==4.0.2,attrs==23.1.0,autopep8==2.0.2,bashlex==0.18,bcrypt==4.0.1,cachetools==5.3.1,certifi==2023.7.22,cffi==1.15.1,chardet==5.1.0,charset-normalizer==3.2.0,click==8.1.6,colorama==0.4.6,coverage==7.2.7,cryptography==41.0.2,decorator==5.1.1,defusedxml==0.7.1,Deprecated==1.2.14,dill==0.3.7,distlib==0.3.7,dockerfile-parse==2.0.1,exceptiongroup==1.1.2,fastcore==1.5.29,filelock==3.12.2,flake8==6.0.0,flexmock==0.11.3,frozenlist==1.4.0,future==0.18.3,ghapi==1.0.4,gssapi==1.8.2,idna==3.4,iniconfig==2.0.0,isort==5.12.0,jira==3.4.1,koji==1.33.1,krb5==0.5.0,lazy-object-proxy==1.9.0,mccabe==0.7.0,multidict==6.0.4,mysql-connector-python==8.1.0,oauthlib==3.2.2,openshift-client==1.0.19,packaging==23.1,paramiko==3.2.0,pip==23.2.1,platformdirs==3.9.1,pluggy==1.2.0,protobuf==4.21.12,pycodestyle==2.10.0,pycparser==2.21,pydantic==1.10.12,pyflakes==3.0.1,pygit2==1.10.1,PyGithub==1.59.0,PyJWT==2.8.0,pylint==2.17.5,PyNaCl==1.5.0,pyproject-api==1.5.3,pyspnego==0.9.1,pytest==7.4.0,python-dateutil==2.8.2,PyYAML==6.0.1,requests==2.31.0,requests-gssapi==1.2.3,requests-kerberos==0.14.0,requests-oauthlib==1.3.1,requests-toolbelt==1.0.0,rh-doozer @ file:///mnt/workspace/jenkins/working/art-tools_doozer_PR-803/.tox/.tmp/package/5/rh-doozer-0.0.0.tar.gz#sha256=e895bbace6e0fe24302779ac2bbd9a6429175e7d943c4f9eb5085c92fd7ba19d,ruamel.yaml==0.17.32,ruamel.yaml.clib==0.2.7,semver==3.0.1,setuptools==68.0.0,six==1.16.0,tenacity==8.2.2,tomli==2.0.1,tomlkit==0.12.1,tox==4.6.4,typing==3.7.4.3,typing_extensions==4.7.1,urllib3==2.0.4,virtualenv==20.24.2,wheel==0.41.0,wrapt==1.15.0,yarl==1.9.2
py38: commands[0]> coverage run --branch --source doozerlib -m unittest discover -t . -s tests/
..................................................................................................................................................s.s....................................s.s...s.s...s..s.s.s....................................../mnt/workspace/jenkins/working/art-tools_doozer_PR-803/doozerlib/repodata.py:175: RuntimeWarning: coroutine 'AsyncMockMixin._execute_mock_call' was never awaited
Coroutine created at (most recent call last)
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 1851, in _run_once
    handle._run()
  File "/usr/lib64/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib64/python3.8/unittest/async_case.py", line 102, in _asyncioLoopRunner
    ret = await awaitable
  File "/usr/lib64/python3.8/unittest/mock.py", line 1342, in patched
    return await func(*newargs, **newkeywargs)
  File "/mnt/workspace/jenkins/working/art-tools_doozer_PR-803/tests/test_repodata.py", line 214, in test_load
    repodata = await loader.load(repo_name, repo_url)
  File "/mnt/workspace/jenkins/working/art-tools_doozer_PR-803/doozerlib/repodata.py", line 175, in load
    resp.raise_for_status()
  File "/usr/lib64/python3.8/unittest/mock.py", line 1081, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib64/python3.8/unittest/mock.py", line 1085, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  resp.raise_for_status()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
......................................................
----------------------------------------------------------------------
Ran 287 tests in 2.543s

OK (skipped=10)
py38: commands[1]> flake8
py38: commands[2]> coverage report
Name                                          Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------
doozerlib/__init__.py                            12      7      2      1    43%
doozerlib/_version.py                             2      2      0      0     0%
doozerlib/assembly.py                           158     20     87     11    84%
doozerlib/assembly_inspector.py                 220    158    128      8    24%
doozerlib/assertion.py                           13      0      6      0   100%
doozerlib/brew.py                               366    206    152      4    40%
doozerlib/build_status_detector.py               85     10     54      3    86%
doozerlib/cli/__init__.py                       122     64     28      0    39%
doozerlib/cli/__main__.py                      1210   1210    480      0     0%
doozerlib/cli/cli_opts.py                        20      3      8      0    89%
doozerlib/cli/config_plashet.py                 536    536    246      0     0%
doozerlib/cli/config_tag_rpms.py                141     20     77     15    82%
doozerlib/cli/detect_embargo.py                 167     35     70      8    75%
doozerlib/cli/get_nightlies.py                  230     59    127      3    71%
doozerlib/cli/images_health.py                   82     30     26      2    59%
doozerlib/cli/images_streams.py                 697    619    306      2     8%
doozerlib/cli/inspect_stream.py                  66     66     28      0     0%
doozerlib/cli/release_calc_upgrade_tests.py      24     24      6      0     0%
doozerlib/cli/release_gen_assembly.py           280    146    120      3    43%
doozerlib/cli/release_gen_payload.py            703    260    300     20    58%
doozerlib/cli/rpms_build.py                     165     59     58      8    57%
doozerlib/cli/rpms_read_config.py                15     15      2      0     0%
doozerlib/cli/scan_sources.py                   185    143    100      2    17%
doozerlib/comment_on_pr.py                       46      0      8      0   100%
doozerlib/config.py                              97     97     44      0     0%
doozerlib/constants.py                           12      0      0      0   100%
doozerlib/coverity.py                           255    225     82      0     9%
doozerlib/dblib.py                              263    160     68      4    35%
doozerlib/distgit.py                           1454    901    690     40    36%
doozerlib/dotconfig.py                           54     43     31      0    13%
doozerlib/exceptions.py                          14      5      0      0    64%
doozerlib/exectools.py                          226    110     80     10    48%
doozerlib/gitdata.py                            171    137     76      0    14%
doozerlib/image.py                              491    303    198      8    33%
doozerlib/logutil.py                              9      0      2      1    91%
doozerlib/metadata.py                           432    151    184     30    61%
doozerlib/model.py                              113     21     36      2    82%
doozerlib/olm/__init__.py                         0      0      0      0   100%
doozerlib/olm/bundle.py                         315    230     72      0    22%
doozerlib/osbs2_builder.py                      119     30     44     19    67%
doozerlib/plashet.py                            134      9     90     15    89%
doozerlib/pushd.py                               22      0      2      0   100%
doozerlib/release_schedule.py                    28     17      8      0    31%
doozerlib/repodata.py                           213     19     84     10    90%
doozerlib/repos.py                              239    110    123     18    48%
doozerlib/rhcos.py                              233     25     84     13    87%
doozerlib/rpm_builder.py                        232     32    123     32    80%
doozerlib/rpm_delivery.py                        16      1      0      0    94%
doozerlib/rpm_utils.py                          143     31     90      9    79%
doozerlib/rpmcfg.py                             151     61     64      8    55%
doozerlib/runtime.py                            907    690    374      7    18%
doozerlib/source_modifications.py               116     33     26      4    68%
doozerlib/state.py                               23     12      8      0    35%
doozerlib/util.py                               455    217    182     13    49%
-------------------------------------------------------------------------------
TOTAL                                         12482   7362   5284    333    38%
.pkg: _exit> python /home/art/.local/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  py38: OK (40.62=setup[33.10]+cmd[4.53,1.24,1.75] seconds)
  congratulations :) (40.93 seconds)

[locriandev]

I left a minor comment, LGTM otherwise. Thanks for catching this!

[thegreyd]

repos.get_content works as expected with ref=branch/commit/tag, I've replaced it now.
But this breaks some of the current expectations we have for data-path like local paths.. so we should think of a long term solution for this in runtime.initialize