traceback during container startup

[dkliban]

Version
quay.io/pulp/pulp latest 728f8846e816 2 weeks ago 2.18 GB

Describe the bug
The following tracebacks are present in the logs when I first start pulp multi service container:

Checking for database migrations
[oneshot] add-workers: Adding workers 2 through 2
[oneshot] add-workers: Workers will be added shortly after the boot finishes
s6-rc: info: service add-workers successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 364, in <module>
    settings = DjangoDynaconf(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 84, in load
    lazy_settings.populate_obj(django_settings_module)
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 115, in __getattr__
    self._setup()
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 174, in _setup
    self._wrapped = Settings(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 253, in __init__
    self.execute_loaders()
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 1026, in execute_loaders
    settings_loader(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/__init__.py", line 167, in settings_loader
    found = obj.find_file(item, project_root=p_root)
  File "/usr/local/lib/python3.8/site-packages/dynaconf/base.py", line 1156, in find_file
    return find_file(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/dynaconf/utils/files.py", line 62, in find_file
    script_dir = os.path.dirname(os.path.abspath(inspect.stack()[-1].filename))
  File "/usr/lib64/python3.8/inspect.py", line 1526, in stack
    return getouterframes(sys._getframe(1), context)
  File "/usr/lib64/python3.8/inspect.py", line 1503, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/usr/lib64/python3.8/inspect.py", line 1473, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/usr/lib64/python3.8/inspect.py", line 708, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/usr/lib64/python3.8/inspect.py", line 737, in getmodule
    file = getabsfile(object, _filename)
  File "/usr/lib64/python3.8/inspect.py", line 721, in getabsfile
    return os.path.normcase(os.path.abspath(_filename))
  File "/usr/lib64/python3.8/posixpath.py", line 379, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

and also a little bit later

Calling /etc/init/postgres-prepare
Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/pulpcore/app/settings.py", line 364, in <module>
    settings = DjangoDynaconf(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 104, in load
    dynaconf.loaders.execute_hooks(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/__init__.py", line 123, in execute_hooks
    hook_module = py_loader.import_from_filename(
  File "/usr/local/lib/python3.8/site-packages/dynaconf/loaders/py_loader.py", line 92, in import_from_filename
    if filename in [item.filename for item in inspect.stack()]:
  File "/usr/lib64/python3.8/inspect.py", line 1526, in stack
    return getouterframes(sys._getframe(1), context)
  File "/usr/lib64/python3.8/inspect.py", line 1503, in getouterframes
    frameinfo = (frame,) + getframeinfo(frame, context)
  File "/usr/lib64/python3.8/inspect.py", line 1473, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/usr/lib64/python3.8/inspect.py", line 708, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/usr/lib64/python3.8/inspect.py", line 737, in getmodule
    file = getabsfile(object, _filename)
  File "/usr/lib64/python3.8/inspect.py", line 721, in getabsfile
    return os.path.normcase(os.path.abspath(_filename))
  File "/usr/lib64/python3.8/posixpath.py", line 379, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

[mikedep333]

Leaving untriaged, @dkliban will look for the existing dynaconf file issue. I think it was related to the python module "click".

[mikedep333]

@dkliban the prior issue! pulp/pulpcore#3310

[decko]

Yeah, those things could be related.
Regarding this issue, I'm using:

localhost/oci_env/pulp                        base        0544cdfddb59 
ghcr.io/pulp/pulp-ci-centos                   latest      58411c36ad1d 

and I'm not seeing any traceback when starting the container.

For me, the click error message disappeared when I updated the pulp-ci-centos container.
Gonna test with the one that @dkliban pointed out.

[decko]

And just to confirm:

quay.io/pulp/pulp  latest      08639dfc0f94  6 hours ago  2.48 GB

and yes, I'm seeing the error messages.

Maybe some differences in the base of the images?
Need to check on the Containerfile of the pulp-ci-centos image and the one for our production images.

[mikedep333]

This should be fixed in dynaconf 3.1.12, when it comes out.
dynaconf/dynaconf#853

[decko]

Since this is duplicated with pulp/pulpcore#3310, should we close it as duplicated?