Key (content_artifact_id, remote_id, pulp_domain_id) already exists during ACS refresh

[ianballou]

Version

  "versions": [
    {
      "component": "core",
      "version": "3.28.5",
      "package": "pulpcore",
      "domain_compatible": true
    },
    {
      "component": "rpm",
      "version": "3.22.3",
      "package": "pulp-rpm",
      "domain_compatible": true
    },
    {
      "component": "python",
      "version": "3.10.0",
      "package": "pulp-python",
      "domain_compatible": false
    },
    {
      "component": "ostree",
      "version": "2.1.1",
      "package": "pulp-ostree",
      "domain_compatible": false
    },
    {
      "component": "file",
      "version": "1.14.3",
      "package": "pulp-file",
      "domain_compatible": true
    },
    {
      "component": "deb",
      "version": "2.21.1",
      "package": "pulp_deb",
      "domain_compatible": false
    },
    {
      "component": "container",
      "version": "2.15.2",
      "package": "pulp-container",
      "domain_compatible": false
    },
    {
      "component": "certguard",
      "version": "1.6.5",
      "package": "pulp-certguard",
      "domain_compatible": true
    },
    {
      "component": "ansible",
      "version": "0.18.0",
      "package": "pulp-ansible",
      "domain_compatible": false
    }
  ],

Describe the bug
When refreshing an ACS with two paths, one failed with the error below:

Error:
Actions::Pulp3::AlternateContentSourceRefreshTest#test_yum_refresh:
RuntimeError: Neutered Exception ForemanTasks::TaskError: Task 3f0bbf30-e8a4-4993-b884-c98ca15aad6d: Katello::Errors::Pulp3Error: 1 subtask(s) failed for task group /pulp/api/v3/task-groups/0189c103-2512-7edd-953e-752f94c795d9/.
Errors:
 {"traceback"=>"  File \"/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py\", line 66, in _execute_task\n    result = func(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 569, in synchronize\n    repo_version = dv.create() or repo.latest_version()\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 161, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py\", line 220, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py\", line 41, in __call__\n    await self.run()\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 311, in run\n    await self._handle_remote_artifacts(batch)\n  File \"/usr/local/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 436, in _handle_remote_artifacts\n    await sync_to_async(RemoteArtifact.objects.bulk_create)(ras_to_create_ordered)\n  File \"/usr/lib/python3.9/site-packages/asgiref/sync.py\", line 448, in __call__\n    ret = await asyncio.wait_for(future, timeout=None)\n  File \"/usr/lib64/python3.9/asyncio/tasks.py\", line 442, in wait_for\n    return await fut\n  File \"/usr/lib64/python3.9/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.9/site-packages/asgiref/sync.py\", line 490, in thread_handler\n    return func(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/manager.py\", line 87, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/query.py\", line 786, in bulk_create\n    returned_columns = self._batched_insert(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/query.py\", line 1831, in _batched_insert\n    self._insert(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/query.py\", line 1805, in _insert\n    return query.get_compiler(using=using).execute_sql(returning_fields)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1822, in execute_sql\n    cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute\n    return self._execute_with_wrappers(\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 80, in _execute_with_wrappers\n    return executor(sql, params, many, context)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/django/db/utils.py\", line 91, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py\", line 89, in _execute\n    return self.cursor.execute(sql, params)\n  File \"/usr/local/lib/python3.9/site-packages/psycopg/cursor.py\", line 723, in execute\n    raise ex.with_traceback(None)\n", "description"=>"duplicate key value violates unique constraint \"core_remoteartifact_content_artifact_id_remo_f11e4ef2_uniq\"\nDETAIL:  Key (content_artifact_id, remote_id, pulp_domain_id)=(0189c0ff-fadc-7c59-8df7-55789e38c170, 0189c103-22e3-779b-a5a7-f924c84c4d50, 0189c0e9-0b5a-78a6-bc20-b705fe6d8247) already exists."}
    foreman-tasks (8.1.2) lib/foreman_tasks.rb:56:in `block in sync_task'
    foreman-tasks (8.1.2) lib/foreman_tasks.rb:55:in `tap'
    foreman-tasks (8.1.2) lib/foreman_tasks.rb:55:in `sync_task'
    /home/vagrant/katello/test/actions/pulp3/orchestration/alternate_content_source_refresh_test.rb:56:in `test_yum_refresh'
    /home/vagrant/katello/test/support/vcr.rb:26:in `block in run'
    /home/vagrant/katello/test/support/vcr.rb:25:in `run'

To Reproduce
Steps to reproduce the behavior:

Refresh ACS with base url 'https://fixtures.pulpproject.org/ ' and subpaths ['rpm-zchunk/', 'rpm-with-modules/']

Expected behavior
Refresh works without error

Additional context
I'm unsure if this will be reproducible easily, it may be due to existing data on my system. I can help with any debugging investigation using my reproducer.

[ianballou]

Unfortunately I'm unable to reproduce this, I think it's a transient failure. Perhaps related to refreshing multiple repositories at once.

[ggainey]

We'll leave this open for now, but it's not under investigation (unless/until we have some idea of a reproducer)