Large memory spike at the very end of repository sync

[jlsherrill]

Version
"rpm": "3.23.0",
"core": "3.38.0"

Describe the bug
We recently set our memory limits to 4G on our pulp worker pods, and this was not enough to sync RHEL 8 Appstream. The sync would be killed, but all the steps would be complete.

I did a local sync and saw memory spike at the very end of the sync up to 3.7 GB. Here's a link to the memory stats: https://docs.google.com/spreadsheets/d/1AE3_QKFdQIO6MUSUgWFMm_-fDEzUI6IYurp3-4AE-fA/edit#gid=0

Its very likely that it spiked over 4GB, but due to sampling frequency, i only saw 3.7.

To Reproduce
Sync https://cdn.redhat.com/content/dist/rhel8/8/x86_64/appstream/os (with certs that let you access it)

Expected behavior
Not a giant memory spike if possible?

Additional context
Here's the full task output when its killed please excuse the funky formatting) :

{
pulp_href: "/pulp/ae86a858/api/v3/tasks/018bb161-8629-749b-ae7a-ce6b3a754c28/"
pulp_created: "2023-11-09T00:00:40.492087Z"
state: "failed"
name: "pulp_rpm.app.tasks.synchronizing.synchronize"
logging_cid: "3d30b27738fb479190dbaca9f8811a94"
started_at: "2023-11-09T00:28:48.278736Z"
finished_at: "2023-11-09T01:31:02.102107Z"
error: {
reason: "Killed by signal 9."
}
worker: "/pulp/default/api/v3/workers/018b9689-c1cb-7034-8e91-d4c4866b87ec/"
progress_reports: [
0: {
message: "Parsed Comps"
code: "sync.parsing.comps"
state: "completed"
total: 68
done: 68
suffix: null
}
1: {
message: "Downloading Metadata Files"
code: "sync.downloading.metadata"
state: "completed"
done: 7
suffix: null
}
2: {
message: "Downloading Artifacts"
code: "sync.downloading.artifacts"
state: "completed"
done: 7
suffix: null
}
3: {
message: "Associating Content"
code: "associating.content"
state: "completed"
done: 36908
suffix: null
}
4: {
message: "Parsed Modulemd"
code: "sync.parsing.modulemds"
state: "completed"
total: 717
done: 717
suffix: null
}
5: {
message: "Parsed Modulemd-defaults"
code: "sync.parsing.modulemd_defaults"
state: "completed"
total: 49
done: 49
suffix: null
}
6: {
message: "Parsed Modulemd Obsolete"
code: "sync.parsing.modulemd_obsoletes"
state: "completed"
total: 0
done: 0
suffix: null
}
7: {
message: "Skipping Packages"
code: "sync.skipped.packages"
state: "completed"
total: 6
done: 6
suffix: null
}
8: {
message: "Parsed Packages"
code: "sync.parsing.packages"
state: "completed"
total: 32880
done: 32880
suffix: null
}
9: {
message: "Un-Associating Content"
code: "unassociating.content"
state: "completed"
done: 0
suffix: null
}
10: {
message: "Parsed Advisories"
code: "sync.parsing.advisories"
state: "completed"
total: 3199
done: 3199
suffix: null
}
]
reserved_resources_record: [
0: "/pulp/ae86a858/api/v3/repositories/rpm/rpm/018b1a52-1006-746f-b465-0aa3ad9840a8/"
1: "shared:/pulp/ae86a858/api/v3/remotes/rpm/rpm/018b1a51-fc00-7f3e-9819-d94696313d83/"
2: "shared:/pulp/default/api/v3/domains/018b1a51-cfd6-7016-8872-6bc88173f27b/"
]
}

[dralley]

I've seen this before, it happens only when doing a complete-mirror mode sync, so I presume it's related to creating the publication.

[jlsherrill]

we're actually not using mirror_mode. here's the completed output of the task, and you can see no publication was created:

{
  "pulp_href": "/pulp/75d8fd70/api/v3/tasks/018bce17-9831-7cb1-adc8-3566f00666b0/",
  "pulp_created": "2023-11-14T13:48:54.707083Z",
  "state": "completed",
  "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
  "logging_cid": "a53ebb89bfb242f299d72f702b583e48",
  "created_by": "/pulp/75d8fd70/api/v3/users/1/",
  "started_at": "2023-11-14T13:48:54.771681Z",
  "finished_at": "2023-11-14T13:54:29.556635Z",
  "error": null,
  "worker": "/pulp/default/api/v3/workers/018bce15-cf48-7bac-9aaf-052b63754017/",
  "parent_task": null,
  "child_tasks": [],
  "task_group": null,
  "progress_reports": [
    {
      "message": "Downloading Metadata Files",
      "code": "sync.downloading.metadata",
      "state": "completed",
      "total": null,
      "done": 7,
      "suffix": null
    },
    {
      "message": "Parsed Modulemd",
      "code": "sync.parsing.modulemds",
      "state": "completed",
      "total": 736,
      "done": 736,
      "suffix": null
    },
    {
      "message": "Parsed Modulemd-defaults",
      "code": "sync.parsing.modulemd_defaults",
      "state": "completed",
      "total": 49,
      "done": 49,
      "suffix": null
    },
    {
      "message": "Parsed Modulemd Obsolete",
      "code": "sync.parsing.modulemd_obsoletes",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Skipping Packages",
      "code": "sync.skipped.packages",
      "state": "completed",
      "total": 0,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Parsed Advisories",
      "code": "sync.parsing.advisories",
      "state": "completed",
      "total": 3392,
      "done": 3392,
      "suffix": null
    },
    {
      "message": "Downloading Artifacts",
      "code": "sync.downloading.artifacts",
      "state": "completed",
      "total": null,
      "done": 1,
      "suffix": null
    },
    {
      "message": "Un-Associating Content",
      "code": "unassociating.content",
      "state": "completed",
      "total": null,
      "done": 0,
      "suffix": null
    },
    {
      "message": "Associating Content",
      "code": "associating.content",
      "state": "completed",
      "total": null,
      "done": 39056,
      "suffix": null
    },
    {
      "message": "Parsed Packages",
      "code": "sync.parsing.packages",
      "state": "completed",
      "total": 34810,
      "done": 34810,
      "suffix": null
    },
    {
      "message": "Parsed Comps",
      "code": "sync.parsing.comps",
      "state": "completed",
      "total": 68,
      "done": 68,
      "suffix": null
    }
  ],
  "created_resources": [
    "/pulp/75d8fd70/api/v3/repositories/rpm/rpm/018bce17-9696-7216-bbab-3c708304eaf1/versions/1/"
  ],
  "reserved_resources_record": [
    "/pulp/75d8fd70/api/v3/repositories/rpm/rpm/018bce17-9696-7216-bbab-3c708304eaf1/",
    "shared:/pulp/75d8fd70/api/v3/remotes/rpm/rpm/018bce17-9392-7bc8-b772-e6de14e0ee20/",
    "shared:/pulp/default/api/v3/domains/018bce17-907e-71ba-a255-965ae2145f5a/"
  ]
}

let me grab the remote configuration

[dralley]

Oh, hmm. Interesting.