pip-compile manager doesn't submit any PRs

[craigds]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

I've added renovate and configured it for pip-compile as best as I can according to the docs, matching three lockfiles.

It seems to find a couple of security vuln alerts from github, and then... it doesn't do anything? No PRs for the security vulns, no PRs for anything else either.

I did get PRs for some non-pip-compile things (docker, docker-compose, buildkite), but I've now disabled those to see if it helps the pip-compile manager (it didn't)

I've configured the update frequency as high as I can find but it hasn't helped.

The dashboard PR says:

Detected dependencies
None detected

There are definitely updates that need doing. if I run pip-compile locally against all three lockfiles I get:

 requirements/local.txt      |  2 +-
 requirements/production.txt | 18 +++++++++---------
 requirements/test.txt       | 12 ++++++------
 3 files changed, 16 insertions(+), 16 deletions(-)

Logs (if relevant)

Full log

It's not really clear to me which bits of the logs are relevant to it not finding any updates to make, but how about this:

Logs

...
DEBUG: Repository config
{
  "fileName": "renovate.json"
  "config": {
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [
      "config:recommended"
    ],
    "pip_requirements": {
      "enabled": false
    },
    "pip_setup": {
      "enabled": false
    },
    "pyenv": {
      "enabled": false
    },
    "buildkite": {
      "enabled": false
    },
    "docker-compose": {
      "enabled": false
    },
    "dockerfile": {
      "enabled": false
    },
    "pip-compile": {
      "fileMatch": [
        "(^|/)requirements/production\\.txt$",
        "(^|/)requirements/test\\.txt$",
        "(^|/)requirements/local\\.txt$"
      ],
      "lockFileMaintenance": {
        "enabled": true,
        "branchTopic": "pip-compile-refresh",
        "commitMessageAction": "Refresh pip-compile outputs",
        "schedule": [
          "at any time"
        ]
      }
    },
    "constraints": {
      "python": "==3.10"
    },
    "branchPrefix": "renovate_",
    "baseBranches": [
      "test-renovate"
    ]
  }
}

...

DEBUG: GitHub vulnerability details
{
  "alerts": {
    "pip/sqlparse": {
      "< 0.5.0": "0.5.0"
    }
  }
}

DEBUG: alert package rules
{
  "alertPackageRules": [
    {
      "matchDatasources": [
        "pypi"
      ],
      "matchPackageNames": [
        "sqlparse"
      ],
      "matchCurrentVersion": "== 0.4.4",
      "matchFileNames": [
        "requirements/local.txt"
      ],
      "allowedVersions": "==0.5.0",
      "prBodyNotes": [
        "### GitHub Vulnerability Alerts",
        "#### [CVE-2024-4340](https://github.com/andialbrecht/sqlparse/security/advisories/GHSA-2m57-hf25-phgg)\n\n### Summary\nPassing a heavily nested list to sqlparse.parse() leads to a Denial of Service due to RecursionError.\n\n### Details + PoC\nRunning the following code will raise Maximum recursion limit exceeded exception:\n```py\nimport sqlparse\nsqlparse.parse('[' * 10000 + ']' * 10000)\n```\nWe expect a traceback of RecursionError:\n```py\nTraceback (most recent call last):\n  File \"trigger_sqlparse_nested_list.py\", line 3, in <module>\n    sqlparse.parse('[' * 10000 + ']' * 10000)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/__init__.py\", line 30, in parse\n    return tuple(parsestream(sql, encoding))\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/filter_stack.py\", line 36, in run\n    stmt = grouping.group(stmt)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 428, in group\n    func(stmt)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 53, in group_brackets\n    _group_matching(tlist, sql.SquareBrackets)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 48, in _group_matching\n    tlist.group_tokens(cls, open_idx, close_idx)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 328, in group_tokens\n    grp = grp_cls(subtokens)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 161, in __init__\n    super().__init__(None, str(self))\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 165, in __str__\n    return ''.join(token.value for token in self.flatten())\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 165, in <genexpr>\n    return ''.join(token.value for token in self.flatten())\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  [Previous line repeated 983 more times]\nRecursionError: maximum recursion depth exceeded\n```\n\n### Fix suggestion\nThe [flatten()](https://github.com/andialbrecht/sqlparse/blob/master/sqlparse/sql.py#L207) function of TokenList class should limit the recursion to a maximal depth:\n```py\nfrom sqlparse.exceptions import SQLParseError\n\nMAX_DEPTH = 100\n\n    def flatten(self, depth=1):\n        \"\"\"Generator yielding ungrouped tokens.\n\n        This method is recursively called for all child tokens.\n        \"\"\"\n    if depth >= MAX_DEPTH:\n        raise SQLParseError('Maximal depth reached')\n        for token in self.tokens:\n            if token.is_group:\n                yield from token.flatten(depth + 1)\n            else:\n                yield token\n```\n\n### Impact\nDenial of Service (the impact depends on the use).\nAnyone parsing a user input with sqlparse.parse() is affected.\n"
      ],
      "isVulnerabilityAlert": true,
      "force": {
        "groupName": null,
        "schedule": [],
        "dependencyDashboardApproval": false,
        "minimumReleaseAge": null,
        "rangeStrategy": "update-lockfile",
        "commitMessageSuffix": "[SECURITY]",
        "branchTopic": "{{{datasource}}}-{{{depName}}}-vulnerability",
        "prCreation": "immediate"
      }
    },
    {
      "matchDatasources": [
        "pypi"
      ],
      "matchPackageNames": [
        "sqlparse"
      ],
      "matchCurrentVersion": "== 0.4.4",
      "matchFileNames": [
        "requirements/production.txt"
      ],
      "allowedVersions": "==0.5.0",
      "prBodyNotes": [
        "### GitHub Vulnerability Alerts",
        "#### [CVE-2024-4340](https://github.com/andialbrecht/sqlparse/security/advisories/GHSA-2m57-hf25-phgg)\n\n### Summary\nPassing a heavily nested list to sqlparse.parse() leads to a Denial of Service due to RecursionError.\n\n### Details + PoC\nRunning the following code will raise Maximum recursion limit exceeded exception:\n```py\nimport sqlparse\nsqlparse.parse('[' * 10000 + ']' * 10000)\n```\nWe expect a traceback of RecursionError:\n```py\nTraceback (most recent call last):\n  File \"trigger_sqlparse_nested_list.py\", line 3, in <module>\n    sqlparse.parse('[' * 10000 + ']' * 10000)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/__init__.py\", line 30, in parse\n    return tuple(parsestream(sql, encoding))\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/filter_stack.py\", line 36, in run\n    stmt = grouping.group(stmt)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 428, in group\n    func(stmt)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 53, in group_brackets\n    _group_matching(tlist, sql.SquareBrackets)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/engine/grouping.py\", line 48, in _group_matching\n    tlist.group_tokens(cls, open_idx, close_idx)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 328, in group_tokens\n    grp = grp_cls(subtokens)\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 161, in __init__\n    super().__init__(None, str(self))\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 165, in __str__\n    return ''.join(token.value for token in self.flatten())\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 165, in <genexpr>\n    return ''.join(token.value for token in self.flatten())\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  File \"/home/uriya/.local/lib/python3.10/site-packages/sqlparse/sql.py\", line 214, in flatten\n    yield from token.flatten()\n  [Previous line repeated 983 more times]\nRecursionError: maximum recursion depth exceeded\n```\n\n### Fix suggestion\nThe [flatten()](https://github.com/andialbrecht/sqlparse/blob/master/sqlparse/sql.py#L207) function of TokenList class should limit the recursion to a maximal depth:\n```py\nfrom sqlparse.exceptions import SQLParseError\n\nMAX_DEPTH = 100\n\n    def flatten(self, depth=1):\n        \"\"\"Generator yielding ungrouped tokens.\n\n        This method is recursively called for all child tokens.\n        \"\"\"\n    if depth >= MAX_DEPTH:\n        raise SQLParseError('Maximal depth reached')\n        for token in self.tokens:\n            if token.is_group:\n                yield from token.flatten(depth + 1)\n            else:\n                yield token\n```\n\n### Impact\nDenial of Service (the impact depends on the use).\nAnyone parsing a user input with sqlparse.parse() is affected.\n"
      ],
      "isVulnerabilityAlert": true,
      "force": {
        "groupName": null,
        "schedule": "[Circular]",
        "dependencyDashboardApproval": false,
        "minimumReleaseAge": null,
        "rangeStrategy": "update-lockfile",
        "commitMessageSuffix": "[SECURITY]",
        "branchTopic": "{{{datasource}}}-{{{depName}}}-vulnerability",
        "prCreation": "immediate"
      }
    }
  ]
}

DEBUG: findIssue(Dependency Dashboard)
DEBUG: Issues cache: synced
DEBUG: Found issue 469
DEBUG: http cache: saving https://api.github.com/repos/koordinates/brainz/issues/469 (etag=W/"77dd19c84b6322e859f70d7428d9ebcd02c940ad6cf257a1c8d7e38a6528f2a8", lastModified=Fri, 10 May 2024 03:13:14 GMT)
DEBUG: baseBranches
{
  "baseBranches": [
    "test-renovate"
  ]
}

DEBUG: baseBranch: test-renovate
{
  "baseBranch": "test-renovate"
}

DEBUG: extract()
{
  "baseBranch": "test-renovate"
}

DEBUG: Cached extract for sha=e487c967635443f340ad3f66e8a8bfead63036cc is valid and can be used
{
  "baseBranch": "test-renovate"
}

DEBUG: Deleted cached dep updates
{
  "baseBranch": "test-renovate"
}

INFO: Dependency extraction complete
{
  "baseBranch": "test-renovate"
  "stats": {
    "managers": {},
    "total": {
      "fileCount": 0,
      "depCount": 0
    }
  }
}

DEBUG: baseBranch: test-renovate
{
  "baseBranch": "test-renovate"
}

DEBUG: PackageFiles.add() - Package file saved for base branch
{
  "baseBranch": "test-renovate"
}

DEBUG: Package releases lookups complete
{
  "baseBranch": "test-renovate"
}

DEBUG: branchifyUpgrades
{
  "baseBranch": "test-renovate"
}

DEBUG: detectSemanticCommits()
{
  "baseBranch": "test-renovate"
}

DEBUG: semanticCommits: returning "enabled" from cache
{
  "baseBranch": "test-renovate"
}

DEBUG: 0 flattened updates found: 
{
  "baseBranch": "test-renovate"
}

DEBUG: Returning 0 branch(es)
{
  "baseBranch": "test-renovate"
}

DEBUG: config.repoIsOnboarded=true
{
  "baseBranch": "test-renovate"
}

DEBUG: packageFiles with updates
{
  "baseBranch": "test-renovate"
  "config": {}
}

DEBUG: detectSemanticCommits()
DEBUG: semanticCommits: returning "enabled" from cache
DEBUG: processRepo()
DEBUG: Processing 0 branches: 
DEBUG: Calculating hourly PRs remaining
DEBUG: currentHourStart=2024-05-10T15:00:00.000+00:00
DEBUG: PR hourly limit remaining: 2
DEBUG: Calculating prConcurrentLimit (10)
DEBUG: 0 PRs are currently open
DEBUG: PR concurrent limit remaining: 10
DEBUG: Calculated maximum PRs remaining this run: 2
DEBUG: PullRequests limit = 2
DEBUG: Calculating hourly PRs remaining
DEBUG: currentHourStart=2024-05-10T15:00:00.000+00:00
DEBUG: PR hourly limit remaining: 2
DEBUG: Calculating branchConcurrentLimit (10)
DEBUG: 0 already existing branches found: 
DEBUG: Branch concurrent limit remaining: 10
DEBUG: Calculated maximum branches remaining this run: 2
DEBUG: Branches limit = 2
DEBUG: ensureDependencyDashboard()
DEBUG: Ensuring Dependency Dashboard
DEBUG: http cache: saving https://api.github.com/repos/koordinates/brainz/issues/469 (etag=W/"77dd19c84b6322e859f70d7428d9ebcd02c940ad6cf257a1c8d7e38a6528f2a8", lastModified=Fri, 10 May 2024 03:13:14 GMT)
DEBUG: No changes to dependency dashboard issue needed
DEBUG: validateReconfigureBranch()
DEBUG: No reconfigure branch found
DEBUG: Removing any stale branches
DEBUG: config.repoIsOnboarded=true
DEBUG: No renovate branches found
DEBUG: PackageFiles.clear() - Package files deleted
DEBUG: Repo is activated
DEBUG: Branch summary
{
  "cacheModified": true
  "baseBranches": [
    {
      "branchName": "test-renovate",
      "sha": "e487c967635443f340ad3f66e8a8bfead63036cc"
    }
  ]
  "branches": []
  "defaultBranch": "master"
  "inactiveBranches": []
}
...

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

To get started, please read our guide on creating a minimal reproduction.

Good luck,

The Renovate team

[craigds]

I figured it out!

Turns out the regexes are javascript regexes, and so they needed forward-slash escaped. I don't know wh;y the log couldn't have said something like no files matched <regex> though.

so for future readers the change that fixed it was:

    "fileMatch": [
-      "(^|/)requirements/production\\.txt$",
-      "(^|/)requirements/test\\.txt$",
-      "(^|/)requirements/local\\.txt$"
+      "(^|/)requirements\\/production\\.txt$",
+      "(^|/)requirements\\/test\\.txt$",
+      "(^|/)requirements\\/local\\.txt$"
    ],