Config file: Support Wildcards in directory

[blakemorgan]

Our repo doesn't have a pom.xml file at / but it does in multiple subdirectories (so /*/). Dependabot didn't recognize the wildcard in the directory name when I put that in the config file. Is it possible to get wildcards recognized in the directory value of the config file?

[greysteil]

Frustratingly, this is a bit more difficult on our backend than it sounds. The answer here is pretty similar to:

• Recursively Check for poms #2177

We'd like to support this, and it's on our roadmap (together with grouping projects), but it's a way off.

[bsideup]

Would be nice to have it!

Here is an example where it fails:
https://github.com/bsideup/liiklus/blob/09d16a867b93ca966d6200c6d62f646d0344065b/settings.gradle#L11

[alexf101]

I wrote a script that auto-generates Dependabot configs using globs 😅

Please feel free to use my awful hacks however you like until Dependabot/Github implement this feature properly.

https://gist.github.com/alexf101/b65cbfe7c5a61df7d925589a71d200cf

[jauderho]

+1 from me.

I have a dockerfiles repo with multiple directories representing different images being built. I was dismayed to find out that dependabot.yml only allows specifying one directory.

Even if we are somehow able to do the following would be a start

directory: |
/dir1
/dir2

[michaelmerrill]

Any update on where this is on the roadmap?

[blakemorgan]

For our teams, the Terraform support with Dependabot greatly increased the need for this feature. We have local modules storing common infrastructure for the environments, then a separate folder for each environment. Sometimes, because of weird dependencies, we have to have two folders for each environment. That's upwards of 10 folders for each project. It would be nicer to have one entry in the config file instead of 10.

[gjermundgaraba]

If someone would like a javascript hack as a workaround:

• handlebar template for dependabot file
• "fast-glob" all the different types you need and generate a new dependabot file:

fb.sync([
        `${repoRoot}/**/package.json`,
        `${repoRoot}/**/go.mod`,
        `${repoRoot}/**/pom.xml`,
        `${repoRoot}/**/Dockerfile`,
        // Stuff to ignore:
        `!${repoRoot}/**/node_modules`,
        `!${repoRoot}/**/vendor`])
        .map(mapItToAStructureYouWantToSendToHandleBar)

For monorepos, it is death trying to maintain the dependabot file by hand.

[adamlui]

Knew someone would say something like you. GitHub programmers are paid six figures, we are not paid to do their job, stop trying to excuse laziness (it's been 5 years literally wake up)

[adamlui]

(it is very easy but don't ever code for free for software that is maintained by paid coders unless your time is utterly valueless or you enjoy lazy others ignoring issues for years getting raises/promotions off your unpaid hard work which is just not smart, no other way to put it)

[abdulapopoola]

Multi-dir configuration is now in public beta!; open to hearing your thoughts as you try out the beta!!

Tagging @carlincherry

[mhemmings]

@abdulapopoola @carlincherry Amazing, thanks!

How does this work with Github Actions, since they have slightly different behaviour with regards to directory? Does it fix the ability to update composite actions? (See https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#github-actions)

i.e. I'm hoping the following should now work:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directories:
      - "./github/workflows"
      - "./github/actions/composite-a"
      - "./github/actions/composite-b"
      - "./github/actions/composite-c"
    schedule:
      interval: "weekly"

[blu3mania]

Multi-dir configuration is now in public beta!; open to hearing your thoughts as you try out the beta!!

Tagging @carlincherry

Question, is the beta available to Enterprise Cloud repos?

[lucacome]

I was trying this out and it doesn't seem to work.

Well, it did group all the different directories in the dependabot tab, but then when there was a dependency update it only updated the first go.mod file in the directories list and not the second one.

[cpswan]

I'm seeing similar problems to @lucacome :(

Now rather than getting 4 PRs in one day I'm going to see 4 PRs over 4 days (for the same dependency change), which isn't an improvement on toil, and makes things worse for timely updates.

What's supposed to happen here is all 4 PRs rolled into one.

It makes me wonder if anybody actually tested the 'beta'?

[carlincherry]

How does this work with Github Actions, since they have slightly different behaviour with regards to directory? Does it fix the ability to update composite actions? (See https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#github-actions)

@mhemmings I would expect this to work with composite actions. Did you experience different behavior?

You can try using multi-directory support and grouping rules; that should work to update composite actions. cc @abdulapopoola

[carlincherry]

Question, is the beta available to Enterprise Cloud repos?

@blu3mania yes, this beta should be available to Enterprise Cloud repos!

[carlincherry]

I was trying this out and it doesn't seem to work.

Well, it did group all the different directories in the dependabot tab, but then when there was a dependency update it only updated the first go.mod file in the directories list and not the second one.

@lucacome thank you for raising this to the team; this is unexpected behavior and we've created a bug which our team will prioritize fixing soon.

cc @abdulapopoola

[carlincherry]

@cpswan do you have grouping enabled? cc @abdulapopoola

[adamlui]

I was trying this out and it doesn't seem to work.
Well, it did group all the different directories in the dependabot tab, but then when there was a dependency update it only updated the first go.mod file in the directories list and not the second one.

@lucacome thank you for raising this to the team; this is unexpected behavior and we've created a bug which our team will prioritize fixing soon.

cc @abdulapopoola

@carlincherry since you appear to have the higher authority to prioritize bugs, can this one be added #9071 as it is quite stale and cost me endless hours of extra manual labor that should not be necessary (dependabot is supposed to automate, not create more work than just typing npm update)

It should be quite easy to fix too, it's happens most often when package.json isn't root so the logic for detecting nested manifests should be revised to rely on "node_modules" as an ancestor dir instead

[cpswan]

@cpswan do you have grouping enabled? cc @abdulapopoola

@carlincherry thanks for the pointer. Adding a group seems to get the behaviour I was expecting. It might be worth being more explicit about that in the documentation. Essentially directories: doesn't work (in a meaningful way) without an accompanying groups:.

Here's my working YAML fragment:

  - package-ecosystem: "docker"
    directories:
      - "/some_dockerfiles/"
      - "/another_place/with_dockerfiles/"
    schedule:
      interval: "daily"
    groups:
      docker:
        patterns:
          - "*"

[ffried]

@cpswan do you have grouping enabled? cc @abdulapopoola

@carlincherry thanks for the pointer. Adding a group seems to get the behaviour I was expecting. It might be worth being more explicit about that in the documentation. Essentially directories: doesn't work (in a meaningful way) without an accompanying groups:.

Wouldn't that mean, that you no longer get multiple PRs for different dependencies?
E.g. if a repositories contains three directories, each with an npm project, and each project having e.g. semver and express as dependencies. When using groups as shown above, this would mean, that there's one big PR containing updates to both semver and express for all three directories.
IMHO there should be two PRs, one updating semver in all three directories and one updating express in all three directories.
Otherwise it gets almost impossible to e.g. ignore an update of a dependency (let's say I only want to update semver but not express).

[cpswan]

Wouldn't that mean, that you no longer get multiple PRs for different dependencies? E.g. if a repositories contains three directories, each with an npm project, and each project having e.g. semver and express as dependencies. When using groups as shown above, this would mean, that there's one big PR containing updates to both semver and express for all three directories. IMHO there should be two PRs, one updating semver in all three directories and one updating express in all three directories. Otherwise it gets almost impossible to e.g. ignore an update of a dependency (let's say I only want to update semver but not express).

@ffried you don't have to use a * pattern. You could define different groups for different dependencies. But there's probably an anti-pattern lurking here where it becomes necessary (for some) to have a huge number of groups (on a per dependency basis).

[ffried]

You could define different groups for different dependencies. But there's probably an anti-pattern lurking here where it becomes necessary (for some) to have a huge number of groups (on a per dependency basis).

@cpswan The anti-pattern starts a bit earlier for me. Having to use groups to get Dependabot to do what it normally does by itself (for single directory setups) sounds wrong to me.

[dhensby]

I'm struggling to get this to work as (I) expected with terraform.

Here's an example of my config:

  - package-ecosystem: "terraform"
    directories:
      - "/env/prod"
      - "/env/test"
      - "/env/dev"
    schedule:
      interval: "weekly"
# I have also tried adding the group as described in previous comments
    groups:
      terraform:
        patterns:
          - "*"

However, I only get updates for the first listed directory (/env/prod) instead of updates to all the listed directories.

---

Edit: of course, as soon as I post this, dependabot opens a PR with grouped changes (I had retriggered dependabot previously, but maybe it was waiting for something more specific to happen).

I will agree with others that having to add the groups feels a bit odd and I'd expect that if there was an update to one dependency in all the listed directory, that would effectively become one "group" eg: Updating azuread provider in /env/prod, /env/test, /env/dev. If there were groups defined, then that would be Updating {group name} in ...

[carlincherry]

There's a world where a user sets up multi-directory configuration but also wants the unique PRs vs. grouped PRs generated; that's part of why we opted for the explicitness of requiring a defined group independently of single directory or multiple-directories configuration. I think we could do a better job documenting the expected behavior though, and I have a PR out to our docs team to get this improvement incorporated into the Dependabot docs

[jan-xyz]

I've been following along and waiting for the official release and can attest that I will have the use case for separate PRs.

[lucacome]

I'm not sure I understand the use case of having multi-directory config but wanting separate PRs, but it seems counterintuitive. Wouldn't it make more sense to have different entries in the config to keep things separate then?

To me this feature is more similar to how the updates for GitHub Actions work now, it doesn't matter how many files I have, the same dependency gets updated in one PR across all the files.

I wouldn't want all the updates in the directory list to be grouped together using groups, I just want a PR updating one dependency at a time no matter how many files it's in.

[Snailedlt]

@lucacome That's interesting. I would definitely prefer updating one dependency at a time for solo projects or projects where the team is small. But if I'm working on a repo where multiple teams have ownership over different parts of the code (this is often the case in microservice repos) I absolutely don't want to touch others' code, since then I'll have to get approval from everyone before it can get merged.

I'd say there are usecases for both. 1 is more common, but 2 is more important to get right.
The best solution imo would be to add an option to dependabot. Maybe a new "updateStrategy" attribute with the following possible values: "multiFile", "singleFile", where "singleFile" is the default behaviour we have now

[ipimentelmoraes]

Hello! We've been testing the multi-directories feature, but we've noticed that, even when using groups, dependabot opened duplicated PRs. First, it opened a grouped PR for multiple directories/dependencies (as expected) and, some hours later, separated PRs for dependencies only of the first directory of the list.

For reference, here's our dependabot.yml:

version: 2
updates:
  - package-ecosystem: "npm"
    directories:
      - "actions/"
      - "infra/"
    schedule:
      interval: "daily"
    versioning-strategy: increase
    open-pull-requests-limit: 15 
    allow:
      - dependency-name: "collie"
      - dependency-name: "zoo"
    groups:
      version-updates:
        applies-to: version-updates
        patterns: 
          - "*"

And the PRs opened:

Are we missing anything? We would appreciate some pointers!

[carlincherry]

We're getting close to wildcard functionality! We have a question about behavior you'd like or expect to see when you have overlapping directories. Give us your thoughts in this straw poll! #9823

Given an existing configuration file like 👇

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "monthly"
    reviewers: ["@person1"]

And a user adds the following entry 👇

  - package-ecosystem: "npm"
    directories:
      - "**/*"
    schedule:
      interval: "daily"
    reviewers: ["@person2"]

What would you expect or want to happen? Answer in the poll! #9823

[dhensby]

There's a world where a user sets up multi-directory configuration but also wants the unique PRs vs. grouped PRs generated; that's part of why we opted for the explicitness of requiring a defined group independently of single directory or multiple-directories configuration. I think we could do a better job documenting the expected behavior though, and I have a PR out to our docs team to get this improvement incorporated into the Dependabot docs

@carlincherry Do you have a link to the docs as the only thing I can find about the directories option is in the blog post from April.

Also, the example in the blog post is:

version: 2
updates:
  - package-ecosystem: "bundler"
    directories:
      - "/frontend"
      - "/backend"
      - "/admin"
    schedule:
      interval: "weekly"

But that results in the behaviour I mentioned above, where dependabot only opens PRs for the first directory and does not try to update the other directories at all. I don't entirely understand why a groups definition is required to get it working in all directories, would you mind elaborating on that (or maybe I have the wrong end of the stick when you say "that's part of why we opted for the explicitness of requiring a defined group")?

Just as @lucacome pointed out, I'd really expect this to work like the github-actions ecosystem updates, where one dependency that can be updated in all the directories would be grouped into a single PR (assuming no groups are configured) - otherwise the directories option isn't adding any functionality beyond what already exists when using directory (other than saving a few lines of config, of course).

For me, the benefit is that we can reduce the number of PRs dependabot makes when the same dependency needs updating in different directories and thus far less likely to hit the open PR limit. If we have to use groups, then we have to create a definition for each individual dependency in config, which isn't a particularly scalable or maintainable solution as it could mean creating hundreds of groups to get that behaviour (and it would need updating every time a new dependency was added). This seems counter to the stated objective in the blog post:

Previously, developers with multiple package manifests for the same ecosystem (e.g. npm, pip, gradle) across multiple directories had to create separate dependabot.yml configurations for each of those directories. This could lead to many duplicated configurations, and high maintenance costs if a developer wished to make a change that spanned multiple directories.