🚀 Feature: Allow defining allow rules for catalog providers

[drodil]

🔖 Feature description

There's already support for rules for catalog locations:

catalog:
  rules:
    - allow: [Component, API, Location, Template]

  locations:
    - type: url
      target: https://github.com/org/example/blob/master/org-data.yaml
      rules:
        - allow: [Group]

However, this is not possible to be defined for providers separately.

🎤 Context

To be able to have more fine-grained control over what kind of entities each provider can produce, the config should allow rules to be defined for providers:

catalog:
  providers:
    github:
      providerId:
        organization: 'backstage'
        catalogPath: '/catalog-info.yaml' 
        filters:
          branch: 'main' 
          repository: '.*' 
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }
      rules:
        - allow: [Component]
    githubOrg:
      id: github
      githubUrl: https://github.com
      orgs: ['organization-1', 'organization-2', 'organization-3']
      schedule:
        initialDelay: { seconds: 30 }
        frequency: { hours: 1 }
        timeout: { minutes: 50 }
      rules:
        - allow: [Group]

✌️ Possible Implementation

No response

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

None

[drodil]

Related #4977 and #5014

[Rugvip]

Generally I agree that this could be good to have in some form, perhaps by just giving access to the standardized rule config and logic to entity providers.

These examples are perhaps not the best though, org data providers are fairly hardcoded to just ingest users and groups. If we don't trust them enough to do that there's a bigger problem than what can be solved in code, we'd need to sandbox them somehow at that point. If what we're looking for is filtering I'm thinking it's better to implement it as such.

The GitHub discovery provider doesn't actually ingest these entities directly. It emits Location entities that are then later read by the catalog, which means you'd need to be forwarding these rules to those locations somehow, or something like that. Lot more discussion around that in #12880, where we ultimately landed on #14584 as the solution.

[drodil]

Yeah, the locations are a bit hard for this. The main reason for this was that I would only like to get groups from Github org data and as the users are coming already from Azure. Of course, it could be possible to add a specific config for the GithubOrgDataProvider but thinking here again more generic way for all (even custom) providers to limit the entity types.

[Rugvip]

@drodil yep thinking it's probably best to keep that behind more explicit config to exclude users tbh. Using rules you might want to trigger an alert if someone tries to ingest a disallowed entity, so it could be good to avoid using it for deliberate filtering.