📁 Catalog Improvement Meta Issue

[Rugvip]

There are a lot of ideas around different ways that we can improve the Software Catalog. I've created this issue as a place to track all of these ideas as to make it easier to get an overview of what's already being discussed. It is also a place where we can synchronize the work between members of the community that want to help drive the catalog forward.

Please suggest things to add if you think that something is missing! 🙏

Catalog Model

• Better docs for extending the catalog: [Catalog] Expanding the "Extending the model" Documentation #14383
• There should be a more streamlined API for defining custom catalog models.
• Transitive relations: catalog model: support additional System relations #7109

New Features

• Ingestion errors are hard to catch and need to be surfaced somehow.
• There is no good UX flow for moving or renaming entities.
• Event-based catalog consumption: [RFC] Catalog Event Stream Endpoint #8219
• Entity modification timestamps: Track the last time when the entity was changed #14181
• Multi-fetch: Catalog REST API for fetching multiple entities with one request #12254, added a by-refs batch endpoint for entities #14354
• Catalog federation: Share Catalog Data Between Multiple Backstage Instances #4888
• Catalog onboarding page: [RFC] Provide a new entrypoint to the onboarding process of new or existing things into the catalog #4741
• Entity statuses: [RFC] Status API #2292
• Location registration should check for conflicting entities: [Catalog] Importing conflicting entities should result in an error #10862
• Deeper integration between the catalog model and search? Making it easier to ingest custom entities for search
• Simple solution for merging entity definitions from multiple sources: 🚀 Feature: Possibility to update entities via UI #16155, 🚀 Feature: Ability to enrich entities created from ingestion of org data #24710

Catalog Ingestion

• Event-driven ingestion: Catalog update loop may anger GitHub's secondary rate limits #8232, [RFC][catalog-backend] Refresh entities based on some pre stored key. #12155, feat: events #13931, feat: add support for refreshKeys #12156, feat: add refresh to EntityProviderConnection #13575, [RFC] Rate Limiting || Backstage <<>> Providers #13531
• Ingestion UX: [RFC] Enhancing experience for Multi-Catalog Import capability with the kind: Location Entities #11445
• Entity validation: [Plugin] entity-validator #9245
• Refer to default branch: Support default branch syntax in GitHub UrlReader #9086
• Completely remove orphaned entities as a concept. Related: Catalog cleanup job (remove orphaned entities) #7860
• Disallow entity deletions: Disallow deletion of non-orphaned entities #8616
• Disallow rogue locations: [RFC] do not allow rogue Locations #13959
• Entity location deduplication: [FileReaderProcessor, GithubDiscoveryProcessor] Detected conflicting entityRef after renaming the entity definition file #14042
• Make it possible to disable refresh cycle: 🚀 Feature: Provide an option to disable refresh cycle in catalog processing #18588
• Distribute the refresh interval decision to processors: 🚀 Feature: Provide an option to disable refresh cycle in catalog processing #18588

Performance Issues

• Catalog processing can be slow and not scale beyond a certain point, Adding the incremental entity provider backend #14356
• Improved pagination: Catalog(Page) pagination #12247, Catalog: paginated entities endpoint #12246
• Facets endpoint is slow: Optimize entity-facets endpoint #13655
• Catalog is slow with large number of entities: Slowness Using Entity Catalog with Large Number of Entities #12829

Bugs

• Provided entities not being deleted: Provider-ingested entities not being eager deleted #12883
• Unexpected sticky relations: Group showing multiple parents in the UI #7872
• Conflicting entities are not detected: [Catalog] Importing conflicting entities should result in an error #10862

[dtuite]

Excellent list @Rugvip .

Based on our experience rolling out Backstage with a few dozen customers, the items I'd highlight from your list are (in order)...

1. Improved pagination (we estimate this will cut ~8 seconds (aka. 85%) off some page loads)
2. Ingestion errors are hard to catch and need to be surfaced somehow (seems like ~50% of customers ask "I can't see my component in the catalog..." at some point).
3. There is no good UX flow for moving or renaming entities (we get support tickets about this quite frequently)
4. Catalog onboarding page (yes but also... "I need to onboard 400 components... what do I do?")
5. Entity validation (helps to address all of the above points)
6. Completely remove orphaned entities as a concept (we've implemented our own garbage collection to work around this).

[taras]

@Rugvip we should add #14356 to the list as well

[angeliski]

I would love to see a more simple way to customize the entity pages/cards.

I know isn't only the catalog, the core components should be get some benefits from this ideia, but I can imagine the big necessity comes from the catalog

Things like: add a new information from metadata, change some information from About. Add more information in tables and things like that. It's possible now, but is very difficult to achieve (from my "not React developer" point of view)

[afscrome]

@Rugvip Can you add #12880 to the list.

[Rugvip]

@afscrome that one is actually just about to be closed by #14584, does that solution cover your use-case as well? If not I think it might be best to outline the missing pieces in a new issue

[bforbis]

I'll also note that one of the issues I've seen is a lack of consistency in rendering Entity reference names in the UI.

On some pages like the top bar and in the graph, it gets rendered using metadata.title which is likely the best human readable option.

However in other cases, it gets rendered as just the metadata.name regardless of whether or not a title exists, like in parent/child links for groups, owner links on the catalog table view, and owner picker dropdowns in the scaffolder.

It seems like this really needs to be made consistent and should prefer metadata.title always to metadata.name. For my use case in particular, metadata.name uses uuids imported from other systems and often are not helpful at all to the end user.

[awanlin]

@bforbis, yeah, that for sure would be nice. As this is intended to be a meta-issue for all the related Catalog issues could you log an issue for this and then share the issue number here, please?

[bforbis]

@awanlin here you go:
#15965

[drodil]

Also, I would like to see an extendable stitcher as there are some cases where the same information comes from two different sources, for example, users might be ingested from multiple systems. Merging the entities should be possible so that metadata from system 1 can be combined with metadata from system 2. This though might be somewhat hard to do due to the async behavior of the providers.

But especially the human-readable naming all over should be fixed by utilizing the metadata.title instead name.

[bforbis]

There is no good UX flow for moving or renaming entities.

Is there an issue for this one yet? I'd like to see what has already been discussed.

[freben]

@bforbis Not really for exactly that, I don't think. There's been a number of reports of troubles with conflict resolution etc during moves, but no issue that comes immediately to mind that talks about specific remediations.

[knowacki23]

Hey everyone,
do you have any ideas about:

• Completely remove orphaned entities as a concept. Related: #7860

I'm asking because I would like to implement removing of orphaned entities.
@anisjonischkeit I've seen in #7860 that you wanted to contribute to this one, do you have any updates on it?

Thanks! 😃

[drodil]

@knowacki23 might also be related to the upcoming debug tool that could potentially allow removing orphan entities from UI #9737

[knowacki23]

@knowacki23 might also be related to the upcoming debug tool that could potentially allow removing orphan entities from UI #9737

From what I understand (please correct me if I'm wrong) this debug tool will add new view tab containing all the orphaned entities, I was hoping for a scheduled task like indexBuilder for the search engine, that will automatically remove non-existing entities from the catalog without any user interaction.

[Remyapraveen]

Hi team, I am not sure on the severity of this issue. Came across this on my newly built Backstage instance. The "Learn more about the Software Catalog" leads to a blank page in Backstage website.

[Rugvip]

@Remyapraveen thank you for reporting! We'll need to add a redirect to https://backstage.io/docs/features/software-catalog/ in the microsite config. Do you want to submit a PR?

[Remyapraveen]

Hi @Rugvip, sure, thank you. Shall submit a PR, but should I just go and create a new Pull Request? Could you please guide me on what to do?

[Rugvip]

Turned out to be a pretty widespread issue. I ended up shipping it in #16708 through #14574 (comment)

[Remyapraveen]

I see. Thanks for letting me know @Rugvip

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.