Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A flood of async_register_repository tasks gets created at startup which slows down the event loop #3535

Open
7 tasks done
bdraco opened this issue Mar 1, 2024 · 11 comments · May be fixed by #3536
Open
7 tasks done

A flood of async_register_repository tasks gets created at startup which slows down the event loop #3535

bdraco opened this issue Mar 1, 2024 · 11 comments · May be fixed by #3536
Labels
issue:backend For issues with the backend/integration

Comments

@bdraco
Copy link
Contributor

bdraco commented Mar 1, 2024 •
edited

System Health details

n/a

Checklist

Describe the issue

A flood of async_register_repository tasks gets created at startup which slows down the event loop

Reproduction steps

restart

Debug logs

2024-02-29 23:04:37.856 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f307148a200> - None
2024-02-29 23:04:37.857 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f307148ae80> - None
2024-02-29 23:04:37.857 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine 
...

snip to fix in 65536 limit, it goes on an on
Screenshot 2024-03-01 at 10 17 50 AM

Diagnostics dump

No response
@bdraco bdraco added the issue:backend For issues with the backend/integration label Mar 1, 2024
@hacs-bot
Copy link

hacs-bot bot commented Mar 1, 2024

Make sure you have read the issue guidelines and that you filled out the entire template.

If you have an issue identical to this, do not add comments like "same here", "i have this too", instead add a 👍 reaction to the issue description. Thanks! 👍

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

I'm not sure if these tasks suspend, but if they don't the problem probably goes away if they get created with create_eager_task (if its available) since they will never get scheduled on the loop

@ludeeus
Copy link
Member

ludeeus commented Mar 1, 2024

If you enable experimental features, that should no longer be a problem.

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

Retesting now

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

Sadly enabling experimental features didn't solve it

2024-03-01 14:38:13.463 WARNING (MainThread) [homeassistant.helpers.storage] Loaded data for hacs.data in 0.022890757012646645
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dde40> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381ddd00> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381ddbc0> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dda80> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dc7c0> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dc540> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dd580> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381ddf80> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de0c0> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de200> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de340> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de480> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de5c0> - None
2024-03-01 14:38:13.466 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de700> - None
2024-03-01 14:38:13.467 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de840> - None
2024-03-01 14:38:13.467 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381de980> - None
2024-03-01 14:38:13.467 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381deac0> - None
2024-03-01 14:38:13.467 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381dec00> - None
2024-03-01 14:38:13.467 WARNING (MainThread) [asyncio.base_events] Create task: <coroutine object HacsBase.async_register_repository at 0x7f93381ded40> - None

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

Restarted twice to be sure

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

integration/custom_components/hacs/utils/data.py

Line 268 in a422090

await asyncio.gather(*register_tasks)

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

git blame says its my fault 🥇

@ludeeus
Copy link
Member

ludeeus commented Mar 1, 2024

Ah 👍 That from the data restore, yes that is still there, its from asyncio.gather.

integration/custom_components/hacs/utils/data.py

Line 268 in a422090

await asyncio.gather(*register_tasks)

@bdraco
Copy link
Contributor Author

bdraco commented Mar 1, 2024

They don't actually suspend so creating them eagerly fixes the issue because they never get scheduled on the loop

@bdraco bdraco linked a pull request Mar 1, 2024 that will close this issue
Open
@ludeeus
Copy link
Member

ludeeus commented Mar 1, 2024

Right, but that does not yet exist.
Did some tests, and this will solve it (by not creating any tasks in the first place), and based on (simple) timing testing its also almost twice as fast.

       for entry, repo_data in repositories.items():
            if (
                entry == "0"
                or self.hacs.repositories.is_registered(repository_id=entry)
                or repo_data.get("category", category) is None
            ):
                continue
            await self.hacs.async_register_repository(
                repository_full_name=repo_data["full_name"],
                category=repo_data.get("category", category),
                check=False,
                repository_id=entry,
            )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
issue:backend For issues with the backend/integration
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid creating async_register_repository tasks to register_unknown_repositories bdraco/integration
2 participants
@bdraco @ludeeus