@csquire do you have any feature toggles enabled? for example panelTitleSearch?

No feature toggles are enabled in our config, we are just using defaults.

We also encountered the same problem, after upgrading our grafana instance from 9.1.7 to 9.5.2, searches are typically taking 30 seconds or longer for non-admins.

Our system has about:

• 4500 dashoards
• 5000 users
• 110000 permission

Slow query sql:
SELECT dashboard.id, dashboard.uid, dashboard.title, dashboard.slug, dashboard_tag.term, dashboard.is_folder, dashboard.folder_id, folder.uid AS folder_uid, folder.slug AS folder_slug, folder.title AS folder_title FROM ( SELECT dashboard.id FROM dashboard WHERE ((dashboard.uid IN (SELECT substr(scope, 16) FROM permission WHERE scope LIKE 'dashboards:uid:%' AND role_id IN(SELECT id FROM role INNER JOIN ( SELECT ur.role_id FROM user_role AS ur WHERE ur.user_id = 5087 AND (ur.org_id = 1 OR ur.org_id = 0) UNION SELECT br.role_id FROM builtin_role AS br WHERE br.role IN ('Viewer') AND (br.org_id = 1 OR br.org_id = 0) ) as all_role ON role.id = all_role.role_id) AND action = 'dashboards:read') AND NOT dashboard.is_folder) OR (dashboard.folder_id IN (SELECT id FROM dashboard as d WHERE d.uid IN (SELECT substr(scope, 13) FROM permission WHERE scope LIKE 'folders:uid:%' AND role_id IN(SELECT id FROM role INNER JOIN ( SELECT ur.role_id FROM user_role AS ur WHERE ur.user_id = 5087 AND (ur.org_id = 1 OR ur.org_id = 0) UNION SELECT br.role_id FROM builtin_role AS br WHERE br.role IN ('Viewer') AND (br.org_id = 1 OR br.org_id = 0) ) as all_role ON role.id = all_role.role_id) AND action = 'dashboards:read')) AND NOT dashboard.is_folder) OR (dashboard.uid IN (SELECT substr(scope, 13) FROM permission WHERE scope LIKE 'folders:uid:%' AND role_id IN(SELECT id FROM role INNER JOIN ( SELECT ur.role_id FROM user_role AS ur WHERE ur.user_id = 5087 AND (ur.org_id = 1 OR ur.org_id = 0) UNION SELECT br.role_id FROM builtin_role AS br WHERE br.role IN ('Viewer') AND (br.org_id = 1 OR br.org_id = 0) ) as all_role ON role.id = all_role.role_id) AND action = 'folders:read') AND dashboard.is_folder)) AND dashboard.org_id=1 AND dashboard.title LIKE '%ba%' ORDER BY dashboard.title ASC LIMIT 100 OFFSET 0) AS ids INNER JOIN dashboard ON ids.id = dashboard.id LEFT OUTER JOIN dashboard AS folder ON folder.id = dashboard.folder_id LEFT OUTER JOIN dashboard_tag ON dashboard.id = dashboard_tag.dashboard_id ORDER BY dashboard.title ASC

@torkelo @papagian @Graham-Moreno

any progress？we also meet same problem.

We have the same problem too. Any progress?

What database are you using for grafana? Based on benchmarks the query is performing bad on mysql 5.7 but has no problem on on either postgres 12+ , mysql 8 or sqlite.

Benchmarks is running with around 80k dashboards and 500k permissions. We are not sure we will do any fixes to mysql 5.7 due to it being EOL soon.

If you are using any other database could you please provide that you are using?

What database are you using for grafana? Based on benchmarks the query is performing bad on mysql 5.7 but has no problem on on either postgres 12+ , mysql 8 or sqlite.

Benchmarks is running with around 80k dashboards and 500k permissions. We are not sure we will do any fixes to mysql 5.7 due to it being EOL soon.

If you are using any other database could you please provide that you are using?

mariadb10.1

We are facing the same issue with MySQL v8

We are on MySQL 5.7. A quick test in our staging environment did show improvement when upgrading to MySQL 8, so we'll schedule an upgrade to the production database and see if it helps.

Apologies, we are also on MySQL v5.7

I have the same issue on 10.1.5 grafana. Backend is MySQLv8 (8.0.31-23 Percona Server)
admin:non-admin search times ratio is between 1:40 to 1:timeout (i.e. over 2 min)

Also seeing this. Just promoted a user who was seeing 20s latency on searches to admin and they immediatly went down to what I as admin see of sub 100ms latencies.