Unified Alerting: contact-points/export/ returns EOF

[andrejshapal]

What happened?

Hello,
When trying to get manifest of contact points I am getting EOF (any option in UI and via API).
https://.../api/v1/provisioning/contact-points/export/
{"message":"EOF","traceID":""}
At the same time, the rest of simmilar exports (dashboard, alert rule) works without any issues.

I turned on debug logging, but did not find anything related.
The errors are:

2024-05-17 09:54:54.899	
{
  "error": "EOF",
  "level": "error",
  "logger": "context",
  "msg": "EOF",
  "orgId": 1,
  "remote_addr": "84.237.247.18",
  "t": "2024-05-17T09:54:54.899384343Z",
  "traceID": "",
  "uname": "andrej",
  "userId": 138
}

2024-05-17 09:54:54.899	
{
  "duration": "3.975834ms",
  "handler": "/api/v1/provisioning/contact-points/export",
  "level": "error",
  "logger": "context",
  "method": "GET",
  "msg": "Request Completed",
  "orgId": 1,
  "path": "/api/v1/provisioning/contact-points/export/",
  "referer": "https://appsupport.grafana-dev.domain.com/alerting/notifications?alertmanager=grafana&search=",
  "remote_addr": "1.2.3.4",
  "size": 30,
  "status": 500,
  "status_source": "server",
  "t": "2024-05-17T09:54:54.899445268Z",
  "time_ms": 3,
  "uname": "andrej",
  "userId": 138
}

At the same time, alerts are working without any issues.
I have tried 10.4.1, 10.4.4 and 11.0.0. All had the same issue.

At the same time, I have instance on 10.4.1 which don't have this issue. The difference between instances: one is deployed with grafana operator and uses postgres backend. The one without issues has sqllite.
But I did not find any difference in db payload:

The config in general is the same in both grafanas except db part.

config:
  log:
    mode: "console"
  log.console:
      level: "debug"
      format: "json"
  auth:
    disable_login_form: "true"
  auth.basic:
    enabled: "true"
  auth.proxy:
      enabled: "true"
      header_property: email
      header_name: X-Pomerium-Claim-Email
      headers: Role:X-Pomerium-Role
      auto_sign_up: "true"
      sync_ttl: "60"
      enable_login_token: "false"
  users:
    auto_assign_org_role: Editor
  alerting:
    enabled: "false"
  unified_alerting:
    enabled: "true"
  dataproxy:
    logging: "false"
    timeout: "300"
  feature_toggles:
    enable: editPanelCSVDragAndDrop, enableDatagridEditing, groupToNestedTableTransformation
  security:
    admin_user: root
    admin_password: secret
  database:
    type: postgres
    host: db-grafana.monitoring.svc.cluster.local
    name: app_dev
    user: app_dev_owner_user
    ssl_mode: require
  dashboards:
    default_home_dashboard_path: /var/lib/grafana/home-dashboard/home.json
  date_formats:
    default_timezone: UTC
  server:
    root_url: https://appsupport.grafana-dev.domain.com/
    router_logging: "false"

I did check db logs for errors and did not find any.

What did you expect to happen?

/api/v1/provisioning/contact-points/export should return manifest

Did this work before?

This is my first usage of this functionality.

How do we reproduce it?

I am not sure. If the issue is in postgres, then it should be replicatable just switching be to psql.

application_support_dev=# SHOW server_version;
-[ RECORD 1 ]--+---------------------------------
server_version | 14.7 (Ubuntu 14.7-1.pgdg22.04+1)

But this looks like something wrong in code (serialization maybe) because simmilar api endpoints works without any issues.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

No response

Grafana platform?

Kubernetes

Datasource(s)?

No response