Backupstore not working - secret ??? not found - even Testing Backupstore

[oglimmer]

Starting point: Backup on S3 was working fine for months.

I needed to rotate the API secret, since then I get "secret ??? not found" for anything.

I have tested:

• S3
• CIFS
• even following https://longhorn.io/docs/1.6.1/snapshots-and-backups/backup-and-restore/set-backup-target/#set-up-a-local-testing-backupstore

• I created the k8s secrets in the longhorn namespace and default namespace
• I removed both (made blank) "Backup Target" and "Backup Target Credential Secret" before settings the values
• I made sure that "Backup Target" ends with a /

Any ideas what to do? (or how to investigate?)

[derekbit]

A support bundle is appreciated for the further investigation.

[oglimmer]

I am happy to provide more information and I can upload files, but the complete support bundle contains too many files with too much private data.
Which files/directories do you need? I guess it's all about the longhorn namespace.

[derekbit]

You can send the support bundle to longhorn-support-bundle@suse.com. The support bundle can only be checked by Longhorn members.

[derekbit]

BTW, have you checked this ticket #8299?

[oglimmer]

this addresses on issue around S3 permission. as I tested it against "Local Testing Backupstore" and this doesn't work as well, I assume that my S3 permissions are good.

[derekbit]

I tested it against "Local Testing Backupstore" and this doesn't work as well

OK, let's check this first. Can you provide the steps and the secret for your local testing backupstore?

[oglimmer]

You can send the support bundle to longhorn-support-bundle@suse.com. The support bundle can only be checked by Longhorn members.

I get this

<longhorn-support-bundle@suse.com>: host smtp.google.com[172.253.62.27] said:
552-5.7.0 This message was blocked because its content presents a potential
552-5.7.0 security issue. To review our message content and attachment
content 552-5.7.0 guidelines, go to 552 5.7.0
https://support.google.com/mail/?p=BlockedMessage
o21-20020ac85a55000000b0043ac84f937bsi4531031qta.77 - gsmtp (in reply to
end of DATA command)

when sending the support-bundle. maybe you really just go for the relevent directories.

how about:

• ./yamls/namespaced/longhorn-system
• ./logs/longhorn-system
  ?