Are vault plugins supposed to work with openbao?

[DrDaveD]

I'm trying to load a couple of plugins compiled for vault with the current openbao main branch and I get errors like:

Error enabling: Error making API request.

URL: POST http://127.0.0.1:8202/v1/sys/mounts/secret/oauth
Code: 400. Errors:

* invalid backend version: 2 errors occurred:
        * Unrecognized remote plugin message:
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake
...

Is it supposed to work at this point? Do the plugins need to be modified to use the openbao libraries and recompiled?

[cipherboy]

@DrDaveD Interesting! I had wanted this to work, and would've thought it should've, but apparently I broke something. Do you mind modifying the code to see if you can get which message is unrecognized?

I'm also curious about versions. I'd expect it to work if you chose an upstream SDK corresponding to Vault 1.14.x release, but not necessarily if you chose later ones... Unless I broke it due to removing too much eventing code....?

[cipherboy]

Weird... So oauth uses:

https://github.com/puppetlabs/vault-plugin-secrets-oauthapp/blob/ed4005c19ac7ab9f6f152379a338168021300a21/go.mod#L178

github.com/hashicorp/go-plugin v1.4.5 // indirect

which is a similar version to what we have in our deps...

openbao/go.mod

Line 61 in a5e299d

github.com/hashicorp/go-plugin v1.4.9

(but different point release...).

However, while the location of the message changed, it is still emitted on a line with less than 4 parts (delineated by |)... However, all three versions in play (ssh uses a different one) should actually emit the line from the remote peer:

const unrecognizedRemotePluginMessage = `Unrecognized remote plugin message: %s
... snip ...

Which means, the remote peer's message is... empty?

Are there any log messages from the server side when this plugin is mounted?

[DrDaveD]

Yes there are debug log messages in /var/log/messages:

May  3 17:40:11 fermicloud732 bao[2108738]: 2024-05-03T17:40:11.984-0500 [DEBUG] auth.ssh.auth_ssh_e5bf4da6.ssh: waiting for RPC address: metadata=true path=/usr/libexec/htvault-config/plugins/vault-plugin-auth-ssh.sh
May  3 17:40:11 fermicloud732 bao[2108738]: 2024-05-03T17:40:11.993-0500 [ERROR] auth.ssh.auth_ssh_e5bf4da6.ssh.vault-plugin-auth-ssh.sh: plugin tls init: metadata=true error="error parsing wrapping token: go-jose/go-jose: compact JWS format must have three parts" timestamp=2024-05-03T17:40:11.993-0500
May  3 17:40:11 fermicloud732 bao[2108738]: 2024-05-03T17:40:11.994-0500 [INFO] auth.ssh.auth_ssh_e5bf4da6.ssh: plugin process exited: metadata=true path=/usr/libexec/htvault-config/plugins/vault-plugin-auth-ssh.sh pid=2108893
May  3 17:40:11 fermicloud732 bao[2108738]: 2024-05-03T17:40:11.994-0500 [ERROR] secrets.system.system_61f5ed54: error occurred during enable credential: path=ssh/
May  3 17:40:11 fermicloud732 bao[2108738]:  error=
May  3 17:40:11 fermicloud732 bao[2108738]:  | invalid backend version: 2 errors occurred:
May  3 17:40:11 fermicloud732 bao[2108738]:  | \t* Unrecognized remote plugin message:

I think the key is the tls error

[DrDaveD]

If I try to execute one of the plugins directly with environment VAULT_BACKEND_PLUGIN=6669da05-b1c8-4f49-97d9-c8e5bed98e20 (as found at the end of sdk/plugin/serve.go then I get as output something that looks very similar:

{"@level":"error","@message":"plugin tls init","@timestamp":"2024-05-03T21:40:50.000249-05:00","error":"error parsing wrapping token: square/go-jose: compact JWS format must have three parts"}

So there's something wrong with the tls wrapper.

[DrDaveD]

Aha! I had a plugin wrapper dump the environment and I saw 3 variables with a BAO prefix:

BAO_PLUGIN_AUTOMTLS_ENABLED=true
BAO_PLUGIN_METADATA_MODE=false
BAO_VERSION=2.0.0

If I copy those to corresponding VAULT variables, I can get the plugins to start.

[cipherboy]

Aha, sorry. This makes a lot more sense now. I'll fix this shortly!

[DrDaveD]

Fix is in #321