Enabling security in CubeFS

[Zorlin]

Hi there,

I want to place some basic restrictions on CubeFS so that machines with access to my local network can't wipe my entire cluster by simply talking to my Master/Resource Manager node.

I followed the directions in the old ChubaoFS documentation (which should likely be merged into the current CubeFS docs, or at least the portions about authnode) and successfully got authnode up and running including TLS certificates, but despite doing this when I use cfs-cli:

wings@blackberry:~$ cat ~/.cfs-cli.json 

{
  "masterAddr": [
    "10.7.1.201:17010"
  ],
  "timeout": 60
}

I have access to all CLI options and can delete volumes and do all kinds of stuff with zero authentication.

How do I lock that down? It's the main thing stopping me from rolling out CubeFS properly!

[Zorlin]

#1857

This pull request appears to contain the necessary code to make this work 😲

It was merged, but the changes don't appear in v3.3.1 yet.

[leonrayang]

Thanks for the reminder, will be included in v3.3.2

[Zorlin]

I was able to take the branch "release-3.3.1" and cherry pick #1857 onto it, then make a custom CubeFS build.

I'm now at the stage where I have a HA master and HA authnode setup, with TLS and everything working. Clients can only do major operations (or any sort of change really) when authenticated with a valid key now.

However, when I attempt to add datanodes, it claims that the key for "DatanodeService" doesn't exist, even though I created it through the cfs-authtool the same way I created the keys for the master (following the old ChubaoFS docs).

[Zorlin]

Logs and details below. The main issue is a key not exists error.

root@cubefs-authnode01:/etc/cubefs# cat data_datanode.json
{
    "id": "DatanodeService",
    "role": "service",
    "caps": "{\"API\":[\"*:*:*\"]}"
}

root@cubefs-authnode01:/etc/cubefs# cfs-authtool api -https -host=10.7.1.151:8443 -ticketfile=ticket_admin.json -data=data_datanode.json -output=key_datanode.json AuthService createkey

body: {"code":0,"msg":"success","data":"5<REDACTED>n"}

root@cubefs-authnode01:/etc/cubefs# cat key_datanode.json
{
  "id": "DatanodeService",
  "auth_key": "<REDACTED=",
  "access_key": "<REDACTED",
  "secret_key": "<REDACTED",
  "create_ts": 1707698243,
  "role": "service",
  "caps": "{\"API\":[\"*:*:*\"]}",
  "auth_id_key": "ey<REDACTED>=="
}

2024/02/11 23:33:16.752274 [INFO ] http_server.go:230: action[AuthenticationInterceptor] parseAndCheckClientKey failed, RequestURI[/dataNode/add?addr=10.7.1.201:17310&zoneName=default&clientIDKey=e<REDACTED>==], err[get ticket from auth node failed, clientIDKey[e<REDACTED>==], err[request error, code[2], msg[action[GetKey], clusterID[rifflabs-ldn] ID:DatanodeService, err:key not exists ]]]
2024/02/11 23:33:16.752283 [INFO ] api_args_parse.go:1508: URL[/dataNode/add?addr=10.7.1.201:17310&zoneName=default&clientIDKey=e<REDACTED>==],remoteAddr[10.7.1.162:47016],response

Config of my datanode.json:

{
  "role": "datanode",
  "listen": "17310",
  "prof": "17320",
  "logDir": "/var/log/cubefs/datanode/log",
  "logLevel": "info",
  "raftHeartbeat": "17330",
  "raftReplica": "17340",
  "raftDir":"/var/lib/cubefs/datanode/raft",
  "exporterPort": 9502,
  "masterAddr": [
     "10.7.1.161:17010",
     "10.7.1.162:17010",
     "10.7.1.163:17010"
  ],
  "disks": [
     "/mnt/brick/cubefs:10737418240"
  ],
  "serviceIDKey": "e<REDACTED>=="
}

[leonrayang]

@M1eyu2018 please take a look at this issue.

[M1eyu2018]

@Zorlin hi, the key of datanode may be not stored in authnode. Please check it by using getKey api as follows:

cfs-authtool api -https -host=10.7.1.151:8443 -ticketfile=ticket_admin.json -data=data_datanode.json -output=key_datanode.json AuthService getkey

And if the key is actually not in authnode, just create key by authtool again.

[Zorlin]

Hi @M1eyu2018

In my case I hit a really nasty bug - the key simply did not exist and no amount of deleting and re-creating it would make it work. I could create it, then attempting to create it again would result in a KeyAlreadyExists error (or something like that, I don't recall, sorry), but even though that suggests it now exists - if I poll the API for that key it doesn't exist.

The only way I found to fix it was wiping my entire authnode cluster and recreating everything. That did work however and I was able to get CubeFS fully online with authnode enabled 🎉

[M1eyu2018]

ok, in your case, if the result of getkey api is also that key not exists, the keystore in authnode may go wrong.

[Zorlin]

Yep, I believe that's what happened.

[Zorlin]

@leonrayang any thoughts?

[M1eyu2018]

@Zorlin hi, the key of datanode may be not stored in authnode. Please check it by using getKey api as follows:

cfs-authtool api -https -host=10.7.1.151:8443 -ticketfile=ticket_admin.json -data=data_datanode.json -output=key_datanode.json AuthService getkey

And if the key is actually not in authnode, just create key by authtool again.

[leonrayang]

Hi there,

I want to place some basic restrictions on CubeFS so that machines with access to my local network can't wipe my entire cluster by simply talking to my Master/Resource Manager node.

I followed the directions in the old ChubaoFS documentation (which should likely be merged into the current CubeFS docs, or at least the portions about authnode) and successfully got authnode up and running including TLS certificates, but despite doing this when I use cfs-cli:

wings@blackberry:~$ cat ~/.cfs-cli.json 

{
  "masterAddr": [
    "10.7.1.201:17010"
  ],
  "timeout": 60
}

I have access to all CLI options and can delete volumes and do all kinds of stuff with zero authentication.

How do I lock that down? It's the main thing stopping me from rolling out CubeFS properly!

@Zorlin The authnode documentation will be merged as soon as possible ：）

[Zorlin]

Thank you!!