Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[freenas-api-nfs] volume expansion failes with 405 ??? #394

Open
markussiebert opened this issue May 5, 2024 · 6 comments
Open

[freenas-api-nfs] volume expansion failes with 405 ??? #394

markussiebert opened this issue May 5, 2024 · 6 comments

Comments

@markussiebert
Copy link

Hi,

I tried to expand a nfs volume - but it fails with:

external-resizer I0505 16:40:23.658274       1 connection.go:200] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Ty
pe":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type"
:7}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":12}}},{"Type":{"Rpc":{"type":13}}}]}                                     
external-resizer I0505 16:40:23.658301       1 connection.go:201] GRPC error: <nil>                                               
external-resizer I0505 16:40:23.658335       1 connection.go:193] GRPC call: /csi.v1.Controller/ControllerExpandVolume            
external-resizer I0505 16:40:23.658343       1 connection.go:194] GRPC request: {"capacity_range":{"required_bytes":25769803776},"
volume_capability":{"AccessType":{"Mount":{"fs_type":"nfs","mount_flags":["noatime","nfsvers=4"]}},"access_mode":{"mode":7}},"volu
me_id":"pvc-f9456e89-d5c9-43ca-816b-4d4b9a45fc30"}                                                                                
csi-driver {"host":"democratic-csi-controller-9d657965b-8rmgd","level":"info","message":"new request - driver: FreeNASApiDriver me
thod: ControllerExpandVolume call: {\"path\":\"/csi.v1.Controller/ControllerExpandVolume\",\"metadata\":{},\"request\":{\"secrets\
":\"redacted\",\"volume_id\":\"pvc-f9456e89-d5c9-43ca-816b-4d4b9a45fc30\",\"capacity_range\":{\"required_bytes\":\"25769803776\",\
"limit_bytes\":\"0\"},\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"mount_flags\":[\
"noatime\",\"nfsvers=4\"],\"fs_type\":\"nfs\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}},\"pendingStatus\":{\"code\"
:0,\"details\":\"OK\"},\"cancelled\":false,\"trailingMetadata\":{}}","service":"democratic-csi","timestamp":"2024-05-05T16:40:23.6
59Z"}                                                                                                                             
csi-driver {"host":"democratic-csi-controller-9d657965b-8rmgd","level":"error","message":"handler error - driver: FreeNASApiDriver
 method: ControllerExpandVolume error: Error: \"<html>\\r\\n<head><title>405 Not Allowed</title></head>\\r\\n<body>\\r\\n<center><
h1>405 Not Allowed</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n\" Error: \"<html>\\r\\n<head><tit
le>405 Not Allowed</title></head>\\r\\n<body>\\r\\n<center><h1>405 Not Allowed</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n
</body>\\r\\n</html>\\r\\n\"\n    at Api.DatasetSet (/home/csi/app/src/driver/freenas/http/api.js:443:11)\n    at process.processT
icksAndRejections (node:internal/process/task_queues:95:5)\n    at async FreeNASApiDriver.ControllerExpandVolume (/home/csi/app/sr
c/driver/freenas/api.js:3238:7)\n    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)","service":"democratic
-csi","timestamp":"2024-05-05T16:40:23.668Z"}                                                                                     
external-resizer I0505 16:40:23.669561       1 connection.go:200] GRPC response: {}                                               
external-resizer I0505 16:40:23.669587       1 connection.go:201] GRPC error: rpc error: code = Internal desc = Error: "<html>\r\n
<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r\n<hr><center>nginx</center>\r
\n</body>\r\n</html>\r\n" Error: "<html>\r\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</
h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"

My fault or a bug?
@Malet
Copy link

Malet commented May 6, 2024

I'm also getting the same issue with the freenas-api-iscsi driver:

Labels:        app.kubernetes.io/instance=opensearch                                                                                                                            
               app.kubernetes.io/name=opensearch                                                                                                                                
Annotations:   pv.kubernetes.io/bind-completed: yes                                                                                                                             
               pv.kubernetes.io/bound-by-controller: yes                                                                                                                        
               volume.beta.kubernetes.io/storage-provisioner: freenas-api-iscsi                                                                                                 
               volume.kubernetes.io/storage-provisioner: freenas-api-iscsi                                                                                                      
Finalizers:    [kubernetes.io/pvc-protection]                                                                                                                                   
Capacity:      50Gi                                                                                                                                                             
Access Modes:  RWO                                                                                                                                                              
VolumeMode:    Filesystem                                                                                                                                                       
Used By:       opensearch-cluster-master-0                                                                                                                                      
Conditions:                                                                                                                                                                     
  Type       Status  LastProbeTime                     LastTransitionTime                Reason  Message                                                                        
  ----       ------  -----------------                 ------------------                ------  -------                                                                        
  Resizing   True    Mon, 01 Jan 0001 00:00:00 +0000   Mon, 06 May 2024 12:40:19 +0100                                                                                          
Events:                                                                                                                                                                         
  Type     Reason              Age              From                                Message                                                                                     
  ----     ------              ----             ----                                -------                                                                                     
  Warning  ExternalExpanding   4s               volume_expand                       waiting for an external controller to expand this PVC                                       
  Normal   Resizing            2s (x2 over 3s)  external-resizer freenas-api-iscsi  External resizer is resizing volume pvc-940180dc-975c-43e9-a54e-f5c80ec1e1f0                
  Warning  VolumeResizeFailed  1s (x2 over 3s)  external-resizer freenas-api-iscsi  resize volume "pvc-940180dc-975c-43e9-a54e-f5c80ec1e1f0" by resizer "freenas-api-iscsi" fail
ed: rpc error: code = Internal desc = Error: "<html>\r\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r\n<hr><center>nginx<
/center>\r\n</body>\r\n</html>\r\n" Error: "<html>\r\n<head><title>405 Not Allowed</title></head>\r\n<body>\r\n<center><h1>405 Not Allowed</h1></center>\r\n<hr><center>nginx</c
enter>\r\n</body>\r\n</html>\r\n"                                                                                                                                               
    at Api.DatasetSet (/home/csi/app/src/driver/freenas/http/api.js:443:11)                                                                                                     
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                               
    at async FreeNASApiDriver.ControllerExpandVolume (/home/csi/app/src/driver/freenas/api.js:3238:7)                                                                           
    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:222:18)

@travisghansen
Copy link
Member

Can you enable debug logging on the controller pod and capture the raw http requests/responses?

@Malet
Copy link

Malet commented May 6, 2024 

csi-driver {"host":"truenas-iscsi-democratic-csi-controller-65ddb589b6-nlvqd","level":"debug","message":"FREENAS HTTP RESPONSE BODY: {\"id\":\"samsung/x/iscsi/pvc-940180dc-975c-43e9-a54e-f5c80ec1e1f0\",\"type\":\"VOLUME\",\"name\":\"samsung/x/iscsi/pvc-940180dc-975c-43e9-a54e-f5c80ec1e1f0\",\"pool\":\"samsung\",\"encrypted\":false,\"encryption_root\":null,\"key_load ...
csi-driver {"host":"truenas-iscsi-democratic-csi-controller-65ddb589b6-nlvqd","level":"info","message":"new response - driver: FreeNASApiDriver method: ControllerExpandVolume response: {\"capacity_bytes\":220117073920,\"node_expansion_required\":true}","service":"democratic-csi","timestamp":"2024-05-06T16:33:47.058Z"}

This seems to be working now debug logging is enabled 😆

@markussiebert
Copy link
Author

crazy - the same at my installation - enabling debug logs leads to working resize and volumes that get status released...

@f1reSong
Copy link

+1, it's working on logLevel debug

@markussiebert
Copy link
Author

But still would consider this a bug XD

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@travisghansen @Malet @markussiebert @f1reSong