Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

使用higress网关访问后端协议为grpc，无法访问。 #906

Open
1 task
ldl-home opened this issue Apr 8, 2024 · 27 comments
Open
1 task

使用higress网关访问后端协议为grpc，无法访问。 #906

ldl-home opened this issue Apr 8, 2024 · 27 comments

Comments

@ldl-home
Copy link

ldl-home commented Apr 8, 2024

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via ASRC(Alibaba Security Response Center) where the issue will be triaged appropriately.

  • I have searched the issues of this repository and believe that this is not a duplicate.

Ⅰ. Issue Description

基于k8s环境使用如下配置创建了一个higress访问的域名，后端服务提供grpc

---
apiVersion: v1
kind: Service
metadata:
  namespace: higress-system
  name: go-ping-s
spec:
  selector:
    app: go-ping-s
  ports:
    - name: grpc
      port: 80
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: higress-system
  annotations:
    higress.io/backend-protocol: "GRPC"
  name: go-ping-s
spec:
  ingressClassName: higress
  rules:
    - host: grpc.test.com
      http:
        paths:
          - backend:
              service:
                name: go-ping-s
                port: 
                  number: 80
            path: /
            pathType: Exact

Ⅱ. Describe what happened

通过 postman的grpc协议调用grpc.test.com其中的一个ping方法。无法到后端服务

日志如下
{"authority":"grpc.test.com:80","bytes_received":"0","bytes_sent":"0","downstream_local_address":"192.168.201.50:80","downstream_remote_address":"192.168.201.103:56441","duration":"0","istio_policy_status":"-","method":"POST","path":"/ping.Ping/Ping","protocol":"HTTP/2","request_id":"dd357754-f453-40aa-be3a-eb5171ee649f","requested_server_name":"-","response_code":"501","response_flags":"NR","route_name":"-","start_time":"2024-04-08T05:47:46.917Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"grpc-node-js/1.8.10","x_forwarded_for":"192.168.201.103"}

Just paste your stack trace here!

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

  1. xxx
  2. xxx
  3. xxx

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

  • Higress version:
  • OS :
  • Others:
@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

"response_flags":"NR" 是找不到路由的意思。能提供一下PostMan的测试请求配置吗？

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

image

syntax = "proto3";

package ping;
option go_package="./ping";

message Request {
string ping = 1;
}

message Response {
string pong = 1;
}

service Ping {
rpc Ping(Request) returns(Response);
}

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

好的。我本地验证一下。

@johnlanni
Copy link
Collaborator

path: /
pathType: Exact

@ldl-home 精确匹配/肯定不行了，你看下日志里的path是/ping.Ping/Ping

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

不是这个原因
image
这个后面我该过了

@johnlanni
Copy link
Collaborator

@ldl-home 改成新配置后，对应的日志发一下，也是501 NR？

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

我本地试了一下，是可以的。

image

syntax = "proto3";

option java_multiple_files = true;
option java_package = "com.realch3cho.grpctest.grpc";
option go_package = ".;common";

message HelloRequestType {
  string name = 1;
}

message HelloResponseType {
  string message = 1;
}

service HelloService {
  rpc SayHello(HelloRequestType) returns (HelloResponseType) {}
}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: GRPC
    higress.io/destination: grpc.static
    higress.io/ignore-path-case: "false"
  creationTimestamp: "2024-03-22T02:08:05Z"
  generation: 2
  labels:
    higress.io/resource-definer: higress
  name: test
  namespace: higress-system
  resourceVersion: "571475"
  uid: 241ae809-8df4-43c1-a94e-edbd65bc0a9a
spec:
  ingressClassName: higress
  rules:
  - http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix

@johnlanni
Copy link
Collaborator

另外看上去是console界面上配置的，ingress yaml也提供下吧，在higress-system命名空间下

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

我有一个问题，8080端口是gRPC的吗？

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024 •
edited by johnlanni

@johnlanni

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-ping
  namespace: higress-system
  uid: 32fda53a-b2b8-4485-8514-c9bbfad45edc
  resourceVersion: '10440242'
  generation: 1
  creationTimestamp: '2024-04-08T07:07:11Z'
  labels:
    higress.io/domain_grpc.iotbull.com: 'true'
    higress.io/resource-definer: higress
  annotations:
    higress.io/destination: go-ping-s.default.svc.cluster.local:8080
    higress.io/ignore-path-case: 'false'
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
  managedFields:
    - manager: Kubernetes Java Client
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-04-08T07:07:11Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:higress.io/destination: {}
            f:higress.io/ignore-path-case: {}
          f:labels:
            .: {}
            f:higress.io/domain_grpc.iotbull.com: {}
            f:higress.io/resource-definer: {}
        f:spec:
          f:ingressClassName: {}
          f:rules: {}
          f:tls: {}
    - manager: node-fetch
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2024-04-08T07:08:21Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:nginx.ingress.kubernetes.io/backend-protocol: {}
  selfLink: /apis/networking.k8s.io/v1/namespaces/higress-system/ingresses/go-ping
status:
  loadBalancer: {}
spec:
  ingressClassName: higress
  tls:
    - hosts:
        - grpc.iotbull.com
      secretName: iotbull
  rules:
    - host: grpc.test.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              resource:
                apiGroup: networking.higress.io
                kind: McpBridge
                name: default

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

这个我也试过 ： higress.io/backend-protocol: GRPC

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

我有一个问题，8080端口是gRPC的吗？

就是自己测试启动的一个grpc服务。

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

ingress 修改成 prefix 之后的 access log 能发一下吗？还是之前的报错吗？

你可以试试 exec 到 higress-gateway 那个 pod 里，执行 curl -svk http://go-ping-s.default.svc.cluster.local:8080，把输出贴一下。

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

这样连接被拒绝了

  • connect to 10.68.5.183 port 8080 failed: Connection refused
  • Failed to connect to go-ping-s.default.svc.cluster.local port 8080 after 4 ms: Connection refused
  • Closing connection 0

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

这样连接被拒绝了

  • connect to 10.68.5.183 port 8080 failed: Connection refused
  • Failed to connect to go-ping-s.default.svc.cluster.local port 8080 after 4 ms: Connection refused
  • Closing connection 0

这就对应了我前面的问题，你的gRPC服务监听的什么端口，是8080吗？

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

我这边又部署了一个新的pod 服务a提供api的接口，通过api加上path方式 访问grpc服务（服务b），直接访问pod IP是可以通的。通过clusterip地址访问不到
可能是server层有问题
image

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024 

go-ping-s.default.svc.cluster.local:8080

嗯，那你要查查kubeproxy什么的了。看一下Service和Endpoints呢，里面有Pod吗？

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

目前测试下来正常了，通过higress还是访问不到
root@k8s-node01:/# curl -svk http://go-ping-s.default.svc.cluster.local:8080

  • Trying 10.68.5.183:8080...
  • Connected to go-ping-s.default.svc.cluster.local (10.68.5.183) port 8080 (#0)

GET / HTTP/1.1
Host: go-ping-s.default.svc.cluster.local:8080
User-Agent: curl/7.81.0
Accept: /

  • Received HTTP/0.9 when not allowed

{"authority":"web.test.com","bytes_received":"0","bytes_sent":"15","downstream_local_address":"192.168.201.50:443","downstream_remote_address":"192.168.201.103:54807","duration":"9054","istio_policy_status":"-","method":"GET","path":"/ping?target=grpc.iotbull.com:443","protocol":"HTTP/1.1","request_id":"043d4be7-5aa9-474f-9bda-c5d58bda32ec","requested_server_name":"web.test.com","response_code":"503","response_flags":"URX","route_name":"web-test","start_time":"2024-04-08T10:29:24.954Z","trace_id":"f6c8e00c3196ec8cf5a37bfe645a6fa7","upstream_cluster":"outbound|80||go-web.default.svc.cluster.local","upstream_host":"172.20.135.157:8888","upstream_local_address":"172.20.85.192:52034","upstream_service_time":"9054","upstream_transport_failure_reason":"-","user_agent":"PostmanRuntime/7.37.0","x_forwarded_for":"192.168.201.103"}

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

URX: The request was rejected because the upstream retry limit (HTTP) or maximum connect attempts (TCP) was reached.
https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage

目测是连不上。上面命令是在Gateway的Pod里做的吗？我看host变了。新的ingress发一下呢？

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

URX: The request was rejected because the upstream retry limit (HTTP) or maximum connect attempts (TCP) was reached.
https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage

目测是连不上。上面命令是在Gateway的Pod里做的吗？我看host变了。新的ingress发一下呢？

通过postman 直接调用的 服务A 的接口，然后通过target指定grpc服务地址。拿到返回值
类似如下
image

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 8, 2024

curl -svk http://go-ping-s.default.svc.cluster.local:8080

建议你 exec 到 higress-gateway 的 pod 里再 curl 一下

@ldl-home
Copy link
Author

ldl-home commented Apr 8, 2024

@CH3CHO 执行结果如下
image

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 9, 2024

@CH3CHO 执行结果如下 image

你这个是在node上执行吧？能kubectl exec到pod里面吗？

@ldl-home
Copy link
Author

ldl-home commented Apr 9, 2024

这个就是在higress-gateway的pod内执行的

@CH3CHO
Copy link
Collaborator

CH3CHO commented Apr 9, 2024

这个就是在higress-gateway的pod内执行的

那这个前面的机器名有点迷惑性啊

@johnlanni
Copy link
Collaborator

{"authority":"web.test.com","bytes_received":"0","bytes_sent":"15","downstream_local_address":"192.168.201.50:443","downstream_remote_address":"192.168.201.103:54807","duration":"9054","istio_policy_status":"-","method":"GET","path":"/ping?target=grpc.iotbull.com:443","protocol":"HTTP/1.1","request_id":"043d4be7-5aa9-474f-9bda-c5d58bda32ec","requested_server_name":"web.test.com","response_code":"503","response_flags":"URX","route_name":"web-test","start_time":"2024-04-08T10:29:24.954Z","trace_id":"f6c8e00c3196ec8cf5a37bfe645a6fa7","upstream_cluster":"outbound|80||go-web.default.svc.cluster.local","upstream_host":"172.20.135.157:8888","upstream_local_address":"172.20.85.192:52034","upstream_service_time":"9054","upstream_transport_failure_reason":"-","user_agent":"PostmanRuntime/7.37.0","x_forwarded_for":"192.168.201.103"}

看日志是HTTP/1.1呀，不是gRPC请求，而且upstream host字段是172.20.135.157:8888，没有请求8080

@johnlanni
Copy link
Collaborator

@ldl-home 有调试通吗，看你上面发的日志不是gRPC请求

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@CH3CHO @johnlanni @ldl-home