VirtualServer GRPC HTTP/2.0" 400

[Rockawear]

I'm seeing HTTP/2.0" 400 This works fine locally using grpc app below but failed on EKS. I do have tls termination configured. The error before in debug mode is saying using uninitialized
https://github.com/grpc/grpc-go/blob/master/examples/features/reflection/server/main.go

values.yaml

controller:
  ## The name of the Ingress Controller daemonset or deployment.
  name: controller

  ## The kind of the Ingress Controller installation - deployment or daemonset.
  kind: deployment

  ## The selectorLabels used to override the default values.
  selectorLabels: {}

  ## Annotations for deployments and daemonsets
  annotations: {}

  ## Deploys the Ingress Controller for NGINX Plus.
  nginxplus: false

  # Timeout in milliseconds which the Ingress Controller will wait for a successful NGINX reload after a change or at the initial start.
  nginxReloadTimeout: 60000

  ## Support for App Protect WAF
  appprotect:
    ## Enable the App Protect WAF module in the Ingress Controller.
    enable: false
    ## Sets log level for App Protect WAF. Allowed values: fatal, error, warn, info, debug, trace
    # logLevel: fatal

  ## Support for App Protect DoS
  appprotectdos:
    ## Enable the App Protect DoS module in the Ingress Controller.
    enable: false
    ## Enable debugging for App Protect DoS.
    debug: false
    ## Max number of nginx processes to support.
    maxWorkers: 0
    ## Max number of ADMD instances.
    maxDaemons: 0
    ## RAM memory size to consume in MB.
    memory: 0

  ## Enables the Ingress Controller pods to use the host's network namespace.
  hostNetwork: false

  ## The hostPort configuration for the Ingress Controller pods.
  hostPort:
    ## Enables hostPort for the Ingress Controller pods.
    enable: false

    ## The HTTP hostPort configuration for the Ingress Controller pods.
    http: 80

    ## The HTTPS hostPort configuration for the Ingress Controller pods.
    https: 443

  containerPort:
    ## The HTTP containerPort configuration for the Ingress Controller pods.
    http: 80

    ## The HTTPS containerPort configuration for the Ingress Controller pods.
    https: 443

  ## DNS policy for the Ingress Controller pods
  dnsPolicy: ClusterFirst

  ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in the ConfigMap via `controller.config.entries`.
  nginxDebug: false

  ## The log level of the Ingress Controller.
  logLevel: 2

  ## A list of custom ports to expose on the NGINX Ingress Controller pod. Follows the conventional Kubernetes yaml syntax for container ports.
  customPorts: []

  image:
    ## The image repository of the Ingress Controller.
    repository: nginx/nginx-ingress

    ## The tag of the Ingress Controller image. If not specified the appVersion from Chart.yaml is used as a tag.
    # tag: "3.3.0"

    ## The digest of the Ingress Controller image.
    ## If digest is specified it has precedence over tag and will be used instead
    # digest: "sha256:CHANGEME"

    ## The pull policy for the Ingress Controller image.
    pullPolicy: IfNotPresent

  ## The lifecycle of the Ingress Controller pods.
  lifecycle: {}

  ## The custom ConfigMap to use instead of the one provided by default
  customConfigMap: ""

  config:
    ## The name of the ConfigMap used by the Ingress Controller.
    ## Autogenerated if not set or set to "".
    # name: nginx-config

    ## The annotations of the Ingress Controller configmap.
    annotations: {}

    ## The entries of the ConfigMap for customizing NGINX configuration.
    entries:
      http2: "true"
      proxy-protocol: "true"
      real-ip-header: "proxy_protocol"
      set-real-ip-from: "0.0.0.0/0"
      log-format:  '$remote_addr - $remote_user [$time_local] "$request" $status $grpc_status $body_bytes_sent "$http_referer"  "$http_user_agent" "$http_x_forwarded_for" "$resource_name" "$resource_type" "$resource_namespace" "$service"'


  ## It is recommended to use your own TLS certificates and keys
  defaultTLS:
    ## The base64-encoded TLS certificate for the default HTTPS server.
    ## Note: It is recommended that you specify your own certificate. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    cert: ""

    ## The base64-encoded TLS key for the default HTTPS server.
    ## Note: It is recommended that you specify your own key. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    key: ""

    ## The secret with a TLS certificate and key for the default HTTPS server.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.defaultTLS.cert` and `controller.defaultTLS.key` parameters.
    ## Note: Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    ## Format: <namespace>/<secret_name>
    secret: ""

  wildcardTLS:
    ## The base64-encoded TLS certificate for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection.
    cert: ""

    ## The base64-encoded TLS key for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection.
    key: ""

    ## The secret with a TLS certificate and key for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.wildcardTLS.cert` and `controller.wildcardTLS.key` parameters.
    ## Format: <namespace>/<secret_name>
    secret: "namespace>/<secret_nam"

  ## The node selector for pod assignment for the Ingress Controller pods.
  # nodeSelector: {}

  ## The termination grace period of the Ingress Controller pod.
  terminationGracePeriodSeconds: 30

  ## HorizontalPodAutoscaling (HPA)
  autoscaling:
    ## Enables HorizontalPodAutoscaling.
    enabled: true
    ## The annotations of the Ingress Controller HorizontalPodAutoscaler.
    annotations: {}
    ## Minimum number of replicas for the HPA.
    minReplicas: 1
    ## Maximum number of replicas for the HPA.
    maxReplicas: 5
    ## The target cpu utilization percentage.
    targetCPUUtilizationPercentage: 50
    ## The target memory utilization percentage.
    targetMemoryUtilizationPercentage: 50

  ## The resources of the Ingress Controller pods.
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  limits:
    cpu: 1
    memory: 1Gi

  ## The tolerations of the Ingress Controller pods.
  tolerations: []

  ## The affinity of the Ingress Controller pods.
  affinity: {}

  ## The topology spread constraints of the Ingress controller pods.
  # topologySpreadConstraints: {}

  ## The additional environment variables to be set on the Ingress Controller pods.
  env: []
  # - name: MY_VAR
  #   value: myvalue

  ## The volumes of the Ingress Controller pods.
  volumes: []
  # - name: extra-conf
  #   configMap:
  #     name: extra-conf

  ## The volumeMounts of the Ingress Controller pods.
  volumeMounts: []
  # - name: extra-conf
  #   mountPath: /etc/nginx/conf.d/extra.conf
  #   subPath: extra.conf

  ## InitContainers for the Ingress Controller pods.
  initContainers: []
  # - name: init-container
  #   image: busybox:1.34
  #   command: ['sh', '-c', 'echo this is initial setup!']

  ## The minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available.
  minReadySeconds: 0

  ## Pod disruption budget for the Ingress Controller pods.
  podDisruptionBudget:
    ## Enables PodDisruptionBudget.
    enabled: false
    ## The annotations of the Ingress Controller pod disruption budget.
    annotations: {}
    ## The number of Ingress Controller pods that should be available. This is a mutually exclusive setting with "maxUnavailable".
    # minAvailable: 1
    ## The number of Ingress Controller pods that can be unavailable. This is a mutually exclusive setting with "minAvailable".
    # maxUnavailable: 1

    ## Strategy used to replace old Pods by new ones. .spec.strategy.type can be "Recreate" or "RollingUpdate" for Deployments, and "OnDelete" or "RollingUpdate" for Daemonsets. "RollingUpdate" is the default value.
  strategy: {}

  ## Extra containers for the Ingress Controller pods.
  extraContainers: []
  # - name: container
  #   image: busybox:1.34
  #   command: ['sh', '-c', 'echo this is a sidecar!']

  ## The number of replicas of the Ingress Controller deployment.
  replicaCount: 1

  # Configures the ingress class the Ingress Controller uses.
  ingressClass:
    ## A class of the Ingress Controller.

    ## IngressClass resource with the name equal to the class must be deployed. Otherwise,
    ## the Ingress Controller will fail to start.
    ## The Ingress Controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class.

    ## The Ingress Controller processes all the resources that do not have the "ingressClassName" field for all versions of kubernetes.
    name: nginx

    ## Creates a new IngressClass object with the name "controller.ingressClass.name". Set to false to use an existing IngressClass with the same name. If you use helm upgrade, do not change the values from the previous release as helm will delete IngressClass objects managed by helm. If you are upgrading from a release earlier than 3.3.0, do not set the value to false.
    create: true

    ## New Ingresses without an ingressClassName field specified will be assigned the class specified in `controller.ingressClass`. Requires "controller.ingressClass.create".
    setAsDefaultIngress: false

  ## Comma separated list of namespaces to watch for Ingress resources. By default the Ingress Controller watches all namespaces. Mutually exclusive with "controller.watchNamespaceLabel".
  watchNamespace: ""

  ## Configures the Ingress Controller to watch only those namespaces with label foo=bar. By default the Ingress Controller watches all namespaces. Mutually exclusive with "controller.watchNamespace".
  watchNamespaceLabel: ""

  ## Comma separated list of namespaces to watch for Secret resources. By default the Ingress Controller watches all namespaces.
  watchSecretNamespace: ""

  ## Enable the custom resources.
  enableCustomResources: true

  ## Enable preview policies. This parameter is deprecated. To enable OIDC Policies please use controller.enableOIDC instead.
  enablePreviewPolicies: false

  ## Enable OIDC policies.
  enableOIDC: false

  ## Include year in log header. This parameter will be removed in release 2.7 and the year will be included by default.
  includeYear: false

  ## Enable TLS Passthrough on port 443. Requires controller.enableCustomResources.
  enableTLSPassthrough: false

  ## Set the port for TLS Passthrough. Requires controller.enableCustomResources and controller.enableTLSPassthrough.
  tlsPassthroughPort: 443

  ## Enable cert manager for Virtual Server resources. Requires controller.enableCustomResources.
  enableCertManager: true

  ## Enable external DNS for Virtual Server resources. Requires controller.enableCustomResources.
  enableExternalDNS: true

  globalConfiguration:
    ## Creates the GlobalConfiguration custom resource. Requires controller.enableCustomResources.
    create: false

    ## The spec of the GlobalConfiguration for defining the global configuration parameters of the Ingress Controller.
    spec: {} ## Ensure both curly brackets are removed when adding listeners in YAML format.
    # listeners:
    # - name: dns-udp
    #   port: 5353
    #   protocol: UDP
    # - name: dns-tcp
    #   port: 5353
    #   protocol: TCP


  ## Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources.
  enableSnippets: true

  ## Add a location based on the value of health-status-uri to the default server. The location responds with the 200 status code for any request.
  ## Useful for external health-checking of the Ingress Controller.
  healthStatus: true

  ## Sets the URI of health status location in the default server. Requires controller.healthStatus.
  healthStatusURI: "/nginx-health"

  nginxStatus:
    ## Enable the NGINX stub_status, or the NGINX Plus API.
    enable: false

    ## Set the port where the NGINX stub_status or the NGINX Plus API is exposed.
    port: 8080

    ## Add IPv4 IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas.
    allowCidrs: "127.0.0.1"

  service:
    ## Creates a service to expose the Ingress Controller pods.
    create: true

    ## The type of service to create for the Ingress Controller.
    type: LoadBalancer

    ## The externalTrafficPolicy of the service. The value Local preserves the client source IP.
    externalTrafficPolicy: Local

    ## The annotations of the Ingress Controller service.
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "external"
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    ## The extra labels of the service.
    extraLabels: {}

    ## The static IP address for the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerIP: ""

    ## The ClusterIP for the Ingress Controller service, autoassigned if not specified.
    clusterIP: ""

    ## The list of external IPs for the Ingress Controller service.
    externalIPs: []

    ## The IP ranges (CIDR) that are allowed to access the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerSourceRanges: []

    ## Whether to automatically allocate NodePorts (only for LoadBalancers).
    # allocateLoadBalancerNodePorts: false

    ## Dual stack preference.
    ## Valid values: SingleStack, PreferDualStack, RequireDualStack
    # ipFamilyPolicy: SingleStack

    ## List of IP families assigned to this service.
    ## Valid values: IPv4, IPv6
    # ipFamilies:
    #   - IPv6

    httpPort:
      ## Enables the HTTP port for the Ingress Controller service.
      enable: false

      ## The HTTP port of the Ingress Controller service.
      port: 80

      ## The custom NodePort for the HTTP port. Requires controller.service.type set to NodePort.
      # nodePort: 80

      ## The HTTP port on the POD where the Ingress Controller service is running.
      targetPort: 80

    httpsPort:
      ## Enables the HTTPS port for the Ingress Controller service.
      enable: true

      ## The HTTPS port of the Ingress Controller service.
      port: 443

      ## The custom NodePort for the HTTPS port. Requires controller.service.type set to NodePort.
      # nodePort: 443

      ## The HTTPS port on the POD where the Ingress Controller service is running.
      targetPort: 443

    ## A list of custom ports to expose through the Ingress Controller service. Follows the conventional Kubernetes yaml syntax for service ports.
    customPorts: []

  serviceAccount:
    ## The annotations of the service account of the Ingress Controller pods.
    annotations: {}

    ## The name of the service account of the Ingress Controller pods. Used for RBAC.
    ## Autogenerated if not set or set to "".
    # name: nginx-ingress

    ## The name of the secret containing docker registry credentials.
    ## Secret must exist in the same namespace as the helm release.
    imagePullSecretName: ""

  reportIngressStatus:
    ## Updates the address field in the status of Ingress resources with an external address of the Ingress Controller.
    ## You must also specify the source of the external address either through an external service via controller.reportIngressStatus.externalService,
    ## controller.reportIngressStatus.ingressLink or the external-status-address entry in the ConfigMap via controller.config.entries.
    ## Note: controller.config.entries.external-status-address takes precedence over the others.
    enable: true

    ## Specifies the name of the service with the type LoadBalancer through which the Ingress Controller is exposed externally.
    ## The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources.
    ## controller.reportIngressStatus.enable must be set to true.
    ## The default is autogenerated and matches the created service (see controller.service.create).
    # externalService: nginx-ingress

    ## Specifies the name of the IngressLink resource, which exposes the Ingress Controller pods via a BIG-IP system.
    ## The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources.
    ## controller.reportIngressStatus.enable must be set to true.
    ingressLink: ""

    ## Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress resources. controller.reportIngressStatus.enable must be set to true.
    enableLeaderElection: true

    ## Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. controller.reportIngressStatus.enableLeaderElection must be set to true.
    ## Autogenerated if not set or set to "".
    # leaderElectionLockName: "nginx-ingress-leader-election"

    ## The annotations of the leader election configmap.
    annotations: {}

  pod:
    ## The annotations of the Ingress Controller pod.
    annotations: {}

    ## The additional extra labels of the Ingress Controller pod.
    extraLabels: {}

  ## The PriorityClass of the Ingress Controller pods.
  # priorityClassName: ""

  readyStatus:
    ## Enables readiness endpoint "/nginx-ready". The endpoint returns a success code when NGINX has loaded all the config after startup.
    enable: true

    ## Set the port where the readiness endpoint is exposed.
    port: 8081

    ## The number of seconds after the Ingress Controller pod has started before readiness probes are initiated.
    initialDelaySeconds: 0

  ## Enable collection of latency metrics for upstreams. Requires prometheus.create.
  enableLatencyMetrics: false

  ## Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack.
  disableIPV6: false

  ## Configure root filesystem as read-only and add volumes for temporary data.
  readOnlyRootFilesystem: false

rbac:
  ## Configures RBAC.
  create: true

prometheus:
  ## Expose NGINX or NGINX Plus metrics in the Prometheus format.
  create: true

  ## Configures the port to scrape the metrics.
  port: 9113

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Prometheus endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

  service:
    ## Creates a ClusterIP Service to expose Prometheus metrics internally
    ## Requires prometheus.create=true
    create: false

    labels:
      service: "nginx-ingress-prometheus-service"

  serviceMonitor:
    ## Creates a serviceMonitor to expose statistics on the kubernetes pods.
    create: false

    ## Kubernetes object labels to attach to the serviceMonitor object.
    labels: {}

    ## A set of labels to allow the selection of endpoints for the ServiceMonitor.
    selectorMatchLabels:
      service: "nginx-ingress-prometheus-service"

    ## A list of endpoints allowed as part of this ServiceMonitor.
    ## Matches on the name of a Service port.
    endpoints:
     - port: prometheus

serviceInsight:
  ## Expose NGINX Plus Service Insight endpoint.
  create: false

  ## Configures the port to expose endpoint.
  port: 9114

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Service Insight endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

nginxServiceMesh:
  ## Enables integration with NGINX Service Mesh.
  enable: false

  ## Enables NGINX Service Mesh workload to route egress traffic through the Ingress Controller.
  ## Requires nginxServiceMesh.enable
  enableEgress: false

Deploy

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: go-grpc
  name: go-grpc
  namespace: grpc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-grpc
  template:
    metadata:
      labels:
        app: go-grpc
    spec:
      containers:
      - image: <myimage> 
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 50Mi
        # imagePullPolicy: Never
        name: go-grpc
        ports:
        - containerPort: 50051
        env:
          - name: GODEBUG
            value: http2debug=2
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: go-grpc
  name: go-grpc
  namespace: grpc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 50051
  selector:
    app: go-grpc
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: grpc-vs
  namespace: grpc
spec:
  host: dev.example.com
  tls:
    secret: grpc-secret #self signed
  upstreams:
  - name: grpc1
    service: go-grpc
    port: 50051
    type: grpc
  routes:
  - path: "/helloworld.Greeter"
    action:
      pass: grpc1

test with grpcurl

grpcurl -insecure -vv dev.example.com:443 list

Expected Behavior:
I should be able to list,describe, or get the service

Current Behavior:
test command above failed with

Failed to dial target host "dev.example.com:443": context deadline exceeded

IC logs

2023/11/03 01:16:00 [warn] 274#274: *4017 using uninitialized "resource_name" variable while logging request, client: 10.0.3.21, server: dev.example.com, request: "PRI * HTTP/2.0"                                               │
│ 2023/11/03 01:16:00 [warn] 274#274: *4017 using uninitialized "resource_type" variable while logging request, client: 10.0.3.21, server: dev.example.com, request: "PRI * HTTP/2.0"                                               │
│ 2023/11/03 01:16:00 [warn] 274#274: *4017 using uninitialized "resource_namespace" variable while logging request, client: 10.0.3.21, server: dev.example.com, request: "PRI * HTTP/2.0"                                          │
│ 2023/11/03 01:16:00 [warn] 274#274: *4017 using uninitialized "service" variable while logging request, client: 10.0.3.21, server: dev.example.com, request: "PRI * HTTP/2.0"                                                     │
│ 10.0.3.21 - - [03/Nov/2023:01:16:00 +0000] "PRI * HTTP/2.0" 400  157 "-"  "-" "-" "" "" "" ""

[Rockawear]

This is a nonissue. User error.

[brianehlert]

Would it be possible for you to share the answer?
There might be others that experience the same and could be helped by your experience.

[Rockawear]

I was trying to use grpcurl tool to send request to upstream grpc that doesn't support reflection. grpcurl seems to use /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo by default. I had to create the proto source files that define the service or need protoset files that grpcurl can use.