400 Bad Request

ram-centime · 2024-02-07T14:30:44Z

ram-centime
Feb 7, 2024

Hi,

NGINX Gateway Fabric v1.0.0 installed in my EKS cluster version 1.28, when I access one of the service in my cluster giving 400 Bad Request - The plain HTTP request was sent to HTTPS port response when I access the service.

curl https://xxx.yyy.com/config-service/common/default/infra/currency-conversion.properties

<title>400 The plain HTTP request was sent to HTTPS port</title>

The plain HTTP request was sent to HTTPS port

nginx/1.25.3

Could you please help me in this regards.

I followed the below installation steps, also attaching the manifests.

openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj '/CN=xxx.yyy.com' -addext 'subjectAltName=DNS:xxx.yyy.com'

kubectl create secret tls centimetls --key tls.key --cert tls.crt

openssl x509 -in tls.crt -out tls.pem -outform PEM

kubectl create configmap centimetls --namespace=func-services --from-file=tls.pem

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/standard-install.yaml
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.1.0/crds.yaml
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.1.0/nginx-gateway.yaml

loadbalancer-aws-nlb.yaml

Source: nginx-gateway-fabric/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
name: nginx-gateway
namespace: nginx-gateway
labels:
app.kubernetes.io/name: nginx-gateway
app.kubernetes.io/instance: nginx-gateway
app.kubernetes.io/version: "1.0.0"
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-2::certificate/
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "owner=devops"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: nginx-gateway
app.kubernetes.io/instance: nginx-gateway
ports: # Update the following ports to match your Gateway Listener ports

name: http
port: 80
protocol: TCP
targetPort: 80
name: https
port: 443
protocol: TCP
targetPort: 443

gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: func-services
namespace: func-services
spec:
gatewayClassName: nginx
listeners:

name: http
protocol: HTTP
port: 80
hostname: xxx.yyy.com
name: https
protocol: HTTPS
port: 443
hostname: xxx.yyy.com
tls:
certificateRefs:
- kind: Secret
  group: ""
  name: centimetls

route.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: config-service
namespace: func-services
spec:
parentRefs:

name: func-services
sectionName: https
hostnames:
xxx.yyy.com
rules:
backendRefs:
- name: config-service
  port: 8443
  matches:
- path:
  type: PathPrefix
  value: /config-service

tls.yaml

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: BackendTLSPolicy
metadata:
name: config-service
namespace: func-services
spec:
targetRef:
kind: Service
name: config-service
group: ""
tls:
caCertRefs:
- kind: ConfigMapReference
name: centimetls
group: ""
hostname: xxx.yyy.com

kubectl get svc config-service -n func-services -o yaml

apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: config-service
meta.helm.sh/release-namespace: func-services
creationTimestamp: "2024-02-06T10:00:52Z"
labels:
app: config-service
app.kubernetes.io/managed-by: Helm
name: config-service
namespace: func-services
resourceVersion: "7727170"
uid: 68b26950-ba58-4b09-b714-725e00021318
spec:
clusterIP: 10.100.126.75
clusterIPs:

10.100.126.75
internalTrafficPolicy: Cluster
ipFamilies:
IPv4
ipFamilyPolicy: SingleStack
ports:
name: http
port: 8443
protocol: TCP
targetPort: 8443
selector:
app: config-service
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}

kubectl get deploy config-service -n func-services -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
meta.helm.sh/release-name: config-service
meta.helm.sh/release-namespace: func-services
creationTimestamp: "2024-02-06T10:00:52Z"
generation: 1
labels:
app: config-service
app.kubernetes.io/managed-by: Helm
name: config-service
namespace: func-services
resourceVersion: "8162390"
uid: 81f2680f-2497-4456-aa05-49fd4f64617c
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: config-service
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
ad.datadoghq.com/config-service.logs: |-
[{
"source": "config-service",
"service": "config-service",
"log_processing_rules": [{
"type": "multi_line",
"name": "log_start_with_date",
"pattern" : "\d{4}-(0?[1-9]|1[012])-(0?[1-9]|[12][0-9]|3[01])"
}]
}]
creationTimestamp: null
labels:
app: config-service
spec:
containers:
- env:
- name: server.servlet.context-path
value: /config-service
- name: spring.profiles.active
value: infra
- name: spring.cloud.config.uri
value: https://xxx.yyy.com/config-service
- name: spring.cloud.config.label
value: infra
- name: HOST_KAFKA
value: http://xxx.yyy.com:9092
- name: DEPLOY_ENV
value: infra
- name: JVM_MEM_OPTS
value: -Xms2048m -Xmx2048m
- name: LOG_LEVEL
value: INFO
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: CENTIME_DD_APM_ENABLED
value: "false"
- name: DD_VERSION
value: config-service-qa-16138
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: DD_ENV
value: infra
- name: DD_SERVICE
value: config-service
- name: DD_PROFILING_ENABLED
value: "false"
- name: DD_LOGS_INJECTION
value: "false"
- name: server.ssl.key-store
value: /usr/lib/jvm/java-certs/lib/security/centime-ssl.p12
- name: server.ssl.key-alias
value: centimessl
- name: server.ssl.key-store-password
value: centimessl
image: .dkr.ecr.us-east-2.amazonaws.com/config-service:qa-16138
imagePullPolicy: IfNotPresent
name: config-service
ports:
- containerPort: 8443
name: http
protocol: TCP
readinessProbe:
failureThreshold: 35
httpGet:
path: /config-service/actuator/health
port: 8443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 250m
memory: 1Gi
requests:
cpu: 200m
memory: 900Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /usr/lib/jvm/java-certs/lib/security
name: centimessl
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: jfrogregcreds
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: centime-infra-common
serviceAccountName: centime-infra-common
terminationGracePeriodSeconds: 30
topologySpreadConstraints:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- config-service
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- config-service
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
volumes:
- name: centimessl
secret:
defaultMode: 420
secretName: centimessl
status:
availableReplicas: 1
conditions:

lastTransitionTime: "2024-02-06T10:00:52Z"
lastUpdateTime: "2024-02-06T10:02:16Z"
message: ReplicaSet "config-service-f6b44dc56" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
lastTransitionTime: "2024-02-07T05:19:09Z"
lastUpdateTime: "2024-02-07T05:19:09Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1

jasonwilliams14 · 2024-02-07T15:39:24Z

jasonwilliams14
Feb 7, 2024
Collaborator

@ram-centime Hello there. From your curl request, looks like you are sending a plain HTTP request to a port that is listening on HTTPS without a redirect configured.

You can try using curl -k HTTPS://<url> to test and verify your current setup.

To have NGINX Gateway fabric automatically do a redirect for HTTP requests to HTTPS, you will need to configure a redirect.
Here is an example:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cafe-tls-redirect
spec:
  parentRefs:
  - name: gateway
    sectionName: http
  hostnames:
  - "cafe.example.com"
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        port: 443

You can see our examples located in this repo and folder:

https://github.com/nginxinc/nginx-gateway-fabric/blob/main/examples/https-termination/cafe-routes.yaml

Also, ensure your gateway has been properly setup to be able to perform a TLS redirect and termination. Our folder has an example of the gateway:

  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: cafe-secret
        namespace: certificate

Let us know if that helps.

7 replies

ram-centime Feb 7, 2024
Author

kubectl exec -it -n nginx-gateway nginx-gateway-c8877d97f-7skts -c nginx -- nginx -T

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

configuration file /etc/nginx/nginx.conf:

load_module /usr/lib/nginx/modules/ngx_http_js_module.so;

worker_processes auto;

pid /var/run/nginx/nginx.pid;
error_log stderr info;

events {
worker_connections 1024;
}

http {
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/mime.types;
js_import /usr/lib/nginx/modules/njs/httpmatches.js;

default_type application/octet-stream;

proxy_headers_hash_bucket_size 512;
proxy_headers_hash_max_size 1024;
server_names_hash_bucket_size 256;
server_names_hash_max_size 1024;
variables_hash_bucket_size 512;
variables_hash_max_size 1024;

sendfile on;
tcp_nopush on;

server {
listen unix:/var/run/nginx/nginx-status.sock;
access_log off;

location /stub_status {
    stub_status;
}

}
}

configuration file /etc/nginx/conf.d/config-version.conf:

server {
listen unix:/var/run/nginx/nginx-config-version.sock;
access_log off;

location /version {
    return 200 6;
}

}

configuration file /etc/nginx/conf.d/http.conf:

upstream func-services_first-app_8080 {
random two least_conn;
zone func-services_first-app_8080 512k;

server 172.30.39.23:8080;
server 172.30.39.184:8080;

}

upstream func-services_config-service_8443 {
random two least_conn;
zone func-services_config-service_8443 512k;

server 172.30.39.240:8443;

}

upstream invalid-backend-ref {
random two least_conn;
zone invalid-backend-ref 512k;

server unix:/var/lib/nginx/nginx-500-server.sock;

}

server {
listen 80 default_server;

default_type text/html;
return 404;

}

server {
listen 80;

server_name api.services.infra.centime.com;


location /_prefix_route0 {
    internal;

    proxy_set_header Host $gw_api_compliant_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_pass http://func-services_first-app_8080$request_uri;
}

location /_prefix_route1 {
    internal;
    return 302 "https://$host$request_uri";

}

location / {
    set $http_matches "[{\"redirectPath\":\"/_prefix_route0\",\"any\":true},{\"redirectPath\":\"/_prefix_route1\",\"any\":true}]";
    js_content httpmatches.redirect;

}

}
server {
listen 443 ssl default_server;

ssl_reject_handshake on;

}

server {
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/ssl_keypair_func-services_centimetls.pem;
ssl_certificate_key /etc/nginx/secrets/ssl_keypair_func-services_centimetls.pem;

if ($ssl_server_name != $host) {
    return 421;
}

server_name api.services.infra.centime.com;


location /config-service/ {

    proxy_set_header Host $gw_api_compliant_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_pass http://func-services_config-service_8443$request_uri;
}

location = /config-service {

    proxy_set_header Host $gw_api_compliant_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_pass http://func-services_config-service_8443$request_uri;
}

location / {
    return 404 "";

}

}

server {
listen unix:/var/lib/nginx/nginx-502-server.sock;
access_log off;

return 502;

}

server {
listen unix:/var/lib/nginx/nginx-500-server.sock;
access_log off;

return 500;

}

Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value

of $host. We prefer $http_host because it contains the original value of the host header, which is required by the

Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use

the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.

map $http_host $gw_api_compliant_host {
'' $host;
default $http_host;
}

Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This

allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

configuration file /etc/nginx/mime.types:

types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;

text/mathml                                      mml;
text/plain                                       txt;
text/vnd.sun.j2me.app-descriptor                 jad;
text/vnd.wap.wml                                 wml;
text/x-component                                 htc;

image/avif                                       avif;
image/png                                        png;
image/svg+xml                                    svg svgz;
image/tiff                                       tif tiff;
image/vnd.wap.wbmp                               wbmp;
image/webp                                       webp;
image/x-icon                                     ico;
image/x-jng                                      jng;
image/x-ms-bmp                                   bmp;

font/woff                                        woff;
font/woff2                                       woff2;

application/java-archive                         jar war ear;
application/json                                 json;
application/mac-binhex40                         hqx;
application/msword                               doc;
application/pdf                                  pdf;
application/postscript                           ps eps ai;
application/rtf                                  rtf;
application/vnd.apple.mpegurl                    m3u8;
application/vnd.google-earth.kml+xml             kml;
application/vnd.google-earth.kmz                 kmz;
application/vnd.ms-excel                         xls;
application/vnd.ms-fontobject                    eot;
application/vnd.ms-powerpoint                    ppt;
application/vnd.oasis.opendocument.graphics      odg;
application/vnd.oasis.opendocument.presentation  odp;
application/vnd.oasis.opendocument.spreadsheet   ods;
application/vnd.oasis.opendocument.text          odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                 pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                 xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                 docx;
application/vnd.wap.wmlc                         wmlc;
application/wasm                                 wasm;
application/x-7z-compressed                      7z;
application/x-cocoa                              cco;
application/x-java-archive-diff                  jardiff;
application/x-java-jnlp-file                     jnlp;
application/x-makeself                           run;
application/x-perl                               pl pm;
application/x-pilot                              prc pdb;
application/x-rar-compressed                     rar;
application/x-redhat-package-manager             rpm;
application/x-sea                                sea;
application/x-shockwave-flash                    swf;
application/x-stuffit                            sit;
application/x-tcl                                tcl tk;
application/x-x509-ca-cert                       der pem crt;
application/x-xpinstall                          xpi;
application/xhtml+xml                            xhtml;
application/xspf+xml                             xspf;
application/zip                                  zip;

application/octet-stream                         bin exe dll;
application/octet-stream                         deb;
application/octet-stream                         dmg;
application/octet-stream                         iso img;
application/octet-stream                         msi msp msm;

audio/midi                                       mid midi kar;
audio/mpeg                                       mp3;
audio/ogg                                        ogg;
audio/x-m4a                                      m4a;
audio/x-realaudio                                ra;

video/3gpp                                       3gpp 3gp;
video/mp2t                                       ts;
video/mp4                                        mp4;
video/mpeg                                       mpeg mpg;
video/quicktime                                  mov;
video/webm                                       webm;
video/x-flv                                      flv;
video/x-m4v                                      m4v;
video/x-mng                                      mng;
video/x-ms-asf                                   asx asf;
video/x-ms-wmv                                   wmv;
video/x-msvideo                                  avi;

}

jasonwilliams14 Feb 7, 2024
Collaborator

@ram-centime i will take a look and review your config. 👍

ram-centime Feb 7, 2024
Author

Sure, Thank you!

jasonwilliams14 Feb 7, 2024
Collaborator

@ram-centime one more thing before I forget and review you configs and setup, can you provide logs from NGF pod? Can you send a few requests with curl, then capture the logs and send them here?

is your backend application expecting a secure HTTP connection on port 8443 ?

ram-centime Feb 8, 2024
Author

is your backend application expecting a secure HTTP connection on port 8443 ? Yes

sjberman · 2024-02-07T19:47:14Z

sjberman
Feb 7, 2024
Collaborator

One thing I'll note is that it appears you are using a BackendTLSPolicy, which is not yet supported. We are currently working on it (see #1487), which will be included in our next release.

Current supported features of the Gateway API are included in our docs

0 replies

jasonwilliams14 · 2024-02-07T19:47:23Z

jasonwilliams14
Feb 7, 2024
Collaborator

@ram-centime Ok. After reviewing all the information you sent me. It looks to be that your upstream application server is expecting an HTTPS proxied connection from NGF to your application. I noticed you are using a backendTLSpolicy, which is why I am making this assumption.

A quick note, backendTLSPolicy is currently experimental in the API Gateway project:
https://gateway-api.sigs.k8s.io/api-types/backendtlspolicy/

That being said, the backendTLSPolicy is currently not implemented yet (we have a PR open for it...).

See here:
#1262

One last note. You will also have to make sure you are using the experimental APIs as well to test the backendTLSPolicy functionality.

let me know if I have your setup correct and if this is helpful.

14 replies

ram-centime Apr 2, 2024
Author

nginx.zip

Please find nginx.conf and http.conf

/etc/nginx/conf.d $ ls -l
total 8
-rw-r--r-- 1 102 1001 143 Apr 2 12:41 config-version.conf
-rw-r--r-- 1 102 1001 3257 Apr 2 12:41 http.conf
/etc/nginx/conf.d $

nginx.conf

load_module /usr/lib/nginx/modules/ngx_http_js_module.so;

worker_processes auto;

pid /var/run/nginx/nginx.pid;
error_log stderr info;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/mime.types;
  js_import /usr/lib/nginx/modules/njs/httpmatches.js;

  default_type application/octet-stream;

  proxy_headers_hash_bucket_size 512;
  proxy_headers_hash_max_size 1024;
  server_names_hash_bucket_size 256;
  server_names_hash_max_size 1024;
  variables_hash_bucket_size 512;
  variables_hash_max_size 1024;

  sendfile on;
  tcp_nopush on;

  server {
    listen unix:/var/run/nginx/nginx-status.sock;
    access_log off;

    location /stub_status {
        stub_status;
    }
  }
}

http.conf

upstream func-services_config-service-internal_8443 {
    random two least_conn;
    zone func-services_config-service-internal_8443 512k;

    server 172.30.39.48:8443;
}

upstream invalid-backend-ref {
    random two least_conn;
    zone invalid-backend-ref 32k;

    server unix:/var/lib/nginx/nginx-500-server.sock;
}



server {
    listen 80 default_server;

    default_type text/html;
    return 404;
}

server {
    listen 80;

    server_name internal.fs.services.infra.centime.com;


    location / {
        return 302 "https://$host$request_uri";
    }

}
server {
    listen 443 ssl default_server;

    ssl_reject_handshake on;
}

server {
    listen 443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_func-services_centimetls.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_func-services_centimetls.pem;

    if ($ssl_server_name != $host) {
        return 421;
    }

    server_name internal.fs.services.infra.centime.com;


    location /config-service/ {
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_http_version 1.1;
        proxy_pass https://func-services_config-service-internal_8443$request_uri;
        proxy_ssl_verify on;
        proxy_ssl_name internal.fs.services.infra.centime.com;
        proxy_ssl_trusted_certificate /etc/nginx/secrets/cert_bundle_func-services_centimetls.crt;
    }

    location = /config-service {
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_upgrade";
        proxy_http_version 1.1;
        proxy_pass https://func-services_config-service-internal_8443$request_uri;
        proxy_ssl_verify on;
        proxy_ssl_name internal.fs.services.infra.centime.com;
        proxy_ssl_trusted_certificate /etc/nginx/secrets/cert_bundle_func-services_centimetls.crt;
    }

    location / {
        return 404 "";
    }

}

server {
    listen unix:/var/lib/nginx/nginx-502-server.sock;
    access_log off;

    return 502;
}

server {
    listen unix:/var/lib/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}



# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

ciarams87 Apr 3, 2024
Maintainer

It looks like NGINX is being configured correctly, meaning that your provided files are indeed correct. I would expect to get the error you are seeing if this line had http instead of https: proxy_pass https://func-services_config-service-internal_8443$request_uri;.

I tried to reproduce using your configuration (as best as possible, as naturally I don't have your upstream certificates) and everything looks fine for me - when I do use your provided ConfigMap, I get the expected upstream SSL verification error:
2024/04/03 12:45:59 [error] 216#216: *24 upstream SSL certificate does not match "cafe.example.com" while SSL handshaking to upstream, client: 172.18.0.4, server: cafe.example.com, request: "GET /config-service HTTP/1.1", upstream: "https://10.244.0.7:8443/config-service", host: "cafe.example.com"
Updating the values to match my environment results in a successful 200 response.

I can only reproduce the error you are facing when I do something like this:

curl --resolve cafe.example.com:$GW_HTTPS_PORT:$GW_IP http://cafe.example.com:$GW_HTTPS_PORT/config-service -k

Is it possible that your NLB is terminating the https traffic and forwarding the unencrypted traffic to NGF? I suspect the status 400 is coming before any traffic is being routed to your backend application.

ram-centime Apr 4, 2024
Author

Yes, even I do suspect the same, does NGINX Gateway Fabric support TLSRoute so in Gateway we can enable mode Passthrough with TLS protocol so the the ssl termination will not happen at Gateway.
Or
Any other solution to fix this

sjberman Apr 4, 2024
Collaborator

We don't currently support TLSRoute, however I don't think this would solve your issue, because you would need passthrough at the NLB, not the Gateway (assuming that this is in fact the issue).

sjberman Apr 8, 2024
Collaborator

One thing we could test is if you send traffic straight to the NGF pod instead of to NLB. Could use port-forwarding and then curl --resolve to use the same hostname but resolve it to the NGF pod IP address.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

400 Bad Request - The plain HTTP request was sent to HTTPS port #1547

{{title}}

Replies: 3 comments 21 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

400 Bad Request - The plain HTTP request was sent to HTTPS port #1547

ram-centime Feb 7, 2024

400 Bad Request

loadbalancer-aws-nlb.yaml

Source: nginx-gateway-fabric/templates/service.yaml

gateway.yaml

route.yaml

tls.yaml

kubectl get svc config-service -n func-services -o yaml

kubectl get deploy config-service -n func-services -o yaml

Replies: 3 comments · 21 replies

jasonwilliams14 Feb 7, 2024 Collaborator

ram-centime Feb 7, 2024 Author

kubectl exec -it -n nginx-gateway nginx-gateway-c8877d97f-7skts -c nginx -- nginx -T

configuration file /etc/nginx/nginx.conf:

configuration file /etc/nginx/conf.d/config-version.conf:

configuration file /etc/nginx/conf.d/http.conf:

Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value

of $host. We prefer $http_host because it contains the original value of the host header, which is required by the

Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use

the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.

Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This

allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.

configuration file /etc/nginx/mime.types:

jasonwilliams14 Feb 7, 2024 Collaborator

ram-centime Feb 7, 2024 Author

jasonwilliams14 Feb 7, 2024 Collaborator

ram-centime Feb 8, 2024 Author

sjberman Feb 7, 2024 Collaborator

jasonwilliams14 Feb 7, 2024 Collaborator

ram-centime Apr 2, 2024 Author

nginx.conf

http.conf

ciarams87 Apr 3, 2024 Maintainer

ram-centime Apr 4, 2024 Author

sjberman Apr 4, 2024 Collaborator

sjberman Apr 8, 2024 Collaborator

ram-centime
Feb 7, 2024

Replies: 3 comments 21 replies

jasonwilliams14
Feb 7, 2024
Collaborator

ram-centime Feb 7, 2024
Author

jasonwilliams14 Feb 7, 2024
Collaborator

ram-centime Feb 7, 2024
Author

jasonwilliams14 Feb 7, 2024
Collaborator

ram-centime Feb 8, 2024
Author

sjberman
Feb 7, 2024
Collaborator

jasonwilliams14
Feb 7, 2024
Collaborator

ram-centime Apr 2, 2024
Author

ciarams87 Apr 3, 2024
Maintainer

ram-centime Apr 4, 2024
Author

sjberman Apr 4, 2024
Collaborator

sjberman Apr 8, 2024
Collaborator