how to configure option httplog in the configmap

[dcowden]

I feel like i must be missing it-- how do i get 'option httplog' into the frontend configuration for the ingress controller?

i found tcp-log-format and https-log-format and http-log-format. However, these seem to be intended to activate custom formats, and further the documentation says they are only used when syslog is in use ( i'm using the default configuration, which is sending data to stdout ).

Our ingress controller has generated this haproxy frontend config:

######
###### HTTPS frontend (tcp mode)
######
frontend httpsfront
    bind *:443
    mode tcp
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    default_backend httpsback-shared-backend

But we want this:

######
###### HTTPS frontend (tcp mode)
######
frontend httpsfront
    bind *:443
    mode tcp
    option httplog
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }
    default_backend httpsback-shared-backend

[dcowden]

as a side note, it appears that /etc/haproxy/template/haproxy.tpml doesnt appear to have a stanza for option httplog, so i'm assuming there's no current way to get it in there:

defaults
log global
{{- if $cfg.LoadServerState }}
load-server-state-from-file global
{{- end }}
maxconn {{ $cfg.MaxConn }}
{{- if $cfg.DrainSupport }}
option persist
{{- else }}
option redispatch
{{- end }}
option dontlognull

I'm trying to do performance analysis, so i need option httplog.

[dcowden]

looking at the source for the template, it seems like there's no way to inject options httplog ,

unless specifying log-format {{ $cfg.HTTPSLogFormat }} will activate it.

For now i'll try a workaround of specifying a custom log format that's as the same as httplog would have given

[dcowden]

Ok, I'm stuck i think.
I tried defining https-log-format, but it doesnt work because It is only used when syslog is used.

My problem is that I need performance/request logging, but I want the data to to stdout ( so that it is in the docker logs ) vs syslog.

Unless i'm missing something, its not possible to configure the ingress controller to send request logging to stdout, because all of the options are conditional on syslog

Is there a workaround/option? Right now the only thing i know to do is to make a derivative image, and change the template in there, which i'd rather not do!

[dcowden]

For those others trying to get timestamps in the docker logs:

The plot thickens-- it appears that haproxy itself wont send data to stderr/stdout-- only syslog.

I found this thread that helps

There i mention of starting up syslog/rsyslogd so that the data appears in stdout, but this doesnt appear implemented in this container.

This discussion is also relevant:

I can't believe it is this difficult to get standard access logs into the docker logs! I feel like i'm missing something easy!

[Scukerman]

HAProxy doesn't support any other logging but syslog.

Check these issues out to find your own way of exposing logs to stdout\stderr:
#110, #74

[dcowden]

Thanks @Scukerman!

I will try the sidecar approach.

[dcowden]

Thanks for the suggestion @Scukerman , I got the sidecar running. I'm seeing some odd behavior, though,i'm hoping someone has seen.

When i first start up the ingress ( with sidecar as described in #110), I get logging perfectly.

However, after about 1 minute or so, no further access logs show up. Restarting the pod restores logging for about the same period of time, then it stops.

There are no errors in access_log or the haproxy log. Anyone have ideas?

[Scukerman]

I don't use the sidecar approach because it's much easier to get all the logs in one place. I run a separate pod with jumanjiman/rsyslog image and use rsyslog-pod.my-namespace:514.
Running this way I haven't seen anything like you described.

P.S. I use Deployment instead of DaemonSet.

[dcowden]

awesome! thanks for the help! i'll try that!

[dcowden]

SOLVED-- for documentation purposes, i've documenting what worked for me.

@Scukerman is right that a centralized deployment of a syslog is best. This prevents the need to restart the ingress pods when changing thigns, which is nice.

Sadly, I couldn't take advantage of that strategy because my ingress controllers use hostNetwork:true, since we are on cni and k8s < 1.9. a service providing in-cluster syslog isn't resolvable by DNS.

I used this sidecar strategy, which works well:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: ingress-lb
  namespace: ops
  labels:
    name: ingress-lb
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      hostNetwork: true
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
      - name: access-log
        image: jumanjiman/rsyslog
        ports:
        - name: udp
          containerPort: 514
          protocol: UDP
      - name: haproxy-ingress
        image: quay.io/jcmoraisjr/haproxy-ingress:v0.6-beta.1
        args:
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --allow-cross-namespace
        - --configmap=$(POD_NAMESPACE)/haproxy-ingress
        - --sort-backends
        - --v=3
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: stat
          containerPort: 1936
        readinessProbe:
          httpGet:
            path: /stats
            port: 1936
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /stats
            port: 1936
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace

Advice for newcomers, I stumbled on the combination of haproxy's log format and yaml escaping. Here's a working configmap, which does a custom log format escaped correctly:

apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-ingress
  namespace: ops
data:
  balance-algorithm: "leastconn"
  drain-support: "true"
  use-proxy-protcol: "true"
  proxy-body-size: "100m"
  ssl-redirect: "true"
  timeout-connect: "5s"
  timeout-http-request: "300s"
  timeout-queue: "300s"
  timeout-server: "300s"
  timeout-server-fin: "300s"
  syslog-endpoint: "localhost:514"
  http-log-format: "%ci:%cp->%b/%s\\ method=%HM\\ uri=%HU\\ rcvms=%TR\\ serverms=%Tr\\ activems=%Ta\\ bytes=%B\\ status=%ST"

Finally, and again for newcomers, here's the manifests to deploy a stand-alone syslog server for haproxy, in the case your ingress controllers are using hostPort instead of hostNetwork. We will move to this approach when we can use hostPort

apiVersion: v1
kind: Service
metadata:
  name: ingress-access-logs
  namespace: ops
  labels:
    k8s-app: ingress-access-logs
spec:
  ports:
  - port: 514
    targetPort: 514
    protocol: UDP
  selector:
    app: ingress-access-logs
---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ingress-access-logs
  namespace: ops
  labels:
    name: ingress-access-logs
spec:
  revisionHistoryLimit: 2
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-access-logs   
    spec:      
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
      - name: access-log
        image: jumanjiman/rsyslog
        ports:
        - name: udp
          containerPort: 514
          protocol: UDP

[Scukerman]

My congrats! But be careful, now all of your ports are outside cuz of hostNetwork: true

[dcowden]

Yep, unfortunately. The ideal situation will be to upgrade so we can do host Port.

Thanks again for your help

[Scukerman]

my ingress controllers use hostNetwork:true, since we are on cni and k8s < 1.9

I use regular deployments, service hostPort and CNI since k8s 1.7. The only problem is I have to use use-proxy-protocol: true to save the incoming IP-address of client.

So what the point in the upgrade? Didn't get it.

[dcowden]

Ah yes. I forget to mention one other constraint we have. Since we operate in the cloud, we have to terminate tls at the ingress controller ( so terminating tls at the upstream elb and then using nodeport doesn't work for us).

Now that I think about it, though, I guess we could still put a service in front of the ingress controller...

[Scukerman]

We're using lb as a service with proxy protocol and we terminate traffic at the ingress controller or pass through to the service.

The only one thing is different - we use bare-metal servers.
UPD: And the provider is OVH.

[dcowden]

Awesome, thanks for point this idea out. this has given me an idea to try!