Promtail 2.2.1 - oom issue

[krimeshshah]

Describe the bug
Since 2 weeks, our loki-promtail pods on some nodes keep on restarting. While going through promtail document, i figured out that we could define max_streams parameter to limit the log streams to be porcessed under limits_config block. But i don find any concret document on how do we determine the value based on our resources available and configured?

To Reproduce
Steps to reproduce the behavior:

1. Started Loki (SHA or version)
2. Started Promtail (SHA or version) to tail '2.2.1'
3. Query: {} term

Expected behavior
A clear and concise description of what you expected to happen.Promtail pods should not keep restarting.
Environment:

• Infrastructure: kubernetes
• Deployment tool: helm + argocd

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.
app: promtail
chart: promtail-2.2.0
heritage: Helm
release: loki
name: loki-promtail
namespace: logging
resourceVersion: "1338712901"
uid: 44cc0fe5-197c-43b1-9f78-733d4042e50f
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: promtail
release: loki
template:
metadata:
annotations:
checksum/config: 186de19a9c684a3e7ce0ee51e507e7552305a90470e6278fe28a83ab2d0b7803
kubectl.kubernetes.io/restartedAt: "2021-07-27T16:27:57Z"
prometheus.io/port: http-metrics
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app: promtail
release: loki
spec:
affinity: {}
containers:
- args:
- -config.file=/etc/promtail/promtail.yaml
- -client.url=http://loki:3100/loki/api/v1/push
env:
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: remote-docker.common.repositories.cloud.sap/grafana/promtail:2.2.1
imagePullPolicy: IfNotPresent
name: promtail
ports:
- containerPort: 3101
name: http-metrics
protocol: TCP
readinessProbe:
failureThreshold: 5
httpGet:
path: /ready
port: http-metrics
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 256Mi
requests:
cpu: 100m
memory: 64Mi
securityContext:
readOnlyRootFilesystem: true
runAsGroup: 0
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/promtail
name: config
- mountPath: /run/promtail
name: run
- mountPath: /var/lib/docker/containers
name: docker
readOnly: true
- mountPath: /var/log/pods
name: pods
readOnly: true
- mountPath: /var/log/journal
name: journal
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: artifactory
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: loki-promtail
serviceAccountName: loki-promtail
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: nvidia.com/gpu
operator: Exists
volumes:
- configMap:
defaultMode: 420
name: loki-promtail
name: config
- hostPath:
path: /run/promtail
type: ""
name: run
- hostPath:
path: /var/lib/docker/containers
type: ""
name: docker
- hostPath:
path: /var/log/pods
type: ""
name: pods
- hostPath:
path: /var/log/journal
type: ""
name: journal
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 11
desiredNumberScheduled: 11
numberAvailable: 11
numberMisscheduled: 0
numberReady: 11
observedGeneration: 40
updatedNumberScheduled: 11