GetInfoRefs() [E] signal: killed #30698
Comments
|
If I can provide any more information, feel free to let me know. I’ll be happy to grab it :) |
|
It seems to be a OOM, a large git repo? |
|
You mentioned NFS in chat. Do you have any latency reports from the app container to the file server? I'm wondering if the timeouts you are seeing could be slowness in reading the data (another way to test would be to increase the git timeouts significantly and then browsing) |
|
Yeah that was my fault, a little preemptive. It seems like rebooting my NFS server resolved the issue for a moment, but it came back. Thanks for the idea @techknowlogick, that actually triggered my memory - I've been seeing repository garbage collection timeouts. I had initially thought that it as just because I was running multiple replicas of Gitea in the Kubernetes deployment, but perhaps all 3x of my replicas were timing out and git garbage collection never truly completed. I changed the following values: I think that the above values (which were set to I'll have to dig into my metrics some more to see if there is something that is collects that data 🤔 |
|
Yeah it looks like the issue is still happening unfortunately. I do not see any retransmissions or errors on the NFS server, or from the client (using It's just the same errors as above, over and over again... Here are the updated deployment values that I'm using Updated values image:
repository: gitea/gitea
tag: 1.21.11
pullPolicy: Always
rootless: true
replicaCount: 3
deployment:
annotations:
reloader.stakater.com/auto: "true"
resources:
requests:
memory: 18G
limits:
memory: 18G
clusterDomain: newcluster.local
podSecurityContext:
fsGroupChangePolicy: "OnRootMismatch"
containerSecurityContext:
capabilities:
add: ["SYS_CHROOT"]
redis-cluster:
enabled: false
postgresql:
enabled: false
postgresql-ha:
enabled: false
persistence:
enabled: true
mount: true
create: false
accessModes:
- ReadWriteMany
claimName: "gitea-main-data"
gitea:
additionalConfigSources:
- secret:
secretName: gitea-config-secret
admin:
existingSecret: gitea-admin-secret
config:
APP_NAME: QSBuYW1lIGNvb2wK Gitea
RUN_MODE: dev
indexer:
ISSUE_INDEXER_TYPE: meilisearch
REPO_INDEXER_ENABLED: false
REPO_INDEXER_TYPE: elasticsearch
REPO_INDEXER_REPO_TYPES: sources,mirrors,templates
MAX_FILE_SIZE: 52428800
repository:
ENABLE_PUSH_CREATE_USER: true
ENABLE_PUSH_CREATE_ORG: true
DEFAULT_BRANCH: master
DEFAULT_PRIVATE: true
repository.upload:
FILE_MAX_SIZE: 102400
MAX_FILES: 300
server:
DISABLE_SSH: true
DOMAIN: gitea.something.com
ROOT_URL: https://gitea.something.com
ENABLE_GZIP: true
LFS_START_SERVER: true
database:
DB_TYPE: postgres
HOST: cluster16-rw.databases.svc.newcluster.local
session:
PROVIDER: db
ui:
THEMES: gitea,arc-green,github-dark,codeberg,anamecool
DEFAULT_THEME: arc-green
attachment:
MAX_SIZE: 102400
MAX_FILES: 300
git.timeout:
GC: 500
actions:
ENABLED: true
service:
REGISTER_EMAIL_CONFIRM: false
REGISTER_MANUAL_CONFIRM: true
DISABLE_REGISTRATION: false
REQUIRE_SIGNIN_VIEW: true
ENABLE_NOTIFY_MAIL: true
ALLOW_ONLY_EXTERNAL_REGISTRATION: false
ENABLE_CAPTCHA: false
DEFAULT_KEEP_EMAIL_PRIVATE: true
DEFAULT_ALLOW_CREATE_ORGANIZATION: true
DEFAULT_ENABLE_TIMETRACKING: true
NO_REPLY_ADDRESS: "noreply-something@gmail.com"
cache:
ENABLED: true
ADAPTER: redis
HOST: redis://:gitea@gitea-redis.gitea.svc.newcluster.local:6379/0?pool_size=100&idle_timeout=180s&
queue:
TYPE: redis
CONN_STR: redis://:gitea@gitea-redis.gitea.svc.newcluster.local:6379/0?pool_size=100&idle_timeout=180s&
cron:
ENABLED: true
TIMEOUT: 360s
cron.update_mirrors:
SCHEDULE: 0 */12 * * *
cron.resync_all_hooks:
ENABLED: true
cron.repo_health_check:
TIMEOUT: 360s
cron.reinit_missing_repos:
ENABLED: true
security:
LOGIN_REMEMBER_DAYS: 180
mailer:
ENABLED: true
webhook:
DELIVER_TIMEOUT: 45s
metrics:
enabled: true |
and here's the gist to go along with those errors above |
|
It really does seem to have something to do with my NFS latency / throughput, I'm testing different things out now... |
|
Yep, this was all because of my 1gig switch being absolutely slammed, and NFS couldn't get through quick enough. |
Description
Hey all,
Lunny told me to create a GitHub issue for this. My Gitea really has become very sluggish, doesn't always respond, and has its logs filled with
context deadline exceeded:I'm running the Gitea Helm Chart on my vanilla Kubernetes Cluster.
Kubernetes YAML Deployment
I've just run my Gitea in
devmode, hopefully I can also provide additional logs forgitoutput.Gitea Version
1.21.11
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
https://gist.github.com/perfectra1n/efb5cfa3752c34d382f0ad4291784d7a
Screenshots
Another log gist:
https://gist.github.com/perfectra1n/9146f98c8d99e87fa97916b8af27dd14
Git Version
No response
Operating System
No response
How are you running Gitea?
I'm running the Gitea Helm Chart on my vanilla Kubernetes Cluster. It it using PostgreSQL (CNPG), and is using democratic-csi to provide storage for the PVC that Gitea is utilizing for data.
Database
PostgreSQL
The text was updated successfully, but these errors were encountered: