Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Couldn't delete HttpChaos because request too large #4076

Open
dearboll23 opened this issue Jun 6, 2023 · 5 comments · May be fixed by #4402
Open

Couldn't delete HttpChaos because request too large #4076

dearboll23 opened this issue Jun 6, 2023 · 5 comments · May be fixed by #4402
Assignees
Labels
type/bug Something isn't working

Comments

@dearboll23
Copy link

dearboll23 commented Jun 6, 2023

Bug Report

What version of Kubernetes are you using?

Client Version: v1.24.2
Server Version: v1.24.11-gke.1000

What version of Chaos Mesh are you using?

ghcr.io/chaos-mesh/chaos-mesh:v2.5.1

What did you do? / Minimal Reproducible Example

I want to delete a HttpChaos CR, but the deletion is hanging over, and the chaos-controller's log shows the target HttpFault resource is very large, with an error message request too large, seems like an error message from etcd log. So I guess the issue is the size of HttpChaos is too large to delete it in etcd server. When I show the HttpChaos as yaml format, the status field is really very long recording all the history of itself.

What did you expect to see?

What did you see instead?

Output of chaosctl

@dearboll23
Copy link
Author

Update. I repaired the large HttpChaos resource and delete it by the following commands.

kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io chaos-mesh-validation-auth
kubectl patch --type=merge  httpchaos delaytest2 -p '{"status":null}’

But I think there should be a limitation for the event recorder to avoid the resource being too large.

@dearboll23 dearboll23 changed the title Couldn't delete HttpFault because request too large Couldn't delete HttpChaos because request too large Jun 7, 2023
@STRRL
Copy link
Member

STRRL commented Jun 13, 2023

Hi @dearboll23 , could you help us to reproduce this issue?

Could you share the yaml definition of the "large HTTPChaos" resource?

@STRRL STRRL self-assigned this Jun 13, 2023
@STRRL STRRL added the type/bug Something isn't working label Jun 13, 2023
@dearboll23
Copy link
Author

The resource is here. Seems like there's no limitation or rotation of the field status.experiment.containerRecords[].

@spencercjh
Copy link
Contributor

I try to reproduce and create the httpchaos locally and failed to delete. It is hung over and there isn't any log about delete in the controller-manager.

@spencercjh
Copy link
Contributor

I think the author of the issue has a good point in saying that the event in status cannot grow indefinitely. I can't even apply the yaml provided by kubectl apply -f with error metadata.annotations: Too long: must have at most 262144 characters.

@Gallardot Gallardot linked a pull request Apr 19, 2024 that will close this issue
10 tasks
@Gallardot Gallardot self-assigned this Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants