Bug: Infra still present after e2e pipeline finishes

[Despire]

some of the infra build by claudie in the e2e pipeline will not be deleted after the pipeline ends with failure, this should not be the case

The root cause needs to be identified, since this migh be a bug in how the infra is being tracked and therefore not deleted

[JKBGIT1]

I ran testing-framework 10 times. In 3 of them, I forgot to specify AUTO_CLEAN_UP, and in the spare 7, I experienced not deleting infra twice.

test-set2 - State lock on terraform destroy, test-set3 hit quota

The test-set2 finished with ERROR for ts2-c-2 and ts2-c-1. These test cases failed in terraformer, but unfortunately I have logs only from the builder.

builder logs
2024-04-17T12:21:00Z ERR ../go/services/builder/domain/usecases/terraformer_caller.go:36 > failed to build [ts2-lb-2-66u4u6r] cluster=ts2-c-2-3ce38dd module=builder project=claudie-test-set2
2024-04-17T12:21:00Z ERR ../go/services/builder/domain/usecases/config_processor.go:213 > Failed to build cluster error="error in Terraformer for cluster ts2-c-2 project claudie-test-set2 : failed to successfully build desired state" cluster=ts2-c-2 module=builder
2024-04-17T12:21:00Z INF ../go/services/builder/domain/usecases/terraformer_caller.go:32 > BuildInfrastructure on Terraformer finished successfully cluster=ts2-c-1-hxv4b2e module=builder project=claudie-test-set2
2024-04-17T12:21:00Z ERR ../go/services/builder/domain/usecases/terraformer_caller.go:36 > failed to build [ts2-lb-1-e1p9tm7] cluster=ts2-c-1-hxv4b2e module=builder project=claudie-test-set2
2024-04-17T12:21:00Z ERR ../go/services/builder/domain/usecases/config_processor.go:213 > Failed to build cluster error="error in Terraformer for cluster ts2-c-1 project claudie-test-set2 : failed to successfully build desired state" cluster=ts2-c-1 module=builder
2024-04-17T12:21:00Z ERR ../go/services/builder/domain/usecases/config_processor.go:251 > Error encountered while processing config error="error in Terraformer for cluster ts2-c-2 project claudie-test-set2 : failed to successfully build desired state" module=builder project=claudie-test-set2

Eventually, the testing-framework failed to delete some of the infrastructure belonging to these test cases, because terraform destroy failed due to the state locking.

terraformer logs

2024-04-17T12:23:35Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-lb-2-66u4u6r": failed to check existence of object claudie-test-set2/ts2-lb-2-66u4u6r: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C710AA2EAC528C, HostID: 4e8d2d37f42f3f7041a0ccf0d2b796d6965258dfd1964cf3039dc68c4d2b8043, NotFound:  module=terraformer
ts2-lb-2-66u4u6r    ╷
ts2-lb-2-66u4u6r    │ Error: Error acquiring the state lock
ts2-lb-2-66u4u6r    │
ts2-lb-2-66u4u6r    │ Error message: ConditionalCheckFailedException: The conditional request
ts2-lb-2-66u4u6r    │ failed
ts2-lb-2-66u4u6r    │ Lock Info:
ts2-lb-2-66u4u6r    │   ID:    	a38938fb-c812-0a49-ead2-6007ea416f09
ts2-lb-2-66u4u6r    │   Path:  	claudie-tf-state-files/claudie-test-set2/ts2-lb-2-66u4u6r
ts2-lb-2-66u4u6r    │   Operation: OperationTypeApply
ts2-lb-2-66u4u6r    │   Who:   	@terraformer-6dc989dd75-gf68l
ts2-lb-2-66u4u6r    │   Version:   1.4.4
ts2-lb-2-66u4u6r    │   Created:   2024-04-17 12:15:02.453067033 +0000 UTC
ts2-lb-2-66u4u6r    │   Info: 	 
ts2-lb-2-66u4u6r    │
ts2-lb-2-66u4u6r    │
ts2-lb-2-66u4u6r    │ Terraform acquires a state lock to protect the state from being written
ts2-lb-2-66u4u6r    │ by multiple users at the same time. Please resolve the issue above and try
ts2-lb-2-66u4u6r    │ again. For most commands, you can disable locking with the "-lock=false"
ts2-lb-2-66u4u6r    │ flag, but this is not recommended.

this error also appeared for ts2-lb-1-e1p9tm7, ts2-lb-2-66u4u6r, ts2-lb-1-e1p9tm7, ts2-c-2-3ce38dd, ts2-lb-2-66u4u6r, ts2-c-1-hxv4b2e.

In this run, some infra wasn't also deleted for test-set3. The test-set3 hit quotas in the genesis cloud. I probably forgot to delete the instances in some of the previous runs. I assume in the ones, where I forgot to specify AUTO_CLEAN_UP.

Another state lock in test-set2

Again, the testing-framework logged an error in the terraformer for test-set2 and failed to delete its infra due to the state lock.

terraformer logs

2024-04-19T11:18:23Z ERR ../go/services/terraformer/server/domain/usecases/build_infrastructure.go:52 > Error encountered while building cluster: error while creating the LB cluster ts2-lb-2 : error while running terraform apply in ts2-lb-2-jiwb57i : failed to execute cmd: terraform apply --auto-approve --parallelism=8: exit status 1 cluster=ts2-lb-2-jiwb57i module=terraformer project=claudie-test-set2
2024-04-19T11:35:52Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-lb-2-jiwb57i": failed to check existence of object claudie-test-set2/ts2-lb-2-jiwb57i: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB382945ADEB, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound:  module=terraformer
2024-04-19T11:37:40Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-lb-2-jiwb57i": failed to check existence of object claudie-test-set2/ts2-lb-2-jiwb57i: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B4AA84D7C, HostID: 82c15dd284042931463fddbb8dc95b4a7d1f899b3149cddbdae41293c967d286, NotFound:  module=terraformer
2024-04-19T11:37:40Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-c-1-fwpt1zm": failed to check existence of object claudie-test-set2/ts2-c-1-fwpt1zm: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B4AB1D1E9, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound:  module=terraformer
2024-04-19T11:37:40Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-c-2-3tgpdmo": failed to check existence of object claudie-test-set2/ts2-c-2-3tgpdmo: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B500FAE6C, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound:  module=terraformer
2024-04-19T11:37:40Z ERR ../go/services/terraformer/server/domain/usecases/destroy_infrastructure.go:93 > Error while destroying the infrastructure for project claudie-test-set2 : failed to check existence of state file for "ts2-c-1-fwpt1zm": failed to check existence of object claudie-test-set2/ts2-c-1-fwpt1zm: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B50105FBF, HostID: 82c15dd284042931463fddbb8dc95b4a7d1f899b3149cddbdae41293c967d286, NotFound:  module=terraformer

builder logs

2024-04-19T11:18:23Z ERR ../go/services/builder/domain/usecases/config_processor.go:213 > Failed to build cluster error="error in Terraformer for cluster ts2-c-2 project claudie-test-set2 : failed to successfully build desired state" cluster=ts2-c-2 module=builder
2024-04-19T11:33:21Z ERR ../go/services/builder/domain/usecases/config_processor.go:251 > Error encountered while processing config error="error in Terraformer for cluster ts2-c-2 project claudie-test-set2 : failed to successfully build desired state" module=builder project=claudie-test-set2
2024-04-19T11:37:40Z ERR ../go/services/builder/domain/usecases/workflow_helpers.go:203 > failed to destroy config error="error in destroy config Terraformer for config ts2-c-2 project claudie-test-set2 : error while destroying infrastructure for cluster ts2-c-2 project claudie-test-set2 : error while calling DestroyInfrastructure on Terraformer: rpc error: code = Unknown desc = error while destroying infrastructure for project claudie-test-set2 : failed to check existence of state file for \"ts2-lb-2-jiwb57i\": failed to check existence of object claudie-test-set2/ts2-lb-2jiwb57i: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB382945ADEB, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound: " module=builder project=claudie-test-set2
2024-04-19T11:37:40Z ERR ../go/services/builder/domain/usecases/workflow_helpers.go:203 > failed to destroy config error="error in destroy config Terraformer for config ts2-c-2 project claudie-test-set2 : error while destroying infrastructure for cluster ts2-c-2 project claudie-test-set2 : error while calling DestroyInfrastructure on Terraformer: rpc error: code = Unknown desc = error while destroying infrastructure for project claudie-test-set2 : failed to check existence of state file for \"ts2-lb-2-jiwb57i\": failed to check existence of object claudie-test-set2/ts2-lb-2-jiwb57i: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B4AA84D7C, HostID: 82c15dd284042931463fddbb8dc95b4a7d1f899b3149cddbdae41293c967d286, NotFound: " module=builder project=claudie-test-set2
2024-04-19T11:37:40Z ERR ../go/services/builder/domain/usecases/workflow_helpers.go:203 > failed to destroy config error="error in destroy config Terraformer for config ts2-c-2 project claudie-test-set2 : error while destroying infrastructure for cluster ts2-c-2 project claudie-test-set2 : error while calling DestroyInfrastructure on Terraformer: rpc error: code = Unknown desc = error while destroying infrastructure for project claudie-test-set2 : failed to check existence of state file for \"ts2-c-2-3tgpdmo\": failed to check existence of object claudie-test-set2/ts2c-2-3tgpdmo: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B500FAE6C, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound: " module=builder project=claudie-test-set2
2024-04-19T11:37:40Z ERR ../go/services/builder/domain/usecases/config_processor.go:55 > Error while destroying config  error="error in destroy config Terraformer for config ts2-c-2 project claudie-test-set2 : error while destroying infrastructure for cluster ts2-c-2 project claudie-test-set2 : error while calling DestroyInfrastructure on Terraformer: rpc error: code = Unknown desc = error while destroying infrastructure for project claudie-test-set2 : failed to check existence of state file for \"ts2-c-2-3tgpdmo\": failed to check existence of object claudie-test-set2/ts2-c-2-3tgpdmo: operation error S3: HeadObject, https response error StatusCode: 404, RequestID: 17C7AB6B500FAE6C, HostID: 6ad3c76aaa5b64d0dd430e4c58b993d21475b1467a88115932d2f992461a7565, NotFound: " module=builder project=claudie-test-set2