Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can the desired target change during deployments? #174

Open
josselinbuils opened this issue Jul 15, 2021 · 4 comments 路 May be fixed by #176
Open

Can the desired target change during deployments? #174

josselinbuils opened this issue Jul 15, 2021 · 4 comments 路 May be fixed by #176

Comments

@josselinbuils
Copy link

Hello,

First thank you for this great tool, it is really helpful 馃榾

We have some timeout issues recently when using it to deploy our website.

After investigation, it seems to be because the "desired_target" changes during the deployment because our auto scaling mechanism but the tool does not seem to take it into account.

We could disable the auto scaling during our deployments but there is a risk for our infrastructure in case of traffic pics.

Is it a known bahavior? If yes, what would be the complexity to make the tool manage that?

Regards,
Josselin
@fabfuel
Copy link
Owner

fabfuel commented Jul 16, 2021

Hi Josselin,

thanks for sharing, it's always great to hear that the tool is helpful - highly appreciated!

Normally what you describe should (馃) not happen, as the ecs deploy command does not change the desired count configuration of the service.

Could you please share a few (simplified) examples of how you deploy (the commands you run) and describe how the "desired_target" (do you mean the "desired count" of the ECS service or another config option?) changes from before to after the deployment?

Thanks
Fabian

@josselinbuils
Copy link
Author

josselinbuils commented Jul 19, 2021 •
edited

Hello,

Here is the kind of command we launch:

ecs deploy ${ECS_CLUSTER_NAME} ${ECS_SERVICE_NAME} -i ${ECS_SERVICE_NAME} ${DOCKER_IMAGE}:${DOCKER_TAG} ${EXTRA_DOCKER_IMAGES} --region ${AWS_REGION} --profile assume-role-ecs --no-deregister --ignore-warnings --rollback --timeout ${ECS_DEPLOY_TIMEOUT} -e ${ECS_SERVICE_NAME} DD_VERSION ${DD_VERSION}"

We are deploying multiple versions of our website and one of them uses a lot of instances (~150).

We have issues with this one because our scaling mechanism seems to change the desired number of containers during the deployments and the tool does not seem to take it into account.

Or maybe there is another issue but we reach the desired number of containers in the AWS console but the ecs-deploy tool timeouts and then rollbacks.

@josselinbuils
Copy link
Author

It seems to be a pagination issue because we use more than 100 instances, one of our SREs will propose a PR to fix it 馃槃

@josselinbuils
Copy link
Author

By the way, is it wanted not to allow having more instances than necessary to deploy ?

Regarding this condition: https://github.com/fabfuel/ecs-deploy/blob/develop/ecs_deploy/ecs.py#L1182.

adrien-f added a commit to adrien-f/ecs-deploy that referenced this issue Jul 19, 2021
@adrien-f
Fix deployment rollbacking with more than 100 tasks
ca08306 
The `list_tasks` call was capped at 100 tasks, if your service had a desired
superior of 100 the deployment would fail.

Fix fabfuel#174

Signed-off-by: Adrien Fillon <adrien.fillon@manomano.com>
@adrien-f adrien-f linked a pull request Jul 19, 2021 that will close this issue
Open
adrien-f added a commit to adrien-f/ecs-deploy that referenced this issue Jul 19, 2021
@adrien-f
Fix deployment rollbacking with more than 100 tasks
6ed6557 
The `list_tasks` call was capped at 100 tasks, if your service had a desired
superior of 100 the deployment would fail.

Fix fabfuel#174

Signed-off-by: Adrien Fillon <adrien.fillon@manomano.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix deployment rollbacking with more than 100 tasks adrien-f/ecs-deploy
2 participants
@fabfuel @josselinbuils