[0.6.0] AWS Security group creation is unreliable

[mrwilby]

I fairly regularly (1 in 10 packs) see an error, which I believe is a timing (or 'eventual consistency') issue in the way AWS crates security groups. I am fairly certain that this is the root cause because I have seen similar issues with my own code - which also creates security groups - when using boto:

==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing SSH access on the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Error launching source instance: The security group 'sg-yyyy1234' does not exist in VPC 'vpc-xxxx1234' (InvalidGroup.NotFound)
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Error launching source instance: The security group 'sg-yyyy1234' does not exist in VPC 'vpc-xxxx1234' (InvalidGroup.NotFound)

Packer has successfully initiated the request to create the security group - but I believe packer is attempting to use the group ID before AWS has finished the group creation.

When I've seen this with my own python code + boto, the issue appears to be caused by AWS taking some small amount of time to actually complete the security group creation. Thus, when calling the EC2 "create security group" API, sometimes EC2 returns "creation complete" before the security group is actually available for use by subsequent packer commands.

I solved this in my code by trying to fetch the security group information in a loop until the first requests succeeds - at which point I can be sure that AWS has finished the security group creation and thus my code can continue to actually use it.

[mrwilby]

NB: I didn't research if the AWS "create security group" API which packer is using is actually meant to be asynchronous or synchronous - so this might also be an AWS EC2 bug (but I am somewhat doubtful)

[CpuID]

This seems to be a duplicate of #878 by the way

I just got it again in 0.6.0 myself also.

[mrwilby]

Agree this is a dupe. Thanks for linking!