I want to filter instances by matching a substring of a tag value

[DrStrangepork]

I'm about ripping my hair out over this. All of my instances have a tag named "Data", and the value of "Data" one of the following: "foo=yes,bar=yes", "foo=yes,bar=no", "foo=no,bar=yes", "foo=no,bar=no", I want to run a --query which returns the InstanceId for any instance that has a tag "Data" value that contains "foo=yes". So something like

aws --region us-east-1 ec2 describe-instances --query 'Reservations[].Instances[].[?contains(Tags[?Key==`Data`].Value, `foo=yes` == `true`)].InstanceId'

But that doesn't work.

[JordonPhillips]

That certainly is a tricky one! I managed it with the following expression:

"Reservations[].Instances[?Tags[?Key == 'Data' && contains(Value, 'foo=yes')][]][].InstanceId"

That's a long one, so I'm going to break it up, piece by piece. For starters, lets use this sample data:

{
    "Reservations": [{"Instances": [
        {
            "InstanceId": "id-target1",
            "Tags": [
                {"Key": "Data", "Value": "foo=yes,bar=no"},
                {"Key": "Name", "Value": "target"}
            ]
        },
        {
            "InstanceId": "id-target2",
            "Tags": [
                {"Key": "Data", "Value": "foo=yes,bar=yes"},
                {"Key": "Name", "Value": "target"}
            ]
        },
        {
            "InstanceId": "id-invalid1",
            "Tags": [
                {"Key": "Data", "Value": "foo=no,bar=no"},
                {"Key": "Name", "Value": "invalid"}
            ]
        },
        {
            "InstanceId": "id-invalid2",
            "Tags": [
                {"Key": "Data", "Value": "foo=no,bar=yes"},
                {"Key": "Name", "Value": "invalid"}
            ]
        }
    ]}]
}

First, we want to get the instances list, so we start with Reservations[].Instances[]:

[
  {
    "Tags": [
      {
        "Value": "foo=yes,bar=no",
        "Key": "Data"
      },
      {
        "Value": "target",
        "Key": "Name"
      }
    ],
    "InstanceId": "id-target1"
  },
  {
    "Tags": [
      {
        "Value": "foo=yes,bar=yes",
        "Key": "Data"
      },
      {
        "Value": "target",
        "Key": "Name"
      }
    ],
    "InstanceId": "id-target2"
  },
  {
    "Tags": [
      {
        "Value": "foo=no,bar=no",
        "Key": "Data"
      },
      {
        "Value": "invalid",
        "Key": "Name"
      }
    ],
    "InstanceId": "id-invalid1"
  },
  {
    "Tags": [
      {
        "Value": "foo=no,bar=yes",
        "Key": "Data"
      },
      {
        "Value": "invalid",
        "Key": "Name"
      }
    ],
    "InstanceId": "id-invalid2"
  }
]

Now we want the tags which have a Key with the value "Data": .Tags[?Key == 'Data'] for the full expression Reservations[].Instances[].Tags[?Key == 'Data'] and the result:

[
  [
    {
      "Value": "foo=yes,bar=no",
      "Key": "Data"
    }
  ],
  [
    {
      "Value": "foo=yes,bar=yes",
      "Key": "Data"
    }
  ],
  [
    {
      "Value": "foo=no,bar=no",
      "Key": "Data"
    }
  ],
  [
    {
      "Value": "foo=no,bar=yes",
      "Key": "Data"
    }
  ]
]

Now we also want the value of that tag to contain "foo=yes" so we add the condition && contains(Value, 'foo=yes') for the full expression Reservations[].Instances[].Tags[?Key == 'Data' && contains(Value, 'foo=yes')] and the result:

[
  [
    {
      "Value": "foo=yes,bar=no",
      "Key": "Data"
    }
  ],
  [
    {
      "Value": "foo=yes,bar=yes",
      "Key": "Data"
    }
  ],
  [],
  []
]

Alright, now lets flatten that list with [] for the full expression Reservations[].Instances[].Tags[?Key == 'Data' && contains(Value, 'foo=yes')][] and the result:

[
  {
    "Value": "foo=yes,bar=no",
    "Key": "Data"
  },
  {
    "Value": "foo=yes,bar=yes",
    "Key": "Data"
  }
]

Alright, so now we have the tags we want, but what we want the full instance data. So lets move that tag filter back as a filter on instances. So the full expression is now Reservations[].Instances[?Tags[?Key == 'Data' && contains(Value, 'foo=yes')][]] with the result:

[
  [
    {
      "Tags": [
        {
          "Value": "foo=yes,bar=no",
          "Key": "Data"
        },
        {
          "Value": "target",
          "Key": "Name"
        }
      ],
      "InstanceId": "id-target1"
    },
    {
      "Tags": [
        {
          "Value": "foo=yes,bar=yes",
          "Key": "Data"
        },
        {
          "Value": "target",
          "Key": "Name"
        }
      ],
      "InstanceId": "id-target2"
    }
  ]
]

To explain a bit more, ? is checking for a truthy value. For those instances that don't have the proper tag, ?Tags[?Key == 'Data' && contains(Value, 'foo=yes')][] will be return null, a falsy value.

Now all that's left to do is flatten that list and select the instance ids with [].InstanceId for the full expression "Reservations[].Instances[?Tags[?Key == 'Data' && contains(Value, 'foo=yes')][]][].InstanceId" and the result:

[
  "id-target1",
  "id-target2"
]

And we're done!

[DrStrangepork]

Wow, awesome! The Tags[?Key == 'Data' && contains(Value, 'foo=yes')] step is where I derailed. Thank you!

[cognitiaclaeves]

** update **

For my purposes, the following was a much easier approach (this is what I get from accessing a thread dated 2+ years ago):

--filter "Name=tag:aws:cloudformation:stack-name,Values=*$1*"
** original message **

This no longer appears to be working. It worked all the way up until the last step where [?Tags was substituted for .Tags .. when I run it for my use-case, I get an empty set (after successfully getting what I expect from
Reservations[].Instances[].Tags[?Key == 'aws:cloudformation:stack-name' && contains(Value, 'target')][]