Bug: list(map) terraform output test doesn't work with regular expression

[luckeyca]

When the terraform output is list(map), the regular expressions in the tests don't work. When the terraform output is a simple map, regular expression works.

working example:

terraform output

       spoke_subnet_route_table_associations = {
         "spokeapp1subnet" = "route_table_id1"
         "spokeapp2subnet" = "route_table_id2"
         "spokedbsubnet" = "route_table_id3"
       }

test case:

  describe 'terraform output spoke_subnet_route_table_associations' do
    subject do
      attribute('output_spoke_subnet_route_table_associations')
    end

    it {
      is_expected.to include(match(/spoke(app1|app2|db)subnet/) => \  
      match(/^router_table_id*/))
    }
  end

NOT working example:

terraform output:

       dmz_all_subnet_flowlogs = [
         {
           "display_name" = "dmz-app-subnet-flowlog"
           "flowlog_id" = "id1324"
           "retention_duration" = 90
         },
         {
           "display_name" = "dmz-firewall-subnet-flowlog"
           "flowlog_id" = "id2342"
           "retention_duration" = 90
         },
         {
           "display_name" = "dmz-proxy-subnet-flowlog"
           "flowlog_id" = "id3434"
           "retention_duration" = 90
         },
       ]

Test Case Format 1:

  describe 'terraform output dmz_all_subnet_flowlogs' do
    subject do
      attribute('output_dmz_all_subnet_flowlogs')
    end

    it {
      is_expected.to include([{
           'display_name' => match(/dmz-(app|firewall|proxy)-subnet-flowlog/)
           'flowlog_id' => match(/^id+*$/)
           'retention_duration' => 90

     }])
    }
  end

Test Case format 2:

  describe 'terraform output spoke_all_subnet_flowlogs' do
    subject do
      attribute('output_spoke_all_subnet_flowlogs')
    end

    it {
      is_expected.to include(match(/^display_name$/) => \
      match(/dmz-(app1|app2|db)-subnet-flowlog/))
      is_expected.to include(match(/^flowlog_id$/) => \
     match(/^id+*$/)
      is_expected.to include(match(/^retention_duration$/) => \
      eq(90))
    }
  end

[luckeyca]

Hi @aaron-lane, after looking into the test results, I think there might be another bug/limitation with kitchen terraform output test. Does it support list and/or list(map/object) as the output results?

with the example above,

the actual terraform output is:

       dmz_all_subnet_flowlogs = [
         {
           "display_name" = "dmz-app-subnet-flowlog"
           "flowlog_id" = "id1324"
           "retention_duration" = 90
         },
         {
           "display_name" = "dmz-firewall-subnet-flowlog"
           "flowlog_id" = "id2342"
           "retention_duration" = 90
         },
         {
           "display_name" = "dmz-proxy-subnet-flowlog"
           "flowlog_id" = "id3434"
           "retention_duration" = 90
         },
       ]

However, in the test results, the 2nd "expected" line below only shows the very first map object in the list which should be the full list of all 3 map objects. Could you please advise if it's the case and if I need to open another bug or feature request ticket? Thanks.

     ×  terraform output spoke_all_subnet_flowlogs is expected to include hash_including("display_name" => "match /dmz-(app1|app2|db)-subnet-flowlog/", "flowlog_id" => "match /^id*/", "retention_duration" => 90)
     expected [{:display_name => "spoke-app1-subnet-flowlog", :flowlog_ocid => "id1234", :retention_duration => 90}]  to include {"display_name" => (match /dmz-(proxy|fw|app)-subnet-flowlog/), "flowlog_ocid" => (match /^id*/), "retention_duration" => 90}

[aaron-lane]

Hi @luckeyca 👋🏼 Thank you for the bug report!

I would expect for the "NOT working example" that your test case would look like this instead:

describe 'terraform output dmz_all_subnet_flowlogs' do
    subject do
      attribute('output_dmz_all_subnet_flowlogs')
    end

    it {
      is_expected.to include(including(
           'display_name' => match(/dmz-(app|firewall|proxy)-subnet-flowlog/)
           'flowlog_id' => match(/^id+*$/)
           'retention_duration' => 90
     ))
    }
  end

Note that the outer include refers to the list of dmz_all_subnet_flowlogs and the inner including refers to one of the map entries in that list. Does that work for you? Also, I don't recommend using match in this way because that will only verify that some entry matches the regular expression, rather than confirming that there is a match for each of app, firewall, and proxy. If you want to verify that all entries exist in the output then I believe that you will need to explicitly write an assertion for each one.

However, in the test results, the 2nd "expected" line below only shows the very first map object in the list which should be the full list of all 3 map objects. Could you please advise if it's the case and if I need to open another bug or feature request ticket?

I will look at adding some additional test coverage to verify if map and list(map) outputs are correctly passed as InSpec attributes.

[luckeyca]

Hi @aaron-lane, I did try the way you suggested. It's not working which was why I asked if the kitchen terraform output supports list(object), map and/or list type terraform output because even with your latest example, there is still no looping.
I 100% agree with that the test case I wrote before has the partial testing issue given there is no obvious looping somewhere.

The workaround, in the meantime, I come up with(still testing, tenant has some other unrelated issues now), are to use native rspec syntax which can do loop easily:

  describe 'terraform output dmz_all_subnet_flowlogs' do
    before(:all) do
      @tf_output = JSON[`terraform \
                        -chdir=test/fixtures/terraform-module-network-dmz \
                        output -json`]
    end

    it 'should have the correct display_name, flowlog_id and retention_duration settings' do
      @tf_output['dmz_all_subnet_flowlogs'] do |flowlog|
        expect(flowlog.each['display_name']).to match(/dmz-(proxy|fw|app)-subnet-flowlog/)
        expect(flowlog.each['flowlog_ocid']).to match(/^id+*$/)
        expect(flowlog.each['retention_duration']).to eq(90)
      end
    end
  end

  describe 'terraform output spoke_all_subnet_flowlogs' do
    before(:all) do
      @tf_output = JSON[`terraform \
                        -chdir=test/fixtures/terraform-module-network-dmz \
                        output -json`]
    end

    it 'should have the correct display_name, flowlog_id and retention_duration settings' do
      @tf_output['dmz_all_subnet_flowlogs'] do |flowlog|
        expect(flowlog.each).to include(
          'display_name' => match(/dmz-(proxy|fw|app)-subnet-flowlog/),
          'flowlog_ocid' => match(/^id+*$/),
          'retention_duration' => 90
        )
      end
    end
  end

[luckeyca]

Hi @aaron-lane, just to let you know I believe map is supported because I have been using it without any issue, just need to confirm if list and list(object) type terraform outputs are supported by kitchen terraform output test. Thanks

> I will look at adding some additional test coverage to verify if `map` and `list(map)` outputs are correctly passed as InSpec attributes.

[aaron-lane]

I've updated the output tests on this branch: https://github.com/newcontext-oss/kitchen-terraform/compare/aaron-lane/output-tests

It appears that support for outputs that are lists of objects (maps) is intact 🕵🏼

[luckeyca]

Hi @aaron-lane Thanks. the changes look good. Which release were they put in as I couldn't find any related changes mentioned in release notes?

[aaron-lane]

Sorry, to clarify, that branch only contains test updates which verify that Kitchen-Terraform does correctly parse list-of-objects Terraform outputs. I will add a specific check around the regular expression logic.