Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

False negative if 'schema' args reversed in stateful check #60

Open
trstephen-amazon opened this issue Apr 30, 2024 · 0 comments
Open
Labels
good first issue Good for newcomers

Comments

@trstephen-amazon
Copy link

trstephen-amazon commented Apr 30, 2024

The --stateful check assumes the --schema arguments are ordered previous then current versions. If the order is flipped then the check will succeed when it should fail.

Reproduction

  1. Download a schema
    aws cloudformation describe-type \
         --type RESOURCE \
         --type-name AWS::QLDB::Stream \
         | jq -r '.Schema | fromjson' > qldb-stream-schema.json
    
  2. Copy and make a breaking change. For example, change .definitions[].Tag.properties.Value.maxLength 255 -> 2
  3. Run a stateful resource check with original/modified (should fail ENSURE_MAXLENGTH_NOT_CONTRACTED)
     guard-rail \
         --schema file://path-to-file/qldb-stream-schema.json \
         --schema file://path-to-file/modified-qldb-stream-schema.json \
         --stateful --format
    
  4. Run a stateful resource check with modified/original
     guard-rail \
         --schema file://path-to-file/modified-qldb-stream-schema.json \
         --schema file://path-to-file/qldb-stream-schema.json \
         --stateful --format
    

What happens: Passing the schema in modified/original order passes despite the breaking change

➜ guard-rail \
    --schema file://path-to-file/modified-qldb-stream-schema.json \
    --schema file://path-to-file/qldb-stream-schema.json \
    --stateful --format
{"maxLength": {"changed": [{"property": "/properties/Tags/*/Value", "old_value": 2, "new_value": 255}]}}

---------
[SKIPPED]:

ENSURE_OLD_PROPERTY_NOT_TURNED_IMMUTABLE
ENSURE_PROPERTY_TYPE_NOT_CHANGED
ENSURE_MAXIMUM_NOT_CONTRACTED
ENSURE_PROPERTY_STRING_PATTERN_NOT_CHANGED
ENSURE_PRIMARY_IDENTIFIER_NOT_CHANGED
ENSURE_OLD_PROPERTY_NOT_REMOVED_FROM_READONLY
ENSURE_NO_MORE_REQUIRED_PROPERTIES
ENSURE_MAXITEMS_NOT_CONTRACTED
ENSURE_MINLENGTH_NOT_CONTRACTED
ENSURE_ENUM_NOT_CHANGED
ENSURE_MINITEMS_NOT_CONTRACTED
ENSURE_MINIMUM_NOT_CONTRACTED
ENSURE_OLD_PROPERTY_NOT_TURNED_WRITEONLY
ENSURE_OLD_PROPERTY_NOT_REMOVED

[PASSED]:

ENSURE_MAXLENGTH_NOT_CONTRACTED

[WARNING]:

What should happen: Check output should be the same (failing) regardless of order of args


I think you may have to scrap the --schema flags and go with args that don't consider order e.g. --current-schema, --original-schema

@ammokhov ammokhov added the good first issue Good for newcomers label Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants