Missing failure messages and ID gone missing error

[luislezcair]

Description of Issue/Question

I was diagnosing an issue in an application and I came across this behavior, which I don't know if it is intended or not. I have a simple test with pre and post snapshots that uses no-diff to compare the output of a command, in this example is "show configuration".

Setup

I have a simple Python script to test this:

main.py

import os
from jnpr.jsnapy import SnapAdmin
from jnpr.junos import Device

CONFIG_FILE = os.path.join('tests', 'test.yml')
PRE_SNAP = os.path.join('pre', 'pre_test.xml')
POST_SNAP = os.path.join('post', 'post_test.xml')

TESTS = {'tests': [CONFIG_FILE]}


if __name__ == "__main__":
    device = Device(host='example.com', user='example', passwd='123456')
    snap = SnapAdmin()
    r = snap.check(TESTS, PRE_SNAP, POST_SNAP, device)

(note: the device is not used, I couldn't find a way to avoid passing one to the check method).

test.yml

show_configuration:
  - command: "show configuration"
  - item:
      xpath: //configuration-information
      id: configuration-output
      tests:
        - no-diff: configuration-output
          info: "ok"
          err: "configuration differs"

pre_test.xml

<configuration-information>
    <configuration-output>
        some config string
    </configuration-output>
</configuration-information

post_test.xml

<configuration-information>
    <configuration-output>
        some different config string
    </configuration-output>
</configuration-information>

What I get from running this script is first this output from the logs, which I don't really understand why those "ids" are missing, or what does it mean:

ID gone missing!!!
ID list '{'id_0': ['some config string']}' is not present in post snapshot
ID gone missing!!!
ID list '{'id_0': ['some different config string']}' is not present in pre snapshot

The other thing is the result of r[0].test_results:

{'show configuration': [{'count': {'fail': 2, 'pass': 0},
                         'failed': [{'id_missing_pre': {'configuration-output': 'some '
                                                                                'different '
                                                                                'config '
                                                                                'string'}},
                                    {'id_missing_post': {'configuration-output': 'some '
                                                                                 'config '
                                                                                 'string'}}],
                         'node_name': 'configuration-output',
                         'passed': [],
                         'result': False,
                         'test_name': 'show_configuration',
                         'testoperation': 'no-diff',
                         'xpath': '//configuration-information'}]}

It says I have two failed tests, when it's actually only one test (PRE and POST snapshots are different). Also those "missing ids" should have my error string which I specified in test.yml: "configuration differs".

I found this strange because when PRE and POST are the same, I get the correct message and the correct number of tests reported:

{'show configuration': [{'count': {'fail': 0, 'pass': 1},
                         'failed': [],
                         'node_name': 'configuration-output',
                         'passed': [{'id': {'configuration-output': 'some '
                                                                    'config '
                                                                    'string'},
                                     'message': 'ok',
                                     'post': {'configuration-output': ['some '
                                                                       'config '
                                                                       'string']},
                                     'post_node_value': ['some config string'],
                                     'pre': {'configuration-output': ['some '
                                                                      'config '
                                                                      'string']},
                                     'pre_node_value': ['some config string']}],
                         'result': True,
                         'test_name': 'show_configuration',
                         'testoperation': 'no-diff',
                         'xpath': '//configuration-information'}]}

It's a bit tedious to have these different formats for the output because I am trying to parse these results and have them displayed with some HTML along with the error message defined in the test.

Maybe I'm doing something wrong or using the library in an unintended way?

Any help will be appreciated. Thanks in advance.

Versions Report

I am using jsnapy 1.3.3 with Python 3.8.1

[luislezcair]

I was thinking it could be useful if Jsnapy could check if there's a difference between the two snapshots for one single element, defined by the Xpath but without an id. It should be useful for simple cases like this, with only one XML node to check and no ID.

[moophat]

This is essentially the same problem (regarding the way these comparators construct a key_unions element to compare two list) as I described in here
#377

But looking at the current state of this repo it seem like no one is willing to look into this. The resolution is actually not that difficult but there must be someone available to merge the fix if it's implemented. I'm trying to contact JTAC on this

[chidanandpujar]

issue is not reproducible in the jsnapy 1.3.7 release, we will close this issue as fixed.

~/jsnapy_test1/jsnapy# cat tests/test.yml 
show_configuration:
  - command: "show configuration"
  - item:
      xpath: //configuration-information
      id: configuration-output
      tests:
        - no-diff: configuration-output
          info: "ok"
          err: "configuration differs"

~/jsnapy_test1/jsnapy# cat pre/pre_test.xml 
<configuration-information>
    <configuration-output>
        some config string
    </configuration-output>
</configuration-information>

~/jsnapy_test1/jsnapy# cat post/post_test.xml 
<configuration-information>
    <configuration-output>
        some different config string
    </configuration-output>
</configuration-information>

~/jsnapy_test1/jsnapy# cat test_issue_364.py 
import os
from jnpr.jsnapy import SnapAdmin
from jnpr.junos import Device

CONFIG_FILE = os.path.join('tests', 'test.yml')
PRE_SNAP = os.path.join('pre', 'pre_test.xml')
POST_SNAP = os.path.join('post', 'post_test.xml')

TESTS = {'tests': [CONFIG_FILE]}


if __name__ == "__main__":
    device = Device(host='x.x.x.x', user='xyz', passwd='xyz')
    snap = SnapAdmin()
    r = snap.check(TESTS, PRE_SNAP, POST_SNAP, device)

python test_issue_364.py 
> /root/jsnapy_test1/venv/lib/python3.10/site-packages/jnpr/jsnapy/check.py(630)generate_test_files()
-> op = Operator()
(Pdb) c
**************************** Device: x.x.x.x ****************************
Tests Included: show_configuration 
************************ Command: show configuration ************************
> /root/jsnapy_test1/venv/lib/python3.10/site-packages/jnpr/jsnapy/check.py(411)compare_reply()
-> top_ignore_null = False
(Pdb) c
> /root/jsnapy_test1/venv/lib/python3.10/site-packages/jnpr/jsnapy/check.py(220)expression_evaluator()
-> testop = self._get_testop(elem_test)
(Pdb) c
> /root/jsnapy_test1/venv/lib/python3.10/site-packages/jnpr/jsnapy/operator.py(76)define_operator()
-> self.log_detail = logdetail
(Pdb) c
configuration differs
configuration differs
FAIL | All "configuration-output" is not same in pre and post snapshot [ 0 value matched / 2 value failed ]
------------------------------- Final Result!! -------------------------------
show_configuration : Failed
Total No of tests passed: 0
Total No of tests failed: 1 
Overall Tests failed!!! 

Thanks
Chidanand