Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New release to support python 3.13 #95

Open
marcindulak opened this issue Oct 24, 2023 · 6 comments
Open

New release to support python 3.13 #95

marcindulak opened this issue Oct 24, 2023 · 6 comments

Comments

@marcindulak
Copy link
Contributor

Originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=2245828

Example error from pytest:

______________________ AggregatorTestCase.test_aggregator ______________________

self = <test_unified2.AggregatorTestCase testMethod=test_aggregator>

    def test_aggregator(self):
    
>       aggregator = unified2.Aggregator()

tests/test_unified2.py:201: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <idstools.unified2.Aggregator object at 0x7f3c78cff500>

    def __init__(self):
>       LOG.warn("idstools.unified2.Aggregator has been deprecated")
E       AttributeError: 'Logger' object has no attribute 'warn'

../../BUILDROOT/python-idstools-0.6.4-11.fc40.x86_64/usr/lib/python3.13/site-packages/idstools/unified2.py:433: AttributeError

Python 3.13 removes logging.warn https://docs.python.org/3.13/whatsnew/3.13.html#logging

This change 6a74ca1 implemented the fix, but it has not been released yet.

Additionally, it may be also helpful to review the SyntaxError: invalid escape sequence warnings. I think those are related to python/cpython#77093. They are not treated as errors yet, but may become one day.

Here is a Dockerfile to trigger them by running pytest: docker build --progress=plain -t py-idstools:latest .

FROM python:3.13-rc-bookworm

RUN set -x \
    && apt-get update \
    && apt-get install -y git

RUN set -x \
    && pip install pytest

RUN set -x \
    && git clone https://github.com/jasonish/py-idstools/ \
    && cd py-idstools \
    && PYTHONPATH=. pytest -v

CMD ["/bin/bash"]

Output

#4 3.499 + PYTHONPATH=. pytest -v
#4 3.880 ============================= test session starts ==============================
#4 3.880 platform linux -- Python 3.13.0a1, pytest-7.4.2, pluggy-1.3.0 -- /usr/local/bin/python
...
#4 8.607 =============================== warnings summary ===============================
#4 8.607 tests/test_rule.py:40
#4 8.607   /py-idstools/tests/test_rule.py:40: SyntaxWarning: invalid escape sequence '\/'
#4 8.607     rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#4 8.607 
#4 8.607 tests/test_rule.py:152
#4 8.607   /py-idstools/tests/test_rule.py:152: SyntaxWarning: invalid escape sequence '\/'
#4 8.607     rule_string = """alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)"""
#4 8.607 
#4 8.607 tests/test_rule.py:172
#4 8.607   /py-idstools/tests/test_rule.py:172: SyntaxWarning: invalid escape sequence '\;'
#4 8.607     rule_string = u"""alert ip any any -> any any (msg:"TEST RULE\; and some"; content:"uid=0|28|root|29|"; tag:session,5,packets; classtype:bad-unknown; sid:10000000; rev:1;)"""
#4 8.607 
#4 8.607 tests/test_rule.py:175
#4 8.607   /py-idstools/tests/test_rule.py:175: SyntaxWarning: invalid escape sequence '\;'
#4 8.607     self.assertEqual(rule.msg, "TEST RULE\; and some")
#4 8.607 
#4 8.607 tests/test_rulecat.py:173
#4 8.607   /py-idstools/tests/test_rulecat.py:173: SyntaxWarning: invalid escape sequence '\|'
#4 8.607     line = '2020757 "\|0d 0a\|" "|ff ff|"'
#4 8.607 
#4 8.607 tests/test_rulecat.py:184
#4 8.607   /py-idstools/tests/test_rulecat.py:184: SyntaxWarning: invalid escape sequence '\|'
#4 8.607     line = 're:classtype:trojan-activity "\|0d 0a\|" "|ff ff|"'
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:225
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:225: SyntaxWarning: invalid escape sequence '\$'
#4 8.607     b = re.sub("\$\{(\d+)\}", "\\\\\\1", b)
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:247
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:247: SyntaxWarning: invalid escape sequence '\w'
#4 8.607     drop_rule = idstools.rule.parse(re.sub("^\w+", "drop", rule.raw))
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:595
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:595: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:\"(.*)\")"),
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:596
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:596: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:(.*?)),.*"),
#4 8.607 
#4 8.607 idstools/scripts/rulecat.py:597
#4 8.607   /py-idstools/idstools/scripts/rulecat.py:597: SyntaxWarning: invalid escape sequence '\s'
#4 8.607     re.compile("\s+(re:(.*))"),
#4 8.607 
#4 8.607 idstools/suricata.py:51
#4 8.607   /py-idstools/idstools/suricata.py:51: SyntaxWarning: invalid escape sequence '\d'
#4 8.607     m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 8.607 
#4 8.607 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
#4 8.607 ================== 76 passed, 2 skipped, 12 warnings in 4.73s ==================
@jasonish
Copy link
Owner

Staging this for release: #96

@jasonish
Copy link
Owner

jasonish commented Nov 1, 2023

0.6.5 released.

@jasonish jasonish closed this as completed Nov 1, 2023
@marcindulak
Copy link
Contributor Author

Have you tested with the Dockerfile build docker build --progress=plain --no-cache --pull -t py-idstools:latest .

FROM python:3.13-rc-bookworm

RUN set -x \
    && apt-get update \
    && apt-get install -y git

RUN set -x \
    && pip install pytest

RUN set -x \
    && git clone https://github.com/jasonish/py-idstools/ \
    && cd py-idstools \
    && git checkout 51aebc019b5f3132aa6d63b5dc3c17b03a2f5c8c \
    && PYTHONPATH=. pytest -Werror -v

CMD ["/bin/bash"]

I'm still gettting some "invalid escape sequence" with 51aebc0

#4 2.751 HEAD is now at 51aebc0 version: 0.6.5
#4 2.752 + PYTHONPATH=. pytest -Werror -v
#4 3.185 ============================= test session starts ==============================
#4 3.185 platform linux -- Python 3.13.0a1, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
#4 3.185 cachedir: .pytest_cache
#4 3.185 rootdir: /py-idstools
#4 3.185 collecting ... collected 33 items / 3 errors
#4 3.989 
#4 3.989 ==================================== ERRORS ====================================
#4 3.989 _____________________ ERROR collecting tests/test_rule.py ______________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:177: in exec_module
#4 3.989     source_stat, co = _rewrite_test(fn, self.config)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:359: in _rewrite_test
#4 3.989     tree = ast.parse(source, filename=strfn)
#4 3.989 /usr/local/lib/python3.13/ast.py:54: in parse
#4 3.989     return compile(source, filename, mode, flags,
#4 3.989 E     File "/py-idstools/tests/test_rule.py", line 40
#4 3.989 E       rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#4 3.989 E                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\/'
#4 3.989 ____________________ ERROR collecting tests/test_rulecat.py ____________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
#4 3.989     exec(co, module.__dict__)
#4 3.989 tests/test_rulecat.py:37: in <module>
#4 3.989     from idstools.scripts import rulecat
#4 3.989 idstools/scripts/rulecat.py:48: in <module>
#4 3.989     import idstools.suricata
#4 3.989 E     File "/py-idstools/idstools/suricata.py", line 51
#4 3.989 E       m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 3.989 E                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\d'
#4 3.989 ___________________ ERROR collecting tests/test_suricata.py ____________________
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/python.py:617: in _importtestmodule
#4 3.989     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/pathlib.py:567: in import_path
#4 3.989     importlib.import_module(module_name)
#4 3.989 /usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
#4 3.989     return _bootstrap._gcd_import(name[level:], package, level)
#4 3.989 <frozen importlib._bootstrap>:1381: in _gcd_import
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1354: in _find_and_load
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:1325: in _find_and_load_unlocked
#4 3.989     ???
#4 3.989 <frozen importlib._bootstrap>:929: in _load_unlocked
#4 3.989     ???
#4 3.989 /usr/local/lib/python3.13/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
#4 3.989     exec(co, module.__dict__)
#4 3.989 tests/test_suricata.py:36: in <module>
#4 3.989     from idstools import suricata
#4 3.989 E     File "/py-idstools/idstools/suricata.py", line 51
#4 3.989 E       m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#4 3.989 E                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#4 3.989 E   SyntaxError: invalid escape sequence '\d'
#4 3.989 =========================== short test summary info ============================
#4 3.989 ERROR tests/test_rule.py
#4 3.989 ERROR tests/test_rulecat.py
#4 3.989 ERROR tests/test_suricata.py
#4 3.989 !!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
#4 3.989 ============================== 3 errors in 0.80s ===============================

When running pytest without -Werror tests succeed, with some SyntaxWarning: invalid escape sequence

#7 2.873 + PYTHONPATH=. pytest -v
#7 3.240 ============================= test session starts ==============================
#7 3.240 platform linux -- Python 3.13.0a1, pytest-7.4.3, pluggy-1.3.0 -- /usr/local/bin/python
#7 3.240 cachedir: .pytest_cache
#7 3.240 rootdir: /py-idstools
#7 3.240 collecting ... collected 78 items
...
#7 8.365 =============================== warnings summary ===============================
#7 8.365 tests/test_rule.py:40
#7 8.365   /py-idstools/tests/test_rule.py:40: SyntaxWarning: invalid escape sequence '\/'
#7 8.365     rule = idstools.rule.parse("""alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)""")
#7 8.365 
#7 8.365 tests/test_rule.py:152
#7 8.365   /py-idstools/tests/test_rule.py:152: SyntaxWarning: invalid escape sequence '\/'
#7 8.365     rule_string = """alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET CURRENT_EVENTS Request to .in FakeAV Campaign June 19 2012 exe or zip"; flow:established,to_server; content:"setup."; fast_pattern:only; http_uri; content:".in|0d 0a|"; flowbits:isset,somebit; flowbits:unset,otherbit; http_header; pcre:"/\/[a-f0-9]{16}\/([a-z0-9]{1,3}\/)?setup\.(exe|zip)$/U"; pcre:"/^Host\x3a\s.+\.in\r?$/Hmi"; metadata:stage,hostile_download; reference:url,isc.sans.edu/diary/+Vulnerabilityqueerprocessbrittleness/13501; classtype:trojan-activity; sid:2014929; rev:1;)"""
#7 8.365 
#7 8.365 tests/test_rule.py:172
#7 8.365   /py-idstools/tests/test_rule.py:172: SyntaxWarning: invalid escape sequence '\;'
#7 8.365     rule_string = u"""alert ip any any -> any any (msg:"TEST RULE\; and some"; content:"uid=0|28|root|29|"; tag:session,5,packets; classtype:bad-unknown; sid:10000000; rev:1;)"""
#7 8.365 
#7 8.365 tests/test_rule.py:175
#7 8.365   /py-idstools/tests/test_rule.py:175: SyntaxWarning: invalid escape sequence '\;'
#7 8.365     self.assertEqual(rule.msg, "TEST RULE\; and some")
#7 8.365 
#7 8.365 idstools/suricata.py:51
#7 8.365   /py-idstools/idstools/suricata.py:51: SyntaxWarning: invalid escape sequence '\d'
#7 8.365     m = re.search("((\d+)\.(\d+)(\.(\d+))?(\w+)?)", str(buf).strip())
#7 8.365 
#7 8.365 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
#7 8.365 ================== 76 passed, 2 skipped, 5 warnings in 5.13s ===================

@jasonish
Copy link
Owner

jasonish commented Nov 2, 2023

Ah, missed actually testing with 3.13 as I typically don't bother with non-released versions, as I'm very much not in the Python world anymore. Looks like easy fixes I can get to soon.

@marcindulak
Copy link
Contributor Author

OK, the good thing is there is no need for new release, as SyntaxWarning: invalid escape sequence in Python 3.13 are still warnings and not errors.

@jasonish
Copy link
Owner

jasonish commented Nov 3, 2023

Will re-open and fix in the near future. Thanks.

@jasonish jasonish reopened this Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jasonish @marcindulak