Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TCPSession rebuild http session bug #4340

Closed
HasegawaAzusa opened this issue Mar 27, 2024 · 2 comments · Fixed by #4224
Closed

TCPSession rebuild http session bug #4340

HasegawaAzusa opened this issue Mar 27, 2024 · 2 comments · Fixed by #4224

Comments

@HasegawaAzusa
Copy link

Brief description

When I tried to analyze a traffic packet, I found that TCPSession was sometimes unable to reconstruct the TCP data.
I tried to analyze why this happened. Through testing, I learned that it seems that TCPSession did not consider Retransmission.
You can notice this Retransmission TCP package in Wireshark. Wireshark also reconstructed it very well, but for TCPSession it seems to treat it as an independent TCP package (or is there a problem with HTTP's tcp_reassemble?)
I am not very familiar with the scapy framework. I can only pinpoint the general problem here. Please improve the session reconstruction function as soon as possible.
PS: I tried the latest development version, but this problem has not been improved.

Scapy version

2.5.0

Python version

3.11

Operating system

Windows10

Additional environment information

No response

How to reproduce

from scapy.all import *
from scapy.layers import http

file = 'tcp.pcapng'
pcap = sniff(offline=file, session=TCPSession)
sessions = pcap.sessions()
for i in sessions.values():
    print(i.summary())

Actual result

Due to the influence of the Retransmission packet, there will be problems in parsing subsequent HTTP packets starting from that packet.

Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
None
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A / Padding
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A / Padding
None
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http PA / HTTP / Raw
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
None
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
None
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
None

Expected result

Request and Response in all sessions can be parsed normally.
The following is just my guess of the possible results

Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51204 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51204 A / Padding
None
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51213 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A / Padding
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51213 A / Padding
None
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http PA / HTTP / Raw
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51220 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51220 A / Padding
None
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51235 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51235 A / Padding
None
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http S
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http A
Ether / IP / TCP / HTTP / 'POST' '/r36XPcbygnfRjLAfFLbfD.php' 'HTTP/1.1' / Raw
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http A
Ether / IP / TCP 172.26.246.36:51243 > 192.168.239.202:http FA
None
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 SA
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A
Ether / IP / TCP / HTTP / 'HTTP/1.1' '200' 'OK' / Raw
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 FA / Padding
Ether / IP / TCP 192.168.239.202:http > 172.26.246.36:51243 A / Padding
None

Related resources

tcp.zip
@HasegawaAzusa
Copy link
Author

I forgot to upload a screenshot.
image
By simply inserting a print statement, it is observed that the Retransmission package is assigned to the next session.
image
I'm wondering if this is a problem with TCPSession or a problem with the TCP reconstruction algorithm?

@HasegawaAzusa
Copy link
Author

While researching how to improve this bug, I discovered a new bug :(
HTTP's tcp_reassemble does not seem to take into account the gzip-compressed Response of packet transmission (it seems that there is also a problem with Request, but I have not tested it).
The following is my guess as to why the bug occurred:
When calculating the length of the HTTP header, this line of code is used for convenience.
http_length = len(data) - len(http_packet.payload.payload)
The following is my guess as to why the bug occurred:
When calculating the length of the HTTP header, this line of code is used for convenience.
http_length = len(data) - len(http_packet.payload.payload)
But because of http_packet=HTTP(data), I don’t understand the operating principle of the scapy framework. I am shocked that the payload is not retained as the original bytes but is automatically decoded! (My here is Encoding: gzip, so the code decompresses the payload)
So the calculation of length is obtained directly from Content-Length, while the calculation of http_length is obtained by subtracting the decoded load from the original byte stream, which results in a length difference here, because this length difference causes the reconstruction in TCPSession Part of the encoded payload will be lost during the process. In my opinion, such a low-level error should not occur.
I tried to calculate the length by looking for \r\n\r\n which is the standard RFC HTTP header ending, i.e.
http_length = len(data[:idx+4]) solves this BUG and proves that the problem does lie in the length, but the solution is not general enough. I think the payload should not be automatically decoded when the HTTP package is created.
image
PS: Here is the code of the latest development version 2.5.0.dev311.

gpotter2 added a commit to gpotter2/scapy that referenced this issue Apr 27, 2024
@gpotter2
Improve TCPSession retransmission handling
94db327 
- fixes secdev#4197
- also fixes secdev#4340: handling of HTTP reconstruction when gzip is involved
@gpotter2 gpotter2 mentioned this issue Apr 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix TCP retransmissions gpotter2/scapy
1 participant
@HasegawaAzusa