in v5.5.4r some url coming broken - with Transparent HTTPS #810
Comments
|
@sezer Odd looks like the host name field is being corrupted. Can you post a full extract of thread id hw573: so that I can look at more trace information. What platform are you using?
…----- On 30 Apr, 2024, at 11:29, Sezer ***@***.***> wrote:
hello [ https://github.com/philipianpearce | @philipianpearce ] ,
in v5.5.4r some url coming broken like bellow when you going eg instagram, when
you open chrome and search instagram, chrome first of all sending request this
url (content-autofill.googleapis.co) actually
(content-autofill.googleapis.com). if e2guardian cant reach dns address, will
be getting error page like proxy refused from google-chrome.
when , dns req, e2guardian log i examine : e2guardian sending dns request broken
host address.
can you check bug or configuration mistake?
thanks in advance.
[
https://github.com/e2guardian/e2guardian/assets/21127737/d691daeb-840e-4f8d-817d-fb7b2625fe63
| e2guardian554-requesterror.png (view on web) ]
[
https://github.com/e2guardian/e2guardian/assets/21127737/d2b44099-9821-4b61-ac9a-19a020ea9d3e
| e2guardian554pcap2.png (view on web) ]
[
https://github.com/e2guardian/e2guardian/assets/21127737/d07c77bb-285b-4a2c-93b6-f58bee1c9c4a
| e2guardian554-requesterror2.png (view on web) ]
—
Reply to this email directly, [
#810 | view it on GitHub ] , or
[
https://github.com/notifications/unsubscribe-auth/ABHZDWILWAMLX4NSPMWY3LTY75XCPAVCNFSM6AAAAABG77PB2WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TCMBZGYZTONY
| unsubscribe ] .
You are receiving this because you were mentioned. Message ID:
***@***.***>
|
|
@szrce FYI |
|
hi, @philipianpearce i did build from freebsd and i installed freebsd pfsense, the installed os like bellow. is it same debug options ?
|
|
maybe help you this log(when i find new log i will be added bellow) , you can search like start words -
@philipianpearce If you need more logs, you can guide me, how can I deep detail log |
|
Can you increase the debuglevel by adding debuglevel = "thttps" to e2guardian.conf and let me see the output? That should help us to see where the corruption is happening. Thanks |
|
debuglog.txt |
|
The debuglevel = thttps does not seem to have worked. |
did you see new files? current log section i did add new. |
|
Can you check e2config.h file? Both DEBUG_LOW and DEBUG_HIGH should be defined to get full debug. |
i think is open? am i wrong? Running in debug_low mode... Built with: '--with-logdir=/var/log' '--with-piddir=/var/run' '--disable-avastd' '--enable-clamd' '--with-debug_high=on' '--with-debug_low=on' '--enable-dnsauth=on' '--disable-icap' '--disable-kavd' '--enable-sslmitm' '--prefix=/usr/local' '--localstatedir=/var' '--mandir=/usr/local/man' '--disable-silent-rules' '--infodir=/usr/local/share/info/' '--build=amd64-portbld-freebsd14.0' 'build_alias=amd64-portbld-freebsd14.0' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -fstack-protector-strong -fno-strict-aliasing -std=c++11 ' 'LDFLAGS= -fstack-protector-strong ' 'LIBS=' 'CPPFLAGS=' 'CC=cc' 'CFLAGS=-O2 -pipe -fstack-protector-strong -fno-strict-aliasing ' 'CPP=cpp' 'PKG_CONFIG=pkgconf' 'PKG_CONFIG_LIBDIR=/root/pfSense/FreeBSD-ports/www/e2guardian/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig' |
/work/e2guardian-5.5.4r # cat e2config.h | grep "DEBUG_*" |
like a this option here it is new log files. |
|
That looks good. It is just that there is no thttps trace in the output file.
What do you have as debuglevel entry?
…----- On 2 May, 2024, at 12:05, Sezer ***@***.***> wrote:
> Can you check e2config.h file? Both DEBUG_LOW and DEBUG_HIGH should be defined
> to get full debug.
i think is open? am i wrong?
Running in debug_low mode...
e2guardian 5.5.4r
Built with: '--with-logdir=/var/log' '--with-piddir=/var/run' '--disable-avastd'
'--enable-clamd' '--with-debug_high=on' '--with-debug_low=on'
'--enable-dnsauth=on' '--disable-icap' '--disable-kavd' '--enable-sslmitm'
'--prefix=/usr/local' '--localstatedir=/var' '--mandir=/usr/local/man'
'--disable-silent-rules' '--infodir=/usr/local/share/info/'
'--build=amd64-portbld-freebsd14.0' 'build_alias=amd64-portbld-freebsd14.0'
'CXX=c++' 'CXXFLAGS=-O2 -pipe -fstack-protector-strong -fno-strict-aliasing
-std=c++11 ' 'LDFLAGS= -fstack-protector-strong ' 'LIBS=' 'CPPFLAGS=' 'CC=cc'
'CFLAGS=-O2 -pipe -fstack-protector-strong -fno-strict-aliasing ' 'CPP=cpp'
'PKG_CONFIG=pkgconf'
'PKG_CONFIG_LIBDIR=/root/pfSense/FreeBSD-ports/www/e2guardian/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig'
—
Reply to this email directly, [
#810 (comment) |
view it on GitHub ] , or [
https://github.com/notifications/unsubscribe-auth/ABHZDWKFJCFDQUTEEKFW3TDZAIMWPAVCNFSM6AAAAABG77PB2WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGIZDKNJYGU
| unsubscribe ] .
You are receiving this because you were mentioned. Message ID:
***@***.***>
|
did you check this one, and other files maybe help you? |
|
OK that's good - looks like an issue with the TLS client hello parsing on transparent HTTPS filtering - I will try and have a look at a fix for it tomorrow. |
philipianpearce
changed the title
|
@szrce I've uploaded to github a test branch v5.5.sni. Can you try this (with debug) and see if this fixes the issue? I have increased the buffer size for the client hello and also corrected the toread so that it will read the full clienthello. I'm am a bit puzzled as there were already limit checks in get_TLS_SNI which should have prevented the garbled SNI being returned. IF it tests OK, I'll roll into v5.5 and v5.6.dev |
|
OK that's good, but why missing __SSLMITM options, configure.ac is it diffrent 5.5.4r. |
|
__SSLMITM options was redundant - is enabled by default in all v5.5 - configure.ac has been tidied to remove redundant options and remove anomalies (e.g. enable-sslmitm was required to load libraries when no longer any MITM conditional code, ssl libraries now enabled by default) Would you sent me debug on v5.5.sni so that I can check it is working as I expect? Thanks. |
|
here it is, if dont send lots request over everythink is ok. but send request average 15 page getting NETERROR, i did add screenshots. and others output screens
|
|
hi @philipianpearce, were you able to check it? I'm still getting errors. |
|
@szrce NETERROR indicates a problem in the DNS lookup or network path to the requested site. If the message 'is not responding' then it means that the site is rejecting (or not completing the TCP handshake) or there is some intermittent network issue in the path to the site. In any case a NETERROR message is reporting an issue from the operating system, so it is not a bug. Thanks for the trace, it shows that clienthello messages are often large, so I need to further increase the buffer size, however the SNI now seem to be being extracted correctly and not being truncated. |
|
@philipianpearce thank you for your information, i will check detail NETERROR problem, cause i am not sure. on the other hand, if you fix clienthello size i can try. thanks. |
|
@szrce I have changed clienthello buffer size to dynamic. (in v5.5.sni) Would you retest? Thanks. |
|
@philipianpearce hi, until 3/5 second, then was killed with your last commit. May 9 10:05:44 pfSense kernel: swap_pager: out of swap space |
|
Odd, have reverted to fixed buffer in latest v5.5.sni. Does that fix it? |
|
yeap is working with revert, there is no swap exit or kernel dumps, but is still get error like bellow
|
|
I have re-checked the SNI logic and added some more checks to ensure we only act on correct format. |
still same check it bellow image, its still coming broke url
|
|
@philipianpearce, is still up? |
|
@philipianpearce I'm experiencing a similar problem. I'm using pfSense 2.7.0 and the latest E2Guardian5 version from the repository in over 20 different locations. When I try to access a web page, it doesn't load at first, but it opens when I refresh the page. I don't encounter this error in Firefox. It only happens in Chrome and Edge browsers. During this time, an error line similar to "127.0.0.1 access denied net error" appears in the access logs. About a month ago, this issue suddenly started occurring in different locations with pfSense 2.4.5 and the previous version of E2Guardian installed. After upgrading all devices to 2.7.0 and the latest E2Guardian repository, the error started to occur less frequently. Sometimes, the page can't be displayed, but it refreshes itself within 1 second and opens, and again, the net error appears in the access logs. |
|
@sehzade58 Thanks for this report, very helpful. Looks like something has changed in Chrome and Edge browsers. May be related to this bug (the time scale matches) but need to confirm. Can you post a small sample of the actual "127.0.0.1 access denied net error" messages? @szrce Are your users using Chrome or Edge? |
yeap same chrome and edge here it is https://we.tl/t-vqUCw6aUKY pcap , client hello request exist but, dosent exist server response. its start 336,346 and, can you share your email address with me sezerceadres@gmail.com if possible |
|
This screenshot contains exactly the error I'm talking about. |
|
hi, @philipianpearce i hope u are well, these are new output for the debug example site: https://img7.mynet.com.tr/anasayfa/img/sprite.png?v=3 the chrome doesn't reach the page saying connection closed, same request copied and send with "postman" the result tls connection closed. e2guardian giving ouput has a img7.mynet... tcp start connection hello length is 4 or 1,6 its seen wrong, look at from wireshark sni hello length 314, but e2guardian side seen only 4 we cannot read correct buffer data, when i change e2guardian/src/ConnectionHandler.cpp Line 3117 in 7e957b3 to its works fine everything, but not sure is correctly way? |
|
Thanks for the trace. If it does, I will change configure.ac so that this always added as an option. I've just posted a new v5.5.sni version where I have rewritten the SNI detection so that it is not reliant on bit-shifting etc and to detect if the ClientHello is ECH. Would you try this, but after you have tried existing code with -funsigned-char. Thanks |
its work with -funsigned-char without new commit but The following problem continues, these cause getting NETERROR logs and slow
if use both them(funsigned-char and new comm) same output connection closed, but connection is very fast other page. by the way when only get new commit its same working |
|
Thanks for the testing, I'll add -funsigned-char to both the v5.5.5 and v5.5.sni branches. |
sounds good, thank you for focus, dont forget slowing(if you use params the page opening slowly), and without the param with new commits its so fast, bu still continue the problem in summary otherwise thanks |
hello @philipianpearce ,
in v5.5.4r some url coming broken like bellow when you going eg instagram, when you open chrome and search instagram, chrome first of all sending request this url (content-autofill.googleapis.co) actually (content-autofill.googleapis.com). if e2guardian cant reach dns address, will be getting error page like proxy refused from google-chrome.
when , dns req, e2guardian log i examine : e2guardian sending dns request broken host address.
can you check bug or configuration mistake?
thanks in advance.
The text was updated successfully, but these errors were encountered: