cloudflare issues: multiple attempts/black screen #5302

billettg · 2023-08-22T13:43:10Z

Describe the bug
Clicking on the "Connect" button under "Desktop" or "Terminal" results in "Disconnected" approximately 9/10 times. Other times it will connect successfully. The disconnection is immediately shown after clicking "Connect".

To Reproduce
Steps to reproduce the behavior:

Go to "Terminal"
Click on "Connect"
See "Disconnected"

Expected behavior
Connects successfully on every attempt.

Screenshots
If applicable, add screenshots to help explain your problem.

Server Software (please complete the following information):

OS: Ubuntu 22.04.3 LTS
Virtualization: NA
Network: WAN/CloudFlare DNS + proxy (not tunnel)
Version: 1.1.10
Node: 12.22.9

Client Device (please complete the following information):

Device: 3 unique devices
OS: macOS, Linux, Windows
Network: Remote over WAN
Browser: Google Chrome/Safari
MeshCentralRouter Version: NA

Additional context
The problem seems to only occur using CloudFlare, so I think that the proxy is causing websocket disconnection. The MeshCentral VM is hosted on the Hetzner platform. Others face the same issue (e.g https://www.reddit.com/r/MeshCentral/comments/15y28x3/random_disconnects_behind_cloudflare/)

Your config.json file

{
  "settings": {
    "cert": "mesh.example.com",
    "wanonly": true,
    "port": 443,
    "aliasport": 443,
    "redirport": 80,
    "rediraliasPort": 80,
    "webrtc": true,
    "wscompression": true,
    "allowlogintoken": true,
    "trustedproxy": "CloudFlare"
  },
  "domains": {
    "": {
      "newaccounts": false,
      "usernameisemail": true,
      "certurl": "https://mesh.example.com"
    }
  }
}

The text was updated successfully, but these errors were encountered:

vesector · 2023-08-22T13:44:33Z

+1

si458 · 2023-08-22T13:49:35Z

it could be webrtc issue? have you tried with webrtc set to false?

si458 · 2023-08-22T13:50:05Z

also nodejs 12 is now EOL, please update node to the latest LTS (18)

vesector · 2023-08-22T14:06:12Z

I have tested with WebRTC disabled and also enabled, having the same behaviour.

One thing to note is that this issue started a few weeks ago.

billettg · 2023-08-22T14:07:57Z

Updated nodejs to 18.17.1 and same issue with or without webrtc enabled. I rebooted the server and checked nodejs version with the --version parameter.

The logs show connected then immediately disconnected. When it works I noticed that "Relay holding: * (::1) Authenticated" shows as well.

WEBREQUEST: (-) /meshrelay.ashx/.websocket?p=1&nodeid=node//iWMDZtF%24QteFec1amvHRz7nA8c4SqcXNrz7d7HN5JYxOTNwrOvK%24WOqkZ9XJjHuq&id=ma8286xee29&rauth=VEcPv1E75FdSda%24JPWUN7XeUM2plL7mDexVJ37kpZ%24v4selPtY3Kb%24Voz%40maz2MGG1E3IP4Pm9VcASvz3YSGyGo9CNWntAzVx3A%24BLwfSdTIPN4by%40o7
COOKIE: Decoded AESGCM cookie: {"ruserid":"user//gareth","x":"BQF0kfBA","time":1692713115000,"dtime":6268}
RELAY: Relay connected: ma8286xee29 (- -> -)
DISPATCH: DispatchEvent [ '', 'user//gareth' ]
RELAY: Relay disconnect: ma8286xee29 (8- --> -)
DISPATCH: DispatchEvent [
'',
'user//gareth',
'node//iWMDZtF$QteFec1amvHRz7nA8c4SqcXNrz7d7HN5JYxOTNwrOvK$WOqkZ9XJjHuq',
'mesh//DvVA69QdUY7bH8nbnnI4X@cSpkJOl$iIExyDvvBtFEudOUwUuEn5C2QNrHepdL6A'
]

jwiener3 · 2023-08-22T14:24:37Z

+1
I see the same issues, immediate disconnect for different hosts at different locations. All the same symptoms as stated above.
I am proxying through Cloudflare, I am not sure if others are doing the same.

hiddenpcmaster · 2023-08-22T14:25:57Z

+1

dooley74 · 2023-08-23T02:44:11Z

Same here, just started a few days ago though. Running through Cloudflare as well.

jirijanata · 2023-08-23T07:52:10Z

+1
Also running through Cloudflare. Actually the problem first occured for me after Upgrade from 1.1.8 to 1.1.10.

supra36 · 2023-08-23T11:09:00Z

+1 same, although it didn't start immediately after 1.1.8=>1.1.10 transition, it started giving issues gradually and now it affects all agents no matter the OS.

Real-time monitoring using graphs works fine, however it fails to establish any other type of connection, let it be webrtc, stream, rdp, terminal or file transfer. MeshCentral Router also fails to RDP in "Configuring remote session" phase.

Also, I'm running mesh through cloudflare too.

si458 · 2023-08-23T11:14:49Z

this sounds to me like a cloudflare issue not a meshcentral issue, as nothing has changed recently to do with proxies or tunneling
can anybody donate a domain/cloudflare details and i could look at it in my freetime?

supra36 · 2023-08-23T11:19:58Z

I took a look and filtered websocket related issues in google search from last 7 days. These are results.

https://community.cloudflare.com/t/websocket-not-stable-some-time-connect-some-time-no/547638

https://community.cloudflare.com/t/websocket-problem/547094

https://community.cloudflare.com/t/protected-web-socket-connection-dies-after-30-secs-cloudflare-ipv6-issue/547127

Unfortunately these haven't been answered.

Also it seems that CloudFlare bottlenecks websocket connections after reaching certain level of connections (or spikes). Does anybody know if CloudFlare started limiting websocket connections just recently or changed anything related to these limits or to how these are handled?

https://developers.cloudflare.com/support/network/using-cloudflare-with-websockets/

vesector · 2023-08-23T11:24:29Z

Great find @supra36, I have the feeling it is Cloudflare related more than the actual update of MC to .10... I am not 100% though.

I did a tcpdump and I saw Cloudflare giving [F] and [R] flags while reproducing the issue.

Would be nice if someone from the community that experiences the same issue and has a Cloudflare business or higher tier subscription opens a ticket with their support to see if this gets an answer from their side... as using the free tier there is no chance of support afaik.

NiceGuyIT · 2023-08-23T12:15:57Z

There's a Cloudflare Developers on Discord. Might be useful to ask around.

A post 3 days ago mentioned websocket disconnects when using IPv6 but not when using IPv4. Someone mentioned disabling IPv6 in Network -> IPv6 Compatibility.

vesector · 2023-08-23T13:23:47Z

Hi @NiceGuyIT,

Thank you for the suggestion.

I have disabled IPv6 through the API and the issue persists.

For anyone that wants to give it a try... I used PowerShell:

$headers=@{}
$headers.Add("Content-Type", "application/json")
$headers.Add("X-Auth-Email", "YOURCFEMAIL")
$headers.Add("X-Auth-Key", "YOURAPIKEY")
$response = Invoke-WebRequest -Uri 'https://api.cloudflare.com/client/v4/zones/YOURZONEID/settings/ipv6' -Method PATCH -Headers $headers -ContentType 'application/json' -Body '{
"value": "off"
}'

supra36 · 2023-08-24T07:40:31Z

Just to let you know, I ditched cloudflare and used let's encrypt. It's now working fine.

billettg · 2023-08-24T07:52:50Z

Just to let you know, I ditched cloudflare and used let's encrypt. It's now working fine.

I decided to provision another Ubuntu server and installed MC 1.1.10 with Let's Encrypt certificate rather than CloudFlare and it also works fine. I wonder what changes have been made on CloudFlare side to be dropping the websocket connections, it must be something recent. I'm sure their support would be able to comment but unfortunately we are on a free tier without any technical support.

vesector · 2023-08-24T09:51:21Z

As per the questioning raised by @frogweh on #5309 he seems to be using Nginx and not CloudFlare and having the same issue... would be good to test that to see if it's reverse proxy in general or if we can completely isolated to CF.
Also, would be good to try with an older version of .10 so we can rule out the update of MC completely. Will try to do this later this week.

si458 · 2023-08-24T10:00:05Z

@iribarrenjg if you read the post #5309 he says he uses a cloudflare certificate and if you read the logs shows heads for cf connecting from, so he is indeed using cloudflare!
i do think the is an issue with cloudflare?

mon5termatt · 2023-08-25T07:54:50Z

Having the same issue, also using cloudflare.

Vista2003 · 2023-08-25T13:29:39Z

+1 on 1.1.10 (Raspberry Pi OS 64 Bit via Cloudflare and Nginx Proxy Manager)

jwiener3 · 2023-08-25T19:33:00Z

Like others here, I changed my DNS records away from Cloudflare's proxy and things started working as expected. Also, this was happening before 1.1.10. I upgraded to 1.1.10 in hopes of it being a MeshCentral issue. Now I am trying to figure out how to lock down my environment without Cloudflare's rules. I know it is possible, but Cloudflare made it easy :).

si458 · 2023-08-25T19:58:48Z

@jwiener3 you can do an IP allow list for agents and clients which might help?

MeshCentral/meshcentral-config-schema.json

Lines 592 to 628 in b1d2d1a

    
           "ipBlockedUserRedirect": { 
        
             "type": "string", 
        
             "default": null, 
        
             "description": "If set, a user from a banned IP address will be redirected to this URL." 
        
           }, 
        
           "userAllowedIP": { 
        
             "type": [ 
        
               "string", 
        
               "array" 
        
             ], 
        
             "default": null, 
        
             "description": "When set, only users from allowed IP address ranges can connect to the server. Example: \"192.168.2.100,192.168.1.0/24\"" 
        
           }, 
        
           "userBlockedIP": { 
        
             "type": [ 
        
               "string", 
        
               "array" 
        
             ], 
        
             "default": null, 
        
             "description": "When set, users from these denied IP address ranges will not be able to connect to the server. Example: \"192.168.2.100,192.168.1.0/24\"" 
        
           }, 
        
           "agentAllowedIP": { 
        
             "type": [ 
        
               "string", 
        
               "array" 
        
             ], 
        
             "default": null, 
        
             "description": "When set, only agents from allowed IP address ranges can connect to the server. Example: \"192.168.2.100,192.168.1.0/24\"" 
        
           }, 
        
           "agentBlockedIP": { 
        
             "type": [ 
        
               "string", 
        
               "array" 
        
             ], 
        
             "default": null, 
        
             "description": "When set, agents from these denied IP address ranges will not be able to connect to the server. Example: \"192.168.2.100,192.168.1.0/24\"" 
        
           },

jwiener3 · 2023-08-25T20:31:11Z

@jwiener3 you can do an IP allow list for agents and clients which might help?

Thanks, I will take a look at that.

mon5termatt · 2023-08-25T20:44:14Z

So I can just disable the proxy checkmark on cloudflare? I have my domain registered with them.

mon5termatt · 2023-08-25T20:45:48Z

jwiener3 · 2023-08-25T21:50:44Z

So I can just disable the proxy checkmark on cloudflare? I have my domain registered with them.

Yes that is what I did, and then I had to open up my ACL on the server that was hosting Meshcentral to allow connections from anywhere, as I only had it allowing connections from cloudflare IP space.

appleimperio · 2023-08-26T15:16:30Z

+1 docker Raspberry pi with Cloudflare tunnels. Today I update the Cloudflare tunnel app and now is not connecting at all

appleimperio · 2024-01-30T19:30:41Z

No Cloudflare tunnel is not working either.

g4youu · 2024-01-30T19:50:16Z

A few weeks back, I utilized the Ngrok tunnel in conjunction with MeshCentral, and it functioned as anticipated. However, the only drawback was that Ngrok had a bandwidth limit of 1GB compare to unlimited Cloudflare tunnel. On the other hand, when I used the Cloudflare tunnel, I faced the usual issue of a black screen.

nealhead · 2024-01-30T19:52:39Z

A few weeks back, I utilized the Ngrok tunnel in conjunction with MeshCentral, and it functioned as anticipated. However, the only drawback was that Ngrok had a bandwidth limit of 1GB compare to unlimited Cloudflare tunnel. On the other hand, when I used the Cloudflare tunnel, I faced the usual issue of a black screen.

Is it really unlimited if it doesn't work? :)

Pshemas · 2024-02-02T14:46:30Z

the same on my end. After I've attempted to move from local MC install and put it on a web server (OVH VPS, Rocky 9, Node 21) I'm getting black screen on remote connection (with a tiny red square in top left corner on start).
The agents connect (both Linux and Windows clients), show statuses and so on, but all I can get is black screen. Would love to get it working with Cloudflare.

david85091 · 2024-02-05T12:06:47Z

i gotta agree! This is a royal pain in the Buttocks! i really do not like opening holes in the firewall

DivMode · 2024-02-08T21:41:25Z

I see a lot of you guys have meshcentral setup with cloudflare tunnels.

Can anyone share their configuration or are there any guides out there to get this setup?

My remote computer is setup at my parents place but they have 5g internet and it looks like using cf tunnels is the best way to get it to work before this issue but Im hoping it will eventually be fixed.

si458 · 2024-02-08T21:58:10Z

@DivMode sadly using cloudflare tunnels is also effected, so even if u used a tunnel to route ur meshcentral server behind a firewall or proxy, you will still experience a black screen issue

Dasno7 · 2024-02-13T19:17:24Z

Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again

si458 · 2024-02-13T19:22:35Z

Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again

From my previous testing, I had a higher success rate just using their paid teir, the first level...

But I haven't tried it recently to verify if that's still the case or not.

Has anybody else got paid cloudflare and clarity if it still works or not?

blaine07 · 2024-02-13T19:27:22Z

Could it somehow be using stream on paid plans?

si458 · 2024-02-13T19:29:09Z

@blaine07 I don't think so?
I think the stream is video with mp4, we just stream jpeg

stoiet · 2024-02-13T21:05:31Z

@si458 Currently we have Pro plan, but we've been experiencing the same issue for weeks. We disabled the proxy as a temporary solution. We do not use MeshCentral but in the case of a different application we experience exactly the same websocket connection issue. Unpredictable, some browsers just cannot initialise the socket connection, but others can. With DNS only CNAME record we have zero issue.

I found this github issue in the Cloudflare Developers discord server, on that thread I have explained more: https://discord.com/channels/595317990191398933/1150828630340022343/1150828630340022343

si458 · 2024-02-13T21:13:04Z

@stoiet so it's a cloudflare issue then!
Great thank you!
I also totally forgot I asked about the issue on cloudflares discord...

Edzilla2000 · 2024-02-13T21:18:25Z

Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again

At this moment you should consider that cloudfare tunnels are just not compatible with meshcentral.
However, if your meshcentral instance cannot be exposed to the internet because you're behind NAT and you can't open port 443, setting up a reverse proxy on any kind of VPS and proxying through that is very easy.

Dasno7 · 2024-02-14T05:35:40Z

Just posted an update on this matter on the cloudfare community page. Hopefully we can get a response there...

https://community.cloudflare.com/t/meshcentral-agents-can-no-longer-connect-using-tunnel/551937/13

Dasno7 · 2024-02-14T05:41:09Z

Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again

At this moment you should consider that cloudfare tunnels are just not compatible with meshcentral. However, if your meshcentral instance cannot be exposed to the internet because you're behind NAT and you can't open port 443, setting up a reverse proxy on any kind of VPS and proxying through that is very easy.

I have setup a reverse proxy on a VPS. If I proxied this through cloudflare while my meshcentral is on DNS only, would my IP address still be hidden from the public? Excuse me for being a noob on the topic

si458 · 2024-02-14T07:45:26Z

Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again

At this moment you should consider that cloudfare tunnels are just not compatible with meshcentral. However, if your meshcentral instance cannot be exposed to the internet because you're behind NAT and you can't open port 443, setting up a reverse proxy on any kind of VPS and proxying through that is very easy.

I have setup a reverse proxy on a VPS. If I proxied this through cloudflare while my meshcentral is on DNS only, would my IP address still be hidden from the public? Excuse me for being a noob on the topic

@Dasno7 you are still proxing through cloudflare so your meshcentral won't work

You would have to set the dns to dns only and point the ip of ur meshcentral dns to the VPS of ur reverse proxy, then forward the reverse proxy to ur meshcentral server, this would hide ur meshcentral ip as others have explained

Matt-CyberGuy · 2024-03-23T03:12:43Z

Not sure when this got fixed, but a friend just casually mentioned to me mesh was working for him over his cloudflare proxy again. I just checked, it's working for us as well. No idea what changed... but definitely didn't change anything. Crossing fingers this isn't temporary this time.

gt2416 · 2024-03-23T05:24:30Z

YES!! It does work AGAIN. I love you all !!
No more do I need to login to my firewall change rules etc when I need to connect to someone outside my office !!!
I am so happy

si458 · 2024-03-23T08:35:26Z

Oh wow will check when bk off hols.
We changed nothing!
I repeat we changed nothing!!!
So clearly cloudflare changed something if its working again!?

temak1238 · 2024-03-23T08:41:51Z

Yes they did change something. I can now see the ip addresses of my clients again, Before the change i got only the ip from the CloudFlare proxy back.

silversword411 · 2024-03-24T00:17:58Z

Cloudflare changes "something" again....7+ months later.

What crap

Goldmaster · 2024-03-24T12:24:40Z

Can confirm that on free tier, the issue has resolved itself. No need for any work from mesh central. I would advise the devs to put a warning in case a similar issue like this happens again.

si458 · 2024-03-24T12:30:10Z

@Goldmaster like this? https://ylianst.github.io/MeshCentral/meshcentral/faq/#im-using-cloudflare-and-im-getting-a-black-screen-but-the-mouse-moves

dooley74 · 2024-03-24T13:26:56Z

Can confirm that on free tier, the issue has resolved itself. No need for any work from mesh central. I would advise the devs to put a warning in case a similar issue like this happens again.

Could you please share your (sanitised) config/json file? I cant get mine to work with Cloudflare again, I dont know what I am doing wrong.

Goldmaster · 2024-03-24T16:19:39Z

@si458 yes, didn't realise that was a page. Only thing that's incorrect is that if one is running cloudflared, then disable proxy will result in argo tunnel error.

@dooley74 my configuration is quite plain and is the default. Make a copy of your current configuration by renaming it to .bk then let mesh central generate a new config

I set up mesh central in unraid https://youtu.be/yHuG5qFsYtk

si458 · 2024-03-24T16:24:40Z

@Goldmaster, yes using cloudflare tunnel is slightly different from using the dns Proxy. But the issue still stands that cloudflare broke something, and it took them months to fix whatever they changed! So let's hope they keep it that way and it doesn't break again!

WhatWhyIT · 2024-04-02T22:27:35Z

This issue appears resolved while using cloudlfare tunnels.
I had a ticket open for about a month(opened March 5th) and never migrated off the tunnel. between then and now, it appears to have gotten resolved.

billettg added the bug label Aug 22, 2023

NiceGuyIT mentioned this issue Aug 23, 2023

Mesh Central web socket relay only connects after multiple tries #5309

Closed

si458 pinned this issue Jan 30, 2024

kevpwhite mentioned this issue Feb 14, 2024

how to work with cloudflare? #2060

Closed

cloudflare issues: multiple attempts/black screen #5302

cloudflare issues: multiple attempts/black screen #5302

Comments

billettg commented Aug 22, 2023 • edited

vesector commented Aug 22, 2023

si458 commented Aug 22, 2023

si458 commented Aug 22, 2023

vesector commented Aug 22, 2023

billettg commented Aug 22, 2023 • edited

jwiener3 commented Aug 22, 2023

hiddenpcmaster commented Aug 22, 2023

dooley74 commented Aug 23, 2023

jirijanata commented Aug 23, 2023 • edited

supra36 commented Aug 23, 2023

si458 commented Aug 23, 2023

supra36 commented Aug 23, 2023 • edited

vesector commented Aug 23, 2023

NiceGuyIT commented Aug 23, 2023

vesector commented Aug 23, 2023 • edited

supra36 commented Aug 24, 2023

billettg commented Aug 24, 2023 • edited

vesector commented Aug 24, 2023

si458 commented Aug 24, 2023

mon5termatt commented Aug 25, 2023

Vista2003 commented Aug 25, 2023 • edited

jwiener3 commented Aug 25, 2023

si458 commented Aug 25, 2023

jwiener3 commented Aug 25, 2023

mon5termatt commented Aug 25, 2023

mon5termatt commented Aug 25, 2023

jwiener3 commented Aug 25, 2023

appleimperio commented Aug 26, 2023

appleimperio commented Jan 30, 2024

g4youu commented Jan 30, 2024

nealhead commented Jan 30, 2024

Pshemas commented Feb 2, 2024 • edited

david85091 commented Feb 5, 2024 • edited

DivMode commented Feb 8, 2024

si458 commented Feb 8, 2024

Dasno7 commented Feb 13, 2024

si458 commented Feb 13, 2024

blaine07 commented Feb 13, 2024

si458 commented Feb 13, 2024

stoiet commented Feb 13, 2024

si458 commented Feb 13, 2024

Edzilla2000 commented Feb 13, 2024

Dasno7 commented Feb 14, 2024

Dasno7 commented Feb 14, 2024

si458 commented Feb 14, 2024

Matt-CyberGuy commented Mar 23, 2024

gt2416 commented Mar 23, 2024

si458 commented Mar 23, 2024 • edited

temak1238 commented Mar 23, 2024

silversword411 commented Mar 24, 2024

Goldmaster commented Mar 24, 2024

si458 commented Mar 24, 2024

dooley74 commented Mar 24, 2024

Goldmaster commented Mar 24, 2024

si458 commented Mar 24, 2024

WhatWhyIT commented Apr 2, 2024

billettg commented Aug 22, 2023 •

edited

billettg commented Aug 22, 2023 •

edited

jirijanata commented Aug 23, 2023 •

edited

supra36 commented Aug 23, 2023 •

edited

vesector commented Aug 23, 2023 •

edited

billettg commented Aug 24, 2023 •

edited

Vista2003 commented Aug 25, 2023 •

edited

Pshemas commented Feb 2, 2024 •

edited

david85091 commented Feb 5, 2024 •

edited

si458 commented Mar 23, 2024 •

edited