Clients cant connect to agentPort, attempted connections show up as "WEBREQUEST: (X.X.X.X) AgentPort: /agent.ashx" in server log

[netw0rk-noob]

I aim for the following setup:
Dashboard / WebUI only accesible from LAN, via a .local domain name with a cert from my LAN CA which is served via nginx on the same host as the meshcentral server.
Meshagents / clients connecting from the internet through my reverse proxy (nginx on another host) which handles TLS with a letsencrypt cert.
Connection scheme:
meshAgent --publicDomain.tld:8443--> Router (NAT) --reverseProxyIP:8443-->reverseProxy (handles TLS) --meshCentralIP:8443--> MeshCentral
admin --localDomain.tld:443--> nginx on MeshCentralHost (handles TLS) --127.0.0.1:4443--> MeshCentral on MeshCentralHost

The admin connections work without any problems. When a meshagent (I tried with windows, debian and android) tries to connect, the meshcentral server (run with --debug agent,web,webrequest) logs the following:
WEBREQUEST: (MeshAgentPublicIP) AgentPort: /agent.ashx
and the client does not connect / show up in dashboard.

server logs at startup:

# node node_modules/meshcentral --debug agent,web,webrequest
MeshCentral HTTP redirection server running on port 81.
MeshCentral v1.1.22, Hybrid (LAN + WAN) mode.
MeshCentral HTTP server running on port 4443, alias port 443.
MeshCentral HTTP agent-only server running on port 8443.
Loaded web certificate from "https://publicdomain.tld:", host: "publicdomain.tld"
  SHA384 cert hash: .....
  SHA384 key hash: .....
WEBREQUEST: (MeshAgentPublicIP) AgentPort: /agent.ashx
[…]

Besides that (but thats a minor problem) the download-links (and script-links, when using the scripted linux-install) do not reflect my changes to config.json regarding agentPort and/or agentAliasPort: The download/script links point to port :443 (which is the aliasPort of the dashboard), not to port :8443 (which is the agentPort) and therfore do not work unless manually edited to point to port :8443.

I assume that this is a configuration mistake on my end and would be very grateful if someone could take a look at my setup.
If any relevant information is missing, please ask for it.

Server Software:

• OS: Debian 12 bookworm
• Virtualization: unprivileged LXC inside of Proxmox VE
• Network: private network (NAT) with Ports 80, 443, 8443 forwared to the aforementioned reverseProxy
• Version: v1.1.22
• Node: v18.19.0

Client Device:

• Device: laptop accessing the Dahboard
• OS: Ubuntu 22.04
• Network: same LAN as meshCentral

Remote Device 1 (external):

• Device: computer running meshAgent
• OS: Windows 7
• Network: different, offsite network

Remote Device 2 (internal):

• Device: laptop running meshAgent
• OS: Debian 12 bookworm
• Network: same LAN as meshCentral

config.json:

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "cert": "publicdomain.tld",
    "WANonly": true,
    "port": 4443,
    "aliasPort": 443,
    "agentPort": 8443,
    "_agentAliasPort": 8443,
    "agentAliasDNS": "publicdomain.tld",
    "tlsOffload": "127.0.0.1,192.168.179.143,::1",
    "mpsPort": 0,
  },
  "_domaindefaults": {
    "__comment__": "Any settings in this section is used as default setting for all domains",
    "title": "MyDefaultTitle",
    "footer": "Default page footer",
    "allowedOrigin": true,
    "newAccounts": false
  },
  "domains": {
    "": {
      "allowedOrigin": [ "publicdomain.tld", "localdomain.tld" ],
      "certUrl": "https://publicdomain.tld:"
    }
  }
}

nginx localdomain.tld.conf on the same host as meshCentral:

server {
    listen 80;
    server_name localdomain.tld;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name localdomain.tld;

    # MeshCentral uses long standing web socket connections, set longer timeouts.
    proxy_send_timeout 330s;
    proxy_read_timeout 330s;

    ssl_certificate	/etc/nginx/certificates/localdomain.tld.pem;
    ssl_certificate_key	/etc/nginx/certificates/localdomain.tld.key.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 5m;
    ssl_dhparam /etc/nginx/dh4096.pem;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # MeshCentral Dashboard
    location / {
	proxy_pass http://127.0.0.1:4443/;
	proxy_read_timeout 90;

	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;

	proxy_hide_header X-Powered-By;

	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
    }
    client_max_body_size 0;
}

nginx publicdomain.tld.conf on the reverseProxyl:

server {
    listen 80;
    server_name publicdomain.tld;
    return 301 https://$host$request_uri;
}

server {
    listen 8443 ssl;
    server_name publicdomain.tld;

    # MeshCentral uses long standing web socket connections, set longer timeouts.
    proxy_send_timeout 330s;
    proxy_read_timeout 330s;

    ssl_certificate	/etc/letsencrypt/live/publicdomain.tld/fullchain.pem;
    ssl_certificate_key	/etc/letsencrypt/live/publicdomain.tld/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 5m;
    ssl_dhparam /etc/nginx/dh4096.pem;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_prefer_server_ciphers on;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    # MeshCentral
    location / {
	proxy_pass http://{{reverseProxyIP}}:8443/;

	proxy_read_timeout 90;

	proxy_set_header Host $host;
	proxy_set_header X-Real-IP $remote_addr;
	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	proxy_set_header X-Forwarded-Proto $scheme;

	proxy_hide_header X-Powered-By;

	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection “upgrade”;
	#
    }
    client_max_body_size 0;
}

[si458]

You have agentaliasport uncommented, you need to remove the _
Then restart meshcentral
Then uninstall/reinstall all ur meshagents as the port number will of changed

Edit: also you haven't set agentPortTls either

[netw0rk-noob]

You have agentaliasport uncommented, you need to remove the _ Then restart meshcentral Then uninstall/reinstall all ur meshagents as the port number will of changed

I did indeed uncomment agentAliasPort, because I assumed that it is not needed if it is the same as agentPort which it would be in my case. (8443) I removed the underscore for testing nonetheless.

Edit: also you haven't set agentPortTls either

Yes, I've set that to false explicitly now, which leads to the following config.js:

{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "cert": "publicdomain.tld",
    "WANonly": true,
    "port": 4443,
    "aliasPort": 443,
    "agentPort": 8443,
    "agentAliasPort": 8443,
    "agentAliasDNS": "publicdomain.tld",
    "agentPortTls": false,
    "tlsOffload": "127.0.0.1,192.168.179.143,::1",
    "mpsPort": 0,
  },
  "_domaindefaults": {
    "__comment__": "Any settings in this section is used as default setting for all domains",
    "title": "MyDefaultTitle",
    "footer": "Default page footer",
    "allowedOrigin": true,
    "newAccounts": false
  },
  "domains": {
    "": {
      "allowedOrigin": [ "publicdomain.tld", "localdomain.tld" ],
      "certUrl": "https://publicdomain.tld:"
    }
  }
}

and the following debug log at startup:

# node node_modules/meshcentral --debug agent,web,webrequest
MeshCentral HTTP redirection server running on port 81.
MeshCentral v1.1.22, Hybrid (LAN + WAN) mode.
MeshCentral HTTP server running on port 4443, alias port 443.
MeshCentral HTTP agent-only server running on port 8443, alias port 8443.
Loaded web certificate from "https://publicdomain.tld:", host: "publicdomain.tld"
  SHA384 cert hash: .....
  SHA384 key hash: .....
WEBREQUEST: (MeshAgentPublicIP) AgentPort: /agent.ashx

The (reinstalled) meshagents (I tried with android + debian) still do not connect and at least the Linux/BSD install commands still does not reflect the agent(Alias)Port 8443.

What I did notice though is the fact that even when I manually correct the Linux/BSD install commands to point to port 8443, they cant successfully download their meshsettings from publicdomain.tld:8443 but get a 404: Not Found. The meshagent is downloaded successfully from publicdomain.tld:8443 though. From the output of sudo -E ./meshinstall.sh https://publicdomain.tld:8443 'A9pqunopaAn5st5sqzvE4rpDxsAtDbvxz9nCFhidhpqwrnympwq32fwF5mwzyon5':
https://publicdomain.tld:8443/meshsettings?id=A9pqunopaAn5st5sqzvE4rpDxsAtDbvxz9nCFhidhpqwrnympwq32fwF5mwzyon5
[…]
ERROR 404: Not Found.

If I try to download the meshsettings from https://localdomain.tld/meshsettings?id=A9pqunopaAn5st5sqzvE4rpDxsAtDbvxz9nCFhidhpqwrnympwq32fwF5mwzyon5, it is available for download.

When I try to connect using the android app (the according connection string does reflect the agentPort 8443, but it also did before the recent changes, it immediately aborts. (i.e. it it does switch to Connected for abourt a quarter of a second when tapping on connect before it flips back to Disconnected again.) The server logs one line with the follwing content for every connection attempt:
WEBREQUEST: (MeshAgentPublicIP) AgentPort: /agent.ashx

[netw0rk-noob]

Anyone got a hint for me, what else I could try to debug this? I havent gotten any further on my own.

One additional information that might be relevant: The Meshcentral server and the reverse proxy for external agent connections are in different Subnets. Routes are set up in both directions, both can ping each other.

[si458]

@netw0rk-noob can u share ur config.json as of today please?
im gunna try have alook when i get chance
it sounds like its getting confused because you want to use 2 different DNS names, one for the web ui and 1 for the agent
then making it more confused as ur using a reverse proxy to do the tls and not meshcentral