Mosh connection issues

[keloin]

Hello,

I might be doing something wrong, but some time ago mosh connections started throwing this error:

NoMoshServerArgs - Did not find mosh server startup message. (Have you installed mosh on your server?)
Use mosh1 for the deprecated (previous) mosh version.

At the same time ssh2 stopped working too.

Mosh1 and ssh continue to work.

Ssh2 -vvv shows this in debug messages:

SSHSession:DEBUG:Attempting authentication with publickey.
SSHSession:DEBUG:Authentication failed

Ssh -vvv shows a lot of debug, but among others I can see this:

ssh_userauth_agent: Trying identity id_rsa
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
packet_send2: packet: wrote [type=50, len=608, padding_size=9, comp=598, payload=598]
ssh_packet_socket_callback: packet: read type 60 [len=576,padding=19,comp=556,payload=556]
ssh_packet_process: Dispatching handler for packet type 60
ssh_userauth_agent: Public key of id_rsa accepted by server
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
packet_send2: packet: wrote [type=50, len=1152, padding_size=17, comp=1134, payload=1134]
ssh_packet_socket_callback: packet: read type 52 [len=16,padding=14,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 52
ssh_packet_userauth_success: Authentication successful

Am I doing something wrong? :-)

Please and thank you!

[carloscabanero]

So ssh2 is deprecated. If ssh works, that is the one that should work. We kept ssh2 for a few years in case people had issues while we migrated from libssh2 to libssh.

On our new mosh version failing, can you send mosh --verbose so we can see what may be going on?

[keloin]

Ha! I see what the problem is:

bootstrapMoshServer: Command output: Failed binding to 10.0.0.101:60000
Error binding to IP 10.0.0.101: bind: Address already in use
Failed binding to 0.0.0.0:60000
Error binding to any interface: bind: Address already in use
Network exception: bind: Address already in use

But in the config for the host’s ports I have 60000:61000.
Don’t know why it tries to use 60000 all the time.

I just checked, if I remove the port config and leave it empty, it iterates and does go 60000, 60001, etc.
But when I have that config there it always tries to use 60000.
Seems like a defect?

Thanks!

[wedge1020]

Seemingly out of the blue, this morning my long-established mosh sessions started terminating, and I have not had much luck re-establishing them. No configuration changes on the remote host, no updates or packages installed. I was working away without issue for 1-2 hours then suddenly this issue started, and has persisted since.

With some manual establishing of LC_ALL I can get ssh to connect through blink with less issue, I can locally mosh (from remote host to remote host), but I cannot mosh TO the remote host from blink on my iPad.

blink> mosh --verbose user@remote
socket_callback_connected: Socket connection callback: 1 (0)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,password
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
bootstrapMoshServer: Trying bootstrap with sequence: [Blink.UseMoshOnPath, Blink.UseMoshOnPath], experimental: BKMoshExperimentalIP(rawValue: 0), family: nil, args: new -s -c 256 -l LANG=en_US.UTF-8
bootstrapMoshServer: Trying Blink.UseMoshOnPath
bootstrapMoshServer: Connecting to ~/.local/blink/mosh-server new -s -c 256 -l LANG=en_US.UTF-8
ssh_packet_global_request: Invalid SSH_MSG_GLOBAL_REQUEST packet
bootstrapMoshServer: Command output: bash: line 1: /home/user/.local/blink/mosh-server: No such file or directory

bootstrapMoshServer: Bootstrap failed with NoMoshServerArgs
bootstrapMoshServer: Trying Blink.UseMoshOnPath
bootstrapMoshServer: Connecting to mosh-server new -s -c 256 -l LANG=en_US.UTF-8
bootstrapMoshServer: Command output: The locale requested by LANG=C.UTF-8 isn't available here.
Running `locale-gen C.UTF-8' may be necessary.

This looks to be the blink local activity. The actual remote connection attempt from this output:

MOSH CONNECT 60001 theKey

mosh-server (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 34929]

startMoshServer: Remote Mosh server bootstrapped with params MoshServerParams(key: "theKey", udpPort: "60001", remoteIP: "remoteIP")

mosh did not make a successful connection to remoteIP:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

(By default, mosh uses a UDP port between 60000 and 61000. The -p option
                                                                        selects a specific UDP port number.)
blink> 

Remote system logs report the following:

==> auth.log <==
Apr 27 19:06:37 remote sshd[35451]: Accepted publickey for user from local_IP port 17917 ssh2: RSA SHA256:theKey
Apr 27 19:06:37 remote sshd[35451]: pam_unix(sshd:session): session opened for user user(uid=1000) by (uid=0)
Apr 27 19:06:37 remote systemd-logind[543]: New session 303 of user user.

==> daemon.log <==
Apr 27 19:06:37 remote systemd[1]: Started Session 303 of user user.

==> auth.log <==
Apr 27 19:06:37 remote sshd[35451]: pam_unix(sshd:session): session closed for user user
Apr 27 19:06:37 remote systemd-logind[543]: Session 303 logged out. Waiting for processes to exit.

It seems as if something has forgotten/lost the locale. I don’t seem to have much flexibility on the blink.sh side of things, and I’ve done what I can on the remote side to assuage its issues, seemingly to no avail.

I have no issue using mosh or ssh from another system. It just seems local to my blink.sh on my iPad.

Full range of MOSH ports open on the firewall.

And was working fine until today (April 27, 2024) until sometime into the 10AM hour (EDT).

Is there some background updating I am not aware of?

I may try removing and reinstalling the blink.sh app to see if it changes the current situation.

[carloscabanero]

Hi @wedge1020

This seems unrelated to the Issue mentioned here. What version of Blink are you running?

In any case, it looks like what has changed is on your server. Mosh needs to have a UTF-8 locale and it looks like your server is not providing it.

If you try running mosh-server new -s -c 256 -l LANG=en_US.UTF-8 on the server manually over SSH, do you get a valid server output?

Seemingly out of the blue, this morning my long-established mosh sessions started terminating, and I have not had much luck re-establishing them. No configuration changes on the remote host, no updates or packages installed. I was working away without issue for 1-2 hours then suddenly this issue started, and has persisted since.

With some manual establishing of LC_ALL I can get ssh to connect through blink with less issue, I can locally mosh (from remote host to remote host), but I cannot mosh TO the remote host from blink on my iPad.

blink> mosh --verbose user@remote
socket_callback_connected: Socket connection callback: 1 (0)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,password
ssh_agent_get_ident_count: Answer type: 12, expected answer: 12
bootstrapMoshServer: Trying bootstrap with sequence: [Blink.UseMoshOnPath, Blink.UseMoshOnPath], experimental: BKMoshExperimentalIP(rawValue: 0), family: nil, args: new -s -c 256 -l LANG=en_US.UTF-8
bootstrapMoshServer: Trying Blink.UseMoshOnPath
bootstrapMoshServer: Connecting to ~/.local/blink/mosh-server new -s -c 256 -l LANG=en_US.UTF-8
ssh_packet_global_request: Invalid SSH_MSG_GLOBAL_REQUEST packet
bootstrapMoshServer: Command output: bash: line 1: /home/user/.local/blink/mosh-server: No such file or directory

bootstrapMoshServer: Bootstrap failed with NoMoshServerArgs
bootstrapMoshServer: Trying Blink.UseMoshOnPath
bootstrapMoshServer: Connecting to mosh-server new -s -c 256 -l LANG=en_US.UTF-8
bootstrapMoshServer: Command output: The locale requested by LANG=C.UTF-8 isn't available here.
Running `locale-gen C.UTF-8' may be necessary.

This looks to be the blink local activity. The actual remote connection attempt from this output:

MOSH CONNECT 60001 theKey

mosh-server (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 34929]

startMoshServer: Remote Mosh server bootstrapped with params MoshServerParams(key: "theKey", udpPort: "60001", remoteIP: "remoteIP")

mosh did not make a successful connection to remoteIP:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

(By default, mosh uses a UDP port between 60000 and 61000. The -p option
                                                                        selects a specific UDP port number.)
blink> 

Remote system logs report the following:

==> auth.log <==
Apr 27 19:06:37 remote sshd[35451]: Accepted publickey for user from local_IP port 17917 ssh2: RSA SHA256:theKey
Apr 27 19:06:37 remote sshd[35451]: pam_unix(sshd:session): session opened for user user(uid=1000) by (uid=0)
Apr 27 19:06:37 remote systemd-logind[543]: New session 303 of user user.

==> daemon.log <==
Apr 27 19:06:37 remote systemd[1]: Started Session 303 of user user.

==> auth.log <==
Apr 27 19:06:37 remote sshd[35451]: pam_unix(sshd:session): session closed for user user
Apr 27 19:06:37 remote systemd-logind[543]: Session 303 logged out. Waiting for processes to exit.

It seems as if something has forgotten/lost the locale. I don’t seem to have much flexibility on the blink.sh side of things, and I’ve done what I can on the remote side to assuage its issues, seemingly to no avail.

I have no issue using mosh or ssh from another system. It just seems local to my blink.sh on my iPad.

Full range of MOSH ports open on the firewall.

And was working fine until today (April 27, 2024) until sometime into the 10AM hour (EDT).

Is there some background updating I am not aware of?

I may try removing and reinstalling the blink.sh app to see if it changes the current situation.

[wedge1020]

Hi @wedge1020

This seems unrelated to the Issue mentioned here. What version of Blink are you running?

Apologies. On scanning the issues I had found some of the symptoms similar, and didn’t want to further pollute the solution space.

Blink: v17.2.2.868 (App Store). Mar 11 2024

In any case, it looks like what has changed is on your server. Mosh needs to have a UTF-8 locale and it looks like your server is not providing it.

Only, I had made no such changes.

When I SSH from blink, it works.

When I MOSH/SSH from my MacBook, it works.

If you try running mosh-server new -s -c 256 -l LANG=en_US.UTF-8 on the server manually over SSH, do you get a valid server output?

I didn’t get to try this; first thing I did this morning was remove the blink app and reinstall it.

Everything seemed to resolve: I can now MOSH in as I had before.

Perhaps some local app data had gotten corrupted?

Certainly odd, as yes, it would otherwise have suggested something had happened to my en_US.UTF-8 locale on my remote machine, despite being able to demonstrate otherwise.