Skip to content

Debugging and Theory

Roger Pack edited this page Jan 25, 2022 · 5 revisions

Page Under Construction

Theory

If you have having problems getting mosh to run, knowing a little bit about how it works can be very helpful.

There are two primary components to mosh: mosh-client and mosh-server. A third component is the mosh wrapper script, which is what you will use to start mosh-client and mosh-server. The mosh wrapper script has the responsibility of using ssh to connect to a remote machine, start mosh-server on the remote machine, gather the necessary connection information (encryption key and remote IP), and then finally starting mosh-client.

The very first packet sent will come from mosh-client. It will be a UDP packet sent to the remote machine on a port specified by mosh-server. One mosh-server receives the UDP packet, it will know where to send the reply (since all incoming packets contain the IP address of the sender). If mosh-client never receives a reply from its initial packet, it will exit with the message "mosh did not make a successful connection to host:port"

Debugging

This page attempts to list some common issues you might see with mosh, and some steps to debug them.

General hangs on connection

When there are problems remotely running the mosh-server process, it can appear that mosh hangs, exits immediately without doing anything, or other non-descriptive failure modes.

Example exiting immediately:

/usr/local/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)

The first thing to try is to ssh to the remote machine in a manner similar to mosh:

ssh -S none -o ProxyCommand='mosh --fake-proxy -- %h %p' -n -tt user@server.com -- 'mosh-server new'

One common problem is a dot file (.cshrc, .bashrc, etc) that prints things when a new shell is launched. Since mosh expects to read something from stdout, this can get confuse mosh). Note: be sure to check both the client and the server!

Nothing received from server on UDP port 60001

This, and other similar messages, indicate some type of network problem that is preventing mosh-client and mosh-server from exchanging packets. One way to debug this is with the netcat utility (sometimes called nc). Another way is with tcpdump.

tcpdump

TODO: put something here

netcat

netcat is a utility to send and receive data over a network. The goal is to use it to try to confirm packets can traverse your network. In the examples below, svr is the server machine you're trying to connect to, and cli is the client machine.

On the server machine, pick an unused port number (TODO: how to use netstat to confirm port is not in use), and tell netcat to listen on that port number. -4 means use IPv4 only, -l mean to listen for connections, -u means to use UDP, and -v means to be verbose.

achin@svr$ nc -4 -u -l -v 60013
nc: listening on 0.0.0.0 60013 ...

(Note: some versions of netcat will require the -p argument to specify a port number)

At this point, netcat is now bound to port 60010, and will print out any data it receives.

On the client machine, use netcat to send a packet to the listening server.

achin@cli$ echo "hello world" | nc -4 -v -u --send-only srv 60013
nc: srv (10.10.1.1) 60013 [60013] open
nc: using datagram socket
achin@cli$

On the server side, if all went as expected, you should see "hello world" printed to the terminal:

achin@srv$ nc -4luv -p 60013
nc: listening on 0.0.0.0 60013 ...
nc: connect to 0.0.0.0 60013 from localhost (10.10.1.2) 54724 [54724]
nc: using datagram socket
hello world

If netcat running on the server didn't print "hello world", then you have confirmed that something is blocking those UDP packets. Check your firewalls

Page Under Construction