“Hmm, this music is kinda meh, I wonder what comes next?”

Surrounded by the crowd under the stage of the RIMAC lawn, I reached inside my pocket and unlocked my phone. I knew by gut feeling that this was probably now going to work, but I tried it anyway. Firefox, Google, …

…

Welp, I knew it. Aside from not having Wi-Fi in the Arena, the data speed also sucks. Hmm, I wonder how the packet loss situation is looking? I open Termux, my Linux command line on Android, and ping the Sun God website:

Wow, 87% with a max ping time of 22 seconds? That’s atrocious. There’s even a duplicate! I had never seen that before. But terrible internet can’t stop me! It’s Mosh time!!!

A little history

My time with Mosh goes way back.

I have a backup server across the globe that I self-hosted with a RPi 3B+ that I bought the final year of high school. (My rationale for taking offsite backup to the extreme is that even if the entire Americas is suddenly erased somehow, my entire data collection would still be intact. Don’t ask me why I care about my data when I myself is in even deeper trouble! We don’t worry about minor non-technical details here.) I let my laptop do the backups automatically during the night and all is well. Except I am my own sysadmin, so I needed to do some maintenance work every so often once I get an email about another vulnerability being disclosed in OpenSSH and that I have to update my server now to avoid it being compromised.

I usually don’t mind putting in a couple quick commands, except that my little RPi happy runs halfway across the globe, and that comes with the problem of having an atrocious round-trip time (RTT) of 150 ms. This means that I can’t even see my typos until 150 ms in the future, after I frantically delete and retype, only to find out that I missed a single character in the middle and have to arrow over to fix it (and then proceed to arrow over one more character than I need. Ugh!!)

That’s not all. An even worse problem is that somewhere along the entire route around the globe and back something often gets wonky and all my packets get lost for minutes at a time. This means that I could never tell if my connection just dropped, or that I just needed to wait a bit. It was infuriating. So when I discovered Mosh, I knew that it’s the thing that I have always wanted.

Mosh

Mosh, or Mobile shell, is, according its website:

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.
Mosh is a replacement for interactive SSH terminals. It’s more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.
Mosh is free software, available for GNU/Linux, BSD, macOS, Solaris, Android, Chrome, and iOS.

In simple terms, it synchronizes the state of the terminal instead of taking the SSH (which works over TCP) approach of just piping the two character streams over the network and allowing your local terminal to figure out what it should look like. This allows it to use UDP instead of TCP to work under terrible networking environments and to send as little data as possible, since it doesn’t need to send all the terminal history when it is no longer visible. It works with the worst network connections, once even allowing one of its users to escape from a trapped elevator.

This allows it to do a lot more cool tricks, too! Since UDP doesn’t work based on connections, your SSH session doesn’t drop even when you roam between different connections. This became really useful to me as I often connect to my servers when going around campus, connecting and disconnecting to the various wireless networks all the time. I could even put my laptop to sleep on one corner of campus, get to the other side, wake it up, and continue my SSH session like it has always been connected!

It also doesn’t have a large backing buffer that the TCP protocol needs to retransmit upon a packet loss, and so your Control+C works instantaneously when you inadvertently cat /dev/urandom. What’s more, it can automatically predict my keystrokes as I type out my command, so I can see them instantaneously, as if there is zero latency between me and the server! It will underline the predicted portion and will retract them if the server reports an inconsistency, but it’s seldom wrong.

So…

$ mosh scc-evans
y5jing@evans:~$

I’m in!

But how exactly does being able to SSH under rough connections help me at the Sun God Festival?

w3m

Who says you need to use a GUI to access the interwebs?

$ apt show w3m

…And…

Ta-da! 🎉

But was it just a fluke?

I tried to open the Sun God website with my phone’s browser instead.

…Yeah, I don’t think so.

I love Mosh!

(Of course, Mosh isn’t magic and can’t handle completely broken networks either:)

But at least it reconnects as soon as it’s back up.

But how does it work?

Check out the “How Mosh works” section on its website! I can’t hope to explain all of it here. It has an in-depth explanation, and if you want the juicy details there’s a paper and everything.