Reaching the relay through an SSH tunnel

The relay can open an SSH connection to a server of yours and ask that server to forward a port back to the relay. You then reach the relay at the server's address, and nothing at home listens for the internet. The relay starts, watches and restarts the ssh process itself.

Choose this when you already have a server with a public address and SSH access. If you have no server, the built-in Cloudflare Tunnel needs none.

What you need #

• The relay host has an ssh program on its PATH (or you name another with command, see below).
• An SSH login on the server that works with a key and no prompt. The relay runs ssh in batch mode, so it cannot type a password or a key passphrase. A key with a passphrase works only if an SSH agent holds it. Test it first from the relay host: ssh relay@relay.example.com must log in without asking anything.
• A free port on the server for the tunnel (remotePort).
• Something on the server that serves that port over HTTPS: a web server or reverse proxy such as Caddy or nginx. The tunnel carries plain HTTP, so without it your token and your conversations cross the internet unencrypted once the port is reachable. See Running the relay behind a reverse proxy.

The relay needs no special sshd setting. With remoteBind at loopback it asks for the forward without a bind address, so the server's own GatewayPorts setting decides where the port opens. At its default, no, that is the server itself only, and the reverse proxy is the only thing that faces the internet. With public the relay asks for all interfaces (*:), and that request takes effect only if the server's sshd allows it (GatewayPorts set to clientspecified or yes).

Set it up #

Add an sshTunnel block to config.json (see Where the file is) and restart the relay with oar restart.

"sshTunnel": {
  "mode": "managed",
  "required": false,
  "user": "relay",
  "host": "relay.example.com",
  "remotePort": 4444,
  "remoteBind": "loopback",
  "identityFile": "~/.ssh/id_ed25519"
}

On the server, send HTTPS to the tunnel port. With Caddy:

relay.example.com {
    reverse_proxy localhost:4444
}

Open https://relay.example.com/ in a browser and sign in with your access token. The relay's start-up banner prints a Remote: line and oar url prints a Tunnel URL: line with a QR code. Both are https:// plus the host value, with no port and no check that the tunnel is up, so they are right only when your proxy serves HTTPS on the standard port.

The sshTunnel keys #

All keys sit in the sshTunnel block. The table agrees with config.json, which has the same list.

The old form "enabled": true still works and means "mode": "managed", but only when mode is not set to disabled or managed.

The relay forwards the server's remotePort to its own port on 127.0.0.1 (3333 unless you changed port). With public the forward is requested as *:remotePort, otherwise as remotePort.

These settings are fixed and have no keys: the relay runs ssh -N with ServerAliveInterval=30, ServerAliveCountMax=3, BatchMode=yes, StrictHostKeyChecking=accept-new and ExitOnForwardFailure=yes. ssh gives up on a connection after three unanswered checks, 30 seconds apart.

If a required value is missing (user, host or a valid remotePort in managed mode), the relay starts as usual, logs which one, and leaves the tunnel off.

How it behaves #

Start. The relay starts the tunnel once it is listening. It counts the tunnel as connected when ssh has run for 1.2 seconds and the server has not refused the forward. The log then says Tunnel up to user@host remote port N.

Reconnecting. When ssh exits, the relay starts it again after a wait that grows: 5, 10, 20, 40 and then 60 seconds, and stays at 60. The count starts over once a connection that lasted more than 30 seconds drops. Stopping the relay stops the tunnel and the retries.

Host keys. A server the relay host has never seen is accepted and remembered on the first connection. After that, a changed host key makes ssh refuse to connect until you remove the old entry from known_hosts.

Required #

With required: true the queue stays paused from start-up until the tunnel is connected, and again whenever it drops. Messages you send wait in the queue and run once the tunnel is up. Use it when the relay is useless to you without the tunnel. A tunnel that cannot start because of a missing value keeps the queue paused too.

When the remote port is taken #

If the server says the port cannot be bound, the usual cause is a leftover session of an earlier tunnel that the server has not noticed is dead. Unless autoReclaimPort is false, the relay then connects to the server a second time and runs a cleanup command there.

The built-in cleanup does this, in order:

1. It checks whether the port is in use (with ss, or else by reading the kernel's list of listening sockets). If not, it stops and reports success.
2. It finds the processes listening on the port (with lsof, or else fuser, whichever exists), sends them a termination signal (kill) and waits one second. A process your login may not signal stays. If the port is free now, it reports success.
3. Only with reclaimStaleSshSessions: true, and only if the port is still in use: it sends a termination signal to every sshd session of your user that has no child processes, then waits one second. These are the sessions an ssh -N -R leaves behind. Sessions that run a shell, an editor's remote server or any other program have children and are skipped, as is the session running the cleanup itself.
4. If the port is still in use, the cleanup exits with code 3, which the relay logs as is still held.

With remoteCleanupCommand set, your command runs instead of all of that, and reclaimStaleSshSessions has no effect. The relay runs it as sh -lc '<command>' over a second ssh login with the same key, in batch mode. It must exit with 0 when the port is free and with another code when it is still held.

After a cleanup that freed the port, the relay retries after one second, up to three times in a row. After a failed cleanup, or with autoReclaimPort off, it uses the normal wait above.

Seeing the tunnel's state #

The SSH tunnel is not shown in the places the Cloudflare tunnel is: oar status and oar doctor only report the Cloudflare tunnel, and the status dot does not turn amber for an SSH tunnel (see The status dot). Look here instead:

• The relay's log. Every line starts with [ssh-tunnel]. It shows the ssh command line it runs, what ssh prints, Tunnel up to ..., Reconnecting in Ns (attempt N)... and the reclaim steps. Where the log is depends on how the relay runs; see Files and folders.
• The start-up banner, with a Remote: line and a Tunnel: mode=loopback (or public) line when the managed tunnel is configured. They show the configuration, not whether the tunnel is up. oar url prints the same address as Tunnel URL:.
• /api/status, which has an sshTunnel object with connected, connectedSince, reconnectAttempts, lastError, blocking, required, mode, valid, host, remotePort and remoteBindMode. Fetch it with your token.

lastError is exit:<code> after ssh ended, remote-forward-bind-failed:<code> after the server refused the port, invalid-config or the missing-value message when the block is incomplete, and empty while the tunnel is up.

Security #

Once the tunnel is up, the relay can be reached from the internet, and the access token is all that stands between a stranger and your agents, which run commands on your machine. So:

• Keep remoteBind at loopback and put HTTPS in front on the server. With public, anyone who can reach the server's tunnel port reaches the relay over plain HTTP.
• The access token is the only lock. See The access token and who can reach the relay and What OAR exposes.
• The relay refuses requests to its internal session-worker paths when they carry the Cloudflare marker header cf-ray or a header you list in tunnelMarkerHeaders. A plain reverse proxy adds none of these, so list the header your proxy adds, if it adds one you would like blocked. See config.json.

Troubleshooting #

Look at the [ssh-tunnel] lines in the log first. Lines starting stderr: are what ssh itself printed.