Troubleshooting

Start here when something does not work. The first sections are the checks that apply to almost everything. The tables after them lead from what you see to the page that has the fix. Most of those pages end with their own troubleshooting section; this page does not repeat them.

First steps #

On the relay's machine, in a terminal:

oar status
oar doctor

1. oar status says whether the relay answers. Read its first lines against the table below.
2. oar doctor prints the install, the config and the provider CLIs it finds. It changes nothing and always exits 0, so read the lines.
3. Look at the browser. The status dot and the banners tell whether the problem is between the browser and the relay or behind the relay.
4. Read the log that belongs to the problem (Logs).
5. Restart the smallest thing that could help (Restarting).

Both commands are described on The oar command.

Reading oar status #

Exit code 3 means the relay is not running, the port is held by something else, or there is no config. Exit code 0 means the relay runs.

Reading oar doctor #

Doctor prints the state folder, not the log folder, and it does not say whether the relay runs: that is oar status. Where the logs are is on Files and folders.

Logs #

All log files, their rotation and the folder in a git checkout are on Files and folders.

In the browser #

• The status dot at the top of the conversation list: grey is Web relay unreachable (the browser cannot reach the relay), green is Web relay reachable, amber is the relay reached through a connected Cloudflare tunnel. The tooltip says more, for example how many workers are busy. The status dot
• The banner CLI is offline — messages will queue until it reconnects and CLI offline in the sidebar mean no session worker has reported in for 10 seconds. On an idle relay this is normal: send a message and the banner clears. If it stays, read the worker log. When the agent is not reachable

Restarting #

Symptoms #

Installing and starting #

Signing in and reaching the relay #

A conversation does not start or the agent does not answer #

Each provider page ends with the failures that provider can produce. A failed reply usually names its cause and often carries a button that fixes it.

A turn is stuck, fails or restarts #

Questions and approvals #

Agents working with other sessions and relays #

Files, previews and sharing #

Notifications and the installed app #

Updates #

The relay ran out of memory Added in 0.9.19 #

The relay starts with a fixed heap limit: 60 % of the machine's RAM, in 64 MB steps, at least 2048 MB and at most 8192 MB, and never below Node.js's own default, but never above the memory there is. In a container with a memory limit, that limit counts instead of the RAM. On a machine with 4 GB that is about 2.3 GB. Each start logs the limit and where it came from:

[relay] heap limit 2368 MB (default: 60% of 3.9 GB RAM)

At start and every 10 minutes the log gets a line such as [relay] memory rss=812 heapUsed=640 heapLimit=2560, and one extra [relay] memory high: … line when the heap in use passes 80 % of the limit.

When the relay runs out of memory, it starts again by itself, as after any crash: up to 2 times in a row, and the 3rd crash in a row stops it with exit code 1, after which the Linux systemd service starts it again. A run of 10 minutes or more starts the count afresh. Before 0.9.19 a relay that ran out of memory stayed down. After an update from an earlier version, the heap limit and this relaunch work only once oar itself has been stopped and started again (Upgrade notes for 0.9.19).

To set the limit yourself, put heapLimitMb (whole MB, at least 512) in config.json and restart the relay. A --max-old-space-size in NODE_OPTIONS, or on the node command line that starts server.js, wins over heapLimitMb. Unlike heapLimitMb, a NODE_OPTIONS set in a shell is lost when something else restarts the relay.

The heap snapshot #

When the heap nears its limit, Node.js writes one heap snapshot (Heap.<date>.<time>.<pid>.….heapsnapshot) to the diagnostics folder in the relay's data folder (Files and folders). The relay creates that folder and logs its path at start; on Linux and macOS only you can open it.

• Disk: a snapshot is about as large as the heap in use, so up to the limit. When the drive of the data folder has less than twice the heap limit free, the relay starts without a snapshot and logs why.
• Memory: writing it can take up to about twice the heap in use, and the relay does not answer while it writes. On a small machine writing it can fail or get the process ended; the relay then ends as it would have without it.
• Reading it: copy it off the machine, open it in Chrome DevTools (Memory → Load) and sort by Retained size to see what holds the memory. A snapshot from an earlier, healthy run shows what grew.

Reporting a problem #

If nothing above helps, file an issue at github.com/oar-sh/oar/issues. Include:

• the output of oar --version;
• the output of oar doctor, and of oar status;
• what you did, what you expected and what happened, with the exact message;
• the relevant lines of the log that fits the problem (Logs), not the whole file.