Images, media and formulas

Replies in OAR are more than text. An agent can show you a screenshot or play a clip it made, formulas come out typeset, code blocks have a Copy button, and you can mark up a screenshot before you send it back. This page covers each of these.

The page that draws a conversation does not carry its own Markdown, formula and code highlighting libraries. Your browser loads them from cdn.jsdelivr.net (Markdown and KaTeX) and cdnjs.cloudflare.com (highlight.js), so it needs to reach those hosts. Without them, replies show as plain text, formulas stay as source, and code blocks are not highlighted.

Media in an agent's reply Added in 0.9.2 #

An agent can put an image, a video or an audio file in its reply by writing a Markdown image whose target is the file's absolute path on the relay's machine:

![the failing page](/home/dev/projects/shop/shots/checkout.png)
![demo clip](C:\Users\dev\Videos\demo.mp4)

You do not upload anything and the agent needs no extra tool. The relay reads the file when your browser asks for it.

Whether an image or a player appears depends on the file name's ending, not on what the agent wrote as the description. The description becomes the tooltip of a player.

Click an image to open it in the viewer, with the other media of the conversation to swipe through. Players keep their own controls. Write the path the way the relay's machine does: /home/… on Linux and macOS, C:\… on Windows.

What can be shown #

• Only for you. The relay serves a file only to a request that carries its access token, which a browser signed in to the relay does. A shared read-only link has no token, so it does not show the media, and a click on an image does nothing there.
• Folders. There is no list of allowed folders on Linux and macOS: any absolute path to a file that the user running the relay can read is served, inside the workspace or not, and .. segments are resolved. On Windows the file must be on one of the drives the relay lists under 📁 Browse files. A reply cannot read a file for the agent, but it makes your browser ask for one, so anything the relay's user can read can appear on your screen.
• Missing file, or a folder. The relay refuses the request, and the picture or player does not load.
• Safe types. HTML and XML files are served as plain text so that they cannot run in the relay's page. SVG images are shown, but cannot run script.
• Size. OAR sets no size limit. Video and audio are streamed in parts, so a player can seek.
• No raw HTML. HTML an agent writes in a reply appears as text, not as a page.

Nothing is copied: if the file changes on disk, the next load shows the new version.

By provider #

Agents have to know the trick. OAR tells them in the instructions it gives the agent, once per provider. That is the only difference: what you see is the same.

OpenAI Image chats have no agent, so there is nothing to tell.

See Providers at a glance for the rest of the provider differences, and Not tools for why this is an instruction and not a tool.

Images you send #

An image you attach shows as a thumbnail in your message. Click it to open the viewer. How images are attached, shrunk before they upload and passed to each provider is on Composer.

Annotating an image Added in 0.9.2 #

Use the highlighter to point at the part of a screenshot you mean.

• Before you send. Click an image in the composer. Images you can mark up carry a 🖍️ badge. GIF and SVG images cannot be marked up.
• From the viewer. Open an image, from a message or from 📁 Browse files, and click 🖍️ Annotate. The button is not there for GIF and SVG, or in a shared view.

An editor titled Annotate image opens over the page, showing the image. Its buttons carry icons only; the names below are their accessible labels.

Strokes are translucent, so text under a mark stays readable. A tap leaves a dot.

Touch. One finger draws. A second finger switches to pinch zoom and drops the stroke you had started. Double-tap to zoom in and double-tap again to fit the image. Mouse. Left-drag draws, the wheel zooms, and a right-button or middle-button drag pans. A double-click zooms.

Saving #

Accept flattens the strokes into a PNG at the image's full size, named like screenshot-annotated.png (the original name, then -annotated.png). The original is never changed. The copy then goes through the same steps as any attachment, so a large one is shrunk before it uploads.

• Started from the composer, the annotated copy takes the place of that attachment, and is uploaded.
• Started from the viewer, the annotated copy is added to the composer as a new attachment, and is uploaded, for your next message. The message already sent stays as it was. If the composer is full, the editor stays open so you can free a slot and accept again.

If the image cannot be opened or saved, a short notice says so (Could not open the annotation editor or Could not save annotations) and the editor stays as it is.

Generated images #

In an OpenAI Image chat the replies are images the model generated. They are described on OpenAI and OpenAI Image.

Formulas #

Mathematics and chemistry are typeset with KaTeX, in replies and in thoughts. Chemical formulas use \ce{…}, from KaTeX's mhchem extension, which the page loads with KaTeX.

A display formula may span several lines. $$ or \[ alone on a line opens one, and the matching $$ or \] closes it. The lines between are joined into one formula. A bare [ on a line of its own, closed by a ] on a line of its own, is read as display math only when the lines between use a TeX command such as \frac, \sum or \ce.

A formula in plain parentheses is also typeset when it clearly is TeX: it is at most 240 characters, is not a web address, and holds a command such as \frac or a script such as x^2 or x_{n}.

Dollar signs and prices #

A single $ opens a formula only when all of this holds:

• the character right after it is not a space;
• the next $ closes it, with a character that is not a space right before it;
• no digit follows the closing $. A digit is allowed when the text between looks like TeX (it holds one of \ ^ _ { } =), so $x^2$3 is a formula.

So it cost $5 and $9 and $4.20, the earlier one $3.80 stay plain text, and $x^2$ is a formula. A $ after a backslash is a plain dollar sign. Dollar signs in code spans and code blocks are never read as formulas.

Invalid input #

A formula KaTeX cannot read does not break the reply: the source text appears in the error color in place of the formula.

Code blocks #

A fenced block with a language name is highlighted. Without one, highlighting guesses the language. Every block has a Copy button in its top right corner. It copies the code only, and the label changes to ✓ Copied for a moment, or to Copy failed. Long lines are not wrapped: the block scrolls sideways.

Links #

• A link to a web address opens in a new tab. The page that opens cannot reach back to OAR.
• A file name or path written in plain text in a reply (not in a code span or a link), with an ending of a known file type such as src/app.js, becomes a link that opens the file in the app's viewer instead of a new tab.
• Only http, https and mailto links and addresses inside OAR are kept. Other kinds are removed.

Other Markdown #

Replies use standard Markdown: headings, bold and italic, strike-through, quotes, bulleted and numbered lists, tables, horizontal rules and inline code. Tables fill the width of the message. HTML in a reply shows as text.