How it works

Remora has three kinds of program and one rule between them: applications and viewers never talk to each other, only to the server. This page follows what travels where, and why it is built that way.

Three parts#

Applications are the programs with something to show: a game, a hardware monitor, a media player, a script. Each one connects to the server on the same machine and hands it two things: a template that says what its panel looks like, and a stream of changes to its state.

The server runs on the PC. It keeps every connected application's template and latest state, tells viewers which applications are running, and relays between the two sides. It exists as a headless program and as a desktop app with a window and a tray icon; they are the same server.

Viewers are the screens: the Android app, or the same viewer running in a browser. A viewer lists the servers it has paired with and the applications on each, draws the one you pick, and sends your taps back.

Because everything goes through the server, an application never needs to know whether a phone is there, how many there are, or what kind. And a phone never needs to know how to reach a game.

One change, there and back#

In words:

1. The application opens a WebSocket to the server and says hello. The server answers welcome.
2. The application sends register with its template. The server checks the template and answers ready.
3. On the phone, someone opens that application. The viewer sends watch; the server sends back a snapshot of the application's whole state, and tells the application it now has one watcher.
4. Something changes. The application sends a state message with a patch. The server applies it to its copy and passes the difference on to the viewer, which redraws only what depends on it.
5. The user presses a button. The viewer sends an action; the server checks it and passes it to the application.
6. The application does whatever the button means, and publishes the result as another state patch.

Every message is a single JSON object in a WebSocket text frame. The only binary frames are file uploads. The full list is in the message reference.

State documents and patches#

Each application has one state document: an ordinary JSON object that it owns and the server holds a copy of.

{
  "hero": { "name": "Vasha", "hp": 412, "hpMax": 500 },
  "potions": { "health": 3 },
  "mode": "exploring"
}

The application never sends drawing instructions. It changes this document by sending a patch, which is the part of the document that is different now:

{ "type": "state", "patch": { "hero": { "hp": 371 } } }

A patch is a JSON Merge Patch: objects merge key by key, anything else replaces what was there, null removes a key, and an array is replaced whole. After the patch above, hero.hp is 371 and everything else is untouched.

Three consequences of the server holding the document:

• A viewer can arrive at any time. When it starts watching, the server sends a snapshot of the whole document, then patches from then on. The application does not have to notice or resend anything.
• Only differences travel. The server compares each patch with what it holds and forwards only the keys whose value actually changed. An application that sends its entire state every tick costs the phone nothing extra.
• A slow phone does not build a queue. The server may merge consecutive patches into one, so a viewer that falls behind gets the latest state, not a backlog.

The server also tells each application how many viewers are watching it. While that is zero, an application can stop publishing, or stop doing the work of gathering its state at all.

State lives only as long as the connection. When an application disconnects, its state is discarded and it leaves every viewer's list.

Templates and bindings#

A template says what an application looks like on a viewer. It is a JSON manifest, plus any images, sounds and fonts it uses. The manifest declares three things:

• the state it expects, with starting values;
• components: the things drawn, from a fixed set of eight (text, image, dial, bar, list, button, toggle, slider);
• scenarios: named arrangements of those components, each with a layout for landscape, portrait, or both.

Components are connected to state by bindings. A binding names a path in the state document, and the component's property takes whatever value is there.

{ "hp": {
    "type": "bar",
    "value": { "bind": "hero.hp" },
    "max": { "bind": "hero.hpMax" }
} }

A binding can also format text ("{hero.hp} / {hero.hpMax}"), or act as a condition ({ "bind": "potions.health", "gt": 0 }) that decides whether a button is enabled, whether a component is visible, or whether a different style applies. Even the choice of scenario is a binding: the application switches the whole layout from exploring to a boss fight by changing one value in state.

That is all a binding can do. There is no arithmetic, no scripting, no way to combine conditions. An application that needs a derived value works it out and publishes it. The schema reference has the exact rules.

Templates with no asset files travel inline, inside register. Templates with assets are packed as a ZIP, called a .remora archive, and identified by the SHA-256 of its bytes. An application registers by that hash, and uploads the archive only if the server asks for it. Viewers fetch the manifest and assets from the server over HTTPS and can cache them indefinitely, because a different template is a different hash.

Actions coming back#

Buttons, toggles, sliders, and list rows that have an action are interactive. When the user touches one, the viewer sends the server an action: a name the template declared, and sometimes a value.

{ "type": "action", "instance": "9fQ2mB7cX1vLr8Td", "action": "drinkPotion", "value": "health" }

The server forwards an action only if all of these hold:

• the viewer is watching that application;
• the application's template declares an interactive component with that action name;
• the value has the type that component produces (a boolean from a toggle, a number in range from a slider).

So a viewer cannot send an application anything its own template did not offer.

An action is a request, not a change. A toggle on the phone does not flip a value in state; it asks the application to. The application decides, changes its own state, and publishes it, and that is what the toggle then shows. If the application does nothing, the control goes back to the value in state within a second. The application's state is always the truth.

Feedback does not wait for any of this. A template can give a component an animation, a sound and a haptic tap, and the viewer plays them the instant it is touched.

Pairing and security#

A phone on your network should not be able to read your applications or press their buttons merely because it is on your network. A viewer has to be paired first, and pairing proves one thing: that its user can see the server's screen.

Pairing codes#

The server shows a pairing code in two forms at once: a QR code, and a short code of eight characters for typing, such as 7K2M-9QXA. The QR code also carries the server's address and the pin of its key, so scanning it is all a phone needs.

A pairing code can be used once. It expires within ten minutes of being shown, and the short code stops working after five wrong attempts.

A successful pairing gives the phone a device ID and a long random token. The server keeps only a hash of the token. Its operator can see the list of paired devices and revoke any of them, which closes that device's connections at once.

Pinned TLS#

Viewers connect over TLS. The server generates its own self-signed certificate the first time it runs, and there is no certificate authority involved. Instead the viewer remembers a pin, a fingerprint of the server's public key, at the moment of pairing: from the QR code, or from the certificate the server presents when the code is typed in. From then on it refuses to talk to anything claiming to be that server with a different key.

The specification asks a viewer pairing by typed code to show the first eight characters of the pin, so that you can compare them with what the server shows.

A browser cannot pin a certificate. For that case a server can also offer the viewer endpoint without TLS, when its operator explicitly turns that on. Tokens then cross the network in the clear, and the server says so where the option is enabled. It is there so that a desktop browser can be a viewer.

What a paired viewer can do#

A paired viewer can read the state of every application registered with that server, and trigger any action their templates declare. That is the whole of it. It cannot reach anything a template does not declare, cannot make the server or an application fetch a URL, and cannot upload anything.

Applications are local#

The application endpoint listens on the loopback address only: a program has to be running on the same machine to register. Any such program may; it is already inside your own session. The server refuses an application connection that carries an Origin header, so a web page open in a browser on that machine cannot register itself.

Applications on other machines are out of scope for version 1.0. Each PC runs its own server, and a viewer can be paired with several.

Templates are data#

A template cannot run code, cannot name a network address, and cannot reach outside its own package and its own state. The viewer draws it with a fixed set of components and loads files from nowhere but the server it is connected to. An SVG image is drawn as a static picture; scripts and outside references inside it are not honoured.

This is why a template from someone you do not know is safe to display. It is also why the specification lets the person running a server install a different template for an application, a reskin, without touching the application: both templates only read the same state paths and send the same action names.

Finding the server#

A server advertises itself on the local network with mDNS, as the service type _remora._tcp. The Android app uses that to show which of your paired servers are online, and to keep track of one whose address has changed.

Nothing depends on it. The pairing QR code carries the address, and an address can always be typed in.

What is on the wire#

For reference, the whole surface:

Both WebSocket endpoints speak the subprotocol remora.v1.json, and both begin with hello and welcome, where the two sides agree a protocol version. Within a major version, changes are additive, and receivers ignore what they do not recognise.

To see all of this from an application's side, with code you can run, go to the tutorial. For every field and message, see the schema reference.