Skip to content
Serhii Kuznetsov
All notes

5 min readRealtimeProtocol designTesting

The refusal that named nothing

A change the room threw away stayed queued for ever: its author kept seeing an edit no one else had. Fixing it was less about the limits than about what a refusal has to carry.

While I was chasing a different bug in my whiteboard, I read a line in the client that I had written myself and never thought about again:

case "error":
    return;

The room sends that message when it throws a change away. The client ignores it — not out of laziness, but because there is nothing it can do with it: the message says something was wrong, not which of your changes it was. So the change stayed in the queue as though it were still on its way. The author kept seeing an edit that nobody else would ever receive, and the connection pill kept saying "Live", because the connection was fine.

Which real edits fell into it

Not hypothetical ones. The room validates everything a client sends, and two of its limits are easy to cross by hand:

  • 500 operations per change. A change carries one operation per shape it touches, so "select all and nudge" on a board of more than five hundred shapes is one change with more than five hundred operations. Refused, silently. Ten seconds of work, gone without a word.
  • 512 KB per message. A change bigger than that does not even get read: the room closes the socket. The client then reconnects, re-sends the same queued change, and is closed again — a loop with no exit, from one paste of a large selection.

There was a third, quieter one: dragging a selection box sent the pointer and the selection on every frame it changed, and the room counts messages per socket — 120 a second, 240 in a burst. On a 165 Hz screen a marquee is 165 messages a second on its own, before anything else that window sends. Whatever the limiter then turned away came back as the same nameless error.

The rule the protocol was missing

Every answer about a change must carry the change's number. That is the whole design change; the rest follows from it.

The parser could not do it, because it parsed all-or-nothing: an invalid change came back as null, and the number went with it. Now it reads the envelope first and the contents after, so a change it cannot accept still comes back named:

case "change": {
    const change = parseChange(message.change);
    if (change) return {type: "change", change};
    // The contents did not pass. The envelope may still say which change this was.
    const n = isRecord(message.change) ? message.change.n : undefined;
    return typeof n === "number" && Number.isSafeInteger(n) && n >= 1 ? {type: "bad-change", n} : null;
}

The room answers a bad-change with the reject it already had for a full board — which the client already knew how to handle: take the change out of the queue, rebuild the view from what the server confirmed, and tell the person. The rate limiter moved to after parsing for the same reason: a message it turns away can now be turned away by number.

Not sending what will be refused

Surfacing the refusal is the safety net. A move of a thousand shapes should not need it — it is a perfectly ordinary thing to do to a board, and the limit exists to stop abuse, not work. So the client divides a change that carries more than the room takes into several that fit.

Where it divides turned out to matter more than that it does. The obvious place is when the edit is made — and it is the wrong one. A drag of six hundred shapes commits six hundred operations on every frame of the drag; those commits merge into one queued change while it is still unsent, which is what keeps a drag to one message every fifty milliseconds. Dividing there would replace that one change with two on every frame, break the merging, and send dozens of messages a second — straight into the rate limit I was fixing.

So the division happens at the last possible moment, on the way out:

while (this.sent < this.pending.length) {
    const change = this.pending[this.sent]!;
    if (!fits(change.ops)) {
        if (change.ops.length > 1) {
            this.divide(this.sent);
            continue;
        }
        // One operation no room will ever take. Sent, it would cost the socket
        // and come back on the next connection, for ever.
        this.reject(this.sent, "Part of that edit was too big for the board, so it has been undone.");
        continue;
    }
    this.transport({type: "change", change});
    this.sent++;
}

Two details in that are load-bearing. Dividing renumbers every change queued behind it: none of them has been sent, so their numbers are still mine to hand out, and they have to stay in ascending order — the room applies a change of mine only if its number is higher than the last one it applied, so a lower number arriving later would be swallowed as a repeat. And a single operation that cannot fit is dropped with an explanation rather than sent: that is the end of the reconnect loop.

Numbers

The end-to-end tests now include a 1,200-shape edit made as one change, and a marquee drag measured for how much presence traffic it produces. Run against the old code, they say plainly what the bugs were:

TestBeforeAfter
1,200 shapes in one edit, seen by the other window5 of 1,2051,205 of 1,205
Presence messages during one marquee drag47within one per 50 ms
Refused change, from the author's sidequeued for everrolled back, with a reason

What I took from it

  • An error that cannot be matched to what caused it is the same as no error at all. The client was right to ignore it; the message was wrong to exist in that shape.
  • Limits belong to the protocol, so both ends have to know them. The room's 500 operations was a server-side constant the client had never heard of.
  • A silent failure needs a way to be seen from the outside. This one could hide because the only thing that showed the queue was hidden while online — which was itself a deliberate decision, made for a good reason, in a different context.

The client half is in src/sync/sync-client.ts, the room's in worker/room.ts.

Keep reading

More notes