5 min readRealtimeProtocol designTesting
The refusal that named nothing
A change the room threw away stayed queued for ever: its author kept seeing an edit no one else had. Fixing it was less about the limits than about what a refusal has to carry.
While I was chasing a different bug in my whiteboard, I read a line in the client that I had written myself and never thought about again:
case "error":
return;
The room sends that message when it throws a change away. The client ignores it — not out of laziness, but because there is nothing it can do with it: the message says something was wrong, not which of your changes it was. So the change stayed in the queue as though it were still on its way. The author kept seeing an edit that nobody else would ever receive, and the connection pill kept saying "Live", because the connection was fine.
Which real edits fell into it
Not hypothetical ones. The room validates everything a client sends, and two of its limits are easy to cross by hand:
- 500 operations per change. A change carries one operation per shape it touches, so "select all and nudge" on a board of more than five hundred shapes is one change with more than five hundred operations. Refused, silently. Ten seconds of work, gone without a word.
- 512 KB per message. A change bigger than that does not even get read: the room closes the socket. The client then reconnects, re-sends the same queued change, and is closed again — a loop with no exit, from one paste of a large selection.
There was a third, quieter one: dragging a selection box sent the pointer and the selection on every frame it changed, and the room counts messages per socket — 120 a second, 240 in a burst. On a 165 Hz screen a marquee is 165 messages a second on its own, before anything else that window sends. Whatever the limiter then turned away came back as the same nameless error.
The rule the protocol was missing
Every answer about a change must carry the change's number. That is the whole design change; the rest follows from it.
The parser could not do it, because it parsed all-or-nothing: an invalid change came back as null,
and the number went with it. Now it reads the envelope first and the contents after, so a change it
cannot accept still comes back named:
case "change": {
const change = parseChange(message.change);
if (change) return {type: "change", change};
// The contents did not pass. The envelope may still say which change this was.
const n = isRecord(message.change) ? message.change.n : undefined;
return typeof n === "number" && Number.isSafeInteger(n) && n >= 1 ? {type: "bad-change", n} : null;
}
The room answers a bad-change with the reject it already had for a full board — which the client
already knew how to handle: take the change out of the queue, rebuild the view from what the server
confirmed, and tell the person. The rate limiter moved to after parsing for the same reason: a
message it turns away can now be turned away by number.
Not sending what will be refused
Surfacing the refusal is the safety net. A move of a thousand shapes should not need it — it is a perfectly ordinary thing to do to a board, and the limit exists to stop abuse, not work. So the client divides a change that carries more than the room takes into several that fit.
Where it divides turned out to matter more than that it does. The obvious place is when the edit is made — and it is the wrong one. A drag of six hundred shapes commits six hundred operations on every frame of the drag; those commits merge into one queued change while it is still unsent, which is what keeps a drag to one message every fifty milliseconds. Dividing there would replace that one change with two on every frame, break the merging, and send dozens of messages a second — straight into the rate limit I was fixing.
So the division happens at the last possible moment, on the way out:
while (this.sent < this.pending.length) {
const change = this.pending[this.sent]!;
if (!fits(change.ops)) {
if (change.ops.length > 1) {
this.divide(this.sent);
continue;
}
// One operation no room will ever take. Sent, it would cost the socket
// and come back on the next connection, for ever.
this.reject(this.sent, "Part of that edit was too big for the board, so it has been undone.");
continue;
}
this.transport({type: "change", change});
this.sent++;
}
Two details in that are load-bearing. Dividing renumbers every change queued behind it: none of them has been sent, so their numbers are still mine to hand out, and they have to stay in ascending order — the room applies a change of mine only if its number is higher than the last one it applied, so a lower number arriving later would be swallowed as a repeat. And a single operation that cannot fit is dropped with an explanation rather than sent: that is the end of the reconnect loop.
Numbers
The end-to-end tests now include a 1,200-shape edit made as one change, and a marquee drag measured for how much presence traffic it produces. Run against the old code, they say plainly what the bugs were:
| Test | Before | After |
|---|---|---|
| 1,200 shapes in one edit, seen by the other window | 5 of 1,205 | 1,205 of 1,205 |
| Presence messages during one marquee drag | 47 | within one per 50 ms |
| Refused change, from the author's side | queued for ever | rolled back, with a reason |
What I took from it
- An error that cannot be matched to what caused it is the same as no error at all. The client was right to ignore it; the message was wrong to exist in that shape.
- Limits belong to the protocol, so both ends have to know them. The room's
500 operationswas a server-side constant the client had never heard of. - A silent failure needs a way to be seen from the outside. This one could hide because the only thing that showed the queue was hidden while online — which was itself a deliberate decision, made for a good reason, in a different context.
The client half is in
src/sync/sync-client.ts,
the room's in worker/room.ts.
Keep reading
More notes
5 min read
Two windows, one client id
A shared whiteboard where one window's edits reached nobody. The protocol was right and its tests were green — the browser had copied something I had assumed belonged to a single tab.
- Realtime
- Browser storage
- Debugging
4 min read
The lag my test browser could not see
Headless Chrome draws at 60 fps whatever the monitor. On a 165 Hz screen that hid a 146 ms frame — and the CSS property behind it was one I had decided was free.
- Performance
- CSS
- Measuring
3 min read
AnimatePresence and the pages that went blank
An exit animation in the App Router left 30 of 40 pages at opacity: 0 — header and footer on screen, content invisible, and nothing a text search could find.
- Next.js
- Framer Motion
- Debugging
