Adding disconnect detection and auto-reconnect to HydroNext's WebSocket

Related project

Background

I put HydroNext behind a layer of Tencent EdgeOne. Everything tied to the client—contest announcements, submission records, judging progress—relies on WebSocket pushes from the server.

The nature of OJ contests means these services have to stay connected for the whole event, and announcements are the clearest case: an admin posts something mid-contest, and it should pop up immediately on every registered contestant's page. But contests run one or two hours as a matter of course, and the long ones can exceed ten hours, while EdgeOne's idle reclamation threshold for WebSocket is 300 seconds. Our system therefore has to keep heartbeats healthy over long connections and also be able to recover from failures.

The system already had heartbeats: farmworker checks every 40 seconds, sends a ping if no frame arrives for 30 seconds, and terminates the connection at terminate seconds of silence. Frame intervals on both sides are usually far below 300 seconds, so keepalive was more than sufficient.

And yet, the moment I hooked up EdgeOne, WebSocket disconnects showed up right away, and nothing that needed a live client connection could be pushed in real time.

So if the problem isn't the "heartbeat," where is it? Traffic on the path keeps intermediate devices from reclaiming the connection, but that doesn't prove the connection still works. If it breaks at an intermediate layer, the browser gets neither close nor error, and readyState stays true forever. The server thinks the connection object is still there, but messages just queue up and are never received. One such occurrence in a ten-hour contest and that machine misses every later announcement, until someone refreshes and the connection reopens—and on route-based pages where refreshing is disabled, that's fatal: it blocks indefinitely.

This post is about the part beyond the "heartbeat": how to know a connection is already dead, and how to replace it.

The five client-facing long connections

All of HydroNext's real-time features use WebSocket.

Judging itself is asynchronous, for instance: a submission gets compiled, checked against test cases, and results stream back one by one—a long-lived WS connection is clearly better than polling.

After the design work, there are five client-facing long connections:

These connections share a few traits that every design decision has to reckon with:

• Extremely long lifetimes. A contest starts at one hour and can run past ten.
• Silence most of the time. The announcement connection may stay quiet for its entire life.
• They can't drop. If the timer connection drops, contest time freezes; if the announcement connection drops, admin notices never arrive.

The old code assumed an event would fire when the connection dropped—but when it fails unexpectedly, no such event exists.

After adding EdgeOne

EdgeOne doesn't enable WebSocket by default, and its maximum resource-retention policy reclaims connections after 300 seconds idle. For long connections that are mostly silent but must stay up for a long time, that's a fatal threat.

All five connections use the same path, so they're all subject to the same reclamation policy. There's one connection between browser and CDN and another between CDN and server; if either segment breaks, all the browser sees is "no new messages."

What the old heartbeat actually did

When I saw the idle timeout, my first instinct was also to add a heartbeat. Before touching anything I dug through the framework layer and found one already existed:

This timer wakes up every 40 seconds. If no frame has arrived for over 30 seconds, it sends a ping; if still nothing after 80 seconds, it just terminates the connection. The client, on receiving ping, replies with pong:

On a healthy path, frames flow each way at intervals under 40 seconds—nowhere near 300—so from the CDN's perspective the connection is always active.

In other words, adding more heartbeats is not a viable way to keep the connection alive once the CDN has closed it. Which means the problem was never the heartbeat at all. So where is it?

Keepalive vs. liveness detection

The word "heartbeat" really covers two separate things.

One is keepalive: as long as the CDN sees traffic on the path, it won't reclaim the connection.

The other is the key to this whole improvement: liveness detection. It's about confirming whether the connection can still send and receive right now; if not, drop it and establish a fresh one.

Hydro only implements the keepalive half. The server sends ping, the client replies with pong. The client never probes on its own, never records the last time it received a message, and has no connection timeout. The only signal it uses to declare a connection dead is onclose.

At this point the two sides are far from equal:

“

The server terminates the connection after 80 seconds without a frameterminate, so it can tell when a client closed—but the client has no way at all to detect that the server or the intermediate link closed.

Meanwhile, plenty of situations can silently kill the connection: a phone switching between Wi-Fi and cellular, the browser being suspended in the background, a laptop lid closing, the server restarting while the client's connection object lingers, a CDN node switch. In all of these the disconnect goes unnoticed, and nothing can be sent.

In these cases the TCP connection is half-open: the browser believes OPEN while the peer has long since vanished. No FIN, no RST, no event of any kind.close might arrive minutes later, or never. Hydro's recovery logic depends entirely on that event—and it may never recover at all.

Seven problems in Hydro's implementation

Pull out the pre-change use-websocket.ts and analyze it, and the issues cluster together.

1. Reconnection lives only inside onclose

onclose never fires, so no reconnect happens. With half-open connections, this is the worst offender.

2. onerror just calls close()

The code looks like it's handling errors, but it really just calls ws.close(), and reconnecting still depends on onclose. Once the path is broken, close() waits for a callback that never comes.

3. No handshake timeout

This line is meant to prevent duplicate connections, but it has no timeout. While the handshake is stuck in CONNECTING—weak network, hijacking proxy, a misbehaving CDN node—this socket holds the slot, and every later reconnect attempt gets blocked by it.

4. Detail page and self-test disable auto-reconnect

The detail page explicitly turns auto-reconnect off. Open a record page that's being judged and one disconnect means the page never updates again. Self-test uses the same hook (components/editor/scratchpad.tsx:139), with the same symptom: output freezes at the moment of disconnect.

There was a rationale at the time: detail page data is snapshot-like, so a refresh gets you the current state. But during contests the proctoring client is in use, and it can't ask users to refresh—the whole system runs on the router.

5. Close reasons aren't distinguished

The server actively closes the connection 30 seconds after a record reaches a terminal state:

Old clients ignore the close code: with auto-reconnect on, any close triggers a rebuild three seconds later.

Every reconnect re-runs prepare(). If the underlying cause persists—insufficient permission, bad parameters, failed origin check—it becomes an endless loop: connect, prepare error, server sends 4000, reconnect three seconds later, repeat.

4001 goes the opposite way: it only appears on the detail page, which happens to have auto-reconnect disabled, so no loop—at the cost of not recovering from ordinary disconnects either.

Both problems share one root: the client doesn't know whether this particular close warrants a reconnect.

6. No handling for browser suspension or clock skew

When the browser is suspended, setInterval gets throttled or paused outright. Close the laptop overnight and on waking Date.now() is already far in the past.

Hydro's code had no timeout logic, so this never surfaced. But the moment you add one, the first thing it does on wake is kill a perfectly healthy connection.

7. The send queue has no context

When the connection is unavailable, messages are queued unconditionally. If the connection is actually dead, they just sit there—the user thinks it was sent, it wasn't, and the UI shows nothing.

Summary

Seven items is a lot to read; here they are as a table:

A table doesn't convey the sense of time, so here's a timeline instead. Say the link breaks for good at t = 300s:

In Hydro's old implementation, nothing happens after the break until the user refreshes. HydroNext's new implementation instead fires a probe at 120s idle, declares failure if nothing comes back within 10s, and then reconnects.

Of these seven problems, only three places actually need changes: how to tell a connection is dead, when not to reconnect, and how to dispose of the old connection and its queued messages after reconnecting.

Where debugging got hard

Once we'd settled on where to look, the path back seemed pretty smooth. But actually spending time pinning down which step the problem came from turned out to be another story.

No errors whatsoever

The whole connection never threw a single exception; the console showed no console.error, and no failed requests either. The browser gave no sign the WS had failed, because from its point of view the WS connection had been OPEN

readyState doesn't mean the connection works

WebSocket.OPEN only says this object is open. On a half-open connection, neither reads nor writes error out right away — you have to wait for the TCP timeout, which usually takes several minutes.

Can't reproduce locally

Exactly. This problem can't be reproduced locally, because there's no CDN locally and the proxy won't reclaim idle connections on its own — leave it running all day and it's fine. It only shows up with a CDN in the path, a long idle period, and a particular combination of network routes.

Takes forever

Testing on the live environment means waiting 120s, 300s, or even longer for a single reproduction, which is far too expensive to do by hand. And whether it reproduces or not, you can't conclude much either way.

The UI lies too

The announcement popup uses connected to control the button's

while connected is driven only by onOpen and onClose. On a half-open connection onClosenever fires, so connectedstays trueThe UI shows "connected" and the button looks clickable, but clicking it does nothing

The CDN's internals are invisible

Its reclaim policy, node switching, whether it sends a FIN — none of it is under our control. All I can see is the outcome, like vibe coding.

The clues that actually helped

These are about all you can rely on:

• DevTools' Network → WS panel. Check when the connection was established and its frame history. A half-open connection looks like frames freezing at some point while the entry never disappears.
• Server logs. The framework will terminate after 80 seconds with no frames; if the logs repeatedly show "connection established, then nothing for a long time", the connection died somewhere in the middle of the chain.
• Compare before and after a refresh. If everything works right after refreshing, the server data is fine and the problem is purely that the client never rebuilt the connection.
• A temporary probe. Periodically print readyState along with "how long since the last frame".

Of all of these, the temporary probe gave the most direct answer. After a quiet spell, if readyState is still 1 while the time since the last frame keeps climbing, you can call the connection dead.

But all these can only confirm the connection really is dead — they can't reproduce it, and they can't prove it's fixed. So we switched tactics: encode the failure mode into a test and let the test wait for it to happen.

Add a heartbeat? Wait, that's not right!

Halfway through debugging, the easiest conclusion is that the CDN is killing the connection and a heartbeat will fix it. But the more we patched, the more wrong it felt: "Hang on — the heartbeat runs into the exact same problem!"

The redesign

Every failure path funnels into fail(): handshake timeout, send throwing, probe timeout, onclose, and onerror

1. Probe first, then judge

The simplest approach is to reconnect after N seconds without a message, but that doesn't work for announcements: going hours without business messages is completely normal.

So we split it into two steps: send a probe frame first, and only declare failure if no reply arrives.

The first stage sends just a single ping, without declaring failure right away; any frame received within 10 seconds counts as healthy. Any inbound frame — not just business JSON. If the server is still sending us heartbeats, that alone proves the link is alive. If we only accepted business messages, the announcement connection would be killed over and over by false alarms.

This probe works only because the server already handles it symmetrically: the framework replies immediately with 'ping' when it receives a client's 'pong' (framework/framework/server.ts:697-698). In effect, we send a frame guaranteed to get an answer and use the answer as proof.

But why 120 seconds? The server's heartbeat cycle is 40 seconds, so 120 seconds is exactly three misses in a row. A healthy connection never reaches this branch; if it does, the inbound direction has been down for at least two minutes. Any shorter and it would collide with the normal heartbeat rhythm, sending healthy connections into the probing state constantly.

The client no longer sends its own heartbeat every 40 seconds because what it's waiting for is precisely the signal that "the server's heartbeat stopped". Probing at the same frequency would just add pointless traffic and state churn.

2. Don't wait for close, drop the old socket proactively

Since the close event may never arrive, don't wait for it. Once failure is determined, just let the old socket go.

The order here matters. First clear wsRef.current so late events from the old socket fail the current(ws) check; then detach the handler so its onclose can't trigger failure handling again; and only then call close(), which at that point is just best-effort cleanup.

This is the most common race in reconnect logic:

Without the current(ws) check, that last late onclose would re-fail() and retire the connection that was just established. Every connection would then die within a single RTT, and the page would sit in the reconnecting state forever.

3. Use close codes to decide whether to reconnect

The server closes connections for different reasons, and the client has to tell them apart:

• 1008: the WebSocket standard's policy violation. Hydro never produces this code itself; it usually comes from the ws library rejecting a frame at the protocol layer (oversized fragments, excessive buffering), or from a proxy or gateway denying it by policy. Reconnecting won't change the outcome.
• 4000: the server throws while handling prepare() or message() (framework/framework/server.ts:307, packages/hydrooj/src/service/server.ts:261).
• 4001: the record reaches a terminal state and the session ends (record.ts:513).

In all three cases reconnecting is pointless and sometimes actively harmful:

1000 is deliberately excluded. Intuitively, 1000 signals a normal closure, so we should stop; but in the real world,1000 can also come from a proxy restart, a graceful server restart, or a CDN node going offline. The transport layer closed cleanly, but the business logic may not be finished.

4. Page events as recovery triggers

Alongside the timer, a few browser events are hooked in:

offline only reports it:

navigator.onLine means false, the connection is very likely unusable; and true doesn't mean the OJ is reachable either — DNS, proxy, CDN, or the server itself could be the problem. So offline just sets a flag, while online fires a single probe, and only rebuilds if the probe fails.

online also happens to fix a small annoyance: it can skip the backoff currently being waited out.

When the network has just come back, there's no point waiting another 3 seconds.

5. Browser suspension and clock skew

Both of these are easy to overlook while writing the code, but they do happen in production.

Clock skew:

If the user changes the time manually, or NTP winds the clock backwards, now - lastReceived goes negative. Without handling that, the detection logic could stall for hours.

Delayed tick after suspension:

When the browser comes back to the foreground, setInterval might fire only once, even though Date.now() has already passed half an hour. Treating that as a probe timeout would immediately kill a healthy connection that just woke up. The approach here: once a delayed tick is detected, send another ping and grant a fresh 10-second window.

The full flow

Putting all of the above together, every failure path converges on a single function:

This function guarantees three things: the old connection is always retire; the termination code blocks any further reconnects and clears the queue and timers; and the onClose callback always runs, with the reconnect scheduled inside finally, so even if the callback throws, recovery is unaffected.

Reconnection itself is deduplicated:

reconnectTimer !== undefined ensures only one pending reconnect task exists at a time.

State is reset in one place once the connection is established:

The queue first flushes the messages that piled up while waiting, then calls onOpen. The business layer resubscribes in onOpen, which keeps it from tangling with the ordering of old messages.

send also gets error handling:

canQueue is set to false on termination, unload, and domain switches, and the queue is cleared. A queue must not outlive its connection context, otherwise messages never sent in domain A could go out over the connection for domain B.

The health-check interval stays in sync with the probe window:

The default probe timeout is 10s and the health check also runs every 10s. Shorten the business window and the checks get more frequent too, with a 1s floor so the main thread doesn't get saturated.

The server has to cooperate

Automatic reconnection has one prerequisite: after reconnecting, the client must be able to get the state it missed. If the server merely accepts the connection again and pushes nothing, the client is connected but knows nothing, so this change was made together with the backend.

Announcements

Every connection runs prepare() and ends with refresh(), re-pushing all unread announcements:

The client doesn't need to catch up separately — reconnecting is itself a sync.

And the acknowledge operation is idempotent:

So a reconnect that causes the same acknowledgement to be sent twice is harmless.

Record details

On every connection, the server needs to push the current record once, and the client only has to allow reconnection.

Record list

The list connection is stateless, so the client has to tell the server which records it's viewing

After receiving rids, the server replays the current state

onOpen doubles as the resubscribe mechanism here.

Contest timing

In this part of the system, timing is bound by a 30s lease.

When the server settles the time, a period only counts if the gap between the previous report and this one is no more than 30 seconds:

The client reports every 10s, so normally it's always within the lease. Once the connection is down for more than 30s, the lease expires and that gap isn't credited retroactively.

So timing is especially sensitive to reconnect latency. A 3-second reconnect interval loses essentially no time; in Hydro's older implementation, though, a half-open connection kept dropping time until the user refreshed the page, and tens of minutes or even hours in between went uncounted.

The client marks its state as unsynced and, right after reconnecting, re-reports whether it's currently in the foreground. The server, in prepare(), validates the registration info and sends back the accumulated time along the way:

Wrapping up

That's all the changes.

In fact, this investigation and these changes introduced no new mechanisms — heartbeats and reconnection already existed back in Hydro. What I added in HydroNext is a "dead connection judgement", which makes the system more robust and supports EdgeOne CDN much better. Right — thinking it over, all these problems came from introducing a CDN, which also shows that the system was never designed with robust handling of client-facing WS connections from the start. Fortunately, this series of checks and handlers has filled that gap.