Skip to content

Make the alarm callback path thread-safe, roll back failed registrations, and stop losing callLater failures - #571

Open
lagergren wants to merge 3 commits into
masterfrom
lagergren/master-clock-callback-registry
Open

Make the alarm callback path thread-safe, roll back failed registrations, and stop losing callLater failures#571
lagergren wants to merge 3 commits into
masterfrom
lagergren/master-clock-callback-registry

Conversation

@lagergren

Copy link
Copy Markdown
Contributor

Three defects on the native alarm/callback path, in three commits. They share a theme: a failure on the timer or startup path that is lost rather than reported.

1. The callback registry is not safe for the thread that reads it

ServiceContext's callback registry was a lazily created HashMap — and the shared Java timer thread reads and removes from it while service threads add to it. A resizing put racing a get can corrupt the table structurally: entries in the wrong bins, unrelated keys lost. The values converge, so this is not "duplicate work"; it is a silent wrong answer from a registry.

Worse, WeakCallback.extractCallback() threw from that timer thread when the context was gone or the id already removed. Since java.util.Timer catches nothing, that throw cancels the process-wide Timer outright (that hazard is the subject of the companion PR; this one removes the known thrower). Cancelled alarms also leaked their registry entries.

Now: an eager final ConcurrentHashMap (ensureCallbackMap() deleted, getCallbackMap() never returns null), extractCallback() returns null instead of throwing, a new discard(), and both timer templates tolerate a null callback on fire and discard the entry on cancel.

2. Registration is not rolled back when startup fails

xLocalClock, xNanosTimer and xRTServer registered a container keep-alive before the operation that could fail. A failed timer schedule or a failed server bind therefore left the container pinned alive forever — it never goes idle, so it never terminates.

Adds rollbackBind/closeServerQuietly, guards registerKeepAlive against double-registration across a stop/start cycle, and plugs a leak where invokeSchedule registered a WeakCallback before addAlarm could throw.

3. callLater's completion handler silences failures it cannot classify

Both ServiceContext.callLater overloads carried:

future.whenComplete((r, x) -> {
    if (x != null) {
        callUnhandledExceptionHandler(((WrapperException) x).getExceptionHandle());
    }
});

((WrapperException) x) is a blind cast, and CompletableFuture swallows anything thrown inside whenComplete. So a wrong cast produces no visible ClassCastException — it produces the absence of the unhandled-exception handler ever running. The original failure is gone.

Stated honestly: this is latent, not currently firing. I traced it rather than assuming. whenComplete is attached to the root future (postRequest returns a plain new CompletableFuture(), not a dependent stage), so no CompletionException wrapping occurs; and both internal completion sites pass hException.getException(), whose declared type is WrapperException. On today's in-tree paths the cast never fires.

It is reachable through the published API, though: callLater returns its future to the caller, and future.cancel(true) completes it with CancellationException. No current caller does that — but it is a public method returning a public future, and the failure mode is total silence.

Fixed by routing both overloads through one reporter built on the runtime's own Utils.translate (already used for exactly this job in xFuture's completion callbacks): unwrap CompletionException/ExecutionException, map WrapperException as before, and render anything else visibly instead of dropping it — with the reporter itself wrapped so it can never throw back into the stage that would swallow it.

Why this one is worth more than a tidy-up

This is the sharpest available argument for typing a failure channel rather than casting it. A blind cast normally announces itself. Here the code being type-punned is the error-handling path, and the swallowing happens in a completion stage — so a wrong guess does not defer a failure to run time, it converts a reported failure into a lost one. That is strictly worse than an ordinary bad cast, and it is invisible by construction.

Tests

All behavioural. No source-text assertions — an earlier draft of this branch used them and they were removed, because they pin the shape of one particular fix rather than the behaviour, and would not have caught the original defects.

NativeCallbackRegistrationTest boots a real NativeContainer, builds a real ServiceContext and native entry Frame, and drives the real runtime classes. Five tests, all red on master. Highlights:

  • registryToleratesConcurrentServiceAndTimerThreadAccess — 50,000 concurrent lookups. 12 failures / 12 runs on master, 0 / 12 after. This one is a genuine race: 12/12 on this machine is not a promise it fails everywhere.
  • failedNanosTimerScheduleIsReportedAndUnwoundexpected: <-3> but was: <-1>; master returned R_NEXT after a failed schedule, handing natural code an alarm that can never fire.
  • extractingAMissingCallbackReturnsNullInsteadOfThrowing — catches the raw IllegalStateException that kills the shared Timer.

HttpServerBindRollbackTest binds real sockets on port 0 (ephemeral, both HTTP and HTTPS), starts both servers, then induces failure at the post-registration point and asserts on Container.isIdle() — the property the leak actually breaks. 10/10 red, 0/10 after.

CallLaterFailureReportingTest — two tests, one of which passes on master by design: it is the regression guard proving the ordinary WrapperException path is untouched, and it is also what empirically confirms the trace above. The failing one shows system-err: '' — the discarded failure leaves no trace at all.

:javatools:test 348 → 356, identical 40 pre-existing skips. xdk:installDist run as its own invocation; all results read from the JUnit XML.

Relationship to the other clock PR

Independent in both directions; either can land first. They conflict textually in xLocalClock.java and xNanosTimer.java, so whichever goes second wants a rebase. This PR removes the throwers we know about; the companion says the timer thread must survive one we do not.

The per-service alarm callback registry was a lazily created plain HashMap.
The owning service put entries on its own thread, but alarm maturation removed
them on the process-wide static Timer thread, with no monitor in common. A put
that resizes the table racing a timer-thread remove can corrupt the map or lose
an entry.

Losing an entry was not a benign outcome: WeakCallback.extractCallback() threw
IllegalStateException when the entry was missing, and that throw happened
inside a TimerTask. An exception escaping a TimerTask kills the shared static
Timer, which silently disables every alarm in every container in the process -
a whole-VM denial of service triggered by a data race.

Alarm cancellation also never removed its registry entry, so a canceled alarm
leaked its captured Frame and FunctionHandle for the lifetime of the service.

- ServiceContext: the registry is now an eager, final ConcurrentHashMap, so the
  cross-thread access pattern is encoded in the type instead of relying on
  timing luck. ensureCallbackMap() is gone; getCallbackMap() never returns null.
- WeakCallback: extractCallback() returns null for a missing callback instead of
  throwing on the timer thread, and a new discard() lets cancellation drop the
  entry.
- xLocalClock/xNanosTimer: alarm firing tolerates a null callback, and alarm
  cancellation discards the registry entry.

NativeCallbackRegistrationTest boots a real NativeContainer over the compiled
system modules and drives the real runtime classes. On the pre-fix sources all
three cases fail: the registry is null before first use, a second extraction
throws IllegalStateException, and the concurrent service/timer-thread drain
loses entries (12 of 12 runs).
…ails

Native callback registration is a lifecycle count that keeps a container alive
while Java-side timer/server work may still call back into it. Three paths
incremented that count and then had no way to give it back if a later startup
step failed, pinning the container forever: idle termination never fires and
join() hangs, even though the native resource was never installed.

LocalClock claimed keep-alive ownership from the Alarm constructor. When
Timer.schedule(...) then failed, the recovery path called cancel(), whose
unregister was gated on TimerTask.cancel() - which reports false for a task
that was never scheduled. The count stayed up. Registration now happens as part
of the schedule attempt via registerKeepAlive(), the alarm remembers the exact
container it registered with, and cancelAfterScheduleFailure() unwinds the
attempt without depending on TimerTask.cancel(). An alarm whose service has
been collected releases its count too, instead of holding it forever.

NanoTimer registered the callback and then caught Throwable around
Timer.schedule(...), swallowing the failure: natural code got back an alarm
that would never fire while the keep-alive count stayed elevated. The scheduler
failure now rolls back the trigger and the registration and propagates;
addAlarm() de-registers the alarm it just added and invokeSchedule() turns it
into an XTC exception, discarding the callback registry entry on the way out.
Keep-alive ownership is tracked per alarm in m_containerRegistered so it always
unwinds against the container it was taken from, and releasing it is idempotent.

The server bind registered the callback before the last startup steps. If
createContext(...) failed, the service context was terminated but the callback
count, both partially configured Java servers, and the thread pool were left
behind. Bind now tracks whether registration happened and runs rollbackBind()
before raising the exception, releasing the count, closing whatever servers were
created, shutting down the executor, and clearing the handle so a retry sees an
unconfigured server.

Both new tests drive the real native "schedule" entry point with the shared
timer forced into a state where scheduling fails, and assert on
Container.isIdle() - the property the leak actually breaks. On the pre-fix
sources the LocalClock case leaves the container pinned, and the NanoTimer case
additionally returns R_NEXT instead of R_EXCEPTION, handing natural code an
alarm that can never fire.
callLater guarantees that any failure of the called function surfaces as an
UnhandledExceptionNotification. It reported that failure from inside a
CompletableFuture completion stage, and cast the throwable straight to
WrapperException:

    future.whenComplete((r, x) -> {
        if (x != null) {
            callUnhandledExceptionHandler(((WrapperException) x).getExceptionHandle());
        }
    });

Any throwable that is not a WrapperException turns that cast into a
ClassCastException raised inside the completion stage, and a completion stage
discards whatever is thrown out of it. The cast failure and the original
failure are both lost, and the handler never runs, so a failure the runtime
promises to surface becomes completely silent - not even a stack trace on
stderr.

On the internal paths x really is always a WrapperException: whenComplete is
attached to the root future returned by postRequest rather than to a dependent
stage, so nothing wraps it in a CompletionException, and both sites that
complete it exceptionally pass ExceptionHandle.getException(), which is typed
WrapperException. The hole is the public contract instead: callLater hands its
CompletableFuture to the caller, and cancelling a CompletableFuture completes
it with CancellationException. No in-tree caller does that today, so this is a
latent defect rather than one presently firing - but it is reachable through
the published API, and the failure mode is silence.

Both overloads now route through reportUnhandledException, which uses the
runtime's own Utils.translate: it unwraps CompletionException/ExecutionException,
maps a WrapperException to its handle exactly as before, and renders anything
else - a cancellation, an interrupt, a native failure - as a visible exception
rather than dropping it. Reporting is wrapped so that the reporter itself can
never throw into the completion stage that would swallow it.

CallLaterFailureReportingTest covers both directions: cancelling the future
fails on the unfixed code, where the handler never runs and stderr stays empty,
while an ordinary raised XTC exception passes before and after, pinning the
common WrapperException path against regression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant