Make the alarm callback path thread-safe, roll back failed registrations, and stop losing callLater failures - #571
Open
lagergren wants to merge 3 commits into
Open
Conversation
The per-service alarm callback registry was a lazily created plain HashMap. The owning service put entries on its own thread, but alarm maturation removed them on the process-wide static Timer thread, with no monitor in common. A put that resizes the table racing a timer-thread remove can corrupt the map or lose an entry. Losing an entry was not a benign outcome: WeakCallback.extractCallback() threw IllegalStateException when the entry was missing, and that throw happened inside a TimerTask. An exception escaping a TimerTask kills the shared static Timer, which silently disables every alarm in every container in the process - a whole-VM denial of service triggered by a data race. Alarm cancellation also never removed its registry entry, so a canceled alarm leaked its captured Frame and FunctionHandle for the lifetime of the service. - ServiceContext: the registry is now an eager, final ConcurrentHashMap, so the cross-thread access pattern is encoded in the type instead of relying on timing luck. ensureCallbackMap() is gone; getCallbackMap() never returns null. - WeakCallback: extractCallback() returns null for a missing callback instead of throwing on the timer thread, and a new discard() lets cancellation drop the entry. - xLocalClock/xNanosTimer: alarm firing tolerates a null callback, and alarm cancellation discards the registry entry. NativeCallbackRegistrationTest boots a real NativeContainer over the compiled system modules and drives the real runtime classes. On the pre-fix sources all three cases fail: the registry is null before first use, a second extraction throws IllegalStateException, and the concurrent service/timer-thread drain loses entries (12 of 12 runs).
…ails Native callback registration is a lifecycle count that keeps a container alive while Java-side timer/server work may still call back into it. Three paths incremented that count and then had no way to give it back if a later startup step failed, pinning the container forever: idle termination never fires and join() hangs, even though the native resource was never installed. LocalClock claimed keep-alive ownership from the Alarm constructor. When Timer.schedule(...) then failed, the recovery path called cancel(), whose unregister was gated on TimerTask.cancel() - which reports false for a task that was never scheduled. The count stayed up. Registration now happens as part of the schedule attempt via registerKeepAlive(), the alarm remembers the exact container it registered with, and cancelAfterScheduleFailure() unwinds the attempt without depending on TimerTask.cancel(). An alarm whose service has been collected releases its count too, instead of holding it forever. NanoTimer registered the callback and then caught Throwable around Timer.schedule(...), swallowing the failure: natural code got back an alarm that would never fire while the keep-alive count stayed elevated. The scheduler failure now rolls back the trigger and the registration and propagates; addAlarm() de-registers the alarm it just added and invokeSchedule() turns it into an XTC exception, discarding the callback registry entry on the way out. Keep-alive ownership is tracked per alarm in m_containerRegistered so it always unwinds against the container it was taken from, and releasing it is idempotent. The server bind registered the callback before the last startup steps. If createContext(...) failed, the service context was terminated but the callback count, both partially configured Java servers, and the thread pool were left behind. Bind now tracks whether registration happened and runs rollbackBind() before raising the exception, releasing the count, closing whatever servers were created, shutting down the executor, and clearing the handle so a retry sees an unconfigured server. Both new tests drive the real native "schedule" entry point with the shared timer forced into a state where scheduling fails, and assert on Container.isIdle() - the property the leak actually breaks. On the pre-fix sources the LocalClock case leaves the container pinned, and the NanoTimer case additionally returns R_NEXT instead of R_EXCEPTION, handing natural code an alarm that can never fire.
callLater guarantees that any failure of the called function surfaces as an
UnhandledExceptionNotification. It reported that failure from inside a
CompletableFuture completion stage, and cast the throwable straight to
WrapperException:
future.whenComplete((r, x) -> {
if (x != null) {
callUnhandledExceptionHandler(((WrapperException) x).getExceptionHandle());
}
});
Any throwable that is not a WrapperException turns that cast into a
ClassCastException raised inside the completion stage, and a completion stage
discards whatever is thrown out of it. The cast failure and the original
failure are both lost, and the handler never runs, so a failure the runtime
promises to surface becomes completely silent - not even a stack trace on
stderr.
On the internal paths x really is always a WrapperException: whenComplete is
attached to the root future returned by postRequest rather than to a dependent
stage, so nothing wraps it in a CompletionException, and both sites that
complete it exceptionally pass ExceptionHandle.getException(), which is typed
WrapperException. The hole is the public contract instead: callLater hands its
CompletableFuture to the caller, and cancelling a CompletableFuture completes
it with CancellationException. No in-tree caller does that today, so this is a
latent defect rather than one presently firing - but it is reachable through
the published API, and the failure mode is silence.
Both overloads now route through reportUnhandledException, which uses the
runtime's own Utils.translate: it unwraps CompletionException/ExecutionException,
maps a WrapperException to its handle exactly as before, and renders anything
else - a cancellation, an interrupt, a native failure - as a visible exception
rather than dropping it. Reporting is wrapped so that the reporter itself can
never throw into the completion stage that would swallow it.
CallLaterFailureReportingTest covers both directions: cancelling the future
fails on the unfixed code, where the handler never runs and stderr stays empty,
while an ordinary raised XTC exception passes before and after, pinning the
common WrapperException path against regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects on the native alarm/callback path, in three commits. They share a theme: a failure on the timer or startup path that is lost rather than reported.
1. The callback registry is not safe for the thread that reads it
ServiceContext's callback registry was a lazily createdHashMap— and the shared Java timer thread reads and removes from it while service threads add to it. A resizingputracing agetcan corrupt the table structurally: entries in the wrong bins, unrelated keys lost. The values converge, so this is not "duplicate work"; it is a silent wrong answer from a registry.Worse,
WeakCallback.extractCallback()threw from that timer thread when the context was gone or the id already removed. Sincejava.util.Timercatches nothing, that throw cancels the process-wideTimeroutright (that hazard is the subject of the companion PR; this one removes the known thrower). Cancelled alarms also leaked their registry entries.Now: an eager
final ConcurrentHashMap(ensureCallbackMap()deleted,getCallbackMap()never returns null),extractCallback()returns null instead of throwing, a newdiscard(), and both timer templates tolerate a null callback on fire and discard the entry on cancel.2. Registration is not rolled back when startup fails
xLocalClock,xNanosTimerandxRTServerregistered a container keep-alive before the operation that could fail. A failed timer schedule or a failed server bind therefore left the container pinned alive forever — it never goes idle, so it never terminates.Adds
rollbackBind/closeServerQuietly, guardsregisterKeepAliveagainst double-registration across a stop/start cycle, and plugs a leak whereinvokeScheduleregistered aWeakCallbackbeforeaddAlarmcould throw.3.
callLater's completion handler silences failures it cannot classifyBoth
ServiceContext.callLateroverloads carried:((WrapperException) x)is a blind cast, andCompletableFutureswallows anything thrown insidewhenComplete. So a wrong cast produces no visibleClassCastException— it produces the absence of the unhandled-exception handler ever running. The original failure is gone.Stated honestly: this is latent, not currently firing. I traced it rather than assuming.
whenCompleteis attached to the root future (postRequestreturns a plainnew CompletableFuture(), not a dependent stage), so noCompletionExceptionwrapping occurs; and both internal completion sites passhException.getException(), whose declared type isWrapperException. On today's in-tree paths the cast never fires.It is reachable through the published API, though:
callLaterreturns its future to the caller, andfuture.cancel(true)completes it withCancellationException. No current caller does that — but it is a public method returning a public future, and the failure mode is total silence.Fixed by routing both overloads through one reporter built on the runtime's own
Utils.translate(already used for exactly this job inxFuture's completion callbacks): unwrapCompletionException/ExecutionException, mapWrapperExceptionas before, and render anything else visibly instead of dropping it — with the reporter itself wrapped so it can never throw back into the stage that would swallow it.Why this one is worth more than a tidy-up
This is the sharpest available argument for typing a failure channel rather than casting it. A blind cast normally announces itself. Here the code being type-punned is the error-handling path, and the swallowing happens in a completion stage — so a wrong guess does not defer a failure to run time, it converts a reported failure into a lost one. That is strictly worse than an ordinary bad cast, and it is invisible by construction.
Tests
All behavioural. No source-text assertions — an earlier draft of this branch used them and they were removed, because they pin the shape of one particular fix rather than the behaviour, and would not have caught the original defects.
NativeCallbackRegistrationTestboots a realNativeContainer, builds a realServiceContextand native entryFrame, and drives the real runtime classes. Five tests, all red on master. Highlights:registryToleratesConcurrentServiceAndTimerThreadAccess— 50,000 concurrent lookups. 12 failures / 12 runs on master, 0 / 12 after. This one is a genuine race: 12/12 on this machine is not a promise it fails everywhere.failedNanosTimerScheduleIsReportedAndUnwound—expected: <-3> but was: <-1>; master returnedR_NEXTafter a failed schedule, handing natural code an alarm that can never fire.extractingAMissingCallbackReturnsNullInsteadOfThrowing— catches the rawIllegalStateExceptionthat kills the sharedTimer.HttpServerBindRollbackTestbinds real sockets on port 0 (ephemeral, both HTTP and HTTPS), starts both servers, then induces failure at the post-registration point and asserts onContainer.isIdle()— the property the leak actually breaks. 10/10 red, 0/10 after.CallLaterFailureReportingTest— two tests, one of which passes on master by design: it is the regression guard proving the ordinaryWrapperExceptionpath is untouched, and it is also what empirically confirms the trace above. The failing one showssystem-err: ''— the discarded failure leaves no trace at all.:javatools:test348 → 356, identical 40 pre-existing skips.xdk:installDistrun as its own invocation; all results read from the JUnit XML.Relationship to the other clock PR
Independent in both directions; either can land first. They conflict textually in
xLocalClock.javaandxNanosTimer.java, so whichever goes second wants a rebase. This PR removes the throwers we know about; the companion says the timer thread must survive one we do not.