[tests] make GetObjectArray peer mismatch self-diagnosing - #12245
Open
jonathanpeppers wants to merge 8 commits into
Open
[tests] make GetObjectArray peer mismatch self-diagnosing#12245jonathanpeppers wants to merge 8 commits into
GetObjectArray peer mismatch self-diagnosing#12245jonathanpeppers wants to merge 8 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes intermittent CoreCLR failures caused by duplicate managed peers being registered for the same Java instance by aligning CoreCLR’s JavaMarshalRegisteredPeers.AddPeer() replacement rules with MonoVM’s semantics.
Changes:
- Update
JavaMarshalRegisteredPeers.AddPeer()to prevent replaceable peers from evicting other replaceable peers, and to replace stale entries when a peer reference is no longer valid. - Optimize
WarnNotReplacing()to avoid expensive logging work unless global reference logging is enabled. - Add deterministic regression tests covering both replacement outcomes.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj | Includes the new peer-registration regression tests in the test project. |
| tests/Mono.Android-Tests/Mono.Android-Tests/Java.Interop/ValueManagerPeerRegistrationTests.cs | Adds deterministic tests validating replaceable/non-replaceable peer replacement semantics. |
| src/Mono.Android/Microsoft.Android.Runtime/JavaMarshalRegisteredPeers.cs | Aligns CoreCLR peer replacement logic with MonoVM, handles stale entries, and reduces logging overhead. |
jonathanpeppers
force-pushed
the
jonathanpeppers-probable-fishstick
branch
from
July 29, 2026 17:05
f1250fa to
fae07a2
Compare
Application.Context initialization thread-safe
`Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArray` is flaky on
CoreCLR (~17 failures / 216 executions), and only on CoreCLR-backed
configurations. `Assert.AreSame()` gives us nothing to work with:
Expected existing Context peer, got Android.AppTests.App.
Expected: same as crc64f5183e500568449d.App@8e34fe2
But was: crc64f5183e500568449d.App@8e34fe2
Both operands render identically, because `Object.ToString()` forwards to
the Java `toString()` and both managed peers wrap the *same* Java
instance. So the message tells us only that the peers differ -- not why.
`JniValueManager.GetPeer()` peeks the registry before creating a new
peer, so a mismatch means `PeekPeer()` missed an entry that
`Application.Context` still holds strongly. Replace the assert with a
failure-only diagnostic that distinguishes the possible causes:
* managed hash codes of both peers, so they can be told apart
* `JniIdentityHashCode`, `PeerReference` and `JniManagedPeerState`
* whether `Application.Context` still returns the captured instance
* `IsSameObject()` on the two peer references
* what `PeekPeer()` returns now, and whether it is the expected peer,
the unexpected one, or nothing at all
* every surfaced peer sharing the identity hash code
That separates "registry entry was replaced by a second `AddPeer()`"
from "entry was evicted" from "weak reference was cleared", which is
the information needed to find the root cause.
Verified on device by temporarily forcing the failure branch; the
diagnostic runs to completion without throwing, and reports the
expected all-`True` values in the healthy case.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
jonathanpeppers
force-pushed
the
jonathanpeppers-probable-fishstick
branch
from
July 29, 2026 18:38
fae07a2 to
5ed818f
Compare
Application.Context initialization thread-safeGetObjectArray peer mismatch self-diagnosing
The self-diagnosing assert added previously caught a real CI failure and
reported:
expected (Application.Context): App@managed-0x109791d PeerReference=0x2d66/G
actual (GetObjectArray ()[0]): App@managed-0x1ee05f PeerReference=0x30aa/G
PeekPeer (expected.PeerReference) => App@managed-0x1ee05f
is expected: False; is actual: True
Surfaced peers with JniIdentityHashCode=0x87e2d7a:
App@managed-0x1ee05f
So the registry ends up holding exactly one peer, and it is *not* the one
`Application.Context` caches. But that end state does not say when the
registry stopped agreeing with `Application.Context`, and the two
candidate stories predict the same final state:
* the entry was still present and got replaced while marshaling, or
* the entry was already gone before this test ran, so `PeekPeer()`
legitimately missed and a fresh peer was created and registered.
Note the first story requires `PeekPeer()` to miss the entry while
`AddPeer()`, microseconds later, finds it live and matching -- both using
the same `Target` and `IsSameObject()` checks. Sampling the registry
before the marshaling distinguishes them directly instead of by argument.
Probe `PeekPeer (Application.Context)` at test entry, before `NewArray()`
and after `NewArray()`, and report the three samples on failure. Record a
description instead of the peer itself, so the probe cannot keep a peer
alive and perturb the GC behaviour under investigation.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
…stration The probe added in the previous commit answered the question it was asked. From a CI failure on `Mono.Android.NET_Tests-CoreCLRTrimmable`: expected (Application.Context): App@managed-0x240d846 PeerReference=0x2d36/G actual (GetObjectArray ()[0]): App@managed-0x31cb802 PeerReference=0x3052/G PeekPeer (Application.Context) over time: at test entry: App@managed-0x31cb802 (is expected: False) before NewArray: App@managed-0x31cb802 (is expected: False) after NewArray: App@managed-0x31cb802 (is expected: False) The registry already held the wrong peer *at test entry*, before this test touched anything. `GetObjectArray()` is not where the bug happens; it is merely the first test that looks. That rules out the marshaling path, and with it every hypothesis framed around `NewArray()`/`GetObjectArray()`. So move the check to where it can name a culprit: an assembly-level `ITestAction` samples `PeekPeer (Application.Context)` after every test and records the first test after which it stops returning the peer that `Application.Context` caches. `GetObjectArray()`'s failure message then reports that test, narrowing ~430 candidates to one and making a local repro possible. `PeekPeer()` only reads the registry -- unlike `GetSurfacedPeers()` it does not drain the collected-peer queue -- so sampling it this often should not perturb the behaviour being measured. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The per-test watcher named a culprit on its first CI run: Registry first diverged after test: Android.AppTests.ApplicationTest.ApplicationContextIsApp That test is the *first* test in the run -- `Android.AppTests` sorts ahead of every other namespace -- and its body is only: Assert.IsTrue (Application.Context is App); Assert.IsTrue (App.Created); which does nothing that could plausibly evict a peer. So the divergence almost certainly predates the test run entirely and happens during app startup. But sampling only in `AfterTest` cannot prove that: it cannot separate "already broken when the run started" from "broken by this test". Sample in `BeforeTest` as well and record which one observed it first, so the report reads `before <test>` or `after <test>`. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The watcher pinned down *when* the registry diverges:
Registry first diverged: before Android.AppTests.ApplicationTest.ApplicationContextIsApp
`before`, on the first test of the run -- so the registry is already wrong
when the test run starts, and the divergence happens during app startup.
The greferences say which way it went: `Application.Context` caches
`0x2d66/G` while the registry holds the later `0x3086/G`, i.e. a newer peer
displaced the one `_context` had already cached.
Two mechanisms produce that, and build 1529081 separates them. That build
tested `f0e746d8c` (a merge of `bfcad439e`), which did contain the
`target.Replaceable && !value.Replaceable` guard -- verified with `git show`
-- and `GetObjectArray` still failed:
* `AddPeer()` *replacing* the entry: blocked by that guard, so ruled out.
* the entry being *evicted*, so a later `PeekPeer()` legitimately misses
and registers a fresh peer: unaffected by the guard, and consistent.
Eviction also fits the rest: exactly one surfaced peer, `Application.Context`
holding the older one, CoreCLR-only (`CollectPeers()` has no MonoVM
counterpart), and GC-timing flakiness.
A context only reaches `CollectedContexts` because the GC bridge reported
its peer collected, so by the time `CollectPeers()` evicts it the peer's
weak reference should already be cleared. Record the cases where it is not
-- a live managed peer, possibly strongly held by `Application.Context` --
and publish the count via `AppContext` so the test can report it without
`InternalsVisibleTo`.
Recording only: the eviction still happens, so this measures the bug rather
than masking it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The eviction hypothesis is dead. Build 1532632 reproduced the failure with
the recorder in place and reported:
Live peers evicted by CollectPeers(): <none>
So `CollectPeers()` never drops a live registration, and the peer
`Application.Context` caches is not being evicted.
Re-reading build 1529081's failure message confirms it was this same
assertion, on a build that did carry the `Replaceable`/`Replaceable` guard.
Taken together with the current dumps, both configurations diverge, but in
opposite directions:
* without the guard, the registry follows the newer peer while
`Application.Context` holds the older one (gref 0x2d66 vs 0x30c6);
* with the guard, the registry keeps the older one instead.
Which one `AddPeer()` picks is therefore not the bug. The invariant that
actually breaks is upstream of that choice: a *second* managed peer is
created for the same Java Application instance at all. Whichever one the
registry settles on, `Application.Context` has already cached the other.
`AddPeer()` sees that happen in exactly two places -- replacing an existing
entry, and appending alongside one whose weak reference was cleared -- so
record both, with the identity of the old and new peer, and report them
from the failing test. Recording only; behaviour is unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
Fixes: #10973 `JnienvArrayMarshaling.GetObjectArray()` intermittently failed on CoreCLR (~17 failures / 216 executions) and never on MonoVM or NativeAOT: Expected existing Context peer, got Android.AppTests.App. Expected: same as crc64f5183e500568449d.App@8e34fe2 But was: crc64f5183e500568449d.App@8e34fe2 Both sides print identically because they are two *managed* peers over the same Java instance, so the assert's own message could not explain it. A test-only diagnostic, iterated over several CI runs, produced this: Registry first diverged: before Android.AppTests.ApplicationTest.ApplicationContextIsApp CollectPeers() evicted live peers: <none> AddPeer() replaced registrations: 1 (most recent: key=0xe45a52a replaced App@managed-0x33c0d9d/0x2d66/G with App@managed-0x11c7a8c/0x2d76/G) AddPeer() appended duplicates: <none> `before` on the run's first test means the registry is already wrong when testing starts, so this happens during app startup; `GetObjectArray()` was only the first test to look. And exactly one replacement occurs in the whole process: a second `App` peer displacing the one `Application.Context` had cached. `Application.Context` caches the peer it first sees and never re-reads it, so replacing that registration orphans the cached reference permanently -- `PeekPeer()` then hands out the replacement for the same Java instance. Whether the run fails is a startup race: if `Application.Context` is read before the replacement it keeps the orphaned peer and fails; if after, it caches the survivor and passes. `AddPeer()` replaced whenever the *existing* peer was `Replaceable`, without checking the new one. MonoVM's `AndroidRuntime.JavaObjectValueManager` also requires the new peer to be non-`Replaceable` -- a "replaceable" instance must not replace another "replaceable" instance, per #9862 -- which is why MonoVM never hit this. Apply the same condition here. Also guard the `WarnNotReplacing()` call with `LogGlobalReferenceMessages`: that branch is now reached during normal startup, and its arguments are evaluated eagerly -- including two `GetJniTypeNameFromInstance()` JNI round-trips -- even though `WriteGlobalReferenceLine()` discards them when logging is off. MonoVM guards its equivalent warning the same way. The diagnostics are kept: they are what turned an unexplainable assert into a located bug, and they let CI confirm the fix directly -- a green run must now report no replacement of the `Application.Context` peer, rather than merely not failing a ~50/50 coin flip. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The `Replaceable`/`Replaceable` guard works -- build 1532806 reports
`AddPeer() replaced registrations: <none>` -- and `GetObjectArray()` still
fails. That also explains build 1529081, which carried the same guard and
failed the same way, and it means the guard is MonoVM parity rather than the
fix for this.
What the failing dump now shows is a registry state that none of the
instrumented paths produced:
CollectPeers() evicted live peers: <none>
AddPeer() replaced registrations: <none>
AddPeer() appended duplicates: <none>
Surfaced peers with JniIdentityHashCode=0x17e2473:
Android.AppTests.App@managed-0xf5e94d
The registry holds the peer that `Application.Context` did *not* cache, yet
nothing replaced, appended or evicted anything. Two possibilities remain,
and both run through code that is currently silent:
* the registry's peer registered *first* and the guard then turned away
the one `Application.Context` went on to cache -- the guard's
"not replacing" branch records nothing once
`WarnNotReplacing()` is skipped for logging being off; or
* the cached peer was registered and then removed by `RemovePeer()`, or
by a `CollectPeers()` eviction of an already-cleared entry, neither of
which is counted.
Record all three. The "declined to replace" entry also establishes
registration order, since the peer that registers first is the one kept.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArraywas un-ignored in #12222 and has been flaky ever since — ~17 failures / 216 executions, exclusively on CoreCLR-backed configurations (Mono: 0/27, NativeAOT: 0/26).The failure message is useless:
Both operands print identically, because
Object.ToString()forwards to the JavatoString()and both managed peers wrap the same Java instance. All we learn is "the peers differ" — not why.Why not just fix it?
I tried. Three separate hypotheses were implemented, pushed, and disproven by CI:
AddPeermissing MonoVM's!value.Replaceableguard — dead end:GetPeer()returnsCreatePeer()'s result to the caller beforeAddPeer()runs, and a deterministic bug can't produce an ~18% flake.Application.Contextunsynchronized lazy init — shipped, and build 1531735 still failed withGetObjectArrayas the only failure, contradicting the hypothesis' own prediction.I also could not reproduce locally in 500,000 iterations — but that "negative control" was worthless, since CI runs x86_64 emulators on macOS while I was on an arm64 physical device. It never reproduced the bug even once, so it can't disprove anything.
Logcat analysis of the failing run ruled out the obvious suspects: test ordering is deterministic (NUnit
ParallelScope.None, strictly alphabetical), and there was no GC, no bridge activity, and no typemap miss in the 32 ms failure window. Notably,Android.AppTests.ApplicationTestsorts first and cachesApplication.Context~35 s and ~430 tests beforeGetObjectArrayruns — so_contextis pinned long beforehand, and the registry must be changing underneath it.What this PR does
Rather than ship a fourth guess, this makes the next CI failure actually explain itself.
JniValueManager.GetPeer()peeks the registry before creating a peer, so a mismatch meansPeekPeer()missed an entry thatApplication.Contextstill holds strongly. The assert is replaced with a failure-only diagnostic that distinguishes the candidate causes:JniIdentityHashCode,PeerReference,JniManagedPeerStateApplication.Contextstill returns the captured instanceIsSameObject()on the two peer referencesPeekPeer()returns now — the expected peer, the unexpected one, or nothingTogether these separate "entry replaced by a second
AddPeer()" from "entry evicted" from "weak reference cleared" — exactly the fork in the road I keep guessing at.Risk / validation
Test-only change; no product code is touched. The diagnostic is built only on the failure path, so passing runs are unaffected.
Verified on device by temporarily forcing the failure branch: the diagnostic runs to completion without throwing (important — a secondary exception here would be worse than the original message) and reports the expected all-
Truevalues in the healthy case:In a real failure those lines diverge, and the divergence names the root cause.
Contributes to #10973.