Skip to content

[tests] make GetObjectArray peer mismatch self-diagnosing - #12245

Open
jonathanpeppers wants to merge 8 commits into
mainfrom
jonathanpeppers-probable-fishstick
Open

[tests] make GetObjectArray peer mismatch self-diagnosing#12245
jonathanpeppers wants to merge 8 commits into
mainfrom
jonathanpeppers-probable-fishstick

Conversation

@jonathanpeppers

@jonathanpeppers jonathanpeppers commented Jul 27, 2026

Copy link
Copy Markdown
Member

Context

Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArray was un-ignored in #12222 and has been flaky ever since — ~17 failures / 216 executions, exclusively on CoreCLR-backed configurations (Mono: 0/27, NativeAOT: 0/26).

The failure message is useless:

Expected existing Context peer, got Android.AppTests.App.
Expected: same as crc64f5183e500568449d.App@8e34fe2
But was:  crc64f5183e500568449d.App@8e34fe2

Both operands print identically, because Object.ToString() forwards to the Java toString() and both managed peers wrap the same Java instance. All we learn is "the peers differ" — not why.

Why not just fix it?

I tried. Three separate hypotheses were implemented, pushed, and disproven by CI:

  1. AddPeer missing MonoVM's !value.Replaceable guard — dead end: GetPeer() returns CreatePeer()'s result to the caller before AddPeer() runs, and a deterministic bug can't produce an ~18% flake.
  2. GC-bridge race — CI went green, but that's a ~52% coin flip on this failure rate; not evidence.
  3. Application.Context unsynchronized lazy init — shipped, and build 1531735 still failed with GetObjectArray as the only failure, contradicting the hypothesis' own prediction.

I also could not reproduce locally in 500,000 iterations — but that "negative control" was worthless, since CI runs x86_64 emulators on macOS while I was on an arm64 physical device. It never reproduced the bug even once, so it can't disprove anything.

Logcat analysis of the failing run ruled out the obvious suspects: test ordering is deterministic (NUnit ParallelScope.None, strictly alphabetical), and there was no GC, no bridge activity, and no typemap miss in the 32 ms failure window. Notably, Android.AppTests.ApplicationTest sorts first and caches Application.Context ~35 s and ~430 tests before GetObjectArray runs — so _context is pinned long beforehand, and the registry must be changing underneath it.

What this PR does

Rather than ship a fourth guess, this makes the next CI failure actually explain itself.

JniValueManager.GetPeer() peeks the registry before creating a peer, so a mismatch means PeekPeer() missed an entry that Application.Context still holds strongly. The assert is replaced with a failure-only diagnostic that distinguishes the candidate causes:

  • managed hash codes of both peers, so they can be told apart at all
  • JniIdentityHashCode, PeerReference, JniManagedPeerState
  • whether Application.Context still returns the captured instance
  • IsSameObject() on the two peer references
  • what PeekPeer() returns now — the expected peer, the unexpected one, or nothing
  • every surfaced peer sharing that identity hash code

Together these separate "entry replaced by a second AddPeer()" from "entry evicted" from "weak reference cleared" — exactly the fork in the road I keep guessing at.

Risk / validation

Test-only change; no product code is touched. The diagnostic is built only on the failure path, so passing runs are unaffected.

Verified on device by temporarily forcing the failure branch: the diagnostic runs to completion without throwing (important — a secondary exception here would be worse than the original message) and reports the expected all-True values in the healthy case:

Expected `Application.Context` and `GetObjectArray ()[0]` to be the same managed peer.
  expected (Application.Context): Android.AppTests.App@managed-0x11181ff
    JniIdentityHashCode=0x116b2e2 PeerReference=0x3aea/G JniManagedPeerState=Activatable, Replaceable
  actual   (GetObjectArray ()[0]): Android.AppTests.App@managed-0x11181ff
    JniIdentityHashCode=0x116b2e2 PeerReference=0x3aea/G JniManagedPeerState=Activatable, Replaceable
  Application.Context still == expected: True
  IsSameObject (expected, actual): True
  PeekPeer (expected.PeerReference) => Android.AppTests.App@managed-0x11181ff
    is expected: True; is actual: True
  Surfaced peers with JniIdentityHashCode=0x116b2e2:
    Android.AppTests.App@managed-0x11181ff (is expected: True; is actual: True)

In a real failure those lines diverge, and the divergence names the root cause.

Contributes to #10973.

Copilot AI review requested due to automatic review settings July 27, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes intermittent CoreCLR failures caused by duplicate managed peers being registered for the same Java instance by aligning CoreCLR’s JavaMarshalRegisteredPeers.AddPeer() replacement rules with MonoVM’s semantics.

Changes:

  • Update JavaMarshalRegisteredPeers.AddPeer() to prevent replaceable peers from evicting other replaceable peers, and to replace stale entries when a peer reference is no longer valid.
  • Optimize WarnNotReplacing() to avoid expensive logging work unless global reference logging is enabled.
  • Add deterministic regression tests covering both replacement outcomes.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/Mono.Android-Tests/Mono.Android-Tests/Mono.Android.NET-Tests.csproj Includes the new peer-registration regression tests in the test project.
tests/Mono.Android-Tests/Mono.Android-Tests/Java.Interop/ValueManagerPeerRegistrationTests.cs Adds deterministic tests validating replaceable/non-replaceable peer replacement semantics.
src/Mono.Android/Microsoft.Android.Runtime/JavaMarshalRegisteredPeers.cs Aligns CoreCLR peer replacement logic with MonoVM, handles stale entries, and reduces logging overhead.

Comment thread src/Mono.Android/Microsoft.Android.Runtime/JavaMarshalRegisteredPeers.cs Outdated
@jonathanpeppers jonathanpeppers added the do-not-merge PR should not be merged. label Jul 27, 2026
@jonathanpeppers
jonathanpeppers force-pushed the jonathanpeppers-probable-fishstick branch from f1250fa to fae07a2 Compare July 29, 2026 17:05
@jonathanpeppers jonathanpeppers changed the title [Mono.Android] fix duplicate peer registration on CoreCLR [Mono.Android] make Application.Context initialization thread-safe Jul 29, 2026
`Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArray` is flaky on
CoreCLR (~17 failures / 216 executions), and only on CoreCLR-backed
configurations. `Assert.AreSame()` gives us nothing to work with:

	Expected existing Context peer, got Android.AppTests.App.
	Expected: same as crc64f5183e500568449d.App@8e34fe2
	But was:  crc64f5183e500568449d.App@8e34fe2

Both operands render identically, because `Object.ToString()` forwards to
the Java `toString()` and both managed peers wrap the *same* Java
instance. So the message tells us only that the peers differ -- not why.

`JniValueManager.GetPeer()` peeks the registry before creating a new
peer, so a mismatch means `PeekPeer()` missed an entry that
`Application.Context` still holds strongly. Replace the assert with a
failure-only diagnostic that distinguishes the possible causes:

  * managed hash codes of both peers, so they can be told apart
  * `JniIdentityHashCode`, `PeerReference` and `JniManagedPeerState`
  * whether `Application.Context` still returns the captured instance
  * `IsSameObject()` on the two peer references
  * what `PeekPeer()` returns now, and whether it is the expected peer,
    the unexpected one, or nothing at all
  * every surfaced peer sharing the identity hash code

That separates "registry entry was replaced by a second `AddPeer()`"
from "entry was evicted" from "weak reference was cleared", which is
the information needed to find the root cause.

Verified on device by temporarily forcing the failure branch; the
diagnostic runs to completion without throwing, and reports the
expected all-`True` values in the healthy case.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
@jonathanpeppers
jonathanpeppers force-pushed the jonathanpeppers-probable-fishstick branch from fae07a2 to 5ed818f Compare July 29, 2026 18:38
@jonathanpeppers jonathanpeppers changed the title [Mono.Android] make Application.Context initialization thread-safe [tests] make GetObjectArray peer mismatch self-diagnosing Jul 29, 2026
jonathanpeppers and others added 7 commits July 29, 2026 16:34
The self-diagnosing assert added previously caught a real CI failure and
reported:

	expected (Application.Context): App@managed-0x109791d PeerReference=0x2d66/G
	actual   (GetObjectArray ()[0]): App@managed-0x1ee05f PeerReference=0x30aa/G
	PeekPeer (expected.PeerReference) => App@managed-0x1ee05f
	  is expected: False; is actual: True
	Surfaced peers with JniIdentityHashCode=0x87e2d7a:
	  App@managed-0x1ee05f

So the registry ends up holding exactly one peer, and it is *not* the one
`Application.Context` caches. But that end state does not say when the
registry stopped agreeing with `Application.Context`, and the two
candidate stories predict the same final state:

  * the entry was still present and got replaced while marshaling, or
  * the entry was already gone before this test ran, so `PeekPeer()`
    legitimately missed and a fresh peer was created and registered.

Note the first story requires `PeekPeer()` to miss the entry while
`AddPeer()`, microseconds later, finds it live and matching -- both using
the same `Target` and `IsSameObject()` checks. Sampling the registry
before the marshaling distinguishes them directly instead of by argument.

Probe `PeekPeer (Application.Context)` at test entry, before `NewArray()`
and after `NewArray()`, and report the three samples on failure. Record a
description instead of the peer itself, so the probe cannot keep a peer
alive and perturb the GC behaviour under investigation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
…stration

The probe added in the previous commit answered the question it was asked.
From a CI failure on `Mono.Android.NET_Tests-CoreCLRTrimmable`:

	expected (Application.Context): App@managed-0x240d846 PeerReference=0x2d36/G
	actual   (GetObjectArray ()[0]): App@managed-0x31cb802 PeerReference=0x3052/G
	PeekPeer (Application.Context) over time:
	  at test entry:      App@managed-0x31cb802 (is expected: False)
	  before NewArray:    App@managed-0x31cb802 (is expected: False)
	  after NewArray:     App@managed-0x31cb802 (is expected: False)

The registry already held the wrong peer *at test entry*, before this test
touched anything. `GetObjectArray()` is not where the bug happens; it is
merely the first test that looks. That rules out the marshaling path, and
with it every hypothesis framed around `NewArray()`/`GetObjectArray()`.

So move the check to where it can name a culprit: an assembly-level
`ITestAction` samples `PeekPeer (Application.Context)` after every test and
records the first test after which it stops returning the peer that
`Application.Context` caches. `GetObjectArray()`'s failure message then
reports that test, narrowing ~430 candidates to one and making a local
repro possible.

`PeekPeer()` only reads the registry -- unlike `GetSurfacedPeers()` it does
not drain the collected-peer queue -- so sampling it this often should not
perturb the behaviour being measured.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The per-test watcher named a culprit on its first CI run:

	Registry first diverged after test: Android.AppTests.ApplicationTest.ApplicationContextIsApp

That test is the *first* test in the run -- `Android.AppTests` sorts ahead
of every other namespace -- and its body is only:

	Assert.IsTrue (Application.Context is App);
	Assert.IsTrue (App.Created);

which does nothing that could plausibly evict a peer. So the divergence
almost certainly predates the test run entirely and happens during app
startup. But sampling only in `AfterTest` cannot prove that: it cannot
separate "already broken when the run started" from "broken by this test".

Sample in `BeforeTest` as well and record which one observed it first, so
the report reads `before <test>` or `after <test>`.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The watcher pinned down *when* the registry diverges:

	Registry first diverged: before Android.AppTests.ApplicationTest.ApplicationContextIsApp

`before`, on the first test of the run -- so the registry is already wrong
when the test run starts, and the divergence happens during app startup.
The greferences say which way it went: `Application.Context` caches
`0x2d66/G` while the registry holds the later `0x3086/G`, i.e. a newer peer
displaced the one `_context` had already cached.

Two mechanisms produce that, and build 1529081 separates them. That build
tested `f0e746d8c` (a merge of `bfcad439e`), which did contain the
`target.Replaceable && !value.Replaceable` guard -- verified with `git show`
-- and `GetObjectArray` still failed:

  * `AddPeer()` *replacing* the entry: blocked by that guard, so ruled out.
  * the entry being *evicted*, so a later `PeekPeer()` legitimately misses
    and registers a fresh peer: unaffected by the guard, and consistent.

Eviction also fits the rest: exactly one surfaced peer, `Application.Context`
holding the older one, CoreCLR-only (`CollectPeers()` has no MonoVM
counterpart), and GC-timing flakiness.

A context only reaches `CollectedContexts` because the GC bridge reported
its peer collected, so by the time `CollectPeers()` evicts it the peer's
weak reference should already be cleared. Record the cases where it is not
-- a live managed peer, possibly strongly held by `Application.Context` --
and publish the count via `AppContext` so the test can report it without
`InternalsVisibleTo`.

Recording only: the eviction still happens, so this measures the bug rather
than masking it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The eviction hypothesis is dead. Build 1532632 reproduced the failure with
the recorder in place and reported:

	Live peers evicted by CollectPeers(): <none>

So `CollectPeers()` never drops a live registration, and the peer
`Application.Context` caches is not being evicted.

Re-reading build 1529081's failure message confirms it was this same
assertion, on a build that did carry the `Replaceable`/`Replaceable` guard.
Taken together with the current dumps, both configurations diverge, but in
opposite directions:

  * without the guard, the registry follows the newer peer while
    `Application.Context` holds the older one (gref 0x2d66 vs 0x30c6);
  * with the guard, the registry keeps the older one instead.

Which one `AddPeer()` picks is therefore not the bug. The invariant that
actually breaks is upstream of that choice: a *second* managed peer is
created for the same Java Application instance at all. Whichever one the
registry settles on, `Application.Context` has already cached the other.

`AddPeer()` sees that happen in exactly two places -- replacing an existing
entry, and appending alongside one whose weak reference was cleared -- so
record both, with the identity of the old and new peer, and report them
from the failing test. Recording only; behaviour is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
Fixes: #10973

`JnienvArrayMarshaling.GetObjectArray()` intermittently failed on CoreCLR
(~17 failures / 216 executions) and never on MonoVM or NativeAOT:

	Expected existing Context peer, got Android.AppTests.App.
	Expected: same as crc64f5183e500568449d.App@8e34fe2
	But was:  crc64f5183e500568449d.App@8e34fe2

Both sides print identically because they are two *managed* peers over the
same Java instance, so the assert's own message could not explain it. A
test-only diagnostic, iterated over several CI runs, produced this:

	Registry first diverged: before Android.AppTests.ApplicationTest.ApplicationContextIsApp
	CollectPeers() evicted live peers:  <none>
	AddPeer() replaced registrations:   1 (most recent: key=0xe45a52a
	    replaced App@managed-0x33c0d9d/0x2d66/G
	    with     App@managed-0x11c7a8c/0x2d76/G)
	AddPeer() appended duplicates:      <none>

`before` on the run's first test means the registry is already wrong when
testing starts, so this happens during app startup; `GetObjectArray()` was
only the first test to look. And exactly one replacement occurs in the whole
process: a second `App` peer displacing the one `Application.Context` had
cached.

`Application.Context` caches the peer it first sees and never re-reads it,
so replacing that registration orphans the cached reference permanently --
`PeekPeer()` then hands out the replacement for the same Java instance.
Whether the run fails is a startup race: if `Application.Context` is read
before the replacement it keeps the orphaned peer and fails; if after, it
caches the survivor and passes.

`AddPeer()` replaced whenever the *existing* peer was `Replaceable`, without
checking the new one. MonoVM's `AndroidRuntime.JavaObjectValueManager` also
requires the new peer to be non-`Replaceable` -- a "replaceable" instance
must not replace another "replaceable" instance, per #9862 --
which is why MonoVM never hit this. Apply the same condition here.

Also guard the `WarnNotReplacing()` call with
`LogGlobalReferenceMessages`: that branch is now reached during normal
startup, and its arguments are evaluated eagerly -- including two
`GetJniTypeNameFromInstance()` JNI round-trips -- even though
`WriteGlobalReferenceLine()` discards them when logging is off. MonoVM
guards its equivalent warning the same way.

The diagnostics are kept: they are what turned an unexplainable assert into
a located bug, and they let CI confirm the fix directly -- a green run must
now report no replacement of the `Application.Context` peer, rather than
merely not failing a ~50/50 coin flip.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
The `Replaceable`/`Replaceable` guard works -- build 1532806 reports
`AddPeer() replaced registrations: <none>` -- and `GetObjectArray()` still
fails. That also explains build 1529081, which carried the same guard and
failed the same way, and it means the guard is MonoVM parity rather than the
fix for this.

What the failing dump now shows is a registry state that none of the
instrumented paths produced:

	CollectPeers() evicted live peers:  <none>
	AddPeer() replaced registrations:   <none>
	AddPeer() appended duplicates:      <none>
	Surfaced peers with JniIdentityHashCode=0x17e2473:
	  Android.AppTests.App@managed-0xf5e94d

The registry holds the peer that `Application.Context` did *not* cache, yet
nothing replaced, appended or evicted anything. Two possibilities remain,
and both run through code that is currently silent:

  * the registry's peer registered *first* and the guard then turned away
    the one `Application.Context` went on to cache -- the guard's
    "not replacing" branch records nothing once
    `WarnNotReplacing()` is skipped for logging being off; or
  * the cached peer was registered and then removed by `RemovePeer()`, or
    by a `CollectPeers()` eviction of an already-cleared entry, neither of
    which is counted.

Record all three. The "declined to replace" entry also establishes
registration order, since the peer that registers first is the one kept.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ce866f68-cb01-44de-bce0-7457f7612b72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge PR should not be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants