Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,57 +2,42 @@

## Goal

Implement and verify the behavior tracked in [#2676](https://github.com/moq-dev/moq/issues/2676)
within the issue's stated scope and boundaries.
A C consumer that closes its session, waits for the terminal status, and
returns from `main` exits cleanly on Linux/glibc. The success path of
`test/smoke/clients/c/subscribe.c` returns instead of calling `_exit` to dodge
teardown.

## Plan

Use the public issue's scope, implementation notes, and acceptance criteria
below as the starting plan. Reconcile paths and assumptions with the current
tree before implementation.

### Issue context

#### Symptom

A C program that links `libmoq.a`, consumes a video track, and then exits normally (returning from `main`, or calling `exit`) can abort during atexit teardown with glibc's priority-protected-mutex assertion in `__pthread_tpp_change_priority`. The process dies with SIGABRT (exit 134) and a core dump *after* the MoQ data path has already worked correctly.

This has only been observed on Linux/glibc.

#### Where it bit us

The C leg of the interop smoke test (`test/smoke/clients/c/subscribe.c`). 17bf0d950 worked around it by having the client `_exit(0)` the moment it has its frame, skipping atexit teardown entirely, with the commit message "Fixes the flaky rust -> c leg". The workaround was never accompanied by a diagnosis, so per CLAUDE.md's Root Cause First this should be treated as a real bug rather than a flake that's been made to go away.

moq-dev/moq#2675 extends the same `_exit` to the client's failure paths, for a separate reason (a use-after-free on `user_data`). It does not touch this.

#### Why this matters beyond the smoke test

The smoke client is not the only C consumer. `cpp/obs` links `libmoq` the same way, and the C API's whole contract is "call `moq_session_close`, wait for the terminal `on_status`, exit". A consumer that follows that contract exactly should not abort. Right now the only thing keeping our own test green is that it skips the C runtime's exit path, which is not advice we can reasonably give an embedder.

#### What is known

- `libmoq.a` statically bundles `moq-video`, whose software H.264 fallback is openh264 (vendored C++; `stdc++` is in `rs/libmoq/native-libs/linux.txt` for exactly that reason).
- Nothing in our Rust code creates a priority-protected mutex. `grep` for `PTHREAD_PRIO`/`pthread_mutexattr`/`SCHED_` across `rs/moq-video` and `rs/libmoq` comes back empty, so the mutex in question comes from a dependency or is not really a PRIO\_PROTECT mutex at all (see below).
- libmoq runs its own tokio runtime on a `LazyLock` thread (`rs/libmoq/src/state.rs`), which is never shut down; its threads are still live when C's atexit handlers and static destructors run.

#### Hypotheses, none confirmed

1. **A `pthread_mutex_t` is used after free during teardown.** `__pthread_tpp_change_priority` is only reached when `m->__data.__kind` says PRIO\_PROTECT. If the mutex memory has already been freed, a garbage `__kind` can route an ordinary unlock/destroy down the TPP path with a nonsense priority, which is precisely what that assertion catches. This would make the assertion a *symptom* of a lifetime bug, not of anything priority-related.
2. **A race between the still-running tokio/openh264 worker threads and C's static destructors.** We never stop the runtime, so a worker can touch state whose destructor already ran.
3. **Something in the static-link arrangement**, e.g. openh264's C++ statics being destroyed while a thread is inside them.

Hypothesis 1 is the one worth checking first, because it is the only one that would also be a live bug for a well-behaved embedder rather than a teardown-ordering nuisance.

#### Suggested investigation

- Reproduce with a minimal C program: connect, consume video, `moq_session_close`, wait for the terminal `on_status`, `return 0`. Loop it to get a hit rate.
- Run it under ASan and under valgrind/helgrind. If hypothesis 1 is right, ASan should name the freed mutex directly.
- Get a backtrace from the core: whether the abort comes from `pthread_mutex_destroy`, `pthread_mutex_unlock`, or a static destructor narrows this a lot.
- Check whether it still reproduces with `moq-video`'s software H.264 fallback out of the link, which would implicate openh264 specifically.

#### Definition of done

A C consumer that returns from `main` after closing its session exits cleanly, and `test/smoke/clients/c/subscribe.c` no longer needs `_exit` to dodge teardown.
A program linking `libmoq.a` that consumed video can abort in atexit teardown
with glibc's priority-protected-mutex assertion, after the data path worked.
The smoke client works around it with `_exit(0)` the moment it has its frame
(17bf0d950, "fixes the flaky rust -> c leg"), with no diagnosis. `cpp/obs`
links `libmoq` the same way and follows the documented contract exactly, so
this is a real bug for embedders, not a flake.

What is known: `libmoq` runs a tokio current-thread runtime on a detached
`libmoq` thread behind a `LazyLock` (`rs/libmoq/src/ffi.rs`) that is never
shut down, and its global `State` is likewise never torn down, so worker
threads are live when C's atexit handlers and C++ static destructors run.
openh264 (vendored C++) is always linked. Nothing in our Rust creates a
priority-protected mutex, so either a `pthread_mutex_t` is used after free and
a garbage `__kind` routes an ordinary unlock down the TPP path, or a still
running thread touches state whose destructor already ran.

- Reproduce with a minimal C program (connect, consume video, close, wait for
the terminal status, `return 0`) in a loop to get a hit rate. Take a
backtrace from the core: `pthread_mutex_destroy`, `unlock`, or a static
destructor narrows it a lot.
- Build `libmoq` under AddressSanitizer (the fuzz recipe in `rs/justfile`
shows how to step off the pinned toolchain for `-Zsanitizer`) and run the
loop; if a freed mutex is the cause ASan names it. Check whether it still
reproduces with openh264 out of the link.
- Fix the mechanism where it lives. If it is teardown ordering, an atexit
handler that quiesces the runtime thread before static destructors run is
the likely shape; do not add a `moq_shutdown` the contract does not ask for.
- Remove `_exit` from the smoke client's success path. The failure paths keep
it for the separate `user_data` lifetime reason from #2675.
Comment on lines +39 to +40

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Drain every callback before returning from the smoke client

When the success path is changed to return, waiting only for moq_session_connect's terminal status does not make the stack-allocated ctx_t safe to destroy. The same pointer is independently retained by moq_origin_consume_announced, moq_consume_catalog, and moq_consume_video, and each registration can use it until its own terminal callback; closing the session terminates only its status registration. A late catalog or frame callback can therefore access the stack or destroyed pthread objects after main returns, making the regression test fail for a second lifetime bug instead of isolating atexit teardown. Extend the plan to close and drain every callback registration before removing _exit.

AGENTS.md reference: AGENTS.md:L148-L152

Useful? React with 👍 / 👎.


## Closes

Expand Down
53 changes: 25 additions & 28 deletions quest/m0/2860-cpp-obs-moq-source-cpp-has-no-test-coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,34 @@

## Goal

Implement and verify the behavior tracked in [#2860](https://github.com/moq-dev/moq/issues/2860)
within the issue's stated scope and boundaries.
`just obs test` exercises the consume path: the connection-epoch and
subscription-refcount bookkeeping in `cpp/obs/src/moq-source.cpp` is pinned by
tests, so a change like #2856 cannot blank OBS sources without a red build.

## Plan

Use the public issue's scope, implementation notes, and acceptance criteria
below as the starting plan. Reconcile paths and assumptions with the current
tree before implementation.

### Issue context

#### Problem

`cpp/obs/test/moq-output-test.cpp` covers the publish path only. Its libmoq stub implements `moq_origin_create`, `moq_origin_close`, and `moq_origin_publish` - nothing on the consume side - so `moq-source.cpp` is exercised by nothing. `just obs test` reports success without touching it.

That matters more than usual for this file: PR CI never compiles the plugin at all (see `cpp/obs` in the root CLAUDE.md), so `just obs test` is the only automated gate the source path could have, and it currently isn't one.

It bit us in #2856. That PR stopped `moq_net::Client::connect` blocking on the initial announce set, which left `moq_source_start_consume` calling `moq_origin_request` (resolve against what is announced *now*) off the session-connected callback. The announcement had not necessarily arrived, so `on_broadcast` got `Unroutable`, the source blanked, and because consumption is started only for connection epoch 1 nothing ever retried it: OBS stays blank until the user restarts the source. It was caught in adversarial review rather than by a test.

#### Suggested coverage

The stub set needs the consume half: `moq_origin_consume_announced` (+`_close`), `moq_origin_request` (+`_close`), `moq_consume_catalog`, `moq_consume_track`, `moq_consume_close`. The orderings worth pinning, all of which the current build cannot check:

- Connected fires, the announcement arrives *later*, and the source still subscribes.
- A broadcast that is never announced: the wait stays pending and `moq_source_disconnect` closes it, firing the terminal exactly once.
- A reconnect (epoch 2) while an epoch-1 delivery is still in flight: the stale delivery is dropped on the generation check and its handle is closed, not leaked.
- Terminal-callback refcounting: `refs` returns to zero on each of the delivered / errored / closed paths.

The generation and `subscription_ref` bookkeeping in that file is exactly the kind of thing the comments say the build can't verify, which is the argument the existing output test already makes for itself.

Found by the Codex adversarial review on https://github.com/moq-dev/moq/pull/2856.
`cpp/obs/test/` holds one file, `moq-output-test.cpp`, whose libmoq stub set is
publish-side only (`moq_origin_publish`, `moq_publish_*`, session and client
connect/close). `moq-source.cpp` is compiled by CMake but appears nowhere under
`test/`, and the `just obs test` recipe compiles exactly two translation units
under ThreadSanitizer. PR CI never runs this, so the source path has no
automated gate at all. It bit in #2856: consumption started from the
connected callback resolved against what was announced *now*, the announcement
had not arrived, and nothing retried; caught in review rather than by a test.

- Add `moq-source-test.cpp` with the consume half of the stub: announced
listing, `moq_origin_request`, catalog and track consume, and their closes,
matching the names in `rs/libmoq/src/api.rs`.
- Pin the orderings the build cannot check: connected fires and the
announcement arrives later, and the source still subscribes; a broadcast
never announced stays pending and `moq_source_disconnect` closes it, firing
the terminal exactly once; a reconnect (epoch 2) while an epoch-1 delivery is
in flight drops the stale delivery on the generation check and closes its
handle; refcounts return to zero on the delivered, errored, and closed paths.
- Wire the new binary into `just obs test` under the same TSan build, and
also run it without TSan from `just obs ci`, which is what `obs.yml`
invokes; `just obs test` is manual, so on its own it cannot make the build
red. Mention both in `doc/bin/obs.md` next to the output test.

## Closes

Expand Down
Original file line number Diff line number Diff line change
@@ -1,37 +1,29 @@
# [S] obs: the plugin targets OBS 31.1.1 while Linux CI links against nixpkgs' 32.1.2
# [S] obs: the plugin targets OBS 31.1.1 while Linux CI links against 32

## Goal

Implement and verify the behavior tracked in [#2868](https://github.com/moq-dev/moq/issues/2868)
within the issue's stated scope and boundaries.
The libobs the released macOS and Windows plugin binaries link against is the
same major as the one the Linux compile gate checks, so a libobs API change
cannot pass CI and surface only in a release build on the platforms with no
gate.

## Plan

Use the public issue's scope, implementation notes, and acceptance criteria
below as the starting plan. Reconcile paths and assumptions with the current
tree before implementation.

### Issue context

Split out of a review finding on #2867 (CodeRabbit flagged the stale pin; the divergence below is what makes it worth its own issue).

#### The two versions

- `cpp/obs/buildspec.json` pins **obs-studio 31.1.1**. That's what the obs-deps download gives a macOS or Windows build, so it's the libobs the *released* plugin binaries link against.
- nixpkgs currently carries **obs-studio 32.1.2**, which is what a Linux `just obs build` and the new `obs.yml` gate link against.
- `flake.nix`'s `libobs-headers` also pins 31.1.1, matching buildspec.json (#2867 added a guard in `just obs check` that fails if those two drift).

So the Linux compile gate and the shipped macOS/Windows binaries are a major OBS release apart. Nothing has broken yet, but that gap is exactly where a libobs API change gets through CI and shows up only in a release build, on the platforms with no compile gate at all.

#### What to do

Bump `cpp/obs/buildspec.json` to the current stable (32.2.2 at time of writing) and move `libobs-headers` in `flake.nix` in the same commit. Needs:

- new hashes for the obs-studio, prebuilt obs-deps and Qt6 archives (macOS + Windows entries), and the nix `fetchzip` hash;
- a check that `libobs/obsconfig.h.in` and `frontend/api/obs-frontend-api.h` are still where `libobs-headers` expects them;
- a real `just obs build` on macOS, since that's the platform the release actually ships and the one PR CI never compiles.

`just obs check` will fail until both pins move together, which is the intended tripwire rather than an obstacle.
`cpp/obs/buildspec.json` pins obs-studio 31.1.1, which is what obs-deps hands
a macOS or Windows build. `flake.nix`'s `libobs-headers` pins the same, and
`just obs check` fails if the two drift. Neither is what Linux links: the
`obs.yml` gate uses nixpkgs' obs-studio, a major release ahead, and nothing
compares that third version to the other two.

- Bump `buildspec.json` and `libobs-headers` together to the current stable,
32.2.2 as of 2026-08-14: new hashes for the obs-studio, prebuilt obs-deps,
and Qt6 archives (macOS and Windows) plus the nix `fetchzip` hash.
- Check `libobs/obsconfig.h.in` and `frontend/api/obs-frontend-api.h` are
still where `libobs-headers` expects them.
- Extend the drift guard to the nixpkgs version `just obs ci` links against,
so the next gap is a failing check rather than a discovery.
- A real `just obs build` on macOS, since that is the platform the release
ships and PR CI never compiles.

## Closes

Expand Down
Original file line number Diff line number Diff line change
@@ -1,45 +1,42 @@
# [M] moqsink: the publication has no generation, so a flush after EOS cannot restart it
# [L] moqsink: a flushing restart after EOS opens a new publication generation

## Goal

Implement and verify the behavior tracked in [#3115](https://github.com/moq-dev/moq/issues/3115)
within the issue's stated scope and boundaries.
`FLUSH_STOP` after `moqsink` has completed EOS resumes data flow, as GStreamer
specifies: the element publishes a fresh broadcast, catalog, and producers
without cycling through `READY`, and a buffer arriving after EOS is either
written into the new generation or refused, never into finalized producers.

## Plan

Rescoped during the 2026-08 grooming: items 2 and 3 landed on main via #2998
(Completion/CompletionHandle) and #3104. What remains is item 1: give the
publication a generation so an element-wide flush after EOS can restart it.

### Issue context

Split out of the adversarial reviews on #3101, #3102 and #3104. Each of those fixes a concrete `moqsink` lifecycle bug, and each one stops at the same wall: **a `moqsink` publication has no notion of a generation.** Once it ends, the only way back is a cycle through `READY`.

Three findings across those PRs are all the same root cause.

##### 1. A flushing seek after the element completed EOS is broken

Once the last pad sends EOS, `maybe_post_eos` finalizes every producer and takes the catalog. There is no way to reopen them. #3104 makes a post-EOS buffer answer `FlowError::Eos` instead of writing into finalized producers, which is an improvement over silently dropping it, but GStreamer specifies that `FLUSH_STOP` clears EOS and data flow resumes. We cannot honour that today.

A correct fix creates a new publication generation on a flushing restart (new broadcast/catalog/producers) rather than treating the first EOS as terminal for the element.

##### 2. `eos_posted` conflates two facts

It answers both "the producers were finalized" and "the EOS message was posted". #3104 gates the flush reset on it and therefore inherits the conflation. #2998 separates the two, which is what its per-pad lifecycle rewrite buys, but that separation does not reproduce standalone.

##### 3. The identity check and the bus post are not atomic

`post_session_error` (#3102) checks whether a session is still current, releases the element lock, then posts. A `PAUSED -> READY -> PAUSED` completing inside `post_message` still lands a stale error on the replacement's bus. The lock cannot be held across the post: `post_message` runs bus sync handlers inline on the calling thread, and a handler that reads an element or pad property would deadlock on it.

The natural remedy (an in-flight posting permit that teardown waits on) has the same problem in a different place: a sync handler calling `set_state(READY)` re-enters teardown on the thread already holding the permit. #2998 hits this too and accepts the window explicitly.

Deferring the post to a main-loop idle source would decouple it, at the cost of changing when the error is delivered and requiring a running main loop.

##### Why this is filed rather than fixed

Each PR is a strict improvement over `main` and none of them can close these without redesigning the publication lifecycle. That redesign overlaps heavily with #2998, so it should be settled once rather than three times.

@arielmol - flagging you since #2998 is the closest thing to a design for this, and its `Completion` state machine is most of the way to a generation. Worth deciding whether generations belong in that PR or in a follow-up on top of it.
#2998 landed `Completion` in `rs/moq-gst/src/sink/session.rs`: a monotonic
per-session state (`Open`, `Eos`, `Failed`) whose first terminal transition
wins, with session identity carried by pointer equality on the handle. That
is the right shape for one generation and deliberately has no way back. In
`sink/imp.rs`, `FlushStart`, `FlushStop`, and `StreamStart` are all guarded on
the live state being open, so once `maybe_finish_locked` finished the
completion they are no-ops, `render` keeps answering `FlowError::Eos`, and the
only reset is `start_session` on the `READY` transition.

Design, settled with the #2998 author: `Completion::Eos` stays terminal for
its generation. The first `FLUSH_STOP` after EOS opens one new generation
globally (a fresh `CompletionHandle`, broadcast, catalog, and producers);
every other pad joins that generation on its next buffer rather than each pad
opening its own, so aggregate EOS membership is one set per generation.

- Separate "the producers were finalized" from "the EOS message was posted";
the per-pad lifecycle from #2998 already distinguishes them for pads, and
the element-level latch (`eos_delivered`) needs the same split so a
generation can post EOS again.
- Pad lifecycles reset into the new generation on join, reusing
`lifecycle.reset()` from `start_session`.
Comment on lines +31 to +32

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Rebuild each producer before its first post-flush buffer

lifecycle.reset() replaces the pad's media state with Pad::new(), clearing both the cached caps and track producer. With the proposed lazy join on the next buffer, a normal post-FLUSH_STOP sequence may resend a segment but not CAPS because CAPS remains sticky, so Pad::push_buffer sees no track and silently drops that buffer. Preserve or replay the pad's sticky caps and reconstruct its producer before accepting the first buffer into the new generation, or the second broadcast can contain no media despite the stated test passing through the join path.

Useful? React with 👍 / 👎.

- The stale-error window in `post_session_error` (identity check, then a bus
post outside the lock, which a sync handler may re-enter) is a separate
delivery problem that generations do not close; leave it as documented.
- Tests: EOS on every pad, then `FLUSH_STOP` and buffers, asserts a second
broadcast with a second catalog and a second EOS; a pad that flushes late
joins the current generation, not a third; the post-EOS buffer without a
flush still answers `Eos`.

## Closes

Expand Down
Loading
Loading