[Bug][Linux] native-main 503 when systemctl exists but the user bus is unreachable (#1612 regression gap)

[lidge-jun]

Client or integration

Codex CLI

Area

Service lifecycle

Summary

OpenCodex 2.26.0 blocks every native OpenAI request on a systemd-less Linux container when systemctl is present on PATH but the user bus is unreachable. The proxy stays live, /healthz returns 200, and ocx ready reports {"ready":true,"status":"ready"}, but every native request fails locally with HTTP 503 before reaching upstream:

{"error":{"message":"OpenCodex local native-main profile maintenance is active; retry this request","type":"server_error","code":"server_is_overloaded"}}

This is the same admission fence as #1612, but that fix does not cover this environment. #1612 was closed by treating a systemctl spawn failure as absent. In this container systemctl spawns fine — it exits non-zero because there is no user bus:

$ command -v systemctl
/usr/bin/systemctl

$ systemctl --user show opencodex-proxy -p LoadState -p ActiveState -p FragmentPath -p NeedDaemonReload
Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined
EXIT=1

inspectSystemd() only maps spawnFailed to absent; a non-zero exit still returns unknown, which re-triggers the native-main block. The /.dockerenv marker is present and PID 1 is tini, so this is exactly the externally-supervised foreground shape src/service.ts recommends (Service: unsupported in Docker, "run ocx start directly").

Reproduction

1. Debian 13 container (PID 1 = tini, /.dockerenv present) with systemd installed but no user bus: /run/user/$UID empty, XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS unset.
2. npm install -g @openai/codex@0.148.0 @bitkyc08/opencodex@2.26.0
3. codex login (ChatGPT), confirm native codex exec works.
4. ocx start, then ocx sync.
5. Send a native request:

$ curl -s -X POST http://127.0.0.1:10100/v1/responses \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5.5","input":"hi","stream":false}'
{"error":{"message":"OpenCodex local native-main profile maintenance is active; retry this request",...}}

codex exec surfaces this as a misleading message, which is what a user actually sees:

ERROR: Selected model is at capacity. Please try a different model.

ocx restart does not clear it; ocx ready still reports ready:true while every request 503s.

Control flow

systemd installed, no user bus (Docker/tini)
-> inspectSystemd() runs `systemctl --user show` -> spawn SUCCEEDS, exit 1 (no bus)
-> service-manager-probe.ts: !spawnFailed, status !== 0 -> unknown("systemctl show exited 1: ...")
-> ownership-preflight.ts: manager.kind === "unknown" -> ownership: "unknown"
-> server/index.ts: blockNativeMainStartupForUnownedServiceHome("ownership-unknown")
-> native-profile-startup.ts: isNativeMainTrafficBlocked() === true
-> auth-context.ts: throw CodexMainProfileDrainingError -> 503

The #1612 fix (if (shown.spawnFailed) return { kind: "absent" }) sits one line above the branch that fires here.

Verified workaround

Removing systemctl from the proxy's PATH makes the probe hit the spawnFailed path fixed in #1612, and native traffic recovers immediately:

$ mkdir -p ~/ocx-nosysd && for f in /usr/bin/*; do b=$(basename $f); [ "$b" = systemctl ] && continue; ln -sf $f ~/ocx-nosysd/$b; done
$ ocx stop
$ setsid env PATH=$HOME/ocx-nosysd:/usr/local/bin ocx start &

$ curl -s -X POST http://127.0.0.1:10100/v1/responses -H 'Content-Type: application/json' -d '{"model":"gpt-5.5","input":"hi"}'
{"detail":"Input must be a list"}          # request reached the handler; no 503

$ codex exec "Reply with exactly: PROXY_OK"
PROXY_OK                                    # tokens used 3,922

Same binary, same config, same login — the only variable is whether systemctl is reachable on PATH. That isolates the probe result as the sole cause.

Setting XDG_RUNTIME_DIR=/run/user/1000 does not help: the directory exists but has no bus socket, so systemctl still exits non-zero ("Failed to connect to user scope bus via local transport: No such file or directory").

Suggested fix

Treat "the question never reached a user bus" the same as an unaskable manager, while keeping fail-closed behavior for real evidence.

1. In inspectSystemd(), classify a no-bus failure as absent rather than unknown. Prefer matching the bus-connection failure specifically (stderr Failed to connect to ... bus), not any non-zero exit.
2. Guard it with the container/foreground signal already used elsewhere (/.dockerenv, PID 1 not systemd), so a real desktop Linux box with a transient bus problem still returns unknown.
3. Keep every other path fail-closed: a present unit, NeedDaemonReload=yes, malformed state, conflicting mirrors, or foreign-home evidence must still yield unknown/foreign.

Regression coverage worth adding:

• Docker + systemctl present + bus unreachable (exit 1) -> native-main admits traffic.
• Docker + systemctl present + unit actually installed for a foreign home -> still blocked.
• Non-Docker Linux + bus unreachable -> still unknown (no widening on real hosts).

Two secondary issues

1. The user-facing error is wrong. A local admission fence is reported to the CLI as ERROR: Selected model is at capacity. Please try a different model. That sends users to change models or wait for quota, when the cause is local and the fix is ocx restore or the PATH workaround. The 503 body is accurate; the CLI-visible mapping is not.

2. ocx ready reports ready while every request 503s. {"ready":true,"status":"ready"} with a 100% failure rate makes readiness unusable as a health signal in exactly the automated/container context where it matters most. Readiness could reflect isNativeMainTrafficBlocked() when native-main is the only configured route.

Also worth noting: ocx status correctly says Service: unsupported in Docker and ocx doctor warns AT RISK after restart (no viable background service; run 'ocx restore'). The product already knows it is in an unsupported-service environment at the moment it blocks traffic for not being able to prove service ownership.

Environment

• OpenCodex 2.26.0 (npm global), Codex CLI 0.148.0
• Debian GNU/Linux 13 (trixie), kernel 6.12.94, x86_64
• Container: /.dockerenv present, PID 1 = tini, /run/user/1000 empty
• systemd installed (/usr/bin/systemctl), user bus unreachable
• Bundled Bun 1.3.14 runtime, port 10100
• Auth: codex login (ChatGPT), native codex exec verified working before ocx sync

[coderabbitai]

🔗 Related PRs

#306 - feat(windows): add tray controls and restart-safety diagnostics [merged]
#403 - feat(grok): auto-configure Grok Build and harden strict-client compatibility [merged]
#569 - Add post-sync readiness endpoint and bounded ocx ready wait [merged]
#576 - fix(codex): warn about stale Codex app-servers after a catalog write [merged]
#770 - fix: research-batch reliability and security-boundary issues [closed]
#1160 - fix(service): preserve supervision across restart [open]
#1647 - fix(tray): let the service wrapper exit 0 on an already-live proxy [open]
#1721 - refactor(adapters): centralize runtime adapter authority - Part 1 [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[lidge-jun]

리뷰 · 우선순위 74 / 80

Docker/tini 컨테이너에서 systemctl은 PATH에 있는데 user bus는 없는 환경이다. ocx start와 ocx sync는 되고 /healthz·ocx ready도 ready인데, 네이티브 OpenAI 요청만 로컬 HTTP 503으로 막힌다. 본문은 OpenCodex local native-main profile maintenance is active; retry this request이고, Codex CLI는 이걸 Selected model is at capacity로 보여 준다. #1612가 고친 건 systemctl spawn 실패를 absent로 보는 길이었고, 이번엔 spawn은 성공하고 exit 1이라 그 우회를 못 탄다. PATH에서 systemctl만 빼면 바로 살아나는 게 원인 분리이고, 기본 native 경로가 컨테이너 설치 직후부터 막혀서 74다. 데스크톱 systemd 호스트의 기본 설치/재부팅 사고는 아니라 80은 아니다.

막히는 지점은 src/service-manager-probe.ts의 inspectSystemd()다. systemctl --user show opencodex-proxy -p LoadState -p ActiveState -p FragmentPath -p NeedDaemonReload를 치고, shown.spawnFailed일 때만 { kind: "absent" }다. shown.status !== 0이면 주석 그대로 “the question never reached the bus”인데도 unknown("systemctl show exited 1: …")으로 돌려보낸다. 리포터 stderr인 Failed to connect to user scope bus via local transport가 바로 그 분기다. XDG_RUNTIME_DIR만 채워도 소켓이 없으면 같은 exit 1이 난다.

그 unknown이 서비스홈을 미소유로 고정한다. src/integrations/native/ownership-preflight.ts의 inspectNativeCodexOwnership()은 manager.kind === "unknown"이면 { ownership: "unknown" }이다. src/server/index.ts의 startServer()는 inspectStartupOwnership() 결과가 owned가 아니면 복구 라이프사이클을 안 타고 blockNativeMainStartupForUnownedServiceHome("ownership-unknown")으로 간다. src/codex/native-profile-startup.ts 스냅샷이 blocked가 되면 isNativeMainTrafficBlocked()가 true고, src/codex/auth-context.ts가 CodexMainProfileDrainingError를 던져 Responses가 503을 만든다. ocx restart는 같은 프로브를 다시 돌릴 뿐이라 풀리지 않는다.

제품은 이미 이 환경을 서비스 불가로 알고 있다. src/service.ts의 platformOps()는 /.dockerenv가 있으면 “Run ocx start directly”로 서비스 설치를 거절하고, status는 unsupported in Docker를 준다. doctor도 백그라운드 서비스가 없다고 경고한다. 그런데 ownership 프로브는 그 신호를 안 보고, systemctl이 PATH에만 있으면 질문을 버스에 던진 뒤 실패를 unknown으로 승격한다. 서비스가 아예 없는 foreground 컨테이너에서 “소유를 증명 못 함”으로 native-main을 닫는 셈이다. 옆 이슈 #1939도 같은 systemctl show exited 1 / D-Bus 실패 계열이다.

부수 증상 두 개도 코드와 맞다. ocx ready는 /readyz의 동기화 게이트만 보고 isNativeMainTrafficBlocked()는 안 본다. 그래서 native-main이 유일한 경로여도 {ready:true}가 나온다. CLI의 “model at capacity”는 로컬 admission 펜스를 업스트림 용량 부족으로 오역한다. 503 바디는 맞고, 사용자에게 보이는 문장이 틀리다.

해결방안

inspectSystemd()에서 버스에 질문이 도달하지 못한 실패를 unknown이 아니라 absent로 분류한다. 아무 non-zero나 열지 말고, stderr의 Failed to connect to … bus / $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined처럼 버스 연결 실패만 본다. 이 완화는 /.dockerenv 또는 PID 1이 systemd가 아닌 컨테이너/foreground 신호와 같이 걸어서, 실제 데스크톱에서 버스가 잠깐 죽은 경우는 지금처럼 unknown으로 남긴다. unit이 있거나 NeedDaemonReload=yes이거나 파편/홈이 어긋나면 계속 fail-closed다. src/service.ts가 이미 가진 Docker 신호를 ownership-preflight까지 넘기면, 서비스 불가 환경에서 “소유 미증명”으로 native를 닫지 않게 맞출 수 있다. 회귀는 세 갈래다. Docker + systemctl 있음 + 버스 없음 → native-main 통과. Docker + 다른 홈 unit 실설치 → 계속 차단. 비Docker 호스트 + 버스 실패 → 계속 unknown. ready는 native-main이 유일한 경로일 때 isNativeMainTrafficBlocked()를 반영하고, CLI 오역은 로컬 503을 용량 부족으로 바꾸지 않게 한다.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

Triage note: this issue is already owned by open PR #2029, not unclaimed.

That PR rewrites the exact function this needs changed — the non-zero branch of inspectSystemd in src/service-manager-probe.ts — for #1939. It currently maps two deterministic no-bus messages to absent:

• Failed to get D-Bus connection: No such file or directory
• System has not been booted with systemd

and deliberately keeps everything else unknown, with a test that pins that choice. Your stderr is a third shape:

Failed to connect to user scope bus via local transport: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined

so it currently falls on the unknown side and the fence stays armed.

Why this is one decision, not two

#1939 and this issue are the same classifier with different symptoms. There, an unprovable service home refuses a sync. Here it closes native-main for the process lifetime:

inspectSystemd -> unknown
  -> ownership-preflight.ts:155      ownership: "unknown"
  -> server/index.ts                 blockNativeMainStartupForUnownedServiceHome("ownership-unknown")
  -> native-profile-startup.ts       isNativeMainTrafficBlocked() == true
  -> auth-context.ts                 CodexMainProfileDrainingError -> 503

Opening a second PR against that function would mean two people making the same fail-closed, security-adjacent call in two places, with whichever lands second silently overwriting the first. So this is deliberately not getting its own code change.

What is happening instead

#2029 is CHANGES_REQUESTED for a good reason — a missing bus does not prove the unit file is absent, and returning absent on stderr alone would let the preflight overwrite a foreign installation. I have posted the containment for that on #2029: consult the unit file on disk when the bus cannot answer, which the same function already knows how to do a few lines below, plus the regression that keeps a foreign definition fenced.

Your case is included in the suggested tests, so when #2029 lands with the containment this closes with it.

Your workaround stands meanwhile

Removing systemctl from the proxy's PATH forces the spawnFailed → absent branch from #1612 and restores native traffic. Thank you for isolating that with a single variable — it is what made the classifier the obvious suspect rather than the fence itself.

Keeping this open and linked rather than closing it as a duplicate: the 503 symptom is distinct enough that someone hitting it should be able to find this thread.