[Bug][Windows]: reboot leaves native-main gate stuck; GPT-only 503 until ocx restart

[hoonysis]

Client or integration

Direct HTTP/API client

Area

Service lifecycle

Summary

On Windows 11 with the Task Scheduler service backend, a reboot/login can leave only the native OpenAI/Codex-login models blocked behind the process-wide native-main startup gate.

The proxy itself is healthy (/healthz returns 200), the model remains in the catalog, and non-native providers such as Anthropic continue returning 200. Native GPT requests fail locally with HTTP 503:

OpenCodex local native-main profile maintenance is active; retry this request

The state does not recover on its own. Running ocx restart (or pressing the dashboard/launcher restart control) immediately restores the same GPT model without reauthentication or configuration changes.

This has recurred after multiple Windows reboots. I expected startup recovery to reacquire native-main ownership and reopen admission automatically.

Reproduction

1. Install OpenCodex as the per-user Windows Task Scheduler service (logon trigger).
2. Use a valid native Codex/OpenAI login and confirm a native GPT model works.
3. Reboot Windows and log in.
4. Wait for the scheduled OpenCodex proxy to start.
5. Confirm:
   • GET /healthz returns 200.
   • Anthropic or another non-native provider still works.
   • A native GPT Responses request (stream: true, store: false, list-form input) returns local HTTP 503 with the native-main maintenance message.
6. Wait/retry: the GPT-only 503 persists.
7. Run ocx restart.
8. Send the same GPT request again. It returns 200 without login or config changes.

The Windows scheduled task currently reports AllowHardTerminate=True and MultipleInstances=IgnoreNew.

Version

2.26.0 (installed and running runtime both 2.26.0)

Codex runtime reported by OpenCodex: 0.148.0

Operating system

Windows 11 Home, version 10.0.26200, build 26200

Provider and model

OpenAI (Codex login / native main) / gpt-5.6-sol

Logs or error output

[17:25:23] opencodex service wrapper start
Previous session (PID <redacted>) did not shut down cleanly. Codex state restored from journal.
opencodex proxy running on http://localhost:10100

# /healthz remained HTTP 200.

17:26:37 | gpt-5.6-sol | openai | 503 | OpenCodex local native-main profile maintenance is active; retry this request
17:26:37 | gpt-5.6-sol | openai | 503 | OpenCodex local native-main profile maintenance is active; retry this request
17:26:48 | gpt-5.6-sol | openai | 503 | OpenCodex local native-main profile maintenance is active; retry this request

# Manual ocx restart:
[17:26:56] child exited with code 1; restarting
[17:27:03] opencodex service wrapper start

17:27:12 | gpt-5.6-sol | openai | 200
17:27:17 | gpt-5.6-sol | openai | 200

The exact internal blocked reason is not exposed in the request log. The installed source suggests the relevant path may be:

• native-profile-startup.ts: owner acquisition/recovery updates the process-wide snapshot to blocked.
• isNativeMainTrafficBlocked() keeps native traffic fenced while that snapshot remains blocked.
• auth-context.ts converts this state to the generic native-main maintenance 503.

The preceding unclean-shutdown/journal-recovery line is consistently associated with the post-reboot failure, but I cannot confirm whether the specific retained reason is recovery-pending, owner-unavailable, or another startup block because it is not logged.

It would help if OpenCodex:

1. logged the concrete native-main block reason;
2. retried or completed recovery after stale journal/ownership cleanup; and
3. ensured the Windows scheduler shutdown/startup path releases or reacquires native-main ownership cleanly.

Screenshots and supporting files

Screenshots are available if needed. They show repeated client-visible “Selected model is at capacity” errors, while OpenCodex request logs reveal the actual native-main maintenance 503.

Redacted configuration

{
  "serviceBackend": "Windows Task Scheduler",
  "trigger": "logon",
  "allowHardTerminate": true,
  "multipleInstances": "IgnoreNew",
  "proxyPort": 10100,
  "nativeProvider": "openai (Codex login)",
  "nativeModel": "gpt-5.6-sol",
  "otherProvidersRemainHealthy": true
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[coderabbitai]

🔗 Related PRs

#780 - fix(service): make Windows scheduler stop actually stop the proxy [merged]
#805 - fix(service): bake WinSW admin env and retry stop-path probes [merged]
#1154 - fix(probe): walk Windows service definition chains (Task Scheduler + WinSW) for ownership [merged]
#1626 - fix(windows): remove native service on fresh scheduler install [open]
#1647 - fix(tray): let the service wrapper exit 0 on an already-live proxy [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[lidge-jun]

리뷰 · 우선순위 78 / 80

Windows 재부팅 뒤에 프록시는 살아 있는데 ChatGPT 네이티브 모델만 503이 나는 버그다. /healthz는 200, Anthropic 같은 비네이티브 경로도 200인데 gpt-5.6-sol만 OpenCodex local native-main profile maintenance is active; retry this request로 막힌다. 혼자 풀리지 않고 ocx restart면 바로 살아난다. 지금 dev가 잡고 있는 native-main / Windows 서비스 / doctor-reclaim 불이랑 정확히 겹쳐서 78이다. 다른 프로바이더는 되고 수동 restart로 복구는 돼서 80은 안 넘긴다.

503은 프록시가 죽은 게 아니다. src/codex/native-profile-startup.ts의 프로세스 전역 게이트다. 스냅샷이 ready가 아니면 blocked이고, reason은 foreign-ownership / ownership-unknown / recovery-pending / manual-recovery / owner-conflict / owner-unavailable / stage-cleanup-required 중 하나다. startNativeMainStartupLifecycle()은 시작하자마자 recovery-pending으로 펜스를 친 뒤 retainNativeMainOwner() + observeOwner()로 수렴한다. acquiring이면 계속 recovery-pending, held면 convergeOwnedStartup(), 그 외는 owner-unavailable 또는 owner-conflict로 굳는다.

요청이 실제로 거절되는 지점은 게이트 파일이 아니라 src/codex/auth-context.ts다. isNativeMainTrafficBlocked()는 reason을 보지 않는다. 서비스홈이 foreign-ownership/ownership-unknown이거나 스냅샷이 blocked면 true다. resolveCodexAuthContext()가 네이티브 메인 트래픽을 ineligible로 보고, 고른 계정이 __main__이거나 풀 대체 계정이 없으면 CodexMainProfileDrainingError를 던진다. Responses/live/search/images/compact가 이걸 받아 codexMainProfileDrainingResponse()로 HTTP 503, server_busy, Retry-After: 1을 만든다. 비네이티브 프로바이더는 이 경로를 안 타니까 healthz/Anthropic이 200인 게 정상이다. 문제는 reason이 요청 로그에 안 나온다는 거다. recovery-pending인지 owner-unavailable인지 ownership-unknown인지 manual-recovery인지 한 줄로 찍어야 재현이 닫힌다.

리포터 로그의 Previous session (PID ) did not shut down cleanly. Codex state restored from journal.는 native-main 암호화 저널이 아니다. src/codex/journal.ts의 ~/.codex/opencodex-journal.json이고, Codex config.toml / 프로필을 복구하는 주입 저널이다. (PID )가 비어 있으면 journal.pid가 비어 있다는 뜻이다. 게이트가 보는 저널은 따로 있다. src/codex/native-profile-store.ts의 probeNativeProfileRecoveryState()가 recoveryBlockPath / native journalPath를 보고 none | journal | manual | unreadable을 준다. journal만 자동 복구하고 나머지는 manual-recovery로 정착한다. 둘을 한 원인으로 쓰면 안 된다. 다만 재부팅 + AllowHardTerminate=True면 둘 다 남을 수는 있다.

스케줄러 첫 기동이 native-main을 아예 안 잡는 건 아니다. 래퍼(src/service.ts)가 OCX_SERVICE=1로 ocx start --port를 띄우고, startServer()는 ownership이 owned면 startNativeMainStartupLifecycle()으로 재획득을 시도한다. inspectNativeCodexOwnership()이 unknown/foreign이면 그때만 복구 없이 blockNativeMainStartupForUnownedServiceHome()으로 영구 503이다. 프로브 타임아웃은 SERVICE_PROBE_TIMEOUT_MS = 2000이다. 오너 락도 게이트를 고정할 수 있다. retainNativeMainOwner()가 CODEX_HOME/.opencodex-native-main.owner.sqlite에 BEGIN IMMEDIATE를 걸고, SQLITE_BUSY면 250ms 재시도, ACL ETIMEDOUT 한 번 더 재시도한 뒤 unavailable으로 끝난다. 하드킬이면 exit hook이 DB를 못 닫는다.

ocx restart가 고치는 이유는 서비스를 지웠다 다시 깔아서가 아니다. 라이브면 POST /api/system/restart → acceptSystemRestart()이고, 슈퍼바이즈드 자식은 process.exit(1)해서 래퍼 :loop가 다시 띄운다. 그래서 두 번째 service wrapper start에는 unclean-journal 줄이 없고, 새 프로세스 + 새 모듈 스냅샷 + 두 번째 inspectStartupOwnership이 생긴다. 스케줄러가 재획득을 안 해서가 아니라, 첫 기동이 fail-closed로 정착한 뒤 혼자 안 풀리는 쪽에 가깝다. 그래서 고칠 것도 명확하다. 막힌 reason을 로그에 남기고, stale journal/ownership을 치운 뒤 복구를 다시 돌리고, 첫 기동이 unknown/unavailable으로 굳으면 한 번 더 재획득하게 만들면 된다. #2107이 Win10+WSL 쪽 쌍둥이다. #780/#805/#1154는 stop/ownership-probe 위생 히스토리고, 이 게이트의 직접 원인은 아니다.

해결방안

src/codex/native-profile-startup.ts에서 blocked로 정착한 순간 reason을 요청 로그와 service.log에 한 줄로 남겨라. recovery-pending / owner-unavailable / ownership-unknown / manual-recovery / owner-conflict를 구분하지 못하면 재부팅 재현이 닫히지 않는다. 스케줄러 첫 기동이 unknown/unavailable으로 굳으면 startNativeMainStartupLifecycle()을 한 번 더 돌려 stale recoveryBlockPath와 CODEX_HOME/.opencodex-native-main.owner.sqlite를 치운 뒤 retainNativeMainOwner() + observeOwner()를 재시도한다. src/service.ts 래퍼의 hard-terminate 경로(AllowHardTerminate=True)는 exit hook이 owner DB를 못 닫으므로, 기동 시 SQLITE_BUSY/빈 PID 저널을 자동 복구 대상으로 보고 manual-recovery로 영구 펜스하지 않게 한다. src/codex/auth-context.ts의 503은 reason을 숨기지 말고, healthz와 달리 native-main 게이트가 열린 뒤에만 ready로 보이게 맞춘다. ocx restart가 고치는 두 번째 inspectStartupOwnership을 첫 기동에도 한 번 더 주는 것이 패치의 핵심이다.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

Phase 1 of this is up as #2121 — it does not fix the sticking fence, it makes the next occurrence diagnosable. Explaining why that ordering, since it is the difference between a fix and a guess.

Your instinct about the source was right, and the ownership verdict is the structural cause:

src/server/index.ts:702-710
  owned         -> startNativeMainStartupLifecycle()
  anything else -> blockNativeMainStartupForUnownedServiceHome(...)   // process lifetime

startServer() takes that verdict once and never revisits it. That is exactly why waiting never helped you and ocx restart always did — restart is the only thing that re-runs the decision.

But there are two paths that produce that verdict, and your report cannot distinguish them:

1. ACL fail-closed. A second ETIMEDOUT in the icacls hardening is terminal — it publishes { status: "unavailable", reason: "lock-unavailable" } at native-main-owner.ts:205-212, and observeOwner() settles to owner-unavailable and stops. Your timing fits: the first 503 is ~74s after wrapper start, past the ~60s owner budget.
2. Probe fail-closed. SERVICE_PROBE_TIMEOUT_MS is 2000ms. A scheduler-only install still runs sc.exe query for WinSW; if that times out with WinSW assets absent, the chain returns unknown rather than absent (service-manager-probe.ts:732-736).

Those want different fixes. So #2121 makes the settled reason say which one, on stdout — the same stream your excerpt was already quoting, since it interleaves [17:25:23] opencodex service wrapper start with the 503 lines:

native-main admission is fenced (reason: ownership-unknown); native model requests return 503 until it clears

One line per distinct reason, not per request — your log shows three 503s in eleven seconds and a real client retries harder than that.

Two things you asked for that deliberately did not happen, because both would have broken something:

• The 503 message is unchanged. claude-messages.ts:818 identifies this exact response by matching that string, to keep the fence a 503 instead of remapping it to an Anthropic 529. Appending the reason there would have started telling Claude Code to back off from an upstream that was never involved.
• No header, and nothing new in /api/logs. That log populates its error text by reading error.message back out of the response body; headers are already gone by then, and the 503 error code is pinned to server_is_overloaded by design. A header would have looked like a fix and shown you nothing.

One detail worth knowing when you next hit this: a 503 with no reason in the log is meaningful. The turn-drain claim race throws the same error while the startup gate reads ready, and that path stays deliberately silent — so a reasonless 503 tells you it is not the startup fence at all. That distinction did not exist before.

What would help. Next time it happens, the wrapper stdout around startup — specifically that one native-main admission is fenced line. That single word decides phase 2, which is making a boot-time unknown retryable while OCX_SERVICE=1 instead of a process-lifetime fence, keeping genuine foreign fail-closed. Two narrower fixes fall out of it regardless: a timed-out sc.exe query with both WinSW xml and exe absent should not mark the machine unknown, and a second ACL ETIMEDOUT should back off and retry so a warm icacls reopens the gate without ocx restart.

Being straight about coverage: this was verified on macOS. owner-unavailable — the branch your report most likely hit — is a Windows icacls path, and it turns out nothing in the suite asserts it today. Which is rather the point of shipping the log line first.

[lidge-jun]

Phase 1 is on dev as of #2121 (fbc6f26a2).

The 503 now names the gate reason that actually settled, on stdout:

native-main admission is fenced (reason: ownership-unknown); native model requests return 503 until it clears

This does not fix the sticking fence and the issue stays open for that. It makes the next occurrence diagnosable, which is the prerequisite: the two candidate triggers settle to different reasons, and until one of them is named in a real report, phase 2 would be aiming at a coin flip.

So the ask is unchanged from my earlier comment — next time it happens, the wrapper stdout around startup, specifically that one native-main admission is fenced line. That word decides whether phase 2 targets the ACL retry or the probe classification.