Skip to content

Fix NVMLError_Unknown: init NVML once, auto reset+retry on failure - #1

Open
tugrultamerc wants to merge 1 commit into
tlockcuff:mainfrom
tugrultamerc:fix/nvml-unknown-error
Open

Fix NVMLError_Unknown: init NVML once, auto reset+retry on failure#1
tugrultamerc wants to merge 1 commit into
tlockcuff:mainfrom
tugrultamerc:fix/nvml-unknown-error

Conversation

@tugrultamerc

Copy link
Copy Markdown

The app called nvmlInit/nvmlShutdown on every request from multiple threadpool workers. After a host driver reload/update this left NVML in a bad state and /gpu returned 500 with NVMLError_Unknown.

  • Initialize NVML once per process (thread-safe), shutdown at exit
  • On NVMLError during gather: shutdown+re-init and retry once
  • On init failure: attempt one reset before giving up
  • Return schema-valid status=error payload instead of an unhandled 500
  • Add smoke tests covering outage, recovery, and retry behavior

…efully

The app called nvmlInit/nvmlShutdown on every request from multiple
threadpool workers. After a host driver reload/update this left NVML in
a bad state and /gpu returned 500 with NVMLError_Unknown.

- Initialize NVML once per process (thread-safe), shutdown at exit
- On NVMLError during gather: shutdown+re-init and retry once
- On init failure: attempt one reset before giving up
- Return schema-valid status=error payload instead of an unhandled 500
- Add smoke tests covering outage, recovery, and retry behavior
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant