From 1625f046972c650660ee3d80eb3a3351385f7a11 Mon Sep 17 00:00:00 2001 From: devashish Date: Sat, 22 Aug 2026 16:16:12 +0530 Subject: [PATCH] feat(h1-nvidia): build custom vLLM images on the target host MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ships the Dockerfile that produces vllm-inx:26.06-py3-patched — the image qwen3.8-27b-fp8 needs — inside the lmstack repo, and teaches 10-stack.yml to build it on the target host when any active model references a locally-tagged image and the tag is not already present. Previously the PR pointed users at ric03uec/system/hosts/inx/vllm/ to build the image themselves. That leaks a dependency on a private- looking sibling repo and breaks the "one command brings up the stack" contract. Now `make up HOST=h1-nvidia` on a DGX Spark with qwen3.8-27b-fp8 active builds the image (thin patch on nvcr.io/nvidia/vllm:26.06-py3 that pins prometheus-fastapi-instrumentator>=8.0.1), then starts the stack. Build-context contract: - Dockerfiles live at hosts//vllm/images/.Dockerfile and produce the tag : where the model YAML's image: is :. Matching-by-tag is a naming rule, not a schema field. - The layout task copies the images/ dir onto the target so docker_image has a build context where the daemon lives. - The build task detects locally-tagged images by "no '/' before the ':'" and skips any tag already present (`docker image ls`). Also: - Sync hosts/h3-template/ansible/10-stack.yml with the h1-nvidia changes so a fresh copy of the template still has the same capabilities. Per-host duplication is deliberate (see AGENTS.md); drift is what T7.2 exists to catch, and the template stays the reference shape. - Exclude images/ from T7.1 alongside models/: image Dockerfiles are per-catalog-entry artifacts, not skeleton pieces a new host must carry. NGC base image note: users may need `docker login nvcr.io` on the target if their account gates nvcr.io/nvidia/vllm:26.06-py3. Documented in AGENTS.md, the model YAML notes, and the website. --- hosts/h1-nvidia/AGENTS.md | 16 +++-- hosts/h1-nvidia/ansible/10-stack.yml | 58 +++++++++++++++++++ .../h1-nvidia/vllm/images/vllm-inx.Dockerfile | 20 +++++++ .../h1-nvidia/vllm/models/qwen3.8-27b-fp8.yml | 18 +++--- hosts/h3-template/ansible/10-stack.yml | 57 ++++++++++++++++++ tests/template_test.sh | 7 ++- website/docs/hosts/h1-nvidia.md | 10 ++-- 7 files changed, 167 insertions(+), 19 deletions(-) create mode 100644 hosts/h1-nvidia/vllm/images/vllm-inx.Dockerfile diff --git a/hosts/h1-nvidia/AGENTS.md b/hosts/h1-nvidia/AGENTS.md index c198e00..b8d1cc5 100644 --- a/hosts/h1-nvidia/AGENTS.md +++ b/hosts/h1-nvidia/AGENTS.md @@ -64,12 +64,16 @@ The tracked defaults in `ansible/vars.yml` stay pinned to the 8 GiB workstation case so a fresh `make up HOST=h1-nvidia` on a small card still works. Do not change them. -The model needs a locally built image: `vllm-inx:26.06-py3-patched` (a -Blackwell-patched aarch64 build of vLLM 0.22.1 with the -`Qwen3_5ForConditionalGeneration` arch, `qwen3_xml` tool parser, `qwen3` -reasoning parser, and `Qwen3_5MTP` module registered). Upstream -`vllm/vllm-openai:*` does not carry these as of 2026-08. Build it from -[`ric03uec/system/hosts/inx/vllm/`](https://github.com/ric03uec/system/tree/main/hosts/inx/vllm). +The model runs on `vllm-inx:26.06-py3-patched` — a thin patch layer +(`prometheus-fastapi-instrumentator>=8.0.1`) on `nvcr.io/nvidia/vllm:26.06-py3`. +NGC's base image ships with `Qwen3_5ForConditionalGeneration`, `qwen3_xml`, +`qwen3`, and `Qwen3_5MTP` already registered; upstream `vllm/vllm-openai:*` +does not carry these as of 2026-08. + +`10-stack.yml` builds the patched image on the target the first time any +active model references the tag, using `vllm/images/vllm-inx.Dockerfile`. +You may need `docker login nvcr.io` on the target beforehand if the base +image is gated for your account. Vision (image-text-to-text) is out of scope; tracked in [ric03uec/system#32](https://github.com/ric03uec/system/issues/32). diff --git a/hosts/h1-nvidia/ansible/10-stack.yml b/hosts/h1-nvidia/ansible/10-stack.yml index e68cbca..4134877 100644 --- a/hosts/h1-nvidia/ansible/10-stack.yml +++ b/hosts/h1-nvidia/ansible/10-stack.yml @@ -90,6 +90,7 @@ - {path: litellm, mode: "0755"} - {path: models, mode: "0755"} - {path: chat-templates, mode: "0755"} + - {path: images, mode: "0755"} loop_control: label: "{{ item.path }}" @@ -116,6 +117,19 @@ loop_control: label: "{{ item | basename }}" + # Dockerfiles ship to the target so the docker_image build below has a + # build context on the host where the daemon lives, not on the control + # machine where the playbook does. + - name: Copy custom image Dockerfiles to the install root + ansible.builtin.copy: + src: "{{ item }}" + dest: "{{ install_root }}/images/{{ item | basename }}" + mode: "0644" + with_fileglob: + - "{{ engine_root }}/images/*.Dockerfile" + loop_control: + label: "{{ item | basename }}" + # ========================================================================= # Secrets — dropped as a template, never overwritten (T5.6) # ========================================================================= @@ -163,6 +177,50 @@ loop_control: label: "{{ item.key }}" + # ========================================================================= + # Local image builds + # + # Any active model whose `image:` tag has no registry prefix (no '/' before + # the ':') is a locally-built image. We build it here from a matching + # Dockerfile under vllm/images/ if the tag is not already present, so a + # first-time `make up` on a fresh host produces a runnable stack in one + # command. Public images fall through to compose's own pull. + # + # Naming contract: the file at `vllm/images/.Dockerfile` produces the + # tag `:` where the model YAML's `image:` is `:`. + # So `vllm-inx:26.06-py3-patched` builds from `images/vllm-inx.Dockerfile`. + # ========================================================================= + - name: Find locally-tagged images referenced by active models + ansible.builtin.set_fact: + local_images: >- + {{ models + | map(attribute='image') + | select('match', '^[^/]+:[^/]+$') + | unique + | list }} + + - name: List images already present on the host + ansible.builtin.command: docker image ls --format '{% raw %}{{ .Repository }}:{{ .Tag }}{% endraw %}' + register: docker_image_list + changed_when: false + when: local_images | length > 0 + + - name: Build any locally-tagged image that is not already present + community.docker.docker_image: + name: "{{ item.split(':')[0] }}" + tag: "{{ item.split(':')[1] }}" + source: build + build: + path: "{{ install_root }}/images" + dockerfile: "{{ item.split(':')[0] }}.Dockerfile" + pull: true + loop: "{{ local_images }}" + loop_control: + label: "{{ item }}" + when: + - local_images | length > 0 + - item not in docker_image_list.stdout_lines + # ========================================================================= # Render # ========================================================================= diff --git a/hosts/h1-nvidia/vllm/images/vllm-inx.Dockerfile b/hosts/h1-nvidia/vllm/images/vllm-inx.Dockerfile new file mode 100644 index 0000000..7a5bbe4 --- /dev/null +++ b/hosts/h1-nvidia/vllm/images/vllm-inx.Dockerfile @@ -0,0 +1,20 @@ +# Thin patch layer on top of the NGC vLLM 26.06-py3 image. +# +# Why this exists: 26.06-py3 ships fastapi >= 0.137 (which restructured routers +# into _IncludedRouter wrappers without a .path attribute) together with +# prometheus-fastapi-instrumentator 8.0.0, which unconditionally reads .path +# in its middleware. Every HTTP request 500s before reaching vLLM's router. +# +# Fixed in prometheus-fastapi-instrumentator 8.0.1 (2026-06-22). Upstream vLLM +# also fixed via PRs #45594 (fastapi cap) and #45629 (monkey-patch), neither of +# which is in the 26.06-py3 build. +# +# Refs: +# - https://github.com/vllm-project/vllm/issues/45596 +# - https://github.com/trallnag/prometheus-fastapi-instrumentator/issues/370 +# +# The ansible playbook builds this on the target host when any active model +# references `vllm-inx:26.06-py3-patched` and the tag is not already present. +# Ported verbatim from ric03uec/system/hosts/inx/vllm/Dockerfile. +FROM nvcr.io/nvidia/vllm:26.06-py3 +RUN pip install --no-cache-dir "prometheus-fastapi-instrumentator>=8.0.1" diff --git a/hosts/h1-nvidia/vllm/models/qwen3.8-27b-fp8.yml b/hosts/h1-nvidia/vllm/models/qwen3.8-27b-fp8.yml index ffc751f..e51bf3c 100644 --- a/hosts/h1-nvidia/vllm/models/qwen3.8-27b-fp8.yml +++ b/hosts/h1-nvidia/vllm/models/qwen3.8-27b-fp8.yml @@ -45,13 +45,17 @@ notes: | specific additions are `engine: vllm` and `tier: 96g`. Image: `vllm-inx:26.06-py3-patched` is not on any public registry. - Build it locally from the Dockerfile shipped in - https://github.com/ric03uec/system/tree/main/hosts/inx/vllm — it - is a Blackwell-patched aarch64 build of vLLM 0.22.1 with the - Qwen3_5ForConditionalGeneration arch, qwen3_xml tool parser, - qwen3 reasoning parser, and Qwen3_5MTP module all registered. - Upstream `vllm/vllm-openai:*` does not carry these on any tag as - of 2026-08. + 10-stack.yml builds it on the target host from + hosts/h1-nvidia/vllm/images/vllm-inx.Dockerfile the first time it + sees this tag referenced by an active model. The Dockerfile is a + thin patch layer on `nvcr.io/nvidia/vllm:26.06-py3` (pins + prometheus-fastapi-instrumentator>=8.0.1); NGC ships that base + with Qwen3_5ForConditionalGeneration, qwen3_xml, qwen3, and + Qwen3_5MTP already registered. Upstream `vllm/vllm-openai:*` does + not carry these on any tag as of 2026-08. + + You may need `docker login nvcr.io` on the target beforehand if + NGC gates the base image behind auth for your account. Architecture (from HF config.json): - Qwen3_5ForConditionalGeneration (dense, w/ vision encoder — diff --git a/hosts/h3-template/ansible/10-stack.yml b/hosts/h3-template/ansible/10-stack.yml index 8d3f2ed..29617b3 100644 --- a/hosts/h3-template/ansible/10-stack.yml +++ b/hosts/h3-template/ansible/10-stack.yml @@ -90,6 +90,7 @@ - {path: litellm, mode: "0755"} - {path: models, mode: "0755"} - {path: chat-templates, mode: "0755"} + - {path: images, mode: "0755"} loop_control: label: "{{ item.path }}" @@ -116,6 +117,19 @@ loop_control: label: "{{ item | basename }}" + # Dockerfiles ship to the target so the docker_image build below has a + # build context on the host where the daemon lives, not on the control + # machine where the playbook does. + - name: Copy custom image Dockerfiles to the install root + ansible.builtin.copy: + src: "{{ item }}" + dest: "{{ install_root }}/images/{{ item | basename }}" + mode: "0644" + with_fileglob: + - "{{ engine_root }}/images/*.Dockerfile" + loop_control: + label: "{{ item | basename }}" + # ========================================================================= # Secrets — dropped as a template, never overwritten (T5.6) # ========================================================================= @@ -163,6 +177,49 @@ loop_control: label: "{{ item.key }}" + # ========================================================================= + # Local image builds + # + # Any active model whose `image:` tag has no registry prefix (no '/' before + # the ':') is a locally-built image. We build it here from a matching + # Dockerfile under vllm/images/ if the tag is not already present, so a + # first-time `make up` on a fresh host produces a runnable stack in one + # command. Public images fall through to compose's own pull. + # + # Naming contract: the file at `vllm/images/.Dockerfile` produces the + # tag `:` where the model YAML's `image:` is `:`. + # ========================================================================= + - name: Find locally-tagged images referenced by active models + ansible.builtin.set_fact: + local_images: >- + {{ models + | map(attribute='image') + | select('match', '^[^/]+:[^/]+$') + | unique + | list }} + + - name: List images already present on the host + ansible.builtin.command: docker image ls --format '{% raw %}{{ .Repository }}:{{ .Tag }}{% endraw %}' + register: docker_image_list + changed_when: false + when: local_images | length > 0 + + - name: Build any locally-tagged image that is not already present + community.docker.docker_image: + name: "{{ item.split(':')[0] }}" + tag: "{{ item.split(':')[1] }}" + source: build + build: + path: "{{ install_root }}/images" + dockerfile: "{{ item.split(':')[0] }}.Dockerfile" + pull: true + loop: "{{ local_images }}" + loop_control: + label: "{{ item }}" + when: + - local_images | length > 0 + - item not in docker_image_list.stdout_lines + # ========================================================================= # Render # ========================================================================= diff --git a/tests/template_test.sh b/tests/template_test.sh index 2321236..2ad1d56 100755 --- a/tests/template_test.sh +++ b/tests/template_test.sh @@ -46,8 +46,10 @@ for template in "${templates[@]}"; do section "$template" # -- T7.1 ----------------------------------------------------------------- - # Model YAMLs are excluded: the template ships none on purpose. AGENTS.md is - # excluded because both have one and the contents are meant to differ. + # Model YAMLs are excluded: the template ships none on purpose. Image build + # Dockerfiles under vllm/images/ are excluded for the same reason — they are + # per-catalog-entry artifacts, not skeleton pieces every host needs. AGENTS.md + # is excluded because both have one and the contents are meant to differ. missing=0 while IFS= read -r ref_file; do rel="${ref_file#"$REFERENCE"/}" @@ -55,6 +57,7 @@ for template in "${templates[@]}"; do done < <( find "$REFERENCE" -type f \ -not -path "$REFERENCE/$ENGINE_DIR/models/*" \ + -not -path "$REFERENCE/$ENGINE_DIR/images/*" \ -not -name 'AGENTS.md' | sort ) [[ $missing -eq 0 ]] && ok "T7.1 has every file $REFERENCE has" diff --git a/website/docs/hosts/h1-nvidia.md b/website/docs/hosts/h1-nvidia.md index fcc8df4..5397ce6 100644 --- a/website/docs/hosts/h1-nvidia.md +++ b/website/docs/hosts/h1-nvidia.md @@ -76,10 +76,12 @@ It sits at tier `96g`, so it is only offered on hosts whose probe reports a usable budget of 96 GiB or more. On a Spark, `/lmstack:analyze` picks it up automatically. The 8 GiB defaults stay untouched for workstations. -The image is not on any public registry — build `vllm-inx:26.06-py3-patched` -locally from -[`ric03uec/system/hosts/inx/vllm/`](https://github.com/ric03uec/system/tree/main/hosts/inx/vllm). -Vision (image-text-to-text) is out of scope; see +The image is built on the target host by `10-stack.yml` the first time it +sees `vllm-inx:26.06-py3-patched` referenced by an active model — it's a +thin patch layer (`prometheus-fastapi-instrumentator>=8.0.1`) on +`nvcr.io/nvidia/vllm:26.06-py3`, shipped in `vllm/images/vllm-inx.Dockerfile`. +You may need `docker login nvcr.io` on the target beforehand if NGC's base +is gated for your account. Vision (image-text-to-text) is out of scope; see [ric03uec/system#32](https://github.com/ric03uec/system/issues/32). ## Things that will bite you