Support pulling CNCF ModelPack models with --source_model oci://

docs/parameters.md

@@ -101,7 +101,7 @@ Shared configuration options for the pull, and pull & start mode. In the presenc
 | Option                      | Value format | Description                                                                                                   |
 |-----------------------------|--------------|---------------------------------------------------------------------------------------------------------------|
 | `--pull`                    | `NA`         | Runs the server in pull mode to download the model from the Hugging Face repository.                          |
-| `--source_model`            | `string`     | Name of the model in the Hugging Face repository. If not set, `model_name` is used.                           |
+| `--source_model`            | `string`     | Name of the model in the Hugging Face repository, or a CNCF ModelPack OCI reference prefixed with `oci://` (for example `oci://ghcr.io/org/model:tag`). If not set, `model_name` is used. See [OCI pull mode](./pull_oci_models.md). |
 | `--model_repository_path`   | `string`     | Directory where all required model files will be saved.                                                       |
 | `--model_name`              | `string`     | Name of the model as exposed externally by the server.                                                        |
 | `--target_device`           | `string`     | Device name to be used to execute inference operations. For `--task text_generation`, accepted values include `CPU`/`GPU`/`NPU`/`AUTO`, `GPU.<index>`, and `HETERO:<...>`/`AUTO:<...>` (for example `AUTO:GPU,CPU`). If not specified, it is auto-detected using available GPU devices, with fallback to `CPU` (NPU must be selected explicitly). See [Automatic target device detection](#automatic-target-device-detection). |
@@ -111,6 +111,14 @@ Shared configuration options for the pull, and pull & start mode. In the presenc
 
 > **NOTE:** If you want to use model that is split into several `.gguf` files, you should specify the filename of the first part only, e.g. `--gguf_filename model-name-00001-of-00002.gguf`.
 
+## Pull Mode Options for CNCF ModelPack (OCI) sources
+
+When `--source_model` is prefixed with `oci://`, the model is pulled from an OCI registry as a [CNCF ModelPack](https://github.com/modelpack/model-spec) artifact instead of from Hugging Face. `--task` is required, and `--gguf_filename` is not accepted. See [OCI pull mode](./pull_oci_models.md) for details.
+
+| Variable        | Value format | Description                                                                                                              |
+|-----------------|--------------|--------------------------------------------------------------------------------------------------------------------------|
+| `LLMMAN_BIN`    | `string`     | Default: `llmman` resolved through `PATH`. Full path to the [llmman](https://github.com/llmmanorg/llmman) executable that performs the registry pull. |
+
 ## Pull Mode Options for optimum-cli mode
 
 When pulling models outside of OpenVINO organization the optimum-cli api is used inside ovms. You can set additional parameters for this mode.

docs/prepare_generative_use_cases.md

@@ -9,6 +9,7 @@ hidden:
 ovms_docs_pull
 ovms_demos_common_export
 ovms_docs_pull_optimum
+ovms_docs_pull_oci
 
 ```
 
@@ -17,3 +18,5 @@ Prepare models using OVMS [pull mode](./pull_hf_models.md) (recommended).
 Prepare models using [python script](../demos/common/export_models/README.md).
 
 Prepare models using OVMS with python [optimum pull mode](./pull_optimum_cli.md).
+
+Prepare models distributed as CNCF ModelPack OCI artifacts using OVMS [OCI pull mode](./pull_oci_models.md).

docs/pull_hf_models.md

@@ -8,6 +8,8 @@ This document describes how to use the OpenVINO Model Server (OVMS) pull feature
 
 > **Note:** Models in IR format must be exported using `optimum-cli`, including tokenizer and detokenizer files (also in IR format), if applicable. If they are missing, add them with `convert_tokenizer --with-detokenizer`.
 
+> **Note:** Models can also be pulled from an OCI registry when they are packaged as [CNCF ModelPack](https://github.com/modelpack/model-spec) artifacts — see [OCI pull mode](./pull_oci_models.md).
+
 ## Pulling pre-configured models
 
 There is a special OVMS mode that pulls a model from Hugging Face without starting the service. It is triggered by the `--pull` parameter. The application exits after the model is downloaded. Without `--pull`, the model is deployed and the server starts.

docs/pull_oci_models.md

@@ -0,0 +1,91 @@
+# OVMS Pull mode for CNCF ModelPack images {#ovms_docs_pull_oci}
+
+Besides Hugging Face Hub, OVMS can pull models that are distributed as OCI artifacts following the [CNCF ModelPack specification](https://github.com/modelpack/model-spec). Because ModelPack images are ordinary OCI artifacts, they can be stored in and served from any OCI registry — Docker Hub, GHCR, quay, Artifactory or a self-hosted registry — with the same tooling, authentication and mirroring you already use for container images.
+
+A model is requested by prefixing `--source_model` with the `oci://` scheme:
+
+```text
+ovms --pull --source_model oci://ghcr.io/<org>/<model>:<tag> --model_repository_path /models --task text_generation
+```
+
+> **Note:** The `oci://` scheme is required. A bare `registry/name:tag` string is indistinguishable from a Hugging Face repository id (`org/model`), so OVMS never guesses.
+
+## Prerequisites
+
+OCI pulling is delegated to [`llmman`](https://github.com/llmmanorg/llmman), which implements registry authentication, the ModelPack media types, resumable blob downloads and a local content-addressed store. Install it and make sure it is on `PATH`:
+
+```text
+curl -fsSL https://raw.githubusercontent.com/llmmanorg/llmman/main/install.sh | sh
+```
+
+If the binary lives outside `PATH`, point OVMS at it with the `LLMMAN_BIN` environment variable:
+
+```text
+export LLMMAN_BIN=/opt/llmman/bin/llmman
+```
+
+Registry credentials are `llmman`'s concern, not OVMS's. Log in once with `llmman login <registry>` and every subsequent `ovms --pull oci://<registry>/...` reuses that session.
+
+## Supported payloads
+
+`llmman resolve` reports the format of the layers in the image, and OVMS reacts to it:
+
+| ModelPack payload | What OVMS does | Extra requirements |
+|---|---|---|
+| OpenVINO IR (`openvino_model.xml` + `.bin`) | Serves it directly out of the `llmman` store; only `graph.pbtxt` is written to the model repository. | none |
+| GGUF | Serves the `.gguf` file directly out of the `llmman` store. | `--task text_generation` only |
+| Hugging Face safetensors | Converts to OpenVINO IR with `optimum-cli` into the model repository, honoring `--weight-format` and `--extra_quantization_params`. | Python dependencies, see [pulling with conversion](./pull_optimum_cli.md) |
+
+Packaging models as OpenVINO IR is recommended: it needs no conversion step and no Python in the serving image.
+
+## Examples
+
+Pull an OpenVINO IR ModelPack image without starting the server:
+
+```text
+ovms --pull --source_model oci://ghcr.io/my-org/phi-3-mini-int8-ov:1.0 --model_repository_path /models --task text_generation
+```
+
+Pull and start in one step, overriding the served model name:
+
+```text
+ovms --rest_port 8000 --source_model oci://ghcr.io/my-org/phi-3-mini-int8-ov:1.0 --model_repository_path /models --model_name phi-3-mini --task text_generation
+```
+
+Pull a GGUF ModelPack image:
+
+```text
+ovms --pull --source_model oci://docker.io/ai/qwen3.5:0.8b --model_repository_path /models --task text_generation
+```
+
+Pull a safetensors ModelPack image and quantize it during the conversion:
+
+```text
+ovms --pull --source_model oci://ghcr.io/my-org/qwen3-8b:1.0 --model_repository_path /models --task text_generation --weight-format int4
+```
+
+## Naming and on-disk layout
+
+The `oci://` scheme is dropped from the served model name, so the reference you typed is what clients use in the `model` field:
+
+```text
+--source_model oci://ghcr.io/my-org/model:1.0   ->   model name "ghcr.io/my-org/model:1.0"
+```
+
+Pass `--model_name` to override it.
+
+Inside `--model_repository_path` the reference additionally has its tag separator replaced, because `:` is not a legal filename character on Windows:
+
+```text
+<model_repository_path>/ghcr.io/my-org/model_1.0/graph.pbtxt
+```
+
+For OpenVINO IR and GGUF payloads that directory holds only `graph.pbtxt` — the weights stay in `llmman`'s content-addressed store and are referenced by absolute path, so pulling the same image for several servables does not duplicate them on disk. Removing a model therefore takes two steps: delete the directory from the model repository, and reclaim the blobs with `llmman rm <reference>`.
+
+## Limitations
+
+- `--task` must be provided explicitly. The task is normally inferred by reading `config.json` from the model source, which for a registry reference would mean pulling the whole image before the command line has even been parsed.
+- `--gguf_filename` is rejected. The layer media types in a ModelPack image already identify the payload, so there is nothing to select.
+- Speculative decoding (`--draft_source_model`) and image generation LoRA adapters (`--source_loras`) are still resolved from Hugging Face, also when the base model comes from a registry.
+
+Check the [parameters page](./parameters.md) for detailed descriptions of configuration options.

src/BUILD

@@ -353,6 +353,8 @@ ovms_cc_library(
     srcs = ["default_task.cpp"],
     deps = [
         "libovms_default_task_detector",
+        "libovmslogging",
+        "libovms_server_settings",
         "libovmsstatus",
         "//src/pull_module:curl_downloader",
         "//src/pull_module:hf_env_vars",
@@ -1962,6 +1964,16 @@ cc_binary(
     # linkstatic = False,  # Use for dynamic linking when necessary
 )
 
+cc_binary(
+    name = "llmman",
+    srcs = [
+        "test/llmman_mock.cpp",
+    ],
+    linkopts = LINKOPTS_ADJUSTED,
+    copts = COMMON_STATIC_LIBS_COPTS,
+    deps = [],
+)
+
 cc_binary(
     name = "ovms",
     srcs = [
@@ -2230,6 +2242,7 @@ cc_test(
     ] + glob(["test/models_config_json/**"]),
     deps = [
             "optimum-cli",
+            "llmman",
             "//src:ovms_lib",
             "//src:libovms_default_task_detector",
             "//src/filesystem:libovmsfilesystemfactory",
@@ -2248,6 +2261,7 @@ cc_test(
             ":libgit2_test",
             ":pull_hf_model_test",
             ":pull_gguf_hf_model_test",
+            ":pull_oci_model_test",
             ":listdirectorymodels_test",
             ":graph_export_test",
             ":lora_graph_export_test",
@@ -2587,6 +2601,25 @@ cc_library(
     local_defines = COMMON_LOCAL_DEFINES,
     copts = COPTS_TESTS,
 )
+cc_library(
+    name = "pull_oci_model_test",
+    linkstatic = 1,
+    alwayslink = True,
+    srcs = ["test/pull_oci_model_test.cpp"],
+    linkopts = [],
+    deps = [
+        ":test_platform_utils",
+        ":test_utils",
+        ":test_test_with_temp_dir",
+        "//src/utils:env_guard",
+        "//src/pull_module:oci_downloader",
+        "//src:ovms_lib",
+        "libovmsstring_utils",
+        "@com_google_googletest//:gtest",
+    ],
+    local_defines = COMMON_LOCAL_DEFINES,
+    copts = COPTS_TESTS,
+)
 cc_library(
     name = "listdirectorymodels_test",
     srcs = ["test/listdirectorymodels_test.cpp"],

src/capi_frontend/server_settings.cpp

@@ -13,6 +13,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 //*****************************************************************************
+#include <algorithm>
 #include <map>
 #include <string>
 
@@ -40,7 +41,31 @@ GraphExportType stringToEnum(const std::string& inString) {
     return (it != stringToType.end()) ? it->second : UNKNOWN_GRAPH;
 }
 bool isOptimumCliDownload(const std::string& sourceModel, std::optional<std::string> ggufFilename) {
-    return !startsWith(toLower(sourceModel), "openvino/") && (ggufFilename == std::nullopt);
+    // OCI references are resolved by the OciDownloader, which decides on its
+    // own whether the resolved payload still needs an optimum-cli conversion.
+    return !isOciDownload(sourceModel) && !startsWith(toLower(sourceModel), "openvino/") && (ggufFilename == std::nullopt);
+}
+
+bool isOciDownload(const std::string& sourceModel) {
+    return startsWith(toLower(sourceModel), OCI_SCHEME);
+}
+
+std::string stripOciScheme(const std::string& sourceModel) {
+    if (!isOciDownload(sourceModel)) {
+        return sourceModel;
+    }
+    return sourceModel.substr(std::string(OCI_SCHEME).size());
+}
+
+std::string localModelDirectoryName(const std::string& sourceModel) {
+    if (!isOciDownload(sourceModel)) {
+        return sourceModel;
+    }
+    std::string name = stripOciScheme(sourceModel);
+    // ':' separates the tag (and, for a non-default registry port, the port).
+    // It is not a legal filename character on Windows.
+    std::replace(name.begin(), name.end(), ':', '_');
+    return name;
 }
 
 }  // namespace ovms

src/capi_frontend/server_settings.hpp

@@ -37,6 +37,7 @@ enum ModelDownlaodType : unsigned int {
     GIT_CLONE_DOWNLOAD,
     OPTIMUM_CLI_DOWNLOAD,
     GGUF_DOWNLOAD,
+    OCI_DOWNLOAD,
     UNKNOWN_DOWNLOAD
 };
 
@@ -84,6 +85,24 @@ std::string enumToString(ConfigExportType type);
 ConfigExportType stringToConfigExportEnum(const std::string& inString);
 bool isOptimumCliDownload(const std::string& sourceModel, std::optional<std::string> ggufFilename);
 
+// CNCF ModelPack (https://github.com/modelpack/model-spec) OCI references.
+//
+// An explicit "oci://" scheme is required rather than guessing from a bare
+// "registry/name:tag" string: that shape is indistinguishable from a
+// HuggingFace repo id ("org/model"), and sniffing it would silently hijack
+// existing --source_model org/model deployments.
+constexpr const char* OCI_SCHEME = "oci://";
+bool isOciDownload(const std::string& sourceModel);
+// "oci://ghcr.io/org/model:tag" -> "ghcr.io/org/model:tag". Other references
+// are returned unchanged.
+std::string stripOciScheme(const std::string& sourceModel);
+// Local directory name a source model is downloaded into, relative to
+// --model_repository_path. Identity for every non-OCI reference; for OCI
+// references the scheme is dropped and the tag separator is replaced so the
+// result is a legal directory name on Windows too, e.g.
+// "oci://ghcr.io/org/model:tag" -> "ghcr.io/org/model_tag".
+std::string localModelDirectoryName(const std::string& sourceModel);
+
 enum OvmsServerMode : int {
     SERVING_MODELS_MODE,
     HF_PULL_MODE,

src/cli_parser.cpp

@@ -226,7 +226,7 @@ std::variant<bool, std::pair<int, std::string>> CLIParser::parse(int argc, char*
             cxxopts::value<bool>()->default_value("false"),
             "PULL_HF")
             ("source_model",
-            "HF source model path",
+            "HF source model path, or a CNCF ModelPack OCI reference prefixed with oci:// (e.g. oci://ghcr.io/org/model:tag). OCI references are pulled with the llmman executable, which must be on PATH or pointed at by LLMMAN_BIN.",
             cxxopts::value<std::string>(),
             "HF_SOURCE")
             ("source_loras",
@@ -804,6 +804,14 @@ void CLIParser::prepareGraph(ServerSettingsImpl& serverSettings, HFSettingsImpl&
         if (result->count("source_loras")) {
             hfSettings.sourceLoras = result->operator[]("source_loras").as<std::string>();
         }
+        if (isOciDownload(hfSettings.sourceModel)) {
+            // The layer media types in a CNCF ModelPack image already describe
+            // the payload, so there is nothing for --gguf_filename to select.
+            if (result->count("gguf_filename")) {
+                throw std::logic_error("--gguf_filename parameter unsupported for oci:// models.");
+            }
+            hfSettings.downloadType = OCI_DOWNLOAD;
+        }
         if ((result->count("weight-format") || result->count("extra_quantization_params")) && isOptimumCliDownload(hfSettings.sourceModel, hfSettings.ggufFilename)) {
             hfSettings.downloadType = OPTIMUM_CLI_DOWNLOAD;
         }
@@ -943,12 +951,14 @@ void CLIParser::prepareGraphStart(HFSettingsImpl& hfSettings, ModelsSettingsImpl
     if (result->count("model_name")) {
         modelsSettings.modelName = result->operator[]("model_name").as<std::string>();
     } else if (!hfSettings.sourceModel.empty()) {
-        modelsSettings.modelName = hfSettings.sourceModel;
+        // For an OCI reference the scheme is dropped so the served name is the
+        // registry reference a user would type, e.g. ghcr.io/org/model:tag.
+        modelsSettings.modelName = stripOciScheme(hfSettings.sourceModel);
     }
 
     // Only override modelPath if it wasn't already set via --model_path
     if (!result->count("model_path")) {
-        modelsSettings.modelPath = FileSystem::joinPath({hfSettings.downloadPath, hfSettings.sourceModel});
+        modelsSettings.modelPath = FileSystem::joinPath({hfSettings.downloadPath, localModelDirectoryName(hfSettings.sourceModel)});
     }
 }
 

src/default_task.cpp

@@ -24,7 +24,9 @@
 
 #include "pull_module/curl_downloader.hpp"
 #include "pull_module/hf_env_vars.hpp"
+#include "capi_frontend/server_settings.hpp"
 #include "default_task_detector.hpp"
+#include "logging.hpp"
 #include "status.hpp"
 
 namespace ovms {
@@ -79,7 +81,7 @@ std::optional<std::string> determineDefaultTaskParameter(const std::optional<std
 
     // Try local model repository path before downloading from HuggingFace
     if (modelRepositoryPath.has_value() && !modelRepositoryPath->empty()) {
-        const auto localModelDir = std::filesystem::path(*modelRepositoryPath) / *sourceModel;
+        const auto localModelDir = std::filesystem::path(*modelRepositoryPath) / localModelDirectoryName(*sourceModel);
         if (std::filesystem::exists(localModelDir)) {
             ModelCatalogContext ctx(localModelDir, *sourceModel);
             const std::string task = detector.detect(ctx);
@@ -90,6 +92,15 @@ std::optional<std::string> determineDefaultTaskParameter(const std::optional<std
         }
     }
 
+    // An OCI reference is not addressable on HuggingFace, and reading the
+    // config out of the image would mean pulling it before the CLI has even
+    // finished parsing. Report "unknown" so the caller asks for an explicit
+    // --task instead.
+    if (isOciDownload(*sourceModel)) {
+        SPDLOG_DEBUG("Task cannot be inferred for OCI reference {} - --task has to be provided explicitly", *sourceModel);
+        return std::nullopt;
+    }
+
     // Download config files from HuggingFace.
     // config.json is tried first (covers transformer-style models).
     // model_index.json is only attempted when config.json is absent — it is

src/pull_module/BUILD

@@ -32,6 +32,7 @@ ovms_cc_library(
     deps =  [
         "@ovms//src:libovmsstatus",
         "@ovms//src:libovmslogging",
+        "@ovms//src:libovms_server_settings",
         "@ovms//src/filesystem:libovmsfilesystem",
         "@ovms//src/filesystem:libovmslocalfilesystem",
     ],
@@ -111,6 +112,26 @@ ovms_cc_library(
     visibility = ["//visibility:public"],
 )
 
+ovms_cc_library(
+    name = "oci_downloader",
+    srcs = ["oci_downloader.cpp"],
+    hdrs = ["oci_downloader.hpp"],
+    deps =  [
+        ":cmd_exec",
+        ":model_downloader",
+        ":optimum_export",
+        "@com_github_tencent_rapidjson//:rapidjson",
+        "@ovms//src/port:rapidjson_document",
+        "@ovms//src:libovmslogging",
+        "@ovms//src:libovmsstatus",
+        "@ovms//src:libovmsstring_utils",
+        "@ovms//src:libovms_server_settings",
+        "@ovms//src/filesystem:libovmsfilesystem",
+        "@ovms//src/filesystem:libovmslocalfilesystem",
+    ],
+    visibility = ["//visibility:public"],
+)
+
 ovms_cc_library(
     name = "hf_pull_model_module",
     srcs = ["hf_pull_model_module.cpp"],
@@ -120,6 +141,7 @@ ovms_cc_library(
         ":curl_downloader",
         ":libgit2",
         "gguf_downloader",
+        ":oci_downloader",
         ":optimum_export",
         "@ovms//src/graph_export:graph_export",
         "@ovms//src:cpp_headers",