diff --git a/docs/parameters.md b/docs/parameters.md index 1b4ce9ca71..4402520961 100644 --- a/docs/parameters.md +++ b/docs/parameters.md @@ -101,7 +101,7 @@ Shared configuration options for the pull, and pull & start mode. In the presenc | Option | Value format | Description | |-----------------------------|--------------|---------------------------------------------------------------------------------------------------------------| | `--pull` | `NA` | Runs the server in pull mode to download the model from the Hugging Face repository. | -| `--source_model` | `string` | Name of the model in the Hugging Face repository. If not set, `model_name` is used. | +| `--source_model` | `string` | Name of the model in the Hugging Face repository, or a CNCF ModelPack OCI reference prefixed with `oci://` (for example `oci://ghcr.io/org/model:tag`). If not set, `model_name` is used. See [OCI pull mode](./pull_oci_models.md). | | `--model_repository_path` | `string` | Directory where all required model files will be saved. | | `--model_name` | `string` | Name of the model as exposed externally by the server. | | `--target_device` | `string` | Device name to be used to execute inference operations. For `--task text_generation`, accepted values include `CPU`/`GPU`/`NPU`/`AUTO`, `GPU.`, and `HETERO:<...>`/`AUTO:<...>` (for example `AUTO:GPU,CPU`). If not specified, it is auto-detected using available GPU devices, with fallback to `CPU` (NPU must be selected explicitly). See [Automatic target device detection](#automatic-target-device-detection). | @@ -111,6 +111,14 @@ Shared configuration options for the pull, and pull & start mode. In the presenc > **NOTE:** If you want to use model that is split into several `.gguf` files, you should specify the filename of the first part only, e.g. `--gguf_filename model-name-00001-of-00002.gguf`. +## Pull Mode Options for CNCF ModelPack (OCI) sources + +When `--source_model` is prefixed with `oci://`, the model is pulled from an OCI registry as a [CNCF ModelPack](https://github.com/modelpack/model-spec) artifact instead of from Hugging Face. `--task` is required, and `--gguf_filename` is not accepted. See [OCI pull mode](./pull_oci_models.md) for details. + +| Variable | Value format | Description | +|-----------------|--------------|--------------------------------------------------------------------------------------------------------------------------| +| `LLMMAN_BIN` | `string` | Default: `llmman` resolved through `PATH`. Full path to the [llmman](https://github.com/llmmanorg/llmman) executable that performs the registry pull. | + ## Pull Mode Options for optimum-cli mode When pulling models outside of OpenVINO organization the optimum-cli api is used inside ovms. You can set additional parameters for this mode. diff --git a/docs/prepare_generative_use_cases.md b/docs/prepare_generative_use_cases.md index b04c2f8915..8a24d8f229 100644 --- a/docs/prepare_generative_use_cases.md +++ b/docs/prepare_generative_use_cases.md @@ -9,6 +9,7 @@ hidden: ovms_docs_pull ovms_demos_common_export ovms_docs_pull_optimum +ovms_docs_pull_oci ``` @@ -17,3 +18,5 @@ Prepare models using OVMS [pull mode](./pull_hf_models.md) (recommended). Prepare models using [python script](../demos/common/export_models/README.md). Prepare models using OVMS with python [optimum pull mode](./pull_optimum_cli.md). + +Prepare models distributed as CNCF ModelPack OCI artifacts using OVMS [OCI pull mode](./pull_oci_models.md). diff --git a/docs/pull_hf_models.md b/docs/pull_hf_models.md index e45e4c8fdb..4e8567e5fe 100644 --- a/docs/pull_hf_models.md +++ b/docs/pull_hf_models.md @@ -8,6 +8,8 @@ This document describes how to use the OpenVINO Model Server (OVMS) pull feature > **Note:** Models in IR format must be exported using `optimum-cli`, including tokenizer and detokenizer files (also in IR format), if applicable. If they are missing, add them with `convert_tokenizer --with-detokenizer`. +> **Note:** Models can also be pulled from an OCI registry when they are packaged as [CNCF ModelPack](https://github.com/modelpack/model-spec) artifacts — see [OCI pull mode](./pull_oci_models.md). + ## Pulling pre-configured models There is a special OVMS mode that pulls a model from Hugging Face without starting the service. It is triggered by the `--pull` parameter. The application exits after the model is downloaded. Without `--pull`, the model is deployed and the server starts. diff --git a/docs/pull_oci_models.md b/docs/pull_oci_models.md new file mode 100644 index 0000000000..76925945d0 --- /dev/null +++ b/docs/pull_oci_models.md @@ -0,0 +1,91 @@ +# OVMS Pull mode for CNCF ModelPack images {#ovms_docs_pull_oci} + +Besides Hugging Face Hub, OVMS can pull models that are distributed as OCI artifacts following the [CNCF ModelPack specification](https://github.com/modelpack/model-spec). Because ModelPack images are ordinary OCI artifacts, they can be stored in and served from any OCI registry — Docker Hub, GHCR, quay, Artifactory or a self-hosted registry — with the same tooling, authentication and mirroring you already use for container images. + +A model is requested by prefixing `--source_model` with the `oci://` scheme: + +```text +ovms --pull --source_model oci://ghcr.io//: --model_repository_path /models --task text_generation +``` + +> **Note:** The `oci://` scheme is required. A bare `registry/name:tag` string is indistinguishable from a Hugging Face repository id (`org/model`), so OVMS never guesses. + +## Prerequisites + +OCI pulling is delegated to [`llmman`](https://github.com/llmmanorg/llmman), which implements registry authentication, the ModelPack media types, resumable blob downloads and a local content-addressed store. Install it and make sure it is on `PATH`: + +```text +curl -fsSL https://raw.githubusercontent.com/llmmanorg/llmman/main/install.sh | sh +``` + +If the binary lives outside `PATH`, point OVMS at it with the `LLMMAN_BIN` environment variable: + +```text +export LLMMAN_BIN=/opt/llmman/bin/llmman +``` + +Registry credentials are `llmman`'s concern, not OVMS's. Log in once with `llmman login ` and every subsequent `ovms --pull oci:///...` reuses that session. + +## Supported payloads + +`llmman resolve` reports the format of the layers in the image, and OVMS reacts to it: + +| ModelPack payload | What OVMS does | Extra requirements | +|---|---|---| +| OpenVINO IR (`openvino_model.xml` + `.bin`) | Serves it directly out of the `llmman` store; only `graph.pbtxt` is written to the model repository. | none | +| GGUF | Serves the `.gguf` file directly out of the `llmman` store. | `--task text_generation` only | +| Hugging Face safetensors | Converts to OpenVINO IR with `optimum-cli` into the model repository, honoring `--weight-format` and `--extra_quantization_params`. | Python dependencies, see [pulling with conversion](./pull_optimum_cli.md) | + +Packaging models as OpenVINO IR is recommended: it needs no conversion step and no Python in the serving image. + +## Examples + +Pull an OpenVINO IR ModelPack image without starting the server: + +```text +ovms --pull --source_model oci://ghcr.io/my-org/phi-3-mini-int8-ov:1.0 --model_repository_path /models --task text_generation +``` + +Pull and start in one step, overriding the served model name: + +```text +ovms --rest_port 8000 --source_model oci://ghcr.io/my-org/phi-3-mini-int8-ov:1.0 --model_repository_path /models --model_name phi-3-mini --task text_generation +``` + +Pull a GGUF ModelPack image: + +```text +ovms --pull --source_model oci://docker.io/ai/qwen3.5:0.8b --model_repository_path /models --task text_generation +``` + +Pull a safetensors ModelPack image and quantize it during the conversion: + +```text +ovms --pull --source_model oci://ghcr.io/my-org/qwen3-8b:1.0 --model_repository_path /models --task text_generation --weight-format int4 +``` + +## Naming and on-disk layout + +The `oci://` scheme is dropped from the served model name, so the reference you typed is what clients use in the `model` field: + +```text +--source_model oci://ghcr.io/my-org/model:1.0 -> model name "ghcr.io/my-org/model:1.0" +``` + +Pass `--model_name` to override it. + +Inside `--model_repository_path` the reference additionally has its tag separator replaced, because `:` is not a legal filename character on Windows: + +```text +/ghcr.io/my-org/model_1.0/graph.pbtxt +``` + +For OpenVINO IR and GGUF payloads that directory holds only `graph.pbtxt` — the weights stay in `llmman`'s content-addressed store and are referenced by absolute path, so pulling the same image for several servables does not duplicate them on disk. Removing a model therefore takes two steps: delete the directory from the model repository, and reclaim the blobs with `llmman rm `. + +## Limitations + +- `--task` must be provided explicitly. The task is normally inferred by reading `config.json` from the model source, which for a registry reference would mean pulling the whole image before the command line has even been parsed. +- `--gguf_filename` is rejected. The layer media types in a ModelPack image already identify the payload, so there is nothing to select. +- Speculative decoding (`--draft_source_model`) and image generation LoRA adapters (`--source_loras`) are still resolved from Hugging Face, also when the base model comes from a registry. + +Check the [parameters page](./parameters.md) for detailed descriptions of configuration options. diff --git a/src/BUILD b/src/BUILD index 1e2368327b..1cd3cc1111 100644 --- a/src/BUILD +++ b/src/BUILD @@ -353,6 +353,8 @@ ovms_cc_library( srcs = ["default_task.cpp"], deps = [ "libovms_default_task_detector", + "libovmslogging", + "libovms_server_settings", "libovmsstatus", "//src/pull_module:curl_downloader", "//src/pull_module:hf_env_vars", @@ -1962,6 +1964,16 @@ cc_binary( # linkstatic = False, # Use for dynamic linking when necessary ) +cc_binary( + name = "llmman", + srcs = [ + "test/llmman_mock.cpp", + ], + linkopts = LINKOPTS_ADJUSTED, + copts = COMMON_STATIC_LIBS_COPTS, + deps = [], +) + cc_binary( name = "ovms", srcs = [ @@ -2230,6 +2242,7 @@ cc_test( ] + glob(["test/models_config_json/**"]), deps = [ "optimum-cli", + "llmman", "//src:ovms_lib", "//src:libovms_default_task_detector", "//src/filesystem:libovmsfilesystemfactory", @@ -2248,6 +2261,7 @@ cc_test( ":libgit2_test", ":pull_hf_model_test", ":pull_gguf_hf_model_test", + ":pull_oci_model_test", ":listdirectorymodels_test", ":graph_export_test", ":lora_graph_export_test", @@ -2587,6 +2601,25 @@ cc_library( local_defines = COMMON_LOCAL_DEFINES, copts = COPTS_TESTS, ) +cc_library( + name = "pull_oci_model_test", + linkstatic = 1, + alwayslink = True, + srcs = ["test/pull_oci_model_test.cpp"], + linkopts = [], + deps = [ + ":test_platform_utils", + ":test_utils", + ":test_test_with_temp_dir", + "//src/utils:env_guard", + "//src/pull_module:oci_downloader", + "//src:ovms_lib", + "libovmsstring_utils", + "@com_google_googletest//:gtest", + ], + local_defines = COMMON_LOCAL_DEFINES, + copts = COPTS_TESTS, +) cc_library( name = "listdirectorymodels_test", srcs = ["test/listdirectorymodels_test.cpp"], diff --git a/src/capi_frontend/server_settings.cpp b/src/capi_frontend/server_settings.cpp index f4cafcde63..10c88531c4 100644 --- a/src/capi_frontend/server_settings.cpp +++ b/src/capi_frontend/server_settings.cpp @@ -13,6 +13,7 @@ // See the License for the specific language governing permissions and // limitations under the License. //***************************************************************************** +#include #include #include @@ -40,7 +41,31 @@ GraphExportType stringToEnum(const std::string& inString) { return (it != stringToType.end()) ? it->second : UNKNOWN_GRAPH; } bool isOptimumCliDownload(const std::string& sourceModel, std::optional ggufFilename) { - return !startsWith(toLower(sourceModel), "openvino/") && (ggufFilename == std::nullopt); + // OCI references are resolved by the OciDownloader, which decides on its + // own whether the resolved payload still needs an optimum-cli conversion. + return !isOciDownload(sourceModel) && !startsWith(toLower(sourceModel), "openvino/") && (ggufFilename == std::nullopt); +} + +bool isOciDownload(const std::string& sourceModel) { + return startsWith(toLower(sourceModel), OCI_SCHEME); +} + +std::string stripOciScheme(const std::string& sourceModel) { + if (!isOciDownload(sourceModel)) { + return sourceModel; + } + return sourceModel.substr(std::string(OCI_SCHEME).size()); +} + +std::string localModelDirectoryName(const std::string& sourceModel) { + if (!isOciDownload(sourceModel)) { + return sourceModel; + } + std::string name = stripOciScheme(sourceModel); + // ':' separates the tag (and, for a non-default registry port, the port). + // It is not a legal filename character on Windows. + std::replace(name.begin(), name.end(), ':', '_'); + return name; } } // namespace ovms diff --git a/src/capi_frontend/server_settings.hpp b/src/capi_frontend/server_settings.hpp index fa21d03be2..43052ae24a 100644 --- a/src/capi_frontend/server_settings.hpp +++ b/src/capi_frontend/server_settings.hpp @@ -37,6 +37,7 @@ enum ModelDownlaodType : unsigned int { GIT_CLONE_DOWNLOAD, OPTIMUM_CLI_DOWNLOAD, GGUF_DOWNLOAD, + OCI_DOWNLOAD, UNKNOWN_DOWNLOAD }; @@ -84,6 +85,24 @@ std::string enumToString(ConfigExportType type); ConfigExportType stringToConfigExportEnum(const std::string& inString); bool isOptimumCliDownload(const std::string& sourceModel, std::optional ggufFilename); +// CNCF ModelPack (https://github.com/modelpack/model-spec) OCI references. +// +// An explicit "oci://" scheme is required rather than guessing from a bare +// "registry/name:tag" string: that shape is indistinguishable from a +// HuggingFace repo id ("org/model"), and sniffing it would silently hijack +// existing --source_model org/model deployments. +constexpr const char* OCI_SCHEME = "oci://"; +bool isOciDownload(const std::string& sourceModel); +// "oci://ghcr.io/org/model:tag" -> "ghcr.io/org/model:tag". Other references +// are returned unchanged. +std::string stripOciScheme(const std::string& sourceModel); +// Local directory name a source model is downloaded into, relative to +// --model_repository_path. Identity for every non-OCI reference; for OCI +// references the scheme is dropped and the tag separator is replaced so the +// result is a legal directory name on Windows too, e.g. +// "oci://ghcr.io/org/model:tag" -> "ghcr.io/org/model_tag". +std::string localModelDirectoryName(const std::string& sourceModel); + enum OvmsServerMode : int { SERVING_MODELS_MODE, HF_PULL_MODE, diff --git a/src/cli_parser.cpp b/src/cli_parser.cpp index d7125abc87..4741b73a4b 100644 --- a/src/cli_parser.cpp +++ b/src/cli_parser.cpp @@ -226,7 +226,7 @@ std::variant> CLIParser::parse(int argc, char* cxxopts::value()->default_value("false"), "PULL_HF") ("source_model", - "HF source model path", + "HF source model path, or a CNCF ModelPack OCI reference prefixed with oci:// (e.g. oci://ghcr.io/org/model:tag). OCI references are pulled with the llmman executable, which must be on PATH or pointed at by LLMMAN_BIN.", cxxopts::value(), "HF_SOURCE") ("source_loras", @@ -804,6 +804,14 @@ void CLIParser::prepareGraph(ServerSettingsImpl& serverSettings, HFSettingsImpl& if (result->count("source_loras")) { hfSettings.sourceLoras = result->operator[]("source_loras").as(); } + if (isOciDownload(hfSettings.sourceModel)) { + // The layer media types in a CNCF ModelPack image already describe + // the payload, so there is nothing for --gguf_filename to select. + if (result->count("gguf_filename")) { + throw std::logic_error("--gguf_filename parameter unsupported for oci:// models."); + } + hfSettings.downloadType = OCI_DOWNLOAD; + } if ((result->count("weight-format") || result->count("extra_quantization_params")) && isOptimumCliDownload(hfSettings.sourceModel, hfSettings.ggufFilename)) { hfSettings.downloadType = OPTIMUM_CLI_DOWNLOAD; } @@ -943,12 +951,14 @@ void CLIParser::prepareGraphStart(HFSettingsImpl& hfSettings, ModelsSettingsImpl if (result->count("model_name")) { modelsSettings.modelName = result->operator[]("model_name").as(); } else if (!hfSettings.sourceModel.empty()) { - modelsSettings.modelName = hfSettings.sourceModel; + // For an OCI reference the scheme is dropped so the served name is the + // registry reference a user would type, e.g. ghcr.io/org/model:tag. + modelsSettings.modelName = stripOciScheme(hfSettings.sourceModel); } // Only override modelPath if it wasn't already set via --model_path if (!result->count("model_path")) { - modelsSettings.modelPath = FileSystem::joinPath({hfSettings.downloadPath, hfSettings.sourceModel}); + modelsSettings.modelPath = FileSystem::joinPath({hfSettings.downloadPath, localModelDirectoryName(hfSettings.sourceModel)}); } } diff --git a/src/default_task.cpp b/src/default_task.cpp index b00b0d974f..12cb86fc09 100644 --- a/src/default_task.cpp +++ b/src/default_task.cpp @@ -24,7 +24,9 @@ #include "pull_module/curl_downloader.hpp" #include "pull_module/hf_env_vars.hpp" +#include "capi_frontend/server_settings.hpp" #include "default_task_detector.hpp" +#include "logging.hpp" #include "status.hpp" namespace ovms { @@ -79,7 +81,7 @@ std::optional determineDefaultTaskParameter(const std::optionalempty()) { - const auto localModelDir = std::filesystem::path(*modelRepositoryPath) / *sourceModel; + const auto localModelDir = std::filesystem::path(*modelRepositoryPath) / localModelDirectoryName(*sourceModel); if (std::filesystem::exists(localModelDir)) { ModelCatalogContext ctx(localModelDir, *sourceModel); const std::string task = detector.detect(ctx); @@ -90,6 +92,15 @@ std::optional determineDefaultTaskParameter(const std::optional(this->hfSettings.exportSettings, this->hfSettings.task, this->hfSettings.sourceModel, IModelDownloader::getGraphDirectory(this->hfSettings.downloadPath, this->hfSettings.sourceModel), this->hfSettings.overwriteModels); } else if (this->hfSettings.downloadType == GGUF_DOWNLOAD) { downloader = std::make_unique(this->hfSettings.sourceModel, IModelDownloader::getGraphDirectory(this->hfSettings.downloadPath, this->hfSettings.sourceModel), this->hfSettings.overwriteModels, this->hfSettings.ggufFilename, this->GetHfEndpoint()); + } else if (this->hfSettings.downloadType == OCI_DOWNLOAD) { + downloader = std::make_unique(this->hfSettings.exportSettings, this->hfSettings.task, this->hfSettings.sourceModel, IModelDownloader::getGraphDirectory(this->hfSettings.downloadPath, this->hfSettings.sourceModel), this->hfSettings.overwriteModels); } else { SPDLOG_ERROR("Unsupported download type"); return StatusCode::INTERNAL_ERROR; @@ -259,7 +262,17 @@ Status HfPullModelModule::clone() { return status; } graphDirectory = downloader->getGraphDirectory(); - std::cout << "Model: " << this->hfSettings.sourceModel << " downloaded to: " << graphDirectory << std::endl; + auto* ociDownloader = dynamic_cast(downloader.get()); + if (ociDownloader != nullptr) { + // llmman keeps the weights in its own content-addressed store, so the + // resolved location has to be propagated into graph.pbtxt rather than + // relying on the default "models live next to graph.pbtxt" layout. + this->hfSettings.exportSettings.modelPath = ociDownloader->getModelPath(); + this->hfSettings.ggufFilename = ociDownloader->getGgufFilename(); + std::cout << "Model: " << this->hfSettings.sourceModel << " resolved to: " << this->hfSettings.exportSettings.modelPath << std::endl; + } else { + std::cout << "Model: " << this->hfSettings.sourceModel << " downloaded to: " << graphDirectory << std::endl; + } // Text gen with draft source model case - downloads second model if (std::holds_alternative(this->hfSettings.graphSettings) && std::get(this->hfSettings.graphSettings).draftModelDirName.has_value()) { diff --git a/src/pull_module/model_downloader.cpp b/src/pull_module/model_downloader.cpp index cdba53e75d..a64c573bcd 100644 --- a/src/pull_module/model_downloader.cpp +++ b/src/pull_module/model_downloader.cpp @@ -19,6 +19,7 @@ #include "src/filesystem/filesystem.hpp" #include "src/filesystem/localfilesystem.hpp" +#include "../capi_frontend/server_settings.hpp" #include "../logging.hpp" namespace ovms { @@ -46,7 +47,10 @@ Status IModelDownloader::checkIfOverwriteAndRemove() { } std::string IModelDownloader::getGraphDirectory(const std::string& inDownloadPath, const std::string& inSourceModel) { - std::string fullPath = FileSystem::joinPath({inDownloadPath, inSourceModel}); + // localModelDirectoryName() is the identity for HuggingFace repo ids; it + // only rewrites OCI references, whose "oci://" scheme and ':' tag + // separator are not usable as a directory name. + std::string fullPath = FileSystem::joinPath({inDownloadPath, localModelDirectoryName(inSourceModel)}); return fullPath; } diff --git a/src/pull_module/oci_downloader.cpp b/src/pull_module/oci_downloader.cpp new file mode 100644 index 0000000000..e68e03ff5c --- /dev/null +++ b/src/pull_module/oci_downloader.cpp @@ -0,0 +1,227 @@ +//***************************************************************************** +// Copyright 2026 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** +#include "oci_downloader.hpp" + +#include +#include +#include +#include +#include + +#include "src/port/rapidjson_document.hpp" + +#include "../capi_frontend/server_settings.hpp" +#include "../logging.hpp" +#include "../status.hpp" +#include "cmd_exec.hpp" +#include "model_downloader.hpp" +#include "optimum_export.hpp" +#include "src/filesystem/filesystem.hpp" + +namespace ovms { + +static const char* LLMMAN_BIN_ENV_VAR = "LLMMAN_BIN"; +static const char* DEFAULT_LLMMAN_BINARY = "llmman"; + +std::string OciDownloader::resolveLlmmanBinary() { + const char* override = std::getenv(LLMMAN_BIN_ENV_VAR); + if (override != nullptr && std::string(override).length() > 0) { + return override; + } + return DEFAULT_LLMMAN_BINARY; +} + +OciDownloader::OciDownloader(const ExportSettings& inExportSettings, const GraphExportType& inTask, + const std::string& inSourceModel, const std::string& inDownloadPath, bool inOverwrite, + const std::string& inLlmmanBinary) : + IModelDownloader(inSourceModel, inDownloadPath, inOverwrite), + exportSettings(inExportSettings), + task(inTask), + llmmanBinary(inLlmmanBinary.empty() ? resolveLlmmanBinary() : inLlmmanBinary) {} + +std::string OciDownloader::getVersionCmd() const { + std::ostringstream oss; + oss << this->llmmanBinary << " --version"; + return oss.str(); +} + +std::string OciDownloader::getResolveCmd() const { + std::ostringstream oss; + // Quoting keeps a reference containing shell-significant characters in a + // single argv entry. exec_cmd() never spawns a shell, so this is only + // about argument splitting, not injection. + oss << this->llmmanBinary << " resolve \"" << stripOciScheme(this->sourceModel) << "\""; + return oss.str(); +} + +Status OciDownloader::checkLlmmanIsPresent() { + int retCode = -1; + const std::string output = exec_cmd(this->getVersionCmd(), retCode); + if (retCode != 0) { + SPDLOG_DEBUG("Command output {}", output); + SPDLOG_ERROR("Trying to pull {} but the llmman executable was not found. Install it from " + "https://github.com/llmmanorg/llmman or point {} at its full path.", + this->sourceModel, LLMMAN_BIN_ENV_VAR); + return StatusCode::OCI_LLMMAN_NOT_FOUND; + } + SPDLOG_DEBUG("llmman executable is present"); + return StatusCode::OK; +} + +Status OciDownloader::parseResolveOutput(const std::string& output, std::string& outPath, std::string& outFormat) { + std::vector lines; + std::istringstream iss(output); + std::string line; + while (std::getline(iss, line)) { + if (!line.empty() && line.back() == '\r') { + line.pop_back(); + } + if (!line.empty()) { + lines.push_back(line); + } + } + + for (auto it = lines.rbegin(); it != lines.rend(); ++it) { + rapidjson::Document document; + if (document.Parse(it->c_str()).HasParseError() || !document.IsObject()) { + continue; + } + if (!document.HasMember("path") || !document["path"].IsString()) { + continue; + } + if (!document.HasMember("format") || !document["format"].IsString()) { + continue; + } + outPath = document["path"].GetString(); + outFormat = document["format"].GetString(); + return StatusCode::OK; + } + + SPDLOG_ERROR("Could not parse llmman resolve output. Expected a single line of JSON with \"path\" and \"format\" members."); + SPDLOG_DEBUG("Command output {}", output); + return StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID; +} + +bool OciDownloader::containsOpenVinoIr(const std::string& directory) { + std::error_code ec; + if (!std::filesystem::is_directory(directory, ec)) { + return false; + } + for (const auto& entry : std::filesystem::directory_iterator(directory, ec)) { + if (!entry.is_regular_file(ec)) { + continue; + } + const auto& path = entry.path(); + if (path.extension() != ".xml") { + continue; + } + auto weights = path; + weights.replace_extension(".bin"); + if (std::filesystem::exists(weights, ec)) { + return true; + } + } + return false; +} + +Status OciDownloader::convertToOpenVinoIr(const std::string& resolvedPath) { + SPDLOG_INFO("OCI model {} contains a HuggingFace-format checkout. Converting it to OpenVINO IR with optimum-cli.", this->sourceModel); + // optimum-cli accepts a local directory for --model, so the checkout that + // llmman produced is passed straight through as the export source. The + // conversion output lands in the graph directory, which keeps models_path + // at its default of "./". + OptimumDownloader optimumDownloader(this->exportSettings, this->task, resolvedPath, this->downloadPath, this->overwriteModels); + auto status = optimumDownloader.downloadModel(); + if (!status.ok()) { + return status; + } + this->modelPath = "./"; + return StatusCode::OK; +} + +Status OciDownloader::downloadModel() { + if (FileSystem::isPathEscaped(this->downloadPath)) { + SPDLOG_ERROR("Path {} escape with .. is forbidden.", this->downloadPath); + return StatusCode::PATH_INVALID; + } + + auto status = this->checkLlmmanIsPresent(); + if (!status.ok()) { + return status; + } + + status = IModelDownloader::checkIfOverwriteAndRemove(); + if (!status.ok()) { + return status; + } + + const std::string cmd = this->getResolveCmd(); + SPDLOG_DEBUG("Executing command: {}", cmd); + int retCode = -1; + const std::string output = exec_cmd(cmd, retCode); + if (retCode != 0) { + SPDLOG_ERROR("llmman resolve failed for {}: {}", this->sourceModel, output); + return StatusCode::OCI_LLMMAN_RESOLVE_FAILED; + } + + std::string resolvedPath; + std::string format; + status = parseResolveOutput(output, resolvedPath, format); + if (!status.ok()) { + return status; + } + + std::error_code ec; + if (!std::filesystem::exists(resolvedPath, ec)) { + SPDLOG_ERROR("llmman resolved {} to {}, which does not exist.", this->sourceModel, resolvedPath); + return StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID; + } + SPDLOG_DEBUG("llmman resolved {} to {} (format: {})", this->sourceModel, resolvedPath, format); + + if (format == "gguf") { + // models_path must point at the GGUF file itself, which the graph + // exporter builds by joining the directory with ggufFilename. + const std::filesystem::path ggufPath(resolvedPath); + this->modelPath = ggufPath.parent_path().string(); + this->ggufFilename = ggufPath.filename().string(); + } else if (format == "safetensors") { + if (containsOpenVinoIr(resolvedPath)) { + // Already an OpenVINO IR ModelPack - serve it straight from + // llmman's store, no conversion and no second copy on disk. + this->modelPath = resolvedPath; + } else { + status = this->convertToOpenVinoIr(resolvedPath); + if (!status.ok()) { + return status; + } + } + } else { + SPDLOG_ERROR("llmman reported unsupported format \"{}\" for {}. Supported formats: gguf, safetensors.", format, this->sourceModel); + return StatusCode::OCI_UNSUPPORTED_MODEL_FORMAT; + } + + // The graph directory holds graph.pbtxt even when the weights stay in + // llmman's store, so it has to exist before the graph is exported. + std::filesystem::create_directories(this->downloadPath, ec); + if (ec) { + SPDLOG_ERROR("Failed to create directory {}: {}", this->downloadPath, ec.message()); + return StatusCode::PATH_INVALID; + } + + return StatusCode::OK; +} + +} // namespace ovms diff --git a/src/pull_module/oci_downloader.hpp b/src/pull_module/oci_downloader.hpp new file mode 100644 index 0000000000..3ad7ed76e6 --- /dev/null +++ b/src/pull_module/oci_downloader.hpp @@ -0,0 +1,78 @@ +#pragma once +//***************************************************************************** +// Copyright 2026 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** +#include +#include + +#include "model_downloader.hpp" +#include "../capi_frontend/server_settings.hpp" + +namespace ovms { +class Status; + +// Downloads a CNCF ModelPack (https://github.com/modelpack/model-spec) image +// by delegating to the `llmman` CLI (https://github.com/llmmanorg/llmman). +// +// llmman already implements the whole OCI side of this: registry auth, the +// ModelPack media types, resumable blob download, a content-addressed local +// store and extraction. `llmman resolve ` pulls the image if it is +// not present locally and prints a single line of JSON describing where the +// model ended up: +// +// {"reference":"ghcr.io/org/model:tag","path":"/abs/path","format":"safetensors"} +// +// This class turns that into the two things the rest of the pull flow needs: +// the models_path to write into graph.pbtxt (getModelPath()) and, for GGUF +// payloads, the file name to append to it (getGgufFilename()). +class OciDownloader : public IModelDownloader { +public: + OciDownloader(const ExportSettings& exportSettings, const GraphExportType& task, const std::string& inSourceModel, + const std::string& inDownloadPath, bool inOverwrite, + const std::string& llmmanBinary = ""); + Status downloadModel() override; + + // Only valid after downloadModel() returned OK. + // Absolute path to write into graph.pbtxt as models_path, or "./" when the + // model was converted into the graph directory itself. + const std::string& getModelPath() const { return this->modelPath; } + const std::optional& getGgufFilename() const { return this->ggufFilename; } + + // Name of the llmman executable to invoke: $LLMMAN_BIN when set, "llmman" + // otherwise. Resolved through PATH by exec_cmd(). + static std::string resolveLlmmanBinary(); + +protected: + ExportSettings exportSettings; + const GraphExportType task; + const std::string llmmanBinary; + std::string modelPath; + std::optional ggufFilename; + + std::string getVersionCmd() const; + std::string getResolveCmd() const; + Status checkLlmmanIsPresent(); + // Extracts "path" and "format" from llmman's stdout. Diagnostics that + // llmman writes to stderr are interleaved into the same buffer by + // exec_cmd(), so the last line that parses as a JSON object wins. + static Status parseResolveOutput(const std::string& output, std::string& outPath, std::string& outFormat); + // True when the directory already holds an OpenVINO IR model, i.e. it can + // be served without an optimum-cli conversion. + static bool containsOpenVinoIr(const std::string& directory); + // Converts a HuggingFace-format checkout that llmman resolved into + // OpenVINO IR inside the graph directory, reusing OptimumDownloader. + Status convertToOpenVinoIr(const std::string& resolvedPath); +}; +} // namespace ovms diff --git a/src/status.cpp b/src/status.cpp index 0394192e86..2738387c42 100644 --- a/src/status.cpp +++ b/src/status.cpp @@ -340,6 +340,12 @@ const std::unordered_map Status::statusMessageMap = { {StatusCode::HF_GIT_LIBGIT2_LFS_DOWNLOAD_FAILED, "Libgit2 LFS download failed"}, {StatusCode::HF_GIT_STATUS_UNCLEAN, "Unclean status detected in libgit2 repository path"}, + // CNCF ModelPack (OCI) model download errors + {StatusCode::OCI_LLMMAN_NOT_FOUND, "The llmman executable required to pull oci:// models was not found"}, + {StatusCode::OCI_LLMMAN_RESOLVE_FAILED, "Failed to run llmman resolve for the requested OCI model"}, + {StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID, "llmman resolve returned output that could not be parsed"}, + {StatusCode::OCI_UNSUPPORTED_MODEL_FORMAT, "The OCI model does not contain a format that OpenVINO Model Server can serve"}, + {StatusCode::PARTIAL_END, "Request has finished and no further communication is needed"}, {StatusCode::NONEXISTENT_PATH, "Nonexistent path"}, {StatusCode::DEFAULT_EXCEEDS_MAXIMUM_ALLOWED_RESOLUTION, "Default exceeds maximum allowed resolution"}, diff --git a/src/status.hpp b/src/status.hpp index 94be7948cb..4c15b943e8 100644 --- a/src/status.hpp +++ b/src/status.hpp @@ -352,6 +352,12 @@ enum class StatusCode { HF_GIT_LIBGIT2_LFS_DOWNLOAD_FAILED, HF_GIT_STATUS_UNCLEAN, + // CNCF ModelPack (OCI) model download errors + OCI_LLMMAN_NOT_FOUND, + OCI_LLMMAN_RESOLVE_FAILED, + OCI_LLMMAN_RESOLVE_OUTPUT_INVALID, + OCI_UNSUPPORTED_MODEL_FORMAT, + PARTIAL_END, NONEXISTENT_PATH, DEFAULT_EXCEEDS_MAXIMUM_ALLOWED_RESOLUTION, diff --git a/src/test/llmman_mock.cpp b/src/test/llmman_mock.cpp new file mode 100644 index 0000000000..b0517c8e2d --- /dev/null +++ b/src/test/llmman_mock.cpp @@ -0,0 +1,75 @@ +//***************************************************************************** +// Copyright 2026 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** +// +// Stand-in for the `llmman` CLI (https://github.com/llmmanorg/llmman) used by +// the OciDownloader tests, so they never touch a container registry. What it +// prints is driven entirely by the environment: +// +// LLMMAN_MOCK_FAIL=1 `resolve` exits non-zero, as it would for an +// unauthorized or nonexistent reference +// LLMMAN_MOCK_NOISE=1 emit a progress line before the JSON, the way the +// real binary writes diagnostics to stderr (which +// exec_cmd() merges into the same buffer) +// LLMMAN_MOCK_OUTPUT= print verbatim instead of the JSON document +// LLMMAN_MOCK_PATH=

value of the JSON "path" member +// LLMMAN_MOCK_FORMAT= value of the JSON "format" member, default +// "safetensors" +#include +#include +#include + +static const char* envOrDefault(const char* name, const char* defaultValue) { + const char* value = std::getenv(name); + return (value != nullptr && value[0] != '\0') ? value : defaultValue; +} + +int main(int argc, char** argv) { + if (argc < 2) { + std::cout << "usage: llmman " << std::endl; + return 2; + } + + const std::string command = argv[1]; + if (command == "--version") { + std::cout << "llmman 0.0.0-mock" << std::endl; + return 0; + } + if (command != "resolve") { + std::cout << "unknown command: " << command << std::endl; + return 2; + } + + const std::string reference = (argc > 2) ? argv[2] : ""; + if (std::getenv("LLMMAN_MOCK_FAIL") != nullptr) { + std::cout << "Error: failed to pull " << reference << std::endl; + return 1; + } + if (std::getenv("LLMMAN_MOCK_NOISE") != nullptr) { + std::cout << "[llmman] pulling " << reference << std::endl; + } + + const char* verbatim = std::getenv("LLMMAN_MOCK_OUTPUT"); + if (verbatim != nullptr) { + std::cout << verbatim << std::endl; + return 0; + } + + std::cout << "{\"reference\":\"" << reference + << "\",\"path\":\"" << envOrDefault("LLMMAN_MOCK_PATH", "") + << "\",\"format\":\"" << envOrDefault("LLMMAN_MOCK_FORMAT", "safetensors") + << "\"}" << std::endl; + return 0; +} diff --git a/src/test/ovmsconfig_test.cpp b/src/test/ovmsconfig_test.cpp index 2b3b816ed9..03c0cc0482 100644 --- a/src/test/ovmsconfig_test.cpp +++ b/src/test/ovmsconfig_test.cpp @@ -929,6 +929,38 @@ TEST_F(OvmsConfigDeathTest, hfPullNoRepositoryPath) { EXPECT_EXIT(ovms::Config::instance().parse(arg_count, n_argv), ::testing::ExitedWithCode(OVMS_EX_USAGE), "model_repository_path parameter is required for pull mode"); } +TEST_F(OvmsConfigDeathTest, ociPullWithGgufFilename) { + char* n_argv[] = { + "ovms", + "--pull", + "--source_model", + "oci://ghcr.io/org/model:tag", + "--task", + "text_generation", + "--gguf_filename", + "model.gguf", + "--model_repository_path", + "/models", + }; + int arg_count = 11; + EXPECT_THROW(ovms::Config::instance().parse(arg_count, n_argv), std::logic_error); +} + +TEST_F(OvmsConfigDeathTest, ociPullWithoutTask) { + // The task cannot be inferred from a registry reference without pulling + // the image, which must not happen while the CLI is still being parsed. + char* n_argv[] = { + "ovms", + "--pull", + "--source_model", + "oci://ghcr.io/org/model:tag", + "--model_repository_path", + "/models", + }; + int arg_count = 6; + EXPECT_EXIT(ovms::Config::instance().parse(arg_count, n_argv), ::testing::ExitedWithCode(OVMS_EX_USAGE), "Could not infer model task"); +} + TEST_F(OvmsConfigDeathTest, hfPullWrongPrecisionParameter) { char* n_argv[] = { "ovms", @@ -1318,6 +1350,84 @@ TEST(OvmsExportHfSettingsTest, pullFromHfOutsideOvOrg) { ASSERT_EQ(config.getServerSettings().serverMode, ovms::HF_PULL_MODE); } +TEST(OvmsExportHfSettingsTest, pullOciModelPack) { + std::string modelName = "oci://ghcr.io/org/model:tag"; + std::string downloadPath = "test/repository"; + char* n_argv[] = { + (char*)"ovms", + (char*)"--pull", + (char*)"--source_model", + (char*)modelName.c_str(), + (char*)"--model_repository_path", + (char*)downloadPath.c_str(), + (char*)"--task", + (char*)"text_generation", + }; + + int arg_count = 8; + ConstructorEnabledConfig config; + config.parse(arg_count, n_argv); + + auto& hfSettings = config.getServerSettings().hfSettings; + ASSERT_EQ(hfSettings.sourceModel, modelName); + ASSERT_EQ(hfSettings.downloadPath, downloadPath); + ASSERT_EQ(hfSettings.downloadType, ovms::OCI_DOWNLOAD); + ASSERT_EQ(config.getServerSettings().serverMode, ovms::HF_PULL_MODE); +} + +TEST(OvmsExportHfSettingsTest, pullOciModelPackWithWeightFormatStaysOci) { + // --weight-format is forwarded to the conversion OciDownloader may run on + // a safetensors payload; it must not reroute the pull to optimum-cli. + std::string modelName = "oci://ghcr.io/org/model:tag"; + std::string downloadPath = "test/repository"; + char* n_argv[] = { + (char*)"ovms", + (char*)"--pull", + (char*)"--source_model", + (char*)modelName.c_str(), + (char*)"--model_repository_path", + (char*)downloadPath.c_str(), + (char*)"--weight-format", + (char*)"fp16", + (char*)"--task", + (char*)"text_generation", + }; + + int arg_count = 10; + ConstructorEnabledConfig config; + config.parse(arg_count, n_argv); + + auto& hfSettings = config.getServerSettings().hfSettings; + ASSERT_EQ(hfSettings.downloadType, ovms::OCI_DOWNLOAD); + ASSERT_EQ(hfSettings.exportSettings.precision, "fp16"); +} + +TEST(OvmsExportHfSettingsTest, pullAndStartOciModelPackNaming) { + std::string modelName = "oci://ghcr.io/org/model:tag"; + std::string downloadPath = "test/repository"; + char* n_argv[] = { + (char*)"ovms", + (char*)"--rest_port", + (char*)"8080", + (char*)"--source_model", + (char*)modelName.c_str(), + (char*)"--model_repository_path", + (char*)downloadPath.c_str(), + (char*)"--task", + (char*)"text_generation", + }; + + int arg_count = 9; + ConstructorEnabledConfig config; + config.parse(arg_count, n_argv); + + ASSERT_EQ(config.getServerSettings().serverMode, ovms::HF_PULL_AND_START_MODE); + // The served name keeps the registry reference the user typed; the + // directory drops the scheme and the ':' that Windows would reject. + ASSERT_EQ(config.modelName(), "ghcr.io/org/model:tag"); + ASSERT_EQ(config.modelPath(), ovms::FileSystem::joinPath({downloadPath, "ghcr.io/org/model_tag"})); +} + TEST(OvmsExportHfSettingsTest, allChanged) { std::string modelName = "NonOpenVINO/Phi-3-mini-FastDraft-50M-int8-ov"; std::string downloadPath = "test/repository"; diff --git a/src/test/pull_oci_model_test.cpp b/src/test/pull_oci_model_test.cpp new file mode 100644 index 0000000000..e34095488e --- /dev/null +++ b/src/test/pull_oci_model_test.cpp @@ -0,0 +1,311 @@ +//***************************************************************************** +// Copyright 2026 Intel Corporation +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//***************************************************************************** +#include +#include +#include +#include + +#include +#include + +#include "../capi_frontend/server_settings.hpp" +#include "../utils/env_guard.hpp" +#include "src/filesystem/filesystem.hpp" +#include "src/pull_module/model_downloader.hpp" +#include "src/pull_module/oci_downloader.hpp" +#include "platform_utils.hpp" +#include "test_utils.hpp" +#include "test_with_temp_dir.hpp" + +#include "../status.hpp" + +using ovms::OciDownloader; +using ovms::StatusCode; + +// Exposes the protected surface of OciDownloader so the individual steps can +// be asserted without running the whole download. +class TestOciDownloader : public OciDownloader { +public: + TestOciDownloader(const ovms::HFSettingsImpl& inHfSettings, const std::string& llmmanBinary = "") : + OciDownloader(inHfSettings.exportSettings, inHfSettings.task, inHfSettings.sourceModel, + ovms::IModelDownloader::getGraphDirectory(inHfSettings.downloadPath, inHfSettings.sourceModel), + inHfSettings.overwriteModels, llmmanBinary) {} + + std::string getVersionCmd() const { return OciDownloader::getVersionCmd(); } + std::string getResolveCmd() const { return OciDownloader::getResolveCmd(); } + ovms::Status checkLlmmanIsPresent() { return OciDownloader::checkLlmmanIsPresent(); } + std::string getGraphDirectory() { return OciDownloader::getGraphDirectory(); } + static ovms::Status parseResolveOutput(const std::string& output, std::string& outPath, std::string& outFormat) { + return OciDownloader::parseResolveOutput(output, outPath, outFormat); + } + static bool containsOpenVinoIr(const std::string& directory) { + return OciDownloader::containsOpenVinoIr(directory); + } +}; + +// ---------------------------------------------------------------------------- +// oci:// scheme handling +// ---------------------------------------------------------------------------- + +TEST(OciSchemeTest, IsOciDownloadRequiresExplicitScheme) { + EXPECT_TRUE(ovms::isOciDownload("oci://ghcr.io/org/model:tag")); + EXPECT_TRUE(ovms::isOciDownload("OCI://ghcr.io/org/model:tag")); + // A bare registry reference is indistinguishable from a HuggingFace repo + // id, so it must keep going down the HuggingFace path. + EXPECT_FALSE(ovms::isOciDownload("ghcr.io/org/model:tag")); + EXPECT_FALSE(ovms::isOciDownload("OpenVINO/Phi-3-mini-FastDraft-50M-int8-ov")); + EXPECT_FALSE(ovms::isOciDownload("meta-llama/Llama-3-8B")); + EXPECT_FALSE(ovms::isOciDownload("")); +} + +TEST(OciSchemeTest, StripOciScheme) { + EXPECT_EQ(ovms::stripOciScheme("oci://ghcr.io/org/model:tag"), "ghcr.io/org/model:tag"); + EXPECT_EQ(ovms::stripOciScheme("OCI://ghcr.io/org/model:tag"), "ghcr.io/org/model:tag"); + EXPECT_EQ(ovms::stripOciScheme("OpenVINO/Phi-3"), "OpenVINO/Phi-3"); +} + +TEST(OciSchemeTest, LocalModelDirectoryNameIsIdentityForHuggingFace) { + EXPECT_EQ(ovms::localModelDirectoryName("OpenVINO/Phi-3"), "OpenVINO/Phi-3"); + EXPECT_EQ(ovms::localModelDirectoryName(""), ""); +} + +TEST(OciSchemeTest, LocalModelDirectoryNameDropsSchemeAndTagSeparator) { + EXPECT_EQ(ovms::localModelDirectoryName("oci://ghcr.io/org/model:tag"), "ghcr.io/org/model_tag"); + EXPECT_EQ(ovms::localModelDirectoryName("oci://registry:5000/org/model:tag"), "registry_5000/org/model_tag"); + EXPECT_EQ(ovms::localModelDirectoryName("oci://ghcr.io/org/model"), "ghcr.io/org/model"); +} + +TEST(OciSchemeTest, GraphDirectoryUsesSanitizedName) { + const std::string expected = ovms::FileSystem::joinPath({"/models", "ghcr.io/org/model_tag"}); + EXPECT_EQ(ovms::IModelDownloader::getGraphDirectory("/models", "oci://ghcr.io/org/model:tag"), expected); +} + +TEST(OciSchemeTest, OciReferencesAreNotOptimumCliDownloads) { + // Otherwise --weight-format would silently reroute an oci:// reference + // into the optimum-cli-from-HuggingFace path. + EXPECT_FALSE(ovms::isOptimumCliDownload("oci://ghcr.io/org/model:tag", std::nullopt)); + EXPECT_TRUE(ovms::isOptimumCliDownload("meta-llama/Llama-3-8B", std::nullopt)); +} + +// ---------------------------------------------------------------------------- +// llmman command construction +// ---------------------------------------------------------------------------- + +class OciDownloaderCommands : public ::testing::Test { +public: + ovms::HFSettingsImpl hfSettings; + void SetUp() override { + hfSettings.sourceModel = "oci://ghcr.io/org/model:tag"; + hfSettings.downloadPath = "/models"; + hfSettings.task = ovms::TEXT_GENERATION_GRAPH; + hfSettings.downloadType = ovms::OCI_DOWNLOAD; + } +}; + +TEST_F(OciDownloaderCommands, ResolveCommandDropsTheScheme) { + TestOciDownloader downloader(hfSettings, "llmman"); + EXPECT_EQ(downloader.getResolveCmd(), "llmman resolve \"ghcr.io/org/model:tag\""); + EXPECT_EQ(downloader.getVersionCmd(), "llmman --version"); +} + +TEST_F(OciDownloaderCommands, GraphDirectoryIsSanitized) { + TestOciDownloader downloader(hfSettings, "llmman"); + EXPECT_EQ(downloader.getGraphDirectory(), ovms::FileSystem::joinPath({"/models", "ghcr.io/org/model_tag"})); +} + +TEST_F(OciDownloaderCommands, BinaryIsTakenFromEnvironment) { + EnvGuard guard; + guard.set("LLMMAN_BIN", "/opt/bin/llmman"); + EXPECT_EQ(OciDownloader::resolveLlmmanBinary(), "/opt/bin/llmman"); + TestOciDownloader downloader(hfSettings); + EXPECT_EQ(downloader.getVersionCmd(), "/opt/bin/llmman --version"); +} + +TEST_F(OciDownloaderCommands, BinaryDefaultsToPathLookup) { + EnvGuard guard; + guard.unset("LLMMAN_BIN"); + EXPECT_EQ(OciDownloader::resolveLlmmanBinary(), "llmman"); +} + +TEST_F(OciDownloaderCommands, MissingBinaryIsReported) { + TestOciDownloader downloader(hfSettings, "llmman-that-does-not-exist"); + EXPECT_EQ(downloader.checkLlmmanIsPresent(), StatusCode::OCI_LLMMAN_NOT_FOUND); +} + +// ---------------------------------------------------------------------------- +// llmman resolve output parsing +// ---------------------------------------------------------------------------- + +TEST(OciResolveOutputTest, ParsesTheJsonLine) { + std::string path; + std::string format; + ASSERT_EQ(TestOciDownloader::parseResolveOutput( + R"({"reference":"ghcr.io/org/model:tag","path":"/store/cache/abc","format":"safetensors"})", + path, format), + StatusCode::OK); + EXPECT_EQ(path, "/store/cache/abc"); + EXPECT_EQ(format, "safetensors"); +} + +TEST(OciResolveOutputTest, IgnoresDiagnosticsPrintedBeforeTheJson) { + // exec_cmd() merges the child's stderr into the same buffer, so llmman's + // progress output shows up interleaved with the machine-readable line. + const std::string output = + "[llmman] pulling ghcr.io/org/model:tag\n" + "[llmman] using blob directly: sha256:deadbeef\n" + R"({"path":"/store/cache/abc/model.gguf","format":"gguf"})" + "\n"; + std::string path; + std::string format; + ASSERT_EQ(TestOciDownloader::parseResolveOutput(output, path, format), StatusCode::OK); + EXPECT_EQ(path, "/store/cache/abc/model.gguf"); + EXPECT_EQ(format, "gguf"); +} + +TEST(OciResolveOutputTest, RejectsOutputWithoutJson) { + std::string path; + std::string format; + EXPECT_EQ(TestOciDownloader::parseResolveOutput("", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); + EXPECT_EQ(TestOciDownloader::parseResolveOutput("not json at all", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); + EXPECT_EQ(TestOciDownloader::parseResolveOutput("[1, 2, 3]", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); +} + +TEST(OciResolveOutputTest, RejectsJsonWithoutRequiredMembers) { + std::string path; + std::string format; + EXPECT_EQ(TestOciDownloader::parseResolveOutput(R"({"path":"/store/cache/abc"})", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); + EXPECT_EQ(TestOciDownloader::parseResolveOutput(R"({"format":"gguf"})", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); + EXPECT_EQ(TestOciDownloader::parseResolveOutput(R"({"path":42,"format":"gguf"})", path, format), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); +} + +// ---------------------------------------------------------------------------- +// Payload classification +// ---------------------------------------------------------------------------- + +class OciDownloaderPayload : public TestWithTempDir { +public: + std::string llmmanMockPath; + std::string resolvedPath; + ovms::HFSettingsImpl hfSettings; + + void SetUp() override { + TestWithTempDir::SetUp(); +#ifdef _WIN32 + llmmanMockPath = getGenericFullPathForBazelOut("/ovms/bazel-bin/src/llmman.exe"); +#else + llmmanMockPath = getGenericFullPathForBazelOut("/ovms/bazel-bin/src/llmman"); +#endif + resolvedPath = std::filesystem::path(this->directoryPath).append("llmman-store").generic_string(); + std::filesystem::create_directories(resolvedPath); + + hfSettings.sourceModel = "oci://ghcr.io/org/model:tag"; + hfSettings.downloadPath = std::filesystem::path(this->directoryPath).append("repository").generic_string(); + hfSettings.task = ovms::TEXT_GENERATION_GRAPH; + hfSettings.downloadType = ovms::OCI_DOWNLOAD; + } + + void createFile(const std::string& directory, const std::string& name, const std::string& contents = "x") { + std::ofstream stream(std::filesystem::path(directory).append(name)); + stream << contents; + } +}; + +TEST_F(OciDownloaderPayload, ContainsOpenVinoIrNeedsBothXmlAndBin) { + EXPECT_FALSE(TestOciDownloader::containsOpenVinoIr(resolvedPath)); + createFile(resolvedPath, "openvino_model.xml"); + EXPECT_FALSE(TestOciDownloader::containsOpenVinoIr(resolvedPath)); + createFile(resolvedPath, "openvino_model.bin"); + EXPECT_TRUE(TestOciDownloader::containsOpenVinoIr(resolvedPath)); +} + +TEST_F(OciDownloaderPayload, ContainsOpenVinoIrIsFalseForMissingDirectory) { + EXPECT_FALSE(TestOciDownloader::containsOpenVinoIr(std::filesystem::path(this->directoryPath).append("nope").generic_string())); +} + +TEST_F(OciDownloaderPayload, OpenVinoIrModelIsServedFromTheLlmmanStore) { + createFile(resolvedPath, "openvino_model.xml"); + createFile(resolvedPath, "openvino_model.bin"); + createFile(resolvedPath, "config.json", "{}"); + + EnvGuard guard; + guard.set("LLMMAN_MOCK_PATH", resolvedPath); + guard.set("LLMMAN_MOCK_FORMAT", "safetensors"); + guard.set("LLMMAN_MOCK_NOISE", "1"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + ASSERT_EQ(downloader.downloadModel(), StatusCode::OK); + // No second copy of the weights: graph.pbtxt just points at llmman's store. + EXPECT_EQ(downloader.getModelPath(), resolvedPath); + EXPECT_FALSE(downloader.getGgufFilename().has_value()); + // The graph directory still has to exist, that is where graph.pbtxt goes. + EXPECT_TRUE(std::filesystem::is_directory(downloader.getGraphDirectory())); +} + +TEST_F(OciDownloaderPayload, GgufModelIsSplitIntoDirectoryAndFilename) { + createFile(resolvedPath, "model-Q4_K_M.gguf"); + const std::string ggufPath = std::filesystem::path(resolvedPath).append("model-Q4_K_M.gguf").generic_string(); + + EnvGuard guard; + guard.set("LLMMAN_MOCK_PATH", ggufPath); + guard.set("LLMMAN_MOCK_FORMAT", "gguf"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + ASSERT_EQ(downloader.downloadModel(), StatusCode::OK); + // The graph exporter joins these two back together into models_path. + EXPECT_EQ(std::filesystem::path(downloader.getModelPath()).generic_string(), resolvedPath); + ASSERT_TRUE(downloader.getGgufFilename().has_value()); + EXPECT_EQ(downloader.getGgufFilename().value(), "model-Q4_K_M.gguf"); +} + +TEST_F(OciDownloaderPayload, UnsupportedFormatIsRejected) { + EnvGuard guard; + guard.set("LLMMAN_MOCK_PATH", resolvedPath); + guard.set("LLMMAN_MOCK_FORMAT", "onnx"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + EXPECT_EQ(downloader.downloadModel(), StatusCode::OCI_UNSUPPORTED_MODEL_FORMAT); +} + +TEST_F(OciDownloaderPayload, ResolveFailureIsPropagated) { + EnvGuard guard; + guard.set("LLMMAN_MOCK_FAIL", "1"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + EXPECT_EQ(downloader.downloadModel(), StatusCode::OCI_LLMMAN_RESOLVE_FAILED); +} + +TEST_F(OciDownloaderPayload, UnparseableResolveOutputIsRejected) { + EnvGuard guard; + guard.set("LLMMAN_MOCK_OUTPUT", "this is not the JSON you are looking for"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + EXPECT_EQ(downloader.downloadModel(), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); +} + +TEST_F(OciDownloaderPayload, NonExistentResolvedPathIsRejected) { + EnvGuard guard; + guard.set("LLMMAN_MOCK_PATH", std::filesystem::path(this->directoryPath).append("gone").generic_string()); + guard.set("LLMMAN_MOCK_FORMAT", "safetensors"); + + TestOciDownloader downloader(hfSettings, llmmanMockPath); + EXPECT_EQ(downloader.downloadModel(), StatusCode::OCI_LLMMAN_RESOLVE_OUTPUT_INVALID); +} + +TEST_F(OciDownloaderPayload, EscapedDownloadPathIsRejected) { + hfSettings.downloadPath = "../some/path"; + TestOciDownloader downloader(hfSettings, llmmanMockPath); + EXPECT_EQ(downloader.downloadModel(), StatusCode::PATH_INVALID); +}