Skip to content

Forward generation parameters to Ollama via options dict - #1993

Open
Panchal-Sahil wants to merge 2 commits into
NVIDIA:mainfrom
Panchal-Sahil:fix/ollama-generation-options
Open

Forward generation parameters to Ollama via options dict#1993
Panchal-Sahil wants to merge 2 commits into
NVIDIA:mainfrom
Panchal-Sahil:fix/ollama-generation-options

Conversation

@Panchal-Sahil

Copy link
Copy Markdown

Fixes #1992.

OllamaGenerator and OllamaGeneratorChat inherit max_tokens, temperature and top_k from
Generator.DEFAULT_PARAMS, accept configured values for them, then send none of them. Ollama takes
generation parameters nested under an options key that neither _call_model populated, so garak
dropped the values before the request left. Only timeout and host survived, because __init__
consumes those when building the client.

This follows the _PARAM_MAP / suppressed_params pattern from #1842:

DEFAULT_PARAMS = Generator.DEFAULT_PARAMS | {
    "timeout": 30,
    "host": "127.0.0.1:11434",
    "suppressed_params": set(),
}

_PARAM_MAP = {
    "max_tokens": ("num_predict", int),
    "temperature": ("temperature", float),
    "top_k": ("top_k", int),
}

__init__ coerces suppressed_params to a set and warns on keys outside _PARAM_MAP, matching
BedrockGenerator. _build_options returns None when the map yields nothing, which the ollama
client treats the same as omitting options.

On structure: BedrockGenerator assembles inferenceConfig inline because it has a single
_call_model. Ollama has two, on OllamaGenerator and OllamaGeneratorChat, so the loop sits in
a _build_options helper on the base class that both share. That also means the OllamaVision
generator proposed in #1764 inherits it without changes. Happy to inline it twice if you would
rather it match Bedrock line for line.

context_len is not mapped to num_ctx. Elsewhere garak treats context_len as a description
of the model's window rather than a request to set one: openai.py:411 and azure.py:113 read it
from a table of known windows, and resources/api/huggingface.py:18 reads it from config.n_ctx.
Those read a value in from what the model already is; num_ctx writes one out to resize the
server's KV cache. Ollama is one of the few backends where setting it is meaningful, so this seemed
like a maintainer call. With _PARAM_MAP in place it is a one-line addition.

This changes default behaviour. Generator.DEFAULT_PARAMS sets max_tokens to 150 rather than
None, so an unconfigured Ollama generator now sends {"num_predict": 150}. Runs that generate
until the model stops today cap near 600 characters after this.

That brings the generator in line with the documented default and with the rest of the tree.
Existing Ollama users will still see shorter responses, and scans compared across the change will
shift.

Verification

  • Supporting configuration
{
    "ollama": {
        "OllamaGeneratorChat": {
            "max_tokens": 10,
            "suppressed_params": ["max_tokens"]
        }
    }
}
  • garak -t ollama -n llama3.1:8b --probes encoding.InjectROT13
  • Run the tests and ensure they pass python -m pytest tests/ (365 passed, 13 skipped in
    tests/generators/)
  • Verify the thing does what it should: with max_tokens: 10, response length drops from a
    median of 323 characters to 39, matching an openai.OpenAICompatible control at 41 against the
    same Ollama server. Wall clock drops from 118.90s to 22.40s.
  • Verify the thing does not do what it should not: adding max_tokens to
    suppressed_params restores the old behaviour on demand, median 334 and max 1535, matching
    unpatched main at 323 and 1711. Suppression takes effect at request-assembly time.
  • Document the thing and how it works: the OllamaGenerator class docstring covers
    suppressed_params with a garak.site.yaml example, following BedrockGenerator.

Four tests assert on the outgoing HTTP request body, covering the chat endpoint, the generate
endpoint, the unconfigured default, and suppression overriding a set attribute. All four fail
against main with KeyError: 'options'.

max_tokens, temperature and top_k were accepted but never sent. Ollama takes
generation parameters under an `options` key that neither _call_model
populated, so values were dropped before the request left garak.

Follows the _PARAM_MAP / suppressed_params pattern from NVIDIA#1842. context_len is
not mapped to num_ctx, since garak treats it as a description of the model's
window elsewhere; left for maintainers to rule on.

Signed-off-by: Panchal-Sahil <panchal.sahil@outlook.com>
@jmartin-tech jmartin-tech added the generators Interfaces with LLMs label Jul 30, 2026

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a nice enhancement. One optional question and a minor testing change noted.

Comment thread tests/generators/test_ollama.py Outdated
Comment thread tests/generators/test_ollama.py Outdated
Comment thread garak/generators/ollama.py
Signed-off-by: Panchal-Sahil <panchal.sahil@outlook.com>
@Panchal-Sahil

Copy link
Copy Markdown
Author

Thanks, all three applied. Both tests now compare against gen.max_tokens.

Good call on seed, added it. It sits in Generator._run_params, and base.generate already seeds
self._rng from it, so prompt selection reproduced while sampling did not. Now both do.

I left the rest of the Options list out. top_p, stop, the penalties and mirostat* are not
attributes on the generator, so mapping them means adding them to DEFAULT_PARAMS first, which
expands what users can configure rather than fixing what garak drops today.

Happy to add any of them here if you would rather they land together.

@jmartin-tech jmartin-tech self-assigned this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

generators Interfaces with LLMs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ollama generator discards max_tokens, temperature, top_k and context_len without warning

2 participants