Forward generation parameters to Ollama via options dict - #1993
Open
Panchal-Sahil wants to merge 2 commits into
Open
Forward generation parameters to Ollama via options dict#1993Panchal-Sahil wants to merge 2 commits into
Panchal-Sahil wants to merge 2 commits into
Conversation
max_tokens, temperature and top_k were accepted but never sent. Ollama takes generation parameters under an `options` key that neither _call_model populated, so values were dropped before the request left garak. Follows the _PARAM_MAP / suppressed_params pattern from NVIDIA#1842. context_len is not mapped to num_ctx, since garak treats it as a description of the model's window elsewhere; left for maintainers to rule on. Signed-off-by: Panchal-Sahil <panchal.sahil@outlook.com>
jmartin-tech
requested changes
Jul 31, 2026
jmartin-tech
left a comment
Collaborator
There was a problem hiding this comment.
This looks like a nice enhancement. One optional question and a minor testing change noted.
Signed-off-by: Panchal-Sahil <panchal.sahil@outlook.com>
Author
|
Thanks, all three applied. Both tests now compare against Good call on I left the rest of the Happy to add any of them here if you would rather they land together. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1992.
OllamaGeneratorandOllamaGeneratorChatinheritmax_tokens,temperatureandtop_kfromGenerator.DEFAULT_PARAMS, accept configured values for them, then send none of them. Ollama takesgeneration parameters nested under an
optionskey that neither_call_modelpopulated, so garakdropped the values before the request left. Only
timeoutandhostsurvived, because__init__consumes those when building the client.
This follows the
_PARAM_MAP/suppressed_paramspattern from #1842:__init__coercessuppressed_paramsto a set and warns on keys outside_PARAM_MAP, matchingBedrockGenerator._build_optionsreturnsNonewhen the map yields nothing, which the ollamaclient treats the same as omitting
options.On structure:
BedrockGeneratorassemblesinferenceConfiginline because it has a single_call_model. Ollama has two, onOllamaGeneratorandOllamaGeneratorChat, so the loop sits ina
_build_optionshelper on the base class that both share. That also means theOllamaVisiongenerator proposed in #1764 inherits it without changes. Happy to inline it twice if you would
rather it match Bedrock line for line.
context_lenis not mapped tonum_ctx. Elsewhere garak treatscontext_lenas a descriptionof the model's window rather than a request to set one:
openai.py:411andazure.py:113read itfrom a table of known windows, and
resources/api/huggingface.py:18reads it fromconfig.n_ctx.Those read a value in from what the model already is;
num_ctxwrites one out to resize theserver's KV cache. Ollama is one of the few backends where setting it is meaningful, so this seemed
like a maintainer call. With
_PARAM_MAPin place it is a one-line addition.This changes default behaviour.
Generator.DEFAULT_PARAMSsetsmax_tokensto 150 rather thanNone, so an unconfigured Ollama generator now sends{"num_predict": 150}. Runs that generateuntil the model stops today cap near 600 characters after this.
That brings the generator in line with the documented default and with the rest of the tree.
Existing Ollama users will still see shorter responses, and scans compared across the change will
shift.
Verification
{ "ollama": { "OllamaGeneratorChat": { "max_tokens": 10, "suppressed_params": ["max_tokens"] } } }garak -t ollama -n llama3.1:8b --probes encoding.InjectROT13python -m pytest tests/(365 passed, 13 skipped intests/generators/)max_tokens: 10, response length drops from amedian of 323 characters to 39, matching an
openai.OpenAICompatiblecontrol at 41 against thesame Ollama server. Wall clock drops from 118.90s to 22.40s.
max_tokenstosuppressed_paramsrestores the old behaviour on demand, median 334 and max 1535, matchingunpatched
mainat 323 and 1711. Suppression takes effect at request-assembly time.OllamaGeneratorclass docstring coverssuppressed_paramswith agarak.site.yamlexample, followingBedrockGenerator.Four tests assert on the outgoing HTTP request body, covering the chat endpoint, the generate
endpoint, the unconfigured default, and suppression overriding a set attribute. All four fail
against
mainwithKeyError: 'options'.