Skip to content

docs: describe how to add a core config option and CLI switch - #1999

Open
ehtishammubarik wants to merge 1 commit into
NVIDIA:mainfrom
ehtishammubarik:docs/adding-config-options
Open

docs: describe how to add a core config option and CLI switch#1999
ehtishammubarik wants to merge 1 commit into
NVIDIA:mainfrom
ehtishammubarik:docs/adding-config-options

Conversation

@ehtishammubarik

Copy link
Copy Markdown

Closes #385

There's currently no documented path for adding a value to garak's core configuration and
exposing it on the command line. The reference docs cover configuration from the user side
(configurable), and plugin authoring via DEFAULT_PARAMS (extending.probe,
extending.generator), but nothing describes the core-side procedure. cli.rst and
_config.rst are autodoc stubs, and cliref.rst is generated --help output.

That gap matters more than it looks, because the procedure fails silently when you get it
wrong. Registering a switch with argparse but not adding its name to the matching
*_params list in _config.py means cli.py drops the value into ignored_params
(cli.py:396-407) — the switch parses, the value shows up in the argparse Namespace, and it
never reaches _config. Nothing is printed above debug log level.

Change

Adds docs/source/extending.cli.rst, "Adding a configuration option", covering:

  • where core config lives — the four sections, their defaults in
    garak/resources/garak.core.yaml, and the *_params lists in garak/_config.py
  • the steps for a new option in an existing section: core config default, *_params entry,
    argparse definition taking its default from the config section, and the test hooks in
    tests/test_config.py (OPTIONS_PARAM / OPTIONS_SOLO)
  • verifying the result with --list_config, and what it means if the value is missing
  • adding a whole new config section: the instance, the *_params string, _store_config(),
    the cli.py dispatch branch, and the garak.core.yaml key
  • a note that cliref.rst is generated by the cliref make target and shouldn't be
    hand-edited

Registered in the index.rst "Extending and Contributing" toctree and linked from
extending.rst.

The content follows the step list posted on #385, with one correction: that list says "add a
new dataclass in _config.py" for a new config group, but current code has all four sections
as instances of the single GarakSubConfig dataclass, so the page describes adding an
instance.

Docs only — three .rst files, no code, no data, no new dependencies.

Verification

Each documented step was checked against current main behaviour rather than taken from the
issue comment. Two scratch experiments were run in a throwaway working tree and reverted
before writing the page:

  • argparse entry alone (--demo_cap, no run_params entry): the value appears in the parsed
    Namespace but _config.run has no such attribute — confirming the silent-drop behaviour
    the page warns about.
  • all three steps applied: --demo_cap 3 --list_config gives run.demo_cap = 3;
    --list_config alone gives 7, the garak.core.yaml default. Both match the page.

Commands run:

  • cd docs/source && python -m sphinx -T -E -b html -d _build/doctrees -D language=en -W ./ html
    — exit 0, build succeeded, 152 files (151 before). Same command as
    .github/workflows/docs.yml, including -W. Re-run from a clean html/_build.
  • python -m pytest tests/test_docs.py -q — exit 0, 684 passed (682 before; the new page
    is picked up by the parametrised docs checks)
  • python -m pytest tests/test_config.py -q — exit 0, 85 passed
  • python -m pytest tests/test_configurable.py -q — exit 0, 10 passed
  • Rendered output checked: extending.cli.html builds, appears in the index toctree, and
    the cross-links to/from extending resolve

Environment: Python 3.12.13, editable install with the tests and lint extras plus
docs/requirements-docs.txt, on main at 5486199.

black and pylint are not applicable — no .py file is changed.

A full python -m pytest tests/ run is slow on this machine and hadn't finished when I wrote
this up — it was still progressing with no failures. Since the diff is three .rst files with
no code changes, the Python test surface is unaffected, and CI covers the full matrix.

Why this isn't a duplicate

No open or closed PR addresses #385 (gh pr list --state all --search "385 in:body" returns
nothing, and no PR in the current open list touches CLI or config documentation). #385 is
unassigned; it was marked stale in September 2025, then un-staled and given dontautoclose by
@leondz. The related open PR #1964 covers #1511 (calibration/bag reference generation) and
doesn't overlap with this.

Limitations

  • The page documents the procedure as it is, including the *_params registration step. document how to add a cli switch using the new config setup #385
    itself notes this "ideally shouldn't need docs" — the underlying duplication between
    argparse definitions and the *_params lists is acknowledged in a comment in cli.py. This
    PR doesn't attempt that refactor.
  • The worked example (run.max_retries) is illustrative; no such option is added.
  • Placement is a judgement call. extending.cli.rst follows the existing
    extending.<topic>.rst naming, but it sits next to two plugin-authoring guides while being
    core-level. Happy to fold it into configurable.rst or extending.rst instead if you'd
    prefer fewer pages.

Note: this contribution was prepared with AI coding-assistant help.

Adds a reference page covering the steps to add a value to garak's core
configuration and expose it on the command line: the core config default,
the section's *_params entry, the argparse definition, and the test hooks.
Also covers adding a new config section, and notes that cliref.rst is
generated.

Registers the page in the index toctree and links it from extending.

Closes NVIDIA#385

Signed-off-by: ehtishammubarik <ehtisham.m7@gmail.com>
@leondz

leondz commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Can you fill in a blank for me - why would there be such a code path?

@ehtishammubarik

Copy link
Copy Markdown
Author

Can you fill in a blank for me - why would there be such a code path?

That path is intentional. argparse returns all CLI arguments in one namespace, but only some belong to config sections. The *_params lists identify those config-backed arguments, while commands such as --version, --list_probes, and specially handled arguments such as probes and detectors bypass config assignment.

The issue documented here is that a genuine config option accidentally omitted from its *_params list follows that same path silently. This PR only documents the existing behavior on main; it does not propose a code change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

document how to add a cli switch using the new config setup

2 participants