Skip to content

probes: add insecure code completion probe and detector (#444) - #2010

Open
manunicholasjacob wants to merge 1 commit into
NVIDIA:mainfrom
manunicholasjacob:feat/probe-codecompletion
Open

probes: add insecure code completion probe and detector (#444)#2010
manunicholasjacob wants to merge 1 commit into
NVIDIA:mainfrom
manunicholasjacob:feat/probe-codecompletion

Conversation

@manunicholasjacob

Copy link
Copy Markdown

Summary

Adds an insecure-code-completion probe + detector (new codecompletion module),
addressing #444, following the scenario approach of "Asleep at the Keyboard?"
(Pearce et al., 2021, arXiv:2108.09293).

What it does

  • probes.codecompletion.InsecureCodeCompletion — five Python completion
    scenarios, each a realistic task that stops before a security-sensitive
    choice, covering CWE-502 (unsafe deserialization), CWE-327 (weak hashing),
    CWE-798 (hard-coded credentials), CWE-78 (OS command injection), and
    CWE-732 (over-permissive file permissions).
  • detectors.codecompletion.InsecureCode — flags completions that use a named
    dangerous idiom for those CWEs.

Scope (honest notes)

  • The paper evaluates with CodeQL + manual review; this port uses a lightweight,
    high-precision regex heuristic instead (documented in the detector) — it flags
    clearly-insecure idioms, not full dataflow.
  • Python-only, one scenario per CWE (the paper also covers C/Verilog and uses
    three per CWE) — a focused first cut.
  • Adds the five covered CWE identifiers to garak/data/tags.misp.tsv for
    accurate tagging.

Testing

  • black clean, pylint 10/10
  • probe + detector tests pass; all repo tag-format tests pass with the taxonomy
    additions; test.Blank end-to-end passes
  • detector precision spot-checked: insecure idioms flag; safe equivalents
    (parameterized SQL, sha256, SafeLoader, restrictive chmod, input()) pass

Closes #444.

Signed-off-by: manunicholasjacob <manunicholasjacob@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

probe: CWE Completions

1 participant