Add structured request-content parsing to Egress Gate

projects/egress-gate/AGENTS.md

@@ -37,10 +37,13 @@ Run focused tests while working and `make check` before handoff.
 
 ## Project map
 
-- `src/egress_gate/gates/`: `Gate`, helper bases, registry, and the regex gate
+- `src/egress_gate/gates/`: `Gate`, helper bases, registry, regex policy models,
+  and regex matching
 - `src/egress_gate/config.py`: strict ordered `gates` and `default_decision`
   policy models
 - `src/egress_gate/request.py`: protobuf-free request and request-mutation models
+- `src/egress_gate/request_content/`: reusable text parsers, strict JSON
+  selection, source-preserving edits, and normalized message blocks
 - `src/egress_gate/result.py`: gate evaluations, five-field findings, provenance,
   traces, metadata, and final results
 - `src/egress_gate/request_processor.py`: shared deadline, current-request
@@ -93,10 +96,11 @@ registration state.
 
 ## Current built-ins and boundaries
 
-This slice ships exactly one built-in. `regex` selects one typed body, path,
-query, or header scan and preserves bounded catalog loading, matching,
-overlap resolution, and detect/deny actions. Body scans also support strict
-UTF-8 replacement. Deterministic network request policy belongs to OpenShell.
+This slice ships exactly one built-in. `regex` selects one typed complete-body,
+structured JSON, normalized message-block, path, query, or header scan and
+preserves bounded catalog loading, matching, overlap resolution, and
+detect/deny actions. Complete-body and structured scans also support bounded
+replacement. Deterministic network request policy belongs to OpenShell.
 Do not add more built-ins speculatively.
 
 The OpenShell wire `Finding` remains the released five-field contract:

projects/egress-gate/README.md

@@ -66,12 +66,20 @@ gates:
 default_decision: allow
 ```
 
-The shipped registry contains exactly `regex`. Its `scan` selects the body,
-path, query, or selected request headers. Each scan contains its `action`.
-Every scan supports `detect` and `deny`. A body scan also supports `replace`.
-The typed configuration prevents unsupported combinations. A replace action
-preserves an explicit body-replacement intent even when the resulting bytes
-equal the input. Add custom trusted gates through `--registry`.
+The shipped registry contains exactly `regex`. Its `scan` selects the complete
+body, selected JSON string fields, normalized JSON message blocks, the path,
+query, or selected request headers. Each scan contains its `action`. Every scan
+supports `detect` and `deny`. Complete-body and structured JSON scans also
+support source-preserving `replace`. The typed configuration prevents
+unsupported combinations. A replace action preserves an explicit
+body-replacement intent even when the resulting bytes equal the input. Add
+custom trusted gates through `--registry`.
+
+Structured scans are explicit policy choices. `json-fields` uses bounded typed
+selectors. `message-blocks` applies a configurable JSON message mapping and can
+filter normalized roles and block kinds. Replacement re-encodes only selected
+JSON string tokens; all unrelated request-body bytes remain unchanged. Existing
+`body` scans continue to inspect and optionally replace the complete UTF-8 body.
 
 Small stateless gates can use the optional `registry.gate` helper. Gates that
 need initialization, helper bases, or typed resources use the full class-based
@@ -120,6 +128,7 @@ timeout failures must deny.
 
 - [Overview](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/index.md)
 - [Configuration](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/configuration.md)
+- [Request-content parsing](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/request-content.md)
 - [Test policies offline](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/evaluation.md)
 - [Operations](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/operations.md)
 - [Gate authoring](https://github.com/NVIDIA/OpenShell-Research/blob/main/projects/egress-gate/docs/gates/custom.md)

projects/egress-gate/docs/architecture/index.md

@@ -18,10 +18,12 @@ Egress Gate has one transport adapter and one protobuf-free pipeline processor.
 | Module | Responsibility |
 | --- | --- |
 | `request.py` | Immutable request, headers, and `RequestMutations` |
+| `request_content/` | Reusable text parsers, strict JSON documents, typed selection, source-preserving edits, and normalized message blocks |
 | `result.py` | Gate evaluations, five-field findings, provenance, traces, and result invariants |
 | `gates/base.py` | Gate lifecycle, capabilities, output validation, and UTF-8 helper |
 | `gates/registry.py` | Trusted registration, exact pipeline schema, resources, discovery, and processor preparation |
-| `gates/regex.py` | Typed scan and action selection, bounded matching, overlap handling, caching, and body replacement |
+| `gates/regex_scans.py` | Typed regex scan and action policy configuration |
+| `gates/regex.py` | Content-parser composition, non-body text adaptation, pattern catalogs, bounded matching, finding aggregation, and gate evaluation |
 | `config.py` | Strict ordered gates and required default decision |
 | `request_processor.py` | Shared deadline, immutable snapshot construction, control flow, aggregation, and provenance |
 | `service/` | Protobuf validation/conversion, worker slots, lifecycle, and wire serialization |
@@ -33,6 +35,21 @@ does not add a second execution path or import the transport adapter.
 Only `service/` imports generated protobuf/gRPC bindings. The pipeline processor
 and gates receive domain values and can be tested offline.
 
+The request body remains canonical immutable bytes. A configured gate can
+interpret the current body snapshot as strict JSON and select text nodes, then
+optionally adapt those nodes to normalized message blocks. These views remain
+local to one gate evaluation. The pipeline processor does not parse bodies or
+cache request state across reusable gate instances.
+
+Regex scan models remain declarative Pydantic configuration. During gate
+preparation, body-based variants compose a reusable `RequestContentParser`:
+`Utf8TextParser`, `JsonFieldsParser`, or `MessageBlocksParser`. Each parser owns
+text extraction and how immutable `TextReplacement` values become bounded body
+bytes. `MessageBlocksParser` composes a `MessageBlockExtractor` to normalize a
+parsed JSON document without conflating that semantic step with body parsing.
+The regex gate only adapts path, query, and header values itself, then applies
+matching and actions uniformly to the text targets it receives.
+
 ## Pipeline execution
 
 <figure class="documentation-figure documentation-figure--wide">

projects/egress-gate/docs/architecture/request-lifecycle.md

@@ -31,12 +31,15 @@ For each configured gate, the Egress Gate pipeline processor:
 
 1. Check the shared deadline.
 2. Pass the current read-only `HttpRequest` snapshot to the gate.
-3. Reconstruct and validate the returned `GateEvaluation`.
-4. Add a content-safe `GateTrace` and `SourcedFinding` values owned by the
+3. When configured by the gate, parse the current body as strict JSON, select
+   bounded string nodes, and optionally normalize those nodes as message
+   blocks.
+4. Reconstruct and validate the returned `GateEvaluation`.
+5. Add a content-safe `GateTrace` and `SourcedFinding` values owned by the
    pipeline processor.
-5. On `proceed`, validate the request mutations and construct the next request
+6. On `proceed`, validate the request mutations and construct the next request
    snapshot.
-6. On terminal `allow` or `deny`, stop without invoking later gates.
+7. On terminal `allow` or `deny`, stop without invoking later gates.
 
 The pipeline processor never changes a request object in place. It keeps the
 first snapshot private, constructs a new snapshot after each validated mutation
@@ -45,6 +48,12 @@ combines these mutations in order. A denied result always has an empty mutation
 set. Body replacement `None` and `b""` remain distinct. Header mutation variants
 use the required `kind` values `write` and `remove`.
 
+Structured JSON replacement works through the same complete-body mutation
+contract. The JSON document renders selected string-token edits in one bounded
+pass while preserving every byte outside those tokens. It returns one complete
+replacement body. A later gate therefore parses the body snapshot produced by
+earlier structured or raw replacements.
+
 If every gate proceeds, `default_decision` controls the result. Default deny
 uses `egress_gate_default_deny`. Default allow has no reason code.
 

projects/egress-gate/docs/configuration.md

@@ -49,10 +49,16 @@ fields, unknown gate types, missing defaults, and duplicate names.
 
 The shipped registry contains only `regex`. See
 [Regex gate](gates/regex.md) for scans, actions, catalogs, and replacement
-templates. `scan.kind` selects the body, path, query, or named headers.
-`scan.action.kind` selects `detect` or `deny`; a body scan can also select
-`replace`. The schema does not permit `replace` for another scan kind. A
-trusted application registry supplies other behavior.
+templates. `scan.kind` selects the complete body, selected JSON string fields,
+normalized message blocks, path, query, or named headers. Every scan supports
+`detect` and `deny`. Complete-body, `json-fields`, and `message-blocks` scans
+also support `replace`; path, query, and header schemas do not. A trusted
+application registry supplies other behavior.
+
+Structured scans do not activate automatically based on headers or request
+contents. Existing `kind: body` policies retain their complete-body UTF-8
+behavior. Choose `json-fields` or `message-blocks` explicitly when a policy
+expects a strict JSON request body.
 
 ## Inspect the installed registry
 

projects/egress-gate/docs/gates/custom.md

@@ -12,6 +12,13 @@ protobuf, or `RequestProcessor` internals. Use the function helper for a small,
 stateless gate. Use the class-based API when a gate needs initialization,
 helper-base behavior, or operational resources.
 
+Custom gates that inspect request-body text can compose the public
+`egress_gate.request_content` surface. Prepared parsers are stateless and safe
+to reuse; each parsed result and its text targets remain local to one
+`evaluate` call. See [Parse request content](../request-content.md) for parser
+selection, typed JSON paths, message mappings, replacement, and a custom-gate
+example.
+
 The repository includes runnable examples for both extension styles:
 
 - [Function-based custom gate](https://github.com/NVIDIA/OpenShell-Research/tree/main/projects/egress-gate/examples/custom-gate)

projects/egress-gate/docs/gates/regex.md

@@ -7,8 +7,9 @@ agent_markdown: true
 # Regex gate
 
 The `regex` gate matches one configured part of the current request. It can
-inspect the body, path, query, or selected header values. It returns audit-safe
-findings with type `regex_match`.
+inspect the complete body, selected JSON string fields, normalized JSON message
+blocks, path, query, or selected header values. It returns audit-safe findings
+with type `regex_match`.
 
 Choose what to scan with `scan.kind`, then choose what to do with
 `scan.action.kind`. This example replaces matches in the request body:
@@ -53,6 +54,92 @@ general regex replacement cannot rewrite arbitrary selected headers. A custom
 gate can return supported header writes or removals when it declares the
 `GateCapability.MUTATE_HEADERS` capability.
 
+## Structured JSON fields
+
+`json-fields` parses the current body as strict UTF-8 JSON and scans only string
+values selected by typed paths. The selectors are part of the general
+[`egress_gate.request_content` contract](../request-content.md), which is also
+available to custom gates. A `json-fields` selector starts at the document
+root:
+
+```yaml title="Selected JSON message content"
+name: message-identifiers
+kind: regex
+scan:
+  kind: json-fields
+  selectors:
+    - segments:
+        - kind: key
+          value: messages
+        - kind: each
+        - kind: key
+          value: content
+  action:
+    kind: replace
+    template: '[{entity}]'
+pattern_catalog: patterns.yaml
+```
+
+Missing paths and non-string terminal values produce no scan text. Overlapping
+selectors select the same JSON string once. Matches cannot span two selected
+string values. Structured replacement re-encodes each selected string token
+and preserves every source byte outside selected tokens, including whitespace,
+number spellings, key order, and escaping in unrelated strings.
+
+The JSON parser rejects invalid UTF-8, malformed JSON, duplicate object keys,
+non-standard constants, invalid Unicode scalar values, and configured parsing
+limits. Invalid UTF-8 produces `body_encoding_invalid`; invalid strict JSON
+produces `body_format_invalid`.
+
+## Normalized message blocks
+
+`message-blocks` builds on the same JSON document. A `json-message-map` selects
+one or more message arrays, reads each message role, and applies relative
+`text_selectors`. Optional `tool_input_selectors` and `tool_output_selectors`
+classify provider- or harness-specific fields explicitly. The mapping requires
+at least one selector across those three groups. The scan can then filter
+normalized roles and block kinds:
+
+```yaml title="Selected user and tool messages"
+name: model-visible-identifiers
+kind: regex
+scan:
+  kind: message-blocks
+  message_mapping:
+    kind: json-message-map
+    messages:
+      segments:
+        - kind: key
+          value: request
+        - kind: key
+          value: messages
+    role_key: role
+    text_selectors:
+      - segments:
+          - kind: key
+            value: content
+      - segments:
+          - kind: key
+            value: content
+          - kind: each
+          - kind: key
+            value: text
+  roles: [system, developer, user, tool]
+  block_kinds: [text, tool_output]
+  action:
+    kind: deny
+pattern_catalog: patterns.yaml
+```
+
+Known roles are `system`, `developer`, `user`, `assistant`, and `tool`; other or
+missing roles normalize to `unknown`. Text selected from a `tool` message is a
+`tool_output`; other selected text is `text`. Message mappings are policy
+configuration, so harness-specific envelopes do not require changes to regex
+matching. A custom gate can reuse the public `RequestContentParser`,
+`MessageBlocksParser`, and `MessageBlockExtractor` surfaces when it needs the
+same text-target and replacement contracts with a different semantic
+extractor.
+
 A catalog can be inline or in a relative `.yaml` or `.yml` file. Relative paths
 resolve from the Egress Gate process working directory, not from the policy
 file. Use an inline catalog when the process does not have a stable working
@@ -74,16 +161,18 @@ detections. Replacement uses deterministic, non-overlapping matches.
 | `deny` | terminal `deny`, findings, `egress_gate_regex_denied` |
 | `replace` | `proceed`, findings, explicit body replacement |
 
-`detect` and `deny` work with every scan kind. `replace` exists only in the
-body scan schema. It cannot be configured for a path, query, or header scan.
+`detect` and `deny` work with every scan kind. `replace` exists in the complete
+body, `json-fields`, and `message-blocks` schemas. It cannot be configured for
+a path, query, or header scan.
 This structure keeps unsupported combinations out of generated schemas and
 editor suggestions. OpenShell middleware results cannot rewrite a request path
 or query. Header replacement is not part of the built-in gate.
 
 The replace action owns its template. It returns a body replacement even when
 there is no match. This preserves the operator's explicit intent to replace the
 current body. Invalid body UTF-8 is a stable `body_encoding_invalid` service
-failure.
+failure. A structured scan whose UTF-8 body is not strict JSON produces the
+stable `body_format_invalid` failure.
 
 The regex gate does not edit the body in place. It returns the replacement in
 `RequestMutations`. The pipeline processor uses it to build the next immutable
@@ -99,6 +188,8 @@ OpenShell body limit.
 | `scan.kind` | Additional fields | Supported `action.kind` values |
 | --- | --- | --- |
 | `body` | none | `detect`, `deny`, `replace` |
+| `json-fields` | non-empty typed `selectors` | `detect`, `deny`, `replace` |
+| `message-blocks` | `message_mapping`; optional `roles` and `block_kinds` | `detect`, `deny`, `replace` |
 | `path` | none | `detect`, `deny` |
 | `query` | none | `detect`, `deny` |
 | `header` | non-empty `names` list | `detect`, `deny` |

projects/egress-gate/docs/index.md

@@ -99,6 +99,7 @@ policy pipeline can run in offline tests.
 ## Further reading
 
 - [Configuration](configuration.md)
+- [Parse request content](request-content.md)
 - [Test policies offline](evaluation.md)
 - [Operations](operations.md)
 - [Gate authoring](gates/custom.md)

projects/egress-gate/docs/reference/limits-and-failures.md

@@ -14,6 +14,12 @@ applies its configured `on_error` behavior when that ceiling expires first.
 | Area | Limit |
 | --- | ---: |
 | Request body | 4 MiB |
+| JSON nesting depth | 128 |
+| JSON value nodes | 100,000 |
+| JSON field selectors per scan | 32 |
+| Message content selectors per mapping | 32, plus the required messages selector |
+| JSON path segments per selector | 32 |
+| Selected JSON nodes or normalized message blocks | 4,096 |
 | Pipeline gates | 10 |
 | Finding groups per gate/result | 32 |
 | Estimated finding wire size | 4 KiB |
@@ -35,7 +41,7 @@ rejected value.
 
 | Condition | Outcome |
 | --- | --- |
-| Invalid phase, envelope, policy, or input encoding | gRPC `INVALID_ARGUMENT` |
+| Invalid phase, envelope, policy, input encoding, or configured JSON format | gRPC `INVALID_ARGUMENT` |
 | Gate contract or unexpected execution failure | gRPC `INTERNAL` |
 | Internal processing deadline or pipeline processor limit | deny, source `runtime_limit`, code `egress_gate_limit_exceeded` |
 | Gate terminal deny | deny, source `gate`, gate-owned reason code |