Add Backport Tool Analysis

[tianyiy-tim]

Issues:

Addresses P425131803

Description of changes:

Currently security fixes (or features) must be manually backported to supported LTS and FIPS branches by manually locating the affected branches from the change in code through either git blame or running tests against each branch. For a single fix this can tedious but still manageable, however when multiple fixes arrive, the coordination overhead complicates this process.

This pull request adds a tool that identifies which currently supported branches are affected by a given commit or collection of commits.

Call-outs:

• The tool only reports its findings from its analysis of the collection of commits and does not perform any cherrypicks automatically.

• Anything that git history cannot settle is flagged for AI to review, never immediately flagged as not affected, avoiding false negatives.

• The model answers through a schema, not prose. It's given one tool, record_verdict, forced with tool_choice, whose input_schema constrains the verdict to yes | no | uncertain and confidence to high | medium | low. Bedrock returns the arguments as a dict already in that shape, so there is no reply text to parse. That removes a whole class of bug rather than guarding it: earlier versions read the verdict out of Markdown, and a reasoning sentence starting with "No" could clear a branch.

  Two Bedrock limits are worth knowing, both confirmed against the live model: "strict": true on a tool is rejected (400 tools.0.custom.strict: Extra inputs are not permitted), and so is output_config with a json_schema (400 output_config.format: Extra inputs are not permitted), which is the response-format style the Bedrock guide shows. So the enums are a strong steer rather than a hard guarantee, and read_verdict still validates: an off-enum value, a missing field, a wrong type, a reply with no record_verdict call, or a reply cut short by the token limit all read as no answer, and no answer leaves the branch flagged.

• A fix that reaches inside crypto/fipsmodule/ gets a FIPS boundary warning after the table. The module is validated as a build of exactly that source, so a backport there has certification consequences this tool cannot judge; all it can do is make sure nobody finds out later. The file list goes into the saved run so publish (Add backport publish command #3415) can carry the same warning into every pull request it opens. Tests and generated files under that path are excluded, since neither is compiled into the module and a warning that fires on those stops meaning anything.

• Locally, the AI pass runs on local Amazon Bedrock credentials, can refer to the README.md for setup inquiry and troubleshooting.

• Currently this is the analyze command only. Applying cherrypicks (Add backport apply command #3414), opening the pull requests (Add backport publish command #3415), CI integration (Run the backport bot in CI #3416) and merge conflict resolution (Add backport resolve command #3417) follow on top of this one, in that order. Each is reviewable on its own diff, and each leaves the tool working.

Testing:

Tested with two layers run from 'util/backport' from an AWS-LC checkout.

Unit tests - 137 cases, no checkout or credentials required:

python3 -m unittest testing.test_engine

They cover the pure helpers (whitespace normalization, C-file detection, comment and boilerplate line filters, source-file selection, test/generated path check, branch ordering), the verdict the model records and every way it can be unreadable, the FIPS boundary check including the paths that must not match it, and the engine itself against a stubbed git: patch-id fingerprinting, cherry-pick trailers, already-patched detection, rename following, every verdict classify_branch can reach, the prompt budget, and the support-window dates.

Replay bench - 39 real AWS-LC fixes across 7 release branches with 157 fix branch cells:

python3 testing/replay_fixes.py --no-ai   # git history only, ~5 min
python3 testing/replay_fixes.py           # with the AI pass, ~20 min

Each fix is replayed in a throwaway sandbox with 'origin/main' pinned to the fix and every branch that received a backport wound back to the commit before it (making the tool unable to see the answer). The sandbox borrows objects from the checkout, so nothing is cloned.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.44%. Comparing base (8ed4798) to head (184f7fd).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3389      +/-   ##
==========================================
+ Coverage   78.22%   78.44%   +0.21%     
==========================================
  Files         698      698              
  Lines      124580   124582       +2     
  Branches    17286    17291       +5     
==========================================
+ Hits        97456    97724     +268     
+ Misses      26199    25934     -265     
+ Partials      925      924       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.