External evaluation request — institutional proof beyond runtime evidence

[riscas028]

We have built a small public-safe demonstrator around one narrow question:

Can technically valid runtime and action evidence be prevented from becoming an institutional conclusion when required documentary, authority, outcome, or institutional proof is missing or contradictory?

N’KEMBA is designed as a downstream institutional reconstruction layer. It does not replace runtime attestation, action receipts or TRACE-style evidence. It preserves upstream proof and tests what an institution can actually prove across documents, actors, versions, decisions, external outcomes and time.

We are particularly interested in interoperability with TRACE-style action receipts and external-fact bindings.

The demonstrator includes five controlled adversarial cases. None should reach PROVED; only the complete bounded control case should.

Demo:
https://nkemba-evidence-demo-v06-q9yzjwn4x-ricardo-crespos-projects.vercel.app

We would value one thing in particular:

Can you identify a case where the demonstrator promotes more institutional certainty than the evidence supports?

Synthetic evidence only. No proprietary reconstruction method is disclosed.

If this request belongs in a different TRACE/AgentTrust channel, please point us to the appropriate place.

[lywinged]

The link in the issue is the protected preview of the deployment: a 302 to vercel.com/sso-api with an SSO nonce cookie, and a signed-in visitor is told a team owner has to approve access. The production alias under the same project name, nkemba-evidence-demo-v06.vercel.app, is public and serves the v0.6 evaluation page the issue describes, so I took the question there.

The page is one inline script of 1,450 characters with no data file and no request. It holds a table with six entries, attacks 1 to 5 and the control, and each entry carries a fixed verdict string, a fixed reason sentence and four fixed layer labels. run(i) copies entry i into the result box. Nothing is read from any evidence, and there is no evidence on the page to read.

So the page's own line, that the demonstrator fails if attacks 1 to 5 ever return PROVED, is true the way a printed page is true: attack 1 returns CONTRADICTED because the string CONTRADICTED is stored under key 1. The control returns PROVED for the same reason, and that is the case you asked for. The demonstrator promotes PROVED on no evidence at all, because the verdict was written before any evidence existed. The reproducibility block and the prior test marked PASS are text in the same file.

nkemba.pt's reconstruction page at /verifier/agentic-payment-reconstruction.html is one step better: it reads a case file and computes two things, whether the settlement beneficiary differs from the intended one and a self-test. Its six findings are still a constant array, and the receipt's signature_status is displayed and never read. Running that page's own script under Node with the authority document set to tampered, or the receipt set to invalid, still prints ESTABLISHED for those layers. Same shape one layer up: the label first, the evidence never.

Two things on the same site are done right. The pack verifier embeds the P-256 key and checks the signature locally, and the invariant line, "tamper-evidence ≠ completeness ≠ institutional truth", is the correct standard. The demonstrator meets it the day a verdict is computed rather than stored. Until then the answer to your question is yes, for every case on the page, and it takes one view-source. What this repository needs from a demonstrator is what it asks of its own reports: an input a reader can change, and a verdict that changes with it. Bring that, as vectors, and the question you asked has a way to come out in your favour.

On what #223 asked for and what landed, since #96 builds on it.

Two requests were made there: fix the licence, and turn the negative cases into conformance vectors. The licence landed the next morning, canonical Apache-2.0 text with a NOTICE, in commits at 08:19 and 08:20 on 27 August. The negative cases landed the same morning as tests/test_negative_cases.py in your repository, six cases, with a table in docs/interoperability-observations.md.

Those six cannot fail. handoff.py writes NOT_PROVED into physical_completion and the six institutional claims as string literals, seven occurrences, and has no branch that sets any of them to another value. I ran it with an input carrying a bank settlement confirmation, a signed approval and an adoption record: seven NOT_PROVED, with the outcome status echoed and unused. The protected fields come out NOT_PROVED for any input, so the assertions on them never fire. A test that cannot fail is not evidence that a boundary holds, which is the standard this project applied to its own drift tests in trace-spec#247. The official-vector test is two halves: the Ed25519 check over the trace-spec 03-utf16-key-order vector is real and catches the tamper, and the boundary block after it asserts a dictionary against the literals it was written with.

Nothing from the six cases reached trace-spec or trace-tests as a vector, which was the form the request named. The four follow-ups on #223 after the answer each restate that the reconstruction method is not disclosed, and the last links to your site.

The channel this project has for that is the vendor-annex label in trace-spec, with trace-spec#231 as the worked example. The Ed25519 half of your vector test can fail, and it shows you already know how to write the kind of test this project accepts.

For the maintainers: nothing here is a defect, a test or a spec reading for this suite, so I would move this out of trace-tests, to the vendor-annex track if the author wants to bring vectors, and close it here.

[riscas028]

Thank you — this was a useful counterexample, and we reproduced the core issue rather than arguing around it.

We have now built v0.7 as an evidence-derived verdict demonstrator. The material change is that verdicts are no longer stored per case. The evaluator exposes mutable evidentiary inputs (runtime/session evidence, action receipt/signature, human authority, document version, external outcome/settlement, and institutional adoption) and recomputes the institutional conclusion on each change.

Public test build:
https://nkemba-evidence-demo-v07-korljk9s1-ricardo-crespos-projects.vercel.app

The current mutation/regression set starts from a complete bounded chain and then changes one material input at a time. Results currently include:

• complete chain → PROVED
• invalid authority → CONTRADICTED
• invalid receipt → CONTRADICTED
• superseded document → CONTRADICTED
• failed settlement → CONTRADICTED
• missing external outcome → NOT PROVED
• conflicting receipts → INCOMPLETE
• missing institutional adoption → NOT PROVED
• superseded institutional decision → CONTRADICTED

All nine currently pass the expected-vs-actual mutation check. This remains a synthetic controlled functional-logic demonstrator, not a claim of TRACE conformance, certification, or independent audit.

Your point about bringing this as actual vectors, and about the vendor-annex track / trace-spec#231 example, is also taken. We will treat that as the appropriate next interoperability direction rather than trying to turn this issue into a trace-tests defect report.

If you find an input combination in v0.7 that still promotes more institutional certainty than the evidence supports, that is exactly the counterexample we want.

[riscas028]

Access correction to my previous reply: I shared a protected Vercel deployment URL. The public v0.7 URL is:

https://nkemba-evidence-demo-v07.vercel.app

We opened this address in an incognito browser without signing in. During that manual UI check:

• From the complete control, “Invalidate receipt” changed the conclusion to CONTRADICTED, with the reason that the action receipt/signature is invalid.
• From the complete control, “Tamper authority” changed the conclusion to CONTRADICTED, with the corresponding authority reason.
• Restoring “Human authority” to “Valid for this decision” directly through the dropdown, without pressing a preset button, returned the conclusion to PROVED while the other inputs remained in the complete-control state.

These are observed behaviours of the synthetic demonstrator, not verification of real signatures or independent evidence. They do not establish that every input combination is correct, resolve the findings about the separate reconstruction page or handoff tests, or constitute TRACE conformance. The request for reproducible vectors remains a separate next step.

Apologies for the inaccessible link in the earlier reply.

[imran-siddique]

@riscas028 you asked for a case where the demonstrator promotes more institutional certainty than the evidence supports. There is one, it is in v0.7, and it is the exact failure the tool exists to disprove.

First, credit where it belongs. @lywinged's finding on v0.6 was correct and precise, and you rebuilt rather than argued. v0.7 is genuinely different: evaluate(s) derives the verdict from the six inputs through a precedence lattice, and runTests() calls that same evaluate() on patched states instead of storing an expected verdict beside each case. That is the right architecture and it is what makes the following finding checkable at all.

The finding

evaluate() enumerates the values that are bad and falls through to PROVED. It never positively requires a dimension to be evidenced. So anything it does not recognise is silently treated as good.

I extracted evaluate() from the public page at nkemba-evidence-demo-v07.vercel.app and called it directly:

complete control (your baseline)        PROVED
EMPTY OBJECT (no dimensions at all)     PROVED
all six = ""                            PROVED
all six = "unknown"                     PROVED
all six = "pending"                     PROVED
authority = "conflict"                  PROVED
runtime  = "conflict"                   PROVED
adoption = "pending"                    PROVED
outcome  = "disputed"                   PROVED
receipt  = "expired"                    PROVED

evaluate({}) returns PROVED, with the reason string:

All required evidentiary dimensions are positively evidenced and mutually consistent for this bounded claim.

No runtime evidence, no receipt, no authority, no document, no outcome, no adoption, and the evaluator states that all of them are positively evidenced. That sentence is false about its own input, and it is the sentence a downstream consumer would carry forward.

Two narrower instances of the same root cause, worth naming because they are reachable from realistic inputs rather than only from empty ones:

• receipt and document each handle a conflict value. authority, runtime, outcome and adoption do not. So conflicting authority evidence, the case your own boundary work cares most about, returns PROVED while conflicting receipts return INCOMPLETE.
• adoption handles rejected and superseded but not pending. An institutional decision still in flight reads as recorded.

Why this is the same class as the v0.6 finding rather than a smaller one

v0.6 asserted verdicts it had not computed. v0.7 computes them, but the computation's default is the strongest verdict available. Both produce a PROVED that no evidence supports; the second is harder to see, which makes it the more dangerous of the two, and it would survive every test in runTests() because every one of those cases patches a recognised bad value onto a complete baseline.

The fix, in one sentence

Invert the default. Require each of the six dimensions to hold an enumerated positive value, and treat everything else, including absent keys and unrecognised strings, as NOT PROVED with a reason naming the dimension. Then add one regression case per dimension asserting that an unrecognised value does not reach PROVED, plus the evaluate({}) case above, because that is the one that proves the default is now closed.

This is not a criticism of the direction. A tool whose entire argument is that technical evidence must not be promoted into institutional proof has to fail closed on the absence of evidence, and once it does, the argument it makes is a much stronger one. It is also the same defect class we have been tracking across our own repositories: a verifier that cannot establish something must say so as a first-class outcome rather than let it fall through to a pass.

[lywinged]

Confirmed, and it extends the v0.6 finding by one level. My enumeration ran evaluate() over the 2,304 combinations of the values the six selects offer and found one PROVED, at the all-good state. That took the form's vocabulary as the function's domain, which it is not: called with an empty object, six empty strings, or conflict on authority, the same function returns PROVED here too. The default branch is the finding, and it sits outside anything the page can produce, so the inverted default is the right fix and evaluate({}) is the regression case that proves it closed.

One correction to my earlier comment: I pointed the mapping at the vendor-annex track. That track is for the party producing the attestation; a consumer of TRACE evidence belongs in docs/crosswalks/ with no sponsor, as answered on trace-spec#274, and the Acta crosswalk is the shape to follow.

[riscas028]

@imran-siddique @lywinged — confirmed. We reproduced the v0.7 default-pass defect and have now closed it in v0.8 rather than special-casing the reported inputs.

The evaluator now fails closed before applying the verdict precedence lattice:

• every one of the six required dimensions must be present;
• its value must be explicitly recognised for that dimension;
• absent keys, empty strings and unrecognised values return NOT PROVED and name the affected dimension;
• PROVED is reachable only when all six dimensions hold their explicit positive values.

We also added regression coverage for the cases you identified, including evaluate({}), empty/unrecognised values, and unrecognised values independently across each required dimension, plus realistic states such as authority/runtime conflict, pending adoption, disputed outcome and expired receipt.

Public v0.8 demonstrator:
https://nkemba.pt/nkemba-evidence-demo-v08.html

Published evaluator:
https://nkemba.pt/demo-evaluator-v08.js

The regression suite is also running in GitHub Actions. The first N'KEMBA v0.8 fail-closed regression run on commit ab6520d218e2a6f17ab363a36a7328ea38902f31 completed successfully.

This remains a synthetic functional-logic demonstrator, not TRACE conformance, certification, or an independent audit. The purpose of v0.8 is narrower: make it mechanically impossible for absence or an unknown evidentiary state to fall through to the strongest institutional verdict.

Thank you for the counterexample. It materially improved the evaluator.

[riscas028]

Confirmed and fixed. Thank you both for isolating the default-allow path and for the crosswalk correction.

The v0.8 evaluator now requires an explicit recognised positive value for every required dimension. Absent keys, null, empty strings, pending states, and unrecognised values return NOT PROVED; defined conflicts return INCOMPLETE; defined adverse evidence returns CONTRADICTED. PROVED is reachable only when all six dimensions are positively evidenced.

The first CI run failed before exercising the logic because the evaluator module was omitted from the repository path expected by the regression script. That packaging error is now corrected in commit 4ce1d220af7e46694f0ea7ebcae304e4f0ffcbc7.

Verification:

• fail-closed regression: 23/23 cases passed, including evaluate({}), null, all-empty, all-unknown, one unrecognised value per dimension, authority conflict, disputed outcome, and pending adoption;
• public-surface security audit: passed.

CI evidence: https://github.com/riscas028/nkemba-platform/actions/runs/33906016043

I will follow the consumer crosswalk route under docs/crosswalks/, using the Acta crosswalk as the structural reference, rather than the vendor-annex track.