Assertion interface refactor by KStocky · Pull Request #140 · KStocky/ShaderTestFramework

KStocky · 2026-08-03T18:35:38Z

Huge refactor to support multiple assertion libraries.

There are docs on how to take advantage of this new facility to provide your own HLSL assertion library

…older

… directory, and namespace

…class

Derived class is the one that depends on the assertion framework

… be an object. It just needs to be move only

…type tests

… instead templating the mock interface. Neat

… people can signal errors at this stage.

…nterface

…s... and also binding to references of the base type... Which is a bit rubbish but I can't think of a good way around it until I have reflection...

…opy and pasted mapping types in place of a mapping template

…at is expected to happen in this part of the interface. ReadbackResults implied that the buffers should be readback to the CPU at that point but that is not the case.

…an actually do readbacks. Add Engine parameter to ProcesReadbacks function so that interfaces can actually process the readbacks.

…tests

…o that they can support byte readers if they wish. This means that the concept of byte readers is removed from both the driver and the fixture, but can still be implemented by implementations of the interface

…lso template the test driver run function and the results struct.

…sentially the assertionsv1 test fixture.

… work. This could be useful in defining how people choose what assertion interface to use in their tests

…amework.

…This then injects what framework to use in tests.

…e assertion library include to shaders compiled with that interface.

github-actions · 2026-08-03T18:36:56Z

Test Results

134 files + 2 134 suites +2 30s ⏱️ -2s
7 066 tests - 15 7 066 ✅ - 15 0 💤 ±0 0 ❌ ±0
22 938 runs - 10 22 938 ✅ - 10 0 💤 ±0 0 ❌ ±0

Results for commit 8882701. ± Comparison against base commit 361cc6b.

This pull request removes 30 and adds 15 tests. Note that renamed tests count towards both.

ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithDispatchThreadId_WHEN_Ran_THEN_IdIsInt3
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithIntId_WHEN_Ran_THEN_IdIsInt
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - ThreadIdRegistrationTests/GIVEN_ThreadIDGivenAsUint_WHEN_ThreadIDQueried_THEN_RegisteredThreadIdIsCorrect
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: No thread id type and non zero id
…

ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macro-expanded include
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithDispatchThreadId_WHEN_Ran_THEN_IdAsExpected
ShaderTestFrameworkTests.exe.global ‑ Scenario: AssertionInterfaceTests - Assertion library virtual path is forwarded to shaders
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id/When: Processed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id/When: Processed/Then: Results has expected sub strings
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero xyz thread id
…

github-actions · 2026-08-03T18:36:56Z

Test Results

134 files + 2 134 suites +2 31s ⏱️ -1s
7 066 tests - 15 7 066 ✅ - 15 0 💤 ±0 0 ❌ ±0
22 938 runs - 10 22 938 ✅ - 10 0 💤 ±0 0 ❌ ±0

Results for commit e219eb7. ± Comparison against base commit 361cc6b.

This pull request removes 30 and adds 15 tests. Note that renamed tests count towards both.

ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithDispatchThreadId_WHEN_Ran_THEN_IdIsInt3
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithIntId_WHEN_Ran_THEN_IdIsInt
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - ThreadIdRegistrationTests/GIVEN_ThreadIDGivenAsUint_WHEN_ThreadIDQueried_THEN_RegisteredThreadIdIsCorrect
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and multi dim threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and single threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: No thread id type and non zero id
…

ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macro-expanded include
ShaderTestFrameworkTests.exe.ShaderTestFixtureBaseFixture ‑ HLSLFrameworkTests - Macros - SCENARIO/GIVEN_ScenarioWithDispatchThreadId_WHEN_Ran_THEN_IdAsExpected
ShaderTestFrameworkTests.exe.global ‑ Scenario: AssertionInterfaceTests - Assertion library virtual path is forwarded to shaders
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId/When: results streamed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - TestRunResults - Stream Operator/Given: Failed Asserts with no data/And given: single assert with no sections and no strings and threadId/When: results streamed/Then: Output is as expected
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id/When: Processed
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero x thread id/When: Processed/Then: Results has expected sub strings
ShaderTestFrameworkTests.exe.global ‑ Scenario: TestDataBufferProcessorTests - ThreadInfoToString/Given: non zero xyz thread id
…

♻️ This comment has been updated with latest results.

…o make it look like an actual browser rather than a bot

… the rate limiting a little less

KStocky added 30 commits August 3, 2026 19:29

Move current code that involves assertions into a new namespace and f…

3b3c25b

…older

Move all code that the old assertion framework uses to a AssertionsV1…

9bf56f2

… directory, and namespace

Committing missed file from the last commit

2a9c21e

Remove file that does not exist from CMakeLists

f9c8687

Fix example 4

2873b44

Renaming and moving shader test fixture enums and structs out of the …

f043f13

…class

Separate ShaderTestFixture into a base and derived class

c80825e

Derived class is the one that depends on the assertion framework

Refactoring ShaderTestDriver to not be an Object. It does not need to…

286963a

… be an object. It just needs to be move only

Starting to write the interface required for assertions

d40cce2

Adding UniqueType test utility for generating types when required in …

370e565

…type tests

Added requirement for resource creation function to assertion interface.

c4b7afd

Adding creation params and constructor requirements to the interface

32c7e7d

Refactored assertion interface type tests to remove copy and paste by…

09aa930

… instead templating the mock interface. Neat

Make the CreateGPUResources function return an expected error so that…

e39fdf4

… people can signal errors at this stage.

Added Enum concepts to concepts header

57799c4

Created a meta function for mapping enum values to types

a9c87e1

Added requirements for the shader binding function to the assertion i…

a03a320

…nterface

Adding type trait for getting the first type from a parameter pack

9209f06

AllSameType concept with some tests

c4a1cec

Improving EnumValsToTypes by removing a lot of unnecessary syntax

d0fef89

Simplify assertion interface test enums

5c4440a

Add extra test to show that UniqueType will allow implicit conversion…

2f67267

…s... and also binding to references of the base type... Which is a bit rubbish but I can't think of a good way around it until I have reflection...

Cleaning up the assertion interface tests again. This time removing c…

72aec27

…opy and pasted mapping types in place of a mapping template

Added ReadbackResults requirement for the assertion interface

6cbd7b4

Rename ReadbackResults to QueueReadbacks to be more explicit about wh…

a0ac01d

…at is expected to happen in this part of the interface. ReadbackResults implied that the buffers should be readback to the CPU at that point but that is not the case.

Adding the process readbacks function to the assertion interface

560b872

Added functions to ScopedShader to query dispatch configuration

cb5c575

Add Context parameter to QueueReadbacks function so that interfaces c…

bcd1fcf

…an actually do readbacks. Add Engine parameter to ProcesReadbacks function so that interfaces can actually process the readbacks.

Removed dispatch thread types. dispatch thread ids are always uint3s now

4b288c2

Make TestRunResults conform to the new assertion interface and write …

ff2b726

…tests

KStocky added 11 commits August 3, 2026 19:31

Make the assertion interface also participate in shader compilation s…

d334803

…o that they can support byte readers if they wish. This means that the concept of byte readers is removed from both the driver and the fixture, but can still be implemented by implementations of the interface

Make the move to templating the fixture on the assertion interface. A…

a6f45b7

…lso template the test driver run function and the results struct.

Take PerTestData by value. There is no reason for this to be a reference

51e6ae5

Move ShaderTestFixture to the assertionsv1 namespace since that is es…

f3a4169

…sentially the assertionsv1 test fixture.

Attempt to fix broken cppref links

c1ff93e

Adding a test for having an include be provided as a define. Seems to…

5a94d09

… work. This could be useful in defining how people choose what assertion interface to use in their tests

Making define include test not dependent on the actual shader test fr…

3bd63c8

…amework.

Adding the assertion library include path to the AssertionInterface. …

0642e42

…This then injects what framework to use in tests.

Adding a test to ensure that the assertion interface is forwarding th…

cebb02e

…e assertion library include to shaders compiled with that interface.

Adding docs for the assertion interface system

6806a29

Further docs updates

8882701

KStocky added 6 commits August 3, 2026 19:43

Making linkspector more resilient to timeouts due to rate limits. Als…

3dd3210

…o make it look like an actual browser rather than a bot

Change all cppref links to the www. version to hopefully fall foul of…

a95359e

… the rate limiting a little less

Upgrading linkspector

409c1cb

Yet another attempt to avoid cppreference rate limiting.

fc9ab2b

Adding linkspector config to docs change filter

e219eb7

Allow 403 as an alive code

aa85631

KStocky merged commit 5057d0b into main Aug 3, 2026
7 checks passed

Navigation Menu

Assertion interface refactor - #140

Assertion interface refactor#140
KStocky merged 47 commits into
mainfrom
AssertionInterfaceRefactor

KStocky commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

KStocky commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Test Results

Uh oh!

github-actions Bot commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Aug 3, 2026 •

edited

Loading