Skip to content

Introduce testing engine along with test schema - #3

Draft
mohammed90 wants to merge 6 commits into
mainfrom
copilot-led-attempt
Draft

Introduce testing engine along with test schema#3
mohammed90 wants to merge 6 commits into
mainfrom
copilot-led-attempt

Conversation

@mohammed90

@mohammed90 mohammed90 commented Nov 30, 2025

Copy link
Copy Markdown
Owner

Given the time shortage and absence of feedback and brain gatherings, I, shamefully, turned to GitHub Copilot. The output is yet to be thoroughly reviewed. I'm putting it out for the world and will circle back later.

Upates #1 (comment)

Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a comprehensive HTTP compliance testing framework generated primarily by GitHub Copilot. The implementation includes four Go-based tools: a test case generator, a server test engine, a client test engine, and an HTML report generator, along with supporting JSON schemas and documentation.

Key additions:

  • Test case generation from RFC requirements with automatic test type inference
  • Server compliance testing engine with HTTP request/response validation
  • Client compliance testing engine with mock server capabilities
  • HTML report generator with interactive dashboard and search functionality

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 26 comments.

Show a summary per file
File Description
test-suite/schema/test-case.json JSON schema defining test case structure with multiple test types (request/response, headers, redirects, authentication)
test-suite/README.md Comprehensive documentation for test suite architecture, usage examples, and compliance reporting format
test-generator/main.go Go tool that converts extracted RFC requirements into structured test case templates with automatic type inference
test-generator/go.mod Go module file for test generator (specifies non-existent Go 1.25)
test-engine/main.go HTTP server compliance test execution engine with validation and reporting capabilities
test-engine/go.mod Go module file for test engine (specifies non-existent Go 1.25)
html-report-generator/main.go Converts JSON compliance reports to interactive HTML dashboards with embedded CSS/JS
html-report-generator/go.mod Go module file for HTML generator (specifies non-existent Go 1.25)
html-report-generator/README.md Documentation for HTML report generator features and usage
client-test-engine/main.go Mock server-based client testing engine for validating HTTP client behavior (currently non-functional due to placeholder implementation)
client-test-engine/go.mod Go module file for client test engine (specifies non-existent Go 1.25)
client-test-engine/README.md Documentation for client testing architecture and mock server design
.github/copilot-instructions.md Comprehensive project documentation covering architecture, workflows, and development patterns

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Comment thread test-generator/main.go
Comment on lines +74 to +690
const htmlTemplate = `<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{.Title}}</title>
<style>
:root {
--primary-color: #2563eb;
--success-color: #10b981;
--warning-color: #f59e0b;
--error-color: #ef4444;
--info-color: #06b6d4;
--bg-color: #f8fafc;
--card-bg: #ffffff;
--text-primary: #1e293b;
--text-secondary: #64748b;
--border-color: #e2e8f0;
}

* {
margin: 0;
padding: 0;
box-sizing: border-box;
}

body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
background-color: var(--bg-color);
color: var(--text-primary);
line-height: 1.6;
}

.container {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}

.header {
background: linear-gradient(135deg, var(--primary-color), #3b82f6);
color: white;
padding: 2rem;
border-radius: 12px;
margin-bottom: 2rem;
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
}

.header h1 {
font-size: 2.5rem;
margin-bottom: 0.5rem;
}

.header-info {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 1rem;
margin-top: 1rem;
opacity: 0.9;
}

.card {
background: var(--card-bg);
border-radius: 12px;
padding: 1.5rem;
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);
border: 1px solid var(--border-color);
margin-bottom: 1.5rem;
}

.card h2 {
color: var(--primary-color);
margin-bottom: 1rem;
display: flex;
align-items: center;
gap: 0.5rem;
}

.summary-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
gap: 1.5rem;
margin-bottom: 2rem;
}

.stat-card {
background: var(--card-bg);
padding: 1.5rem;
border-radius: 12px;
text-align: center;
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.05);
border: 1px solid var(--border-color);
position: relative;
overflow: hidden;
}

.stat-card::before {
content: '';
position: absolute;
top: 0;
left: 0;
right: 0;
height: 4px;
background: var(--accent-color, var(--primary-color));
}

.stat-value {
font-size: 2.5rem;
font-weight: bold;
margin-bottom: 0.5rem;
}

.stat-label {
color: var(--text-secondary);
font-weight: 500;
}

.compliance-score {
--accent-color: var(--success-color);
}

.compliance-score .stat-value {
color: var(--success-color);
}

.progress-bar {
width: 100%;
height: 8px;
background: var(--border-color);
border-radius: 4px;
overflow: hidden;
margin: 1rem 0;
}

.progress-fill {
height: 100%;
background: linear-gradient(90deg, var(--success-color), var(--info-color));
border-radius: 4px;
transition: width 0.3s ease;
}

.tabs {
display: flex;
background: var(--card-bg);
border-radius: 12px 12px 0 0;
border: 1px solid var(--border-color);
border-bottom: none;
overflow: hidden;
}

.tab {
flex: 1;
padding: 1rem;
text-align: center;
cursor: pointer;
background: transparent;
border: none;
font-size: 1rem;
font-weight: 500;
color: var(--text-secondary);
transition: all 0.2s;
}

.tab.active {
background: var(--primary-color);
color: white;
}

.tab-content {
background: var(--card-bg);
border: 1px solid var(--border-color);
border-radius: 0 0 12px 12px;
padding: 1.5rem;
min-height: 400px;
}

.tab-panel {
display: none;
}

.tab-panel.active {
display: block;
}

.test-result {
padding: 1rem;
border-radius: 8px;
margin-bottom: 1rem;
border-left: 4px solid;
}

.test-result.pass {
background: rgba(16, 185, 129, 0.1);
border-left-color: var(--success-color);
}

.test-result.fail {
background: rgba(239, 68, 68, 0.1);
border-left-color: var(--error-color);
}

.test-result.skip {
background: rgba(245, 158, 11, 0.1);
border-left-color: var(--warning-color);
}

.test-result.error {
background: rgba(239, 68, 68, 0.15);
border-left-color: var(--error-color);
}

.test-header {
display: flex;
justify-content: between;
align-items: flex-start;
gap: 1rem;
margin-bottom: 0.5rem;
}

.test-id {
font-family: 'Monaco', 'Consolas', monospace;
font-size: 0.9rem;
background: rgba(0, 0, 0, 0.05);
padding: 0.25rem 0.5rem;
border-radius: 4px;
}

.test-status {
padding: 0.25rem 0.75rem;
border-radius: 20px;
font-size: 0.8rem;
font-weight: 600;
text-transform: uppercase;
}

.status-pass { background: var(--success-color); color: white; }
.status-fail { background: var(--error-color); color: white; }
.status-skip { background: var(--warning-color); color: white; }
.status-error { background: var(--error-color); color: white; }

.severity-badge {
padding: 0.25rem 0.5rem;
border-radius: 4px;
font-size: 0.8rem;
font-weight: 600;
}

.severity-ERROR { background: #fef2f2; color: #dc2626; }
.severity-WARNING { background: #fffbeb; color: #d97706; }
.severity-INFO { background: #f0f9ff; color: #0284c7; }

.test-message {
margin: 0.5rem 0;
color: var(--text-secondary);
}

.test-details {
margin-top: 1rem;
padding: 1rem;
background: rgba(0, 0, 0, 0.02);
border-radius: 6px;
font-family: 'Monaco', 'Consolas', monospace;
font-size: 0.85rem;
}

.detail-item {
margin-bottom: 0.25rem;
}

.detail-key {
color: var(--primary-color);
font-weight: 600;
}

.stats-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 1rem;
}

.stat-item {
display: flex;
justify-content: space-between;
align-items: center;
padding: 0.75rem;
background: rgba(0, 0, 0, 0.02);
border-radius: 6px;
}

.search-box {
width: 100%;
padding: 0.75rem;
border: 1px solid var(--border-color);
border-radius: 8px;
font-size: 1rem;
margin-bottom: 1rem;
}

.search-box:focus {
outline: none;
border-color: var(--primary-color);
box-shadow: 0 0 0 3px rgba(37, 99, 235, 0.1);
}

.no-results {
text-align: center;
color: var(--text-secondary);
padding: 2rem;
}

@media (max-width: 768px) {
.container {
padding: 1rem;
}

.header h1 {
font-size: 2rem;
}

.summary-grid {
grid-template-columns: 1fr;
}

.test-header {
flex-direction: column;
align-items: flex-start;
}
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<h1>{{.Title}}</h1>
<div class="header-info">
<div><strong>Target:</strong> {{.Report.Target}}</div>
<div><strong>RFCs:</strong> {{range $i, $rfc := .Report.RFCs}}{{if $i}}, {{end}}RFC {{$rfc}}{{end}}</div>
<div><strong>Generated:</strong> {{.GeneratedAt}}</div>
<div><strong>Version:</strong> {{.Report.Version}}</div>
</div>
</div>

<div class="summary-grid">
<div class="stat-card compliance-score">
<div class="stat-value">{{printf "%.1f" .Report.Summary.Compliance}}%</div>
<div class="stat-label">Compliance Score</div>
<div class="progress-bar">
<div class="progress-fill" style="width: {{.Report.Summary.Compliance}}%"></div>
</div>
</div>

<div class="stat-card">
<div class="stat-value" style="color: var(--success-color)">{{.Report.Summary.Passed}}</div>
<div class="stat-label">Tests Passed</div>
</div>

<div class="stat-card">
<div class="stat-value" style="color: var(--error-color)">{{.Report.Summary.Failed}}</div>
<div class="stat-label">Tests Failed</div>
</div>

<div class="stat-card">
<div class="stat-value" style="color: var(--text-secondary)">{{.Report.Summary.TotalTests}}</div>
<div class="stat-label">Total Tests</div>
</div>
</div>

<div class="card">
<div class="tabs">
<button class="tab active" onclick="showTab('overview')">Overview</button>
<button class="tab" onclick="showTab('passed')">Passed ({{len .PassedTests}})</button>
<button class="tab" onclick="showTab('failed')">Failed ({{len .FailedTests}})</button>
<button class="tab" onclick="showTab('skipped')">Skipped ({{len .SkippedTests}})</button>
<button class="tab" onclick="showTab('stats')">Statistics</button>
</div>

<div class="tab-content">
<div id="overview" class="tab-panel active">
<h2>📊 Test Results Overview</h2>

{{if .FailedTests}}
<h3 style="color: var(--error-color); margin-top: 2rem;">❌ Critical Failures</h3>
{{range .FailedTests}}
{{if eq .Severity "ERROR"}}
<div class="test-result fail">
<div class="test-header">
<div>
<div class="test-id">{{.TestID}}</div>
<div class="test-message">{{.Message}}</div>
</div>
<div style="display: flex; gap: 0.5rem; align-items: center;">
<span class="severity-badge severity-{{.Severity}}">{{.Severity}}</span>
<span class="test-status status-{{.Status | lower}}">{{.Status}}</span>
</div>
</div>
</div>
{{end}}
{{end}}
{{end}}

{{if .PassedTests}}
<h3 style="color: var(--success-color); margin-top: 2rem;">✅ Recent Passes</h3>
{{range slice .PassedTests 0 5}}
<div class="test-result pass">
<div class="test-header">
<div>
<div class="test-id">{{.TestID}}</div>
<div class="test-message">{{.Message}}</div>
</div>
<div style="display: flex; gap: 0.5rem; align-items: center;">
<span class="severity-badge severity-{{.Severity}}">{{.Severity}}</span>
<span class="test-status status-{{.Status | lower}}">{{.Status}}</span>
</div>
</div>
</div>
{{end}}
{{end}}
</div>

<div id="passed" class="tab-panel">
<input type="text" class="search-box" placeholder="Search passed tests..." onkeyup="filterTests('passed', this.value)">
<div id="passed-results">
{{range .PassedTests}}
<div class="test-result pass" data-search="{{.TestID | lower}} {{.Message | lower}}">
<div class="test-header">
<div>
<div class="test-id">{{.TestID}}</div>
<div class="test-message">{{.Message}}</div>
</div>
<div style="display: flex; gap: 0.5rem; align-items: center;">
<span class="severity-badge severity-{{.Severity}}">{{.Severity}}</span>
<span class="test-status status-{{.Status | lower}}">{{.Status}}</span>
</div>
</div>
{{if .Details}}
<div class="test-details">
{{range $key, $value := .Details}}
<div class="detail-item"><span class="detail-key">{{$key}}:</span> {{$value}}</div>
{{end}}
</div>
{{end}}
</div>
{{end}}
</div>
</div>

<div id="failed" class="tab-panel">
<input type="text" class="search-box" placeholder="Search failed tests..." onkeyup="filterTests('failed', this.value)">
<div id="failed-results">
{{range .FailedTests}}
<div class="test-result fail" data-search="{{.TestID | lower}} {{.Message | lower}}">
<div class="test-header">
<div>
<div class="test-id">{{.TestID}}</div>
<div class="test-message">{{.Message}}</div>
</div>
<div style="display: flex; gap: 0.5rem; align-items: center;">
<span class="severity-badge severity-{{.Severity}}">{{.Severity}}</span>
<span class="test-status status-{{.Status | lower}}">{{.Status}}</span>
</div>
</div>
{{if .Details}}
<div class="test-details">
{{range $key, $value := .Details}}
<div class="detail-item"><span class="detail-key">{{$key}}:</span> {{$value}}</div>
{{end}}
</div>
{{end}}
</div>
{{end}}
</div>
</div>

<div id="skipped" class="tab-panel">
<input type="text" class="search-box" placeholder="Search skipped tests..." onkeyup="filterTests('skipped', this.value)">
<div id="skipped-results">
{{range .SkippedTests}}
<div class="test-result skip" data-search="{{.TestID | lower}} {{.Message | lower}}">
<div class="test-header">
<div>
<div class="test-id">{{.TestID}}</div>
<div class="test-message">{{.Message}}</div>
</div>
<div style="display: flex; gap: 0.5rem; align-items: center;">
<span class="severity-badge severity-{{.Severity}}">{{.Severity}}</span>
<span class="test-status status-{{.Status | lower}}">{{.Status}}</span>
</div>
</div>
</div>
{{end}}
</div>
</div>

<div id="stats" class="tab-panel">
<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 2rem;">
<div>
<h3>📊 By Severity</h3>
<div class="stats-grid">
{{range .SeverityStats}}
<div class="stat-item">
<span>{{.Name}}</span>
<strong>{{.Count}}</strong>
</div>
{{end}}
</div>
</div>

<div>
<h3>📋 By Requirement</h3>
<div class="stats-grid">
{{range .RequirementStats}}
<div class="stat-item">
<span>{{.Name}}</span>
<strong>{{.Count}}</strong>
</div>
{{end}}
</div>
</div>

<div>
<h3>📄 By RFC</h3>
<div class="stats-grid">
{{range .RFCStats}}
<div class="stat-item">
<span>RFC {{.Name}}</span>
<strong>{{.Count}}</strong>
</div>
{{end}}
</div>
</div>
</div>
</div>
</div>
</div>
</div>

<script>
function showTab(tabName) {
// Hide all tab panels
const panels = document.querySelectorAll('.tab-panel');
panels.forEach(panel => panel.classList.remove('active'));

// Remove active class from all tabs
const tabs = document.querySelectorAll('.tab');
tabs.forEach(tab => tab.classList.remove('active'));

// Show selected tab panel
document.getElementById(tabName).classList.add('active');

// Add active class to clicked tab
event.target.classList.add('active');
}

function filterTests(category, searchTerm) {
const resultsContainer = document.getElementById(category + '-results');
const testResults = resultsContainer.querySelectorAll('.test-result');
const searchLower = searchTerm.toLowerCase();

let visibleCount = 0;
testResults.forEach(test => {
const searchData = test.getAttribute('data-search');
if (searchData.includes(searchLower)) {
test.style.display = 'block';
visibleCount++;
} else {
test.style.display = 'none';
}
});

// Show/hide no results message
let noResultsMsg = resultsContainer.querySelector('.no-results');
if (visibleCount === 0 && searchTerm.length > 0) {
if (!noResultsMsg) {
noResultsMsg = document.createElement('div');
noResultsMsg.className = 'no-results';
noResultsMsg.textContent = 'No tests found matching your search.';
resultsContainer.appendChild(noResultsMsg);
}
noResultsMsg.style.display = 'block';
} else if (noResultsMsg) {
noResultsMsg.style.display = 'none';
}
}

// Add smooth scrolling for better UX
document.addEventListener('DOMContentLoaded', function() {
// Animate progress bar
const progressFill = document.querySelector('.progress-fill');
if (progressFill) {
progressFill.style.width = '0%';
setTimeout(() => {
progressFill.style.width = progressFill.getAttribute('style').match(/width:\s*([^;]+)/)[1];
}, 500);
}

// Add tooltips to severity badges
const badges = document.querySelectorAll('.severity-badge');
badges.forEach(badge => {
const severity = badge.textContent;
let tooltip = '';
switch(severity) {
case 'ERROR':
tooltip = 'Critical compliance failure - MUST be fixed';
break;
case 'WARNING':
tooltip = 'Recommended practice - SHOULD be addressed';
break;
case 'INFO':
tooltip = 'Optional feature - MAY be implemented';
break;
}
badge.title = tooltip;
});
});
</script>
</body>
</html>`

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Embedding a 600+ line HTML template as a string constant makes the code difficult to maintain. Consider moving the HTML template to a separate file and using embed.FS or reading it at runtime. This would improve code organization and make the template easier to edit and test.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot open a new pull request to apply changes based on this feedback

Comment thread test-suite/README.md
- Validates status codes, headers, and body content
- Supports alternative validation scenarios

### 2. HTTP Header Tests (`http_header`)

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing whitespace at the end of the line.

Suggested change
### 2. HTTP Header Tests (`http_header`)
### 2. HTTP Header Tests (`http_header`)

Copilot uses AI. Check for mistakes.
```bash
# Example client commands
curl -v -L {url} # Basic curl with redirects
wget --server-response {url} # Wget with verbose output

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing whitespace at the end of the line.

Suggested change
wget --server-response {url} # Wget with verbose output
wget --server-response {url} # Wget with verbose output

Copilot uses AI. Check for mistakes.
Comment thread test-generator/main.go
Comment thread test-suite/README.md
{
"id": "rfc9110-section-9.3.2-1-HEAD-001",
"rfc": "9110",
"section": "section-9.3.2-1",

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing whitespace at the end of the line.

Suggested change
"section": "section-9.3.2-1",
"section": "section-9.3.2-1",

Copilot uses AI. Check for mistakes.
make full-workflow RFC_NUMBER=9110 TARGET_URL="https://httpbin.org"
```
1. Extract requirements from RFC XML (`rfc-requirements-extractor/`)
2. Generate test cases from requirements (`test-generator/`)

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing whitespace at the end of the line.

Suggested change
2. Generate test cases from requirements (`test-generator/`)
2. Generate test cases from requirements (`test-generator/`)

Copilot uses AI. Check for mistakes.
Comment thread test-engine/main.go

client := &http.Client{
Timeout: timeout,
}

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HTTP client does not disable redirect following, which could cause issues for tests that expect specific behavior. Consider adding a custom CheckRedirect policy to control redirect behavior explicitly: client.CheckRedirect = func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse } to prevent automatic redirect following if the test needs to validate redirect responses.

Suggested change
}
}
// Prevent automatic redirect following to allow validation of redirect responses
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
}

Copilot uses AI. Check for mistakes.
Comment thread test-generator/main.go
Comment thread test-suite/README.md
{
"summary": {
"total_tests": 150,
"passed": 140,

Copilot AI Nov 30, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing whitespace at the end of the line.

Suggested change
"passed": 140,
"passed": 140,

Copilot uses AI. Check for mistakes.
@mohammed90 mohammed90 changed the title Copilot led attempt Introduce testing engine along with test schema Dec 1, 2025

Copilot AI commented Dec 2, 2025

Copy link
Copy Markdown

@mohammed90 I've opened a new pull request, #4, to work on those changes. Once the pull request is ready, I'll request review from you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

help wanted Extra attention is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants