Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions ROUND2_DEVLOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# ROUND2_DEVLOG

## 2026-05-20 10:15 — Started Round 2
Read the assignment PDF fully before touching code. Sketched the full flow on paper first because the requirements touched storage, scheduling, email delivery, and UI diff rendering together.

## 2026-05-20 10:45 — Decided on architecture
Chose to extend the existing Supabase-backed report flow instead of introducing a separate audit service. Wanted to keep the Round 2 feature integrated directly into the Round 1 data model.

## 2026-05-20 11:20 — Persistent snapshot implementation
Added pricing snapshot persistence to reports. Stored:
- input stack
- audit result
- pricing snapshot
- pricing version
- user email

Used deterministic pricing snapshots so historical audits remain reproducible.

## 2026-05-20 12:30 — First blocker
Hit Supabase migration issues because some RLS policies already existed from Round 1. Accidentally reran schema creation instead of isolated ALTER TABLE commands. Switched to incremental migrations instead.

## 2026-05-20 13:15 — Re-audit engine planning
Spent time deciding whether to store regenerated audits as new rows or generate them dynamically. Chose dynamic regeneration to avoid duplicated audit records and stale secondary snapshots.

## 2026-05-20 14:20 — Pricing diff engine
Built pricing comparison logic between stored snapshots and current pricing data. Added support for:
- price changes
- added tools
- removed tools

## 2026-05-20 15:40 — Recommendation diff system
Implemented structural recommendation diffing so the UI can show:
- added recommendations
- removed recommendations
- changed recommendations
- unchanged recommendations

Wanted the diff to feel understandable instead of just dumping JSON changes.

## 2026-05-20 17:10 — Testing re-audit orchestration
Built orchestration flow:
stored audits → pricing detection → re-run audit → diff generation → grouped notifications

Initially triggered too many unnecessary re-audits because score-only checks were insufficient. Added recommendation-level validation before sending notifications.

## 2026-05-20 18:30 — Grouped notification logic
Implemented user-level grouping to avoid sending multiple emails to the same user when several audits were affected by one pricing change.

## 2026-05-20 20:00 — Wrote automated tests
Added tests for:
- pricing change detection
- recommendation diffs
- grouped notifications
- API routes
- email flow

Focused more on deterministic backend logic than UI snapshot testing due to time constraints.

## 2026-05-20 23:15 — Slept
Slept approximately 23:15 → 07:00. Did not want to continue debugging exhausted because the assignment heavily depends on reasoning quality.

## 2026-05-21 08:40 — Email integration
Integrated Resend for transactional emails. Initially hit domain verification issues because the sender domain was not verified.

## 2026-05-21 09:20 — Email delivery fix
Switched sender temporarily to onboarding@resend.dev for reliable testing within the time constraint. Verified delivery successfully.

## 2026-05-21 10:30 — Re-audit UI implementation
Built side-by-side comparison UI for old vs new audit results. Added:
- savings delta
- score delta
- recommendation badges
- muted unchanged sections

## 2026-05-21 11:10 — Temporary debug route
Added a temporary debug API route to manually trigger invalidation detection locally while testing pricing changes.

## 2026-05-21 11:20 — CI failures
GitHub Actions failed due to stricter lint/typecheck rules than local environment. Fixed:
- unescaped entities
- explicit any usages
- unused variables

## 2026-05-21 12:00 — Removed debug route
Deleted temporary debug endpoint after verifying the full workflow worked end-to-end.

## 2026-05-21 12:40 — Production verification
Verified production flow manually:
1. Generate audit
2. Store report in Supabase
3. Submit email
4. Trigger pricing detection
5. Receive notification email
6. Open re-audit diff page

## 2026-05-21 13:00 — Final cleanup
Reviewed commit history, PR structure, and deployment stability. Avoided adding additional features after the core workflow stabilized.
120 changes: 120 additions & 0 deletions ROUND2_PR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
## What this PR does

This PR adds a complete live re-audit workflow to StackAudit. Audits are now persisted with pricing snapshots, monitored for pricing drift, and automatically re-evaluated when tool pricing changes. Affected users receive a consolidated email notification with a one-click re-audit link that opens a visual diff between the old and updated audit.

The feature turns StackAudit from a one-time calculator into a continuously updated audit system.

---

## Why

AI tooling pricing changes frequently, especially across products like Cursor, Claude, ChatGPT, and Copilot. A static audit becomes stale quickly and can produce outdated recommendations or inaccurate savings estimates.

This feature assumes users care less about a snapshot score and more about staying continuously optimized as the tooling market evolves.

---

## How it works

### Persistent Storage
Each generated audit now stores:
- audit id
- user email
- input stack JSON
- audit result JSON
- pricing snapshot JSON
- pricing version
- timestamp

These are persisted in Supabase and linked to the public report URL.

### Pricing Change Detection
A deterministic backend engine compares historic pricing snapshots against the current pricing source of truth.

When pricing changes:
- the original audit is re-run
- recommendations are diffed
- score delta + savings delta are calculated
- unchanged audits are ignored to prevent spam

### Notification Flow
Affected audits are grouped by user email.

One consolidated email per user is sent through Resend containing:
- what pricing changed
- affected recommendations
- updated savings impact
- a direct re-audit link

### Re-Audit Diff View
The `/re-audit/[id]` route dynamically compares:
- original audit
- newly generated audit

The UI highlights:
- added recommendations
- removed recommendations
- changed savings
- score delta
- total savings delta

---

## What I cut

- I did not add unsubscribe links because the value/effort ratio under the 36h constraint favored shipping the complete diff workflow first.
- I skipped scheduled cron automation and used a manual trigger endpoint (`/api/detect-changes`) because it was faster to verify end-to-end reliably during development.
- I did not build a full admin dashboard for pricing-change analytics or email metrics.
- I skipped persistent storage of re-audit versions to avoid duplicating large audit payloads unnecessarily during the time window.
- I did not add CSV import support for audit inputs because the assignment emphasized the pricing-change lifecycle more than ingestion UX.

---

## How to test it manually

1. Run the application locally or open the deployed URL.
2. Generate a new audit report.
3. Submit an email using the lead capture form.
4. Verify the audit row is persisted in Supabase with:
- pricing snapshot
- audit result
- user email
5. Modify a value in:
`lib/pricing/current-pricing.ts`
6. Trigger:
`POST /api/detect-changes`
7. Verify:
- affected audit count increases
- email is received
8. Click the re-audit link from the email.
9. Verify the diff page displays:
- old vs new recommendations
- score delta
- savings delta
- pricing changes

---

## What's tested

Automated tests included:
- pricing snapshot diff detection
- recommendation diff generation
- grouped notification batching
- re-audit generation logic
- detect-invalidated-audits orchestration
- detect-changes API endpoint
- resend email workflow

All tests pass successfully along with:
- `npm run lint`
- `npm test`
- `npx tsc --noEmit`

---

## Open questions / risks

- Pricing data is currently maintained manually in a centralized pricing file. A production system would likely require automated vendor scraping or admin tooling.
- Large-scale re-audits could create spikes in email volume without batching/rate-limiting infrastructure.
- The diff engine currently assumes recommendation IDs remain stable across audit-engine versions.
34 changes: 34 additions & 0 deletions ROUND2_REFLECTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# ROUND2_REFLECTION

## 1. What was the most uncomfortable trade-off you made because of the time pressure?

The biggest trade-off was intentionally avoiding a fully automated scheduled cron system and using a manually triggerable detection endpoint instead. I originally explored Vercel Cron, but I realized quickly that spending several hours debugging deployment-specific scheduling issues would risk the core feature stability.

I prioritized deterministic re-audit correctness over infrastructure polish. The important thing for this assignment was making sure the actual workflow worked reliably end-to-end:
stored audit → pricing invalidation → email notification → visual diff.

That meant accepting a simpler trigger mechanism while protecting the quality of the detection engine and diff generation logic. If this were production software with more time, I would absolutely automate scheduling and add retry/monitoring infrastructure around email delivery and detection jobs.

## 2. If we extended the deadline by another 24 hours right now, what's the first thing you'd do?

The first thing I would do is redesign the pricing configuration system into a proper versioned pricing registry with change history tracking.

Right now, pricing snapshots are deterministic and reliable, but the current setup still assumes a relatively small static pricing dataset. With another 24 hours, I would create:
- structured pricing version records
- historical change logs
- admin tooling for updating prices safely
- audit replay tooling against any historical pricing version

That would make the re-audit system much more maintainable long-term and reduce the risk of accidental pricing mutations affecting historical comparisons.

I deliberately postponed this because the assignment reward was clearly weighted toward execution quality and complete workflow delivery rather than infrastructure sophistication.

## 3. Looking back at your Round 1 codebase as a now-experienced user of it: what's one thing your Round 1 self made harder for your Round 2 self?

My Round 1 self scattered pricing assumptions across multiple parts of the application instead of centralizing them behind a single pricing abstraction.

That became painful in Round 2 because the re-audit system depends entirely on deterministic historical pricing snapshots. I had to refactor several places where pricing values were effectively duplicated or indirectly embedded inside recommendation logic.

The biggest lesson for me was that systems become significantly harder to evolve when configuration and business logic are tightly coupled. Round 1 was optimized for shipping quickly. Round 2 forced me to think much more carefully about reproducibility, auditability, and long-term maintainability.

If I rebuilt Round 1 today, pricing would have been isolated behind a dedicated versioned pricing module from the beginning.
12 changes: 6 additions & 6 deletions USER_INTERVIEWS.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@

## Interview 1 — Karthik R. (Friend’s Brother)
## Interview 1 — Bholenath R. (Friend’s Brother)
**Role:** Programmer Analyst @ Cognizant
**Team Size:** ~18 developers in internal banking project
**Call Duration:** ~15 mins on Google Meet

Karthik said their team started using multiple AI tools separately without any planning. Some developers preferred GitHub Copilot while others switched to Cursor after seeing YouTube videos and Twitter posts.
Bholenath said their team started using multiple AI tools separately without any planning. Some developers preferred GitHub Copilot while others switched to Cursor after seeing YouTube videos and Twitter posts.

> “Honestly nobody even knows which tools are officially approved anymore.”

Expand All @@ -27,12 +27,12 @@ Instead of saying “cancel immediately,” the report became more cautious and

---

## Interview 2 — Sai Teja (College Senior)
## Interview 2 — Rahul (College Senior)
**Role:** Data Analyst @ TCS
**Team Size:** ~12 people
**Conversation:** Discord call (~10–12 mins)

Sai Teja mainly works with dashboards, Excel automation, SQL, and internal reporting. He said their team recently started experimenting with ChatGPT Team licenses while some employees still used Gemini and Claude separately.
Rahul mainly works with dashboards, Excel automation, SQL, and internal reporting. He said their team recently started experimenting with ChatGPT Team licenses while some employees still used Gemini and Claude separately.

> “Most people don’t use these tools daily. They use them heavily for one week and forget about them.”

Expand All @@ -55,12 +55,12 @@ I simplified several recommendation cards because the earlier wording felt too t

---

## Interview 3 — Akhil S. (Friend Running Small Agency)
## Interview 3 — Sai S. (Friend Running Small Agency)
**Role:** Freelance Designer + Small Agency Owner
**Team Size:** 4 people
**Conversation:** In person

Akhil’s team uses Midjourney, ChatGPT Plus, Canva Pro, and Runway occasionally for client work. Since the team is small, he tracks expenses personally.
Sai’s team uses Midjourney, ChatGPT Plus, Canva Pro, and Runway occasionally for client work. Since the team is small, he tracks expenses personally.

> “Subscriptions are annoying because every tool slowly adds AI features and charges separately.”

Expand Down
5 changes: 5 additions & 0 deletions app/actions/audit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,23 @@ import { persistReport, REPORT_STORAGE_PREFIX } from "@/lib/supabase/db";
import type { AuditInputSchema } from "@/lib/schemas/audit";
import type { FullAuditReport } from "@/lib/audit-engine/types";

import { createPricingSnapshot } from "@/lib/pricing/current-pricing";

export { REPORT_STORAGE_PREFIX };

export async function runAuditClient(
data: AuditInputSchema
): Promise<{ reportId: string }> {
const result = runAuditEngine(data);
const reportId = crypto.randomUUID().replace(/-/g, "").slice(0, 16);
const snapshot = createPricingSnapshot();

const report: FullAuditReport = {
id: reportId,
timestamp: new Date().toISOString(),
input: data,
pricingSnapshot: snapshot,
pricingVersion: snapshot.version,
...result,
};

Expand Down
45 changes: 45 additions & 0 deletions app/api/detect-changes/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import { NextResponse } from "next/server";
import { detectInvalidatedAudits } from "@/lib/re-audit/detect-invalidated-audits";
import { sendReauditEmail } from "@/lib/email/send-reaudit-email";

export const dynamic = "force-dynamic";

export async function POST() {
try {
const groupedNotifications = await detectInvalidatedAudits();

let emailsSent = 0;

// Process sending in parallel
await Promise.all(
groupedNotifications.map(async (notification) => {
const sent = await sendReauditEmail(notification);
if (sent) emailsSent++;
})
);

const affectedAuditsCount = groupedNotifications.reduce(
(acc, group) => acc + group.affectedAudits.length,
0
);

return NextResponse.json(
{
success: true,
affectedUsersCount: groupedNotifications.length,
affectedAuditsCount,
emailsSent,
},
{ status: 200 }
);
} catch (error) {
console.error("[Detect Changes Endpoint] Error:", error);
return NextResponse.json(
{
success: false,
error: "Failed to process detect changes trigger",
},
{ status: 500 }
);
}
}
Loading
Loading