Add docs/why-newweaver.md — rationale for the dev/data team

[jirhiker]

What

Adds docs/why-newweaver.md, a technical explainer for the dev/data team on why NewWeaver exists and how it fits the DataServices stack. Audience was surprised + skeptical about the rewrite; this is the shared picture going forward — persuade by substance, not spin.

Contents

• Naming — NewWeaver is the internal codebase name; public = Weaver. Currently at newweaver.newmexicowaterdata.org, moving to weaver.newmexicowaterdata.org.
• What it is — client-only display surface, no backend, four standards-based adapters, config-driven catalog (new dataset = one entry).
• Old Weaver → NewWeaver — comparison table: PrimeReact fight → shadcn/DSDS; duplicated multi-agency integration → DIE owns it; auth/private-section surface → none needed (Ocotillo serves collections); stale exports → live reads.
• DSDS angle — the contribute/consume boundary with DataServicesDesignSystem; generic primitives up, 16 app-coupled components stay.
• Ecosystem — Mermaid diagram + prose: Aqueduct → DIE (Dagster sources → combine → geoserver) → Ocotillo/GeoServer (Terraform-managed) → NewWeaver reads live over OGC.
• Planning dashboard — /planning region decision-support, powered by DIE die: products, client-only.
• Get involved + Glossary.

Notes for reviewers

• Every non-obvious claim links to the repo file that backs it (README, SPEC, config.ts, codebase-map, planning.ts).
• DIE's Dagster publish pipeline is grounded in the DataIntegrationEngine repo orchestration/.
• Team-knowledge framing is flagged as such (legacy pain points, Aqueduct ingest, Terraform-managed GeoServer — not verifiable in a checked repo). Please correct anything off, especially the comparison table and glossary.

🤖 Generated with Claude Code

[nick-tenorio-nmt]

On the hosted DIE section: while you didn't explicitly state it, are you hinting the that long term the CLI tool is deprecated in favor of the hosted model? Or will a local DIE too always be available for download?

[jirhiker]

On the hosted DIE section: while you didn't explicitly state it, are you hinting the that long term the CLI tool is deprecated in favor of the hosted model? Or will a local DIE too always be available for download?

thats an excellent question. Im leaning towards deprecating the CLI but I could also see it continuing to live if its found to still be useful. Maybe it requires its own fork where DataServices maintain the Dagster version and the CLI is an open source project that the community maintains

[nick-tenorio-nmt]

I would say if the amount of overhead is minimal in maintaining the human CLI, then maintain for a period time. If the CLI is "expensive" to maintain, or if it is holding DIE back on its potential as a hosted machine-to-machine service, then maybe deprecate.

[nick-tenorio-nmt]

on the "what new weaver is" section, you mention four thin clients. however in the "ecosystem fit" digram, there are 5 inputs shown. Maybe the extra one is the two OGC API features (one for USGS one for ocotillo), so perhaps you were referring to type rather than source. perhaps some minor clarification is all that is needed.

[jirhiker]

Addressed two review points in the latest push:

• CLI deprecation (re: comments from @nick-tenorio-nmt & @jirhiker): reframed the hosted-DIE note from an overstated "not being deprecated" into the actual open trade-off — keep the CLI while it's cheap to maintain and useful, deprecate if it's expensive or if a human CLI holds DIE back as a machine-to-machine service, or split it into a community-maintained fork. States plainly that the long-term call is undecided and not something the doc settles. (commit ac5f88c)
• "Four clients" vs 5 diagram inputs (re: @nick-tenorio-nmt): clarified the four adapters are per protocol, not per source — the single OGC API Features client backs USGS, Ocotillo, and GeoServer — so 5 source inputs over 4 adapters is expected. (commit 3ee8550)

🤖 Addressed by Claude Code

[chasetmartin]

Notes from a sync with @jirhiker that I think belong near this documentation or at least can be discussed here: mostly upstream questions this doc surfaces but doesn't settle, plus two concrete asks.

On the ecosystem-fit section: the OGC Features layers from Ocotillo don't carry any time-series data, so the path for Ocotillo observations reaching DIE isn't actually pinned down - presumably via Aqueduct or a similar STA server + DAG that Ocotillo owns, but that should be an explicit decision rather than something that emerges.

Related scaling question for the integrated data products from DIE in this proposal: attaching every measurement to a feature would explode the GeoJSON. For manual measurements, observations as a JSON array on the feature is probably fine; for continuous data, requests get very large, and the likely answer is pre-computed files for bulk pulls with the API covering single-well requests. We want one service to do everything, but manual and continuous may genuinely need different serving strategies — worth acknowledging that split here, since it shapes any export UX.

There's also the one-FROST-vs-partition question that came out of the Aqueduct discovery: single shared instance or an Ocotillo vs Aqueduct split (if STA is chosen to serve Ocotillo continuous data)? More context for that decision - WellPy could feed the bureau side with QC control, so it belongs on the ecosystem-governance track and should be flagged for work to be done.

Two asks. First, legacy Weaver links like location/MG-030 are in the wild — bookmarks, reports, emails, and people reach well details through them. At cutover those shouldn't 404; even a message page or an alternate route to that location's info (needs user research). This feels like it belongs in the "what must be kept" section alongside the AMP requirements, and on a cutover checklist.

Second, I think we need a single diagram of ALL sources and downstream apps in the Data Services ecosystem that gets as granular as distinguishing manual water levels from continuous data, and clarifies known Ocotillo vs Aqueduct boundaries if they are defined. It would be nice to get into other details sources such as water chemistry data. An encompassing diagram would carry a lot of weight for the internal team and external communication. I think it would also help us identify plenty of pain points or TODO integrations/maintenance for all products in the ecosystem. This is probably worth its own ticket in a sprint.

None of this blocks the doc, just wanted to write it out here.

[chasetmartin]

Also I should have noted in the above comment: I am working on using the content Jake has created here to help in drafting basic roadmaps for the entire Aqueduct - DIE - Weaver ecosystem: the idea being that we can all look them over, iterate, and use them as a kickoff point for user research and formally planning next steps for this product area.

[jirhiker]

Addressed @chasetmartin's notes in commit 1494ac5 (nothing here blocked the doc, but the actionable pieces are now captured in it):

• New "Open upstream questions (not settled here)" section — records the four items as explicitly-unsettled ecosystem/governance calls: (1) how Ocotillo observations reach DIE given Ocotillo Features carry no time-series (Aqueduct vs an STA server + DAG Ocotillo owns — flagged as needing an explicit decision); (2) manual-vs-continuous serving split (observations-array-on-feature for manual, pre-computed bulk files + API-single-well for continuous) and that it shapes export UX; (3) one-shared-FROST vs Ocotillo/Aqueduct partition + WellPy QC, flagged for the governance track; (4) the ask for a single granular all-sources/all-apps diagram (manual vs continuous, chemistry, Ocotillo/Aqueduct boundaries) called out as its own sprint ticket, distinct from the read-scoped diagram in the doc.
• Legacy links — added location/{id} link preservation to the "what must be kept" section (must not 404 at cutover; redirect or message page; exact behavior needs user research) and seeded a cutover checklist. Noted it's net-new: no location/{id} route exists in src/router.tsx today.
• Roadmap note — framed the open-questions section as feedstock for the Aqueduct – DIE – Weaver roadmap drafting / user-research kickoff, per your follow-up, rather than as settled conclusions.

🤖 Addressed by Claude Code