refactor: reorganize the repo into packages/ (core + server)

[oc-engteam]

What & why

Reorganize the repository into a top-level packages/ layout so future non-Go clients (e.g. a VSCode
extension) have a natural home alongside the Go code, without disturbing the Go build.

New layout:

• packages/core/ — Go module github.com/codetrek/haystack/core (the search/index library).
• packages/server/ — Go module github.com/codetrek/haystack/server (the server + CLI app; holds
  internal/, cmd/, go.mod, go.sum, .go-cov.toml, config.example.yaml, Dockerfile.test).
• Root keeps the workspace glue: go.work (uses ./packages/server + ./packages/core), regenerated
  vendor/, build.go, Makefile, deps/, docs/, git-hooks/, scripts/, .github/.

Module paths intentionally do not carry a packages/ segment (…/core, …/server) — the repo is
built as a Go workspace and released as a binary, so this is consistent and go build/go test/
go run build.go all resolve via go.work.

Plumbing repaired for the new layout

The bulk of the churn is repointing everything that referenced the old flat layout. Fixed and each
verified by running the real command:

• CI (ci.yml) — Core job working-directory: core → packages/core (all three steps).
• Coverage gates — the App-job script (.github/workflows/scripts/test_and_coverage.sh) now cds
  into packages/server (it previously targeted the repo root, then briefly packages/core), so the
  server module is tested + coverage-gated again; packages/server/.go-cov.toml project/excludes
  fixed to the server module. Verified: server go-cov exits 0 over server/… packages.
• Docker isolation test — packages/server/Dockerfile.test reworked for the workspace (base image
  golang:1.24 to satisfy the go1.24.2 toolchain; drops the now-broken root COPY go.mod go.sum and
  relies on the mounted vendor/); module-targeted go test in the Dockerfile CMD and the Makefile
  test-safe-race override. Verified: docker build + offline (--network=none) go test pass.
• Git-LFS — .gitattributes retargeted to packages/core/tokenizer/fstcjk/dict.fst (so the
  go:embed'd FST is not shipped as a 132-byte pointer on fresh clones). Verified via git check-attr.
• Makefile / README — go test ./... / go run ./ / go build ./ (root is no longer a module)
  → module-targeted forms; README quickstart updated (cd haystack, cp packages/server/config.example.yaml …,
  go run ./packages/server/cmd/haystack …, make build/make test).
• Misc — deleted the orphaned duplicate packages/core/Dockerfile.test; packages/core/.gitignore
.core/ → .coverage/; release.yml Go 1.23 → 1.24.2 (+ setup-go@v5); .vscode/launch.json
  and root .gitignore repointed; stale doc comments refreshed.

Verification

• Both modules go build ./... / go vet / go test -short / gofmt -l green under the workspace.
• Core and server go-cov gates pass (exit 0).
• packages/server/Dockerfile.test builds and runs its test offline against the vendored deps.
• go run build.go produces the release binaries; make build works.

🤖 Generated with Claude Code
via Happy