diff --git a/CHANGELOG.md b/CHANGELOG.md index 228cde5..c07c6dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ own line, so there is no single global version number. Each module keeps its own - [`state`](state/CHANGELOG.md) — statechart engine (v1.x, frozen contract) - [`state/expr`](state/expr/CHANGELOG.md) — CEL-backed guards +- [`source`](source/CHANGELOG.md) — ingress consume engine (v1.x, frozen contract). Per-inlet changelogs: + [`kafka`](source/kafka/CHANGELOG.md), [`statemachine`](source/statemachine/CHANGELOG.md). ## Tooling @@ -29,8 +31,6 @@ own line, so there is no single global version number. Each module keeps its own under `sink//CHANGELOG.md` (bridge, cloudwatch, dynamo, eventbridge, file, firehose, gcppubsub, http, kafka, kinesis, nats, otel, prometheus, redis, s3, slog, sns, sql, sqs, statsd, timestream). -- `source` — ingress. Per-inlet changelogs: - [`kafka`](source/kafka/CHANGELOG.md), [`statemachine`](source/statemachine/CHANGELOG.md). ## Examples diff --git a/README.md b/README.md index 3479c36..3819995 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,7 @@ stability label. | `cmd/crucible` | Headless IR CLI: lint, render, diff, validate, eject. | v0.1.0 | | `telemetry` | Vendor-neutral tracing/metrics seam, plus `slog`, `otel`, `datadog` adapters. | experimental | | `sink` | Egress fan-out, fire-and-forget. 20+ destinations: SQL, Dynamo, S3, Kafka, NATS, Redis, StatsD, … | experimental | -| `source` | Ingress: consume streams and drive machines, ack on durable transition. Inlets: Kafka, JetStream, Redis, CloudEvents, CDC; opt-in retry/DLQ/idempotency/schema middleware. | experimental | +| `source` | Ingress: consume streams and drive machines, ack on durable transition. Inlets: Kafka, JetStream, Redis, CloudEvents, CDC; opt-in retry/DLQ/idempotency/schema middleware. | v1.0.0 (stable) | | `durable` | Durable-execution runtime: record and replay to survive a crash. | experimental | | `cluster` | Distribution runtime: remote actors, supervision, and live migration over a pluggable transport (in-memory transport tested; gRPC `transport` module behind the same interface). | experimental | | `wasm` | Run state behaviors as WebAssembly: polyglot guards over a JSON ABI via wazero. | experimental | diff --git a/STABILITY.md b/STABILITY.md index 6b2280a..dc1db00 100644 --- a/STABILITY.md +++ b/STABILITY.md @@ -10,11 +10,11 @@ each module currently carries. | Label | What it promises | | --- | --- | -| **stable (v1.x)** | Frozen public contract. Breaking changes only in a new major version. `state` is here. | +| **stable (v1.x)** | Frozen public contract. Breaking changes only in a new major version. `state` and `source` are here. | | **stable contract (pre-1.0)** | The module version is still `v0.x`, but a named part of its behavior is a committed contract that will not break silently. `state/expr` commits its expression *semantics* this way. | | **released (v0.x)** | Released and usable, versioned independently, free to move at its own pace. Being pre-1.0, a minor release may still break. `gen` and `cmd/crucible` are here. | | **advisory** | Ships inside a stable module but sits *outside* its frozen contract and may change in a minor release. The `state` subpackages (`analysis`, `evolution`, `conformance`, `verify`) are advisory. | -| **experimental** | Usable, tested, and benchmarked, but the API may change before it reaches v1. Pin a version and expect churn. The IO edges (`sink`, `source`) and host-side runtimes (`durable`, `cluster`, `transport`, `wasm`, `telemetry`) are experimental. | +| **experimental** | Usable, tested, and benchmarked, but the API may change before it reaches v1. Pin a version and expect churn. The IO edge (`sink`) and host-side runtimes (`durable`, `cluster`, `transport`, `wasm`, `telemetry`) are experimental. | | **planned** | Not yet implemented. `broker` is planned. | ## Using pre-1.0 modules @@ -30,3 +30,9 @@ Promoting an experimental module to v1 means committing to a frozen public contr under the same terms as `state`. The graduation criteria and a per-module compatibility matrix are tracked in [issue #179](https://github.com/stablekernel/crucible/issues/179). + +`source` graduated at v1.0.0 (with its Kafka inlet released as +`source/kafka` v1.0.0): the neutral surface is frozen, the delivery and drain +contracts are pinned by regression tests, and the frozen-contract terms are +recorded in [`source/CHANGELOG.md`](source/CHANGELOG.md) and +[`source/kafka/CHANGELOG.md`](source/kafka/CHANGELOG.md). diff --git a/source/CHANGELOG.md b/source/CHANGELOG.md new file mode 100644 index 0000000..5adb7ca --- /dev/null +++ b/source/CHANGELOG.md @@ -0,0 +1,50 @@ +# Changelog + +All notable changes to `crucible/source` are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [1.0.0] — 2026-08-23 + +The first stable release. It freezes the neutral ingress surface — `Inlet`, +`Subscription`, `Message`/`Headers`/`Cursor`, `Handler`/`BatchHandler`/`Result` +with the `Ack`/`Nak`/`NakAfter`/`Term`/`Reject`/`Skip`/`InProgress`/`Manual` +vocabulary, `Middleware`/`Chain`, the `Hopper` engine, every `Option`, the +capability interfaces (`Seekable`, `ConsumerGroups`, `PartitionOrdered`, +`LagReporter`, `Transactional`, `Batched`, `Idempotent`), and the sentinel +errors — under the same frozen-contract terms as `state`: from 1.0.0 onward, +new capabilities arrive as additive packages, modules, and options rather than +breaking changes. + +### Delivery contract (frozen) + +- Delivery is at-least-once within a live subscription: a message is acked only + after its handler reports success, and `Nak` redelivers. Across process + restarts and rebalances, backends whose redelivery rides persisted offsets + (Kafka) resume from the last committed position, which a concurrently + committed higher offset can advance past a nacked record; those backends + redeliver exactly within a session and best-effort across restarts. Each + adapter documents its precise semantics (see `source/kafka/README.md`). +- `Subscription.Close` begins a graceful drain: after it returns, `Next`/ + `NextBatch` never yield new messages — only already-buffered ones — and once + in-flight messages settle, `Next` reports `ErrDrained`. +- Duplicate `Settle` calls for one message are safe: they never corrupt + in-flight accounting or double-advance a backend position. + +### Added (this cycle) + +- `ErrDrained` drain semantics pinned by regression tests for both `Next` and + `NextBatch` (close-then-Next, redelivery storm with bounded goroutines, + drain-on-cancel with a mid-flight backend settle, duplicate-settle + idempotence). + +### Changed + +- `ActionNak` documentation now states per-backend redelivery semantics in one + voice with the adapters: JetStream naks natively (with optional delay); + Kafka pauses and re-seeks the partition so the record is fetched again + in-session, honoring `Result.Requeue` as a pause delay. + +[Unreleased]: https://github.com/stablekernel/crucible/tree/main/source +[1.0.0]: https://github.com/stablekernel/crucible/releases/tag/source%2Fv1.0.0 diff --git a/source/kafka/CHANGELOG.md b/source/kafka/CHANGELOG.md index 29b1978..324fa36 100644 --- a/source/kafka/CHANGELOG.md +++ b/source/kafka/CHANGELOG.md @@ -5,33 +5,83 @@ All notable changes to `crucible/source/kafka` are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [1.0.0] — 2026-08-23 + +The first stable release. It freezes the adapter surface — `Inlet` with `New` +and the `With*` options, the `Subscription` and its capability +implementations, the ack model below, and the sentinel errors — with the +neutral `source` seam carrying no franz-go types. The typed power seams +(`WithSASL`, `WithBalancer`, `WithClientOptions`, `WithClient`, `Inlet.As`, +`Message.As`) deliberately expose franz-go and may track franz-go releases; +everything else follows the frozen-contract terms of `source` v1.0.0. ### Added - Kafka source ingress adapter: an `Inlet` (built with `New` and functional options — `WithSeedBrokers`, `WithSASL`, `WithTLS`, `WithBalancer`, `WithClientID`, `WithDLQTopic`, `WithMaxPollRecords`, `WithTransactional`, - `WithClientOptions`, `WithClient`) opening a `source.Subscription` over - franz-go. The consume loop polls with `PollRecords`, hands records to the - engine as a neutral `source.Message`, and settles per the handler `Result`: - Ack marks for commit (`AutoCommitMarks`, commit-after-process), Nak declines - to mark (best-effort delay via pause + re-seek + resume), Term produces to the - configured dead-letter topic then marks, InProgress and Manual are no-ops. -- Capability interfaces, discovered by type assertion with no vendor types in - the exported surface: `Seekable` (live `SetOffsets`, time seeks via - `ListOffsets`), `ConsumerGroups` (assign/revoke/lost hooks with - drain-and-commit on revoke), `PartitionOrdered`, `LagReporter`, and + `WithStartOffset`, `WithClientOptions`, `WithClient`) opening a + `source.Subscription` over franz-go. The consume loop polls with + `PollRecords`, hands records to the engine as a neutral `source.Message`, + and settles per the handler `Result` (ack model below). +- Capability interfaces, discovered by type assertion: `Seekable` (live + `SetOffsets`, time seeks via `ListOffsets`; partitions enumerated from + committed offsets with broker-metadata fallback, so seeking works on a cold + group), `ConsumerGroups` (assign/revoke/lost hooks with drain-and-commit on + revoke), `PartitionOrdered`, `LagReporter` (end-minus-committed across + committed partitions; `ErrNoCommittedOffsets` on a cold group), and `Transactional` (Kafka EOS). `BlockRebalanceOnPoll` provides a safe processing window. The underlying `*kgo.Client` and `*kgo.Record` are reachable only through `As`. - Exactly-once consume-process-produce. `WithTransactional(id)` builds a - `GroupTransactSession` with read-committed fetch isolation and no auto-commit, - and the subscription's `source.Transactional.Begin(ctx, m, fn)` runs `fn` - inside a producer transaction: records produced through the handed `source.Tx` - are flushed and `m`'s consumed offset is committed in one atomic unit, or the - transaction aborts (on a work error or a rebalance fence) and the input is - redelivered. Produced records are neutral `source.ProducedRecord` values; no - franz-go type crosses the seam. + `GroupTransactSession` with read-committed fetch isolation and no + auto-commit, and `Begin` runs fn inside a producer transaction: records + produced through the handed `source.Tx` are flushed and `m`'s consumed + offset is committed in one atomic unit, or the transaction aborts (on a + work error or a rebalance fence) and the input is redelivered. Poison flows + through Begin too (see Term below). +- `WithStartOffset(StartEarliest | StartLatest)`: typed cold-start policy for + a brand-new consumer group (default earliest, franz-go's default); + partitions with commits always resume from the commit. +- Sentinel errors `ErrNoCommittedOffsets` and `ErrTermInsideTransaction`. + +### Ack model (frozen) + +- `Ack` marks the record for commit (commit-after-process); marks commit on + graceful drain and on revoke. +- `Nak` (plain or `NakAfter(d)`) never marks: the partition is paused, + re-seeked to the record's offset, resumed — deterministic redelivery within + the live subscription, with `d` as the pause delay (best-effort; + head-of-line-blocks the partition; concurrent commits can pass the nacked + offset, so cross-restart redelivery is best-effort). +- `Term` produces to the dead-letter topic, then marks. On a transactional + subscription, direct `Term` reports `ErrTermInsideTransaction`; route + poison through `Begin` (produce the DLQ record via `Tx`, return success) so + the DLQ write and the offset commit are atomic. +- `InProgress`/`Manual` are no-ops. Duplicate settles are safe; duplicate + marks are broker-idempotent. + +### Changed + +- `Nak` redelivery: plain `Nak` now re-seeks immediately (previously it only + declined to mark, deferring redelivery to the next restart/rebalance). This + makes the core at-least-once claim true within a live subscription. +- `SeekToStart/SeekToEnd/SeekToTime` no longer silently no-op before the first + offset commit (metadata fallback); `Lag` errors with + `ErrNoCommittedOffsets` instead of returning 0 for a cold group. +- After `Subscription.Close`, `Next`/`NextBatch` never poll again: they yield + only already-buffered records, then block until in-flight settles land and + return `source.ErrDrained` (previously a closed subscription with in-flight + records could yield new records). +- Package documentation corrects the vendor-boundary claim: no franz-go type + crosses the neutral seam; typed option seams deliberately expose franz-go. + +### Removed + +- The integration test's `testcontainers-go`, `redpanda`, and `kadm` + requirements left this module's `go.mod`: the RedPanda end-to-end leg now + lives in the test-only `source/kafka/integration` module, so downstream + module graphs inherit only franz-go. [Unreleased]: https://github.com/stablekernel/crucible/tree/main/source/kafka +[1.0.0]: https://github.com/stablekernel/crucible/releases/tag/source%2Fkafka%2Fv1.0.0 diff --git a/source/kafka/README.md b/source/kafka/README.md index ce5d3f5..9d6ed67 100644 --- a/source/kafka/README.md +++ b/source/kafka/README.md @@ -113,7 +113,21 @@ one is injected with `WithClient`, in which case it is the caller's. ## Stability -Experimental (pre-v1). The API may change until the suite locks v1.0.0. +Stable at v1.0.0: the adapter surface is frozen (see the +[CHANGELOG](CHANGELOG.md) for the frozen-contract terms). The typed power +seams (`WithSASL`, `WithBalancer`, `WithClientOptions`, `WithClient`) may +track franz-go releases. + +## Setup notes + +- The dead-letter topic must exist before the first `Term` settles: the + adapter's DLQ producer does not enable topic auto-creation. Create it up + front (as the integration suite does with `CreateTopics`). +- Managed Kafka (e.g. MSK): authenticate by constructing franz-go + `sasl.Mechanism` values (`pkg/sasl/plain`, `.../scram`, `.../oauthbearer`, + or the AWS MSK IAM token provider) and passing them via `WithSASL` with + `WithTLS`; this means importing franz-go directly for the mechanism + constructors, which is expected. ## License diff --git a/source/kafka/kafka.go b/source/kafka/kafka.go index e6344c2..460d9fa 100644 --- a/source/kafka/kafka.go +++ b/source/kafka/kafka.go @@ -63,7 +63,9 @@ // // # Stability // -// Experimental (pre-v1); the API may change until the suite locks v1.0.0. +// Stable at v1.0.0: the adapter surface is frozen (see CHANGELOG.md for the +// frozen-contract terms). The typed power seams ([WithSASL], [WithBalancer], +// [WithClientOptions], [WithClient]) may track franz-go releases. package kafka import (