Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ own line, so there is no single global version number. Each module keeps its own

- [`state`](state/CHANGELOG.md) — statechart engine (v1.x, frozen contract)
- [`state/expr`](state/expr/CHANGELOG.md) — CEL-backed guards
- [`source`](source/CHANGELOG.md) — ingress consume engine (v1.x, frozen contract). Per-inlet changelogs:
[`kafka`](source/kafka/CHANGELOG.md), [`statemachine`](source/statemachine/CHANGELOG.md).

## Tooling

Expand All @@ -29,8 +31,6 @@ own line, so there is no single global version number. Each module keeps its own
under `sink/<name>/CHANGELOG.md` (bridge, cloudwatch, dynamo, eventbridge, file,
firehose, gcppubsub, http, kafka, kinesis, nats, otel, prometheus, redis, s3, slog,
sns, sql, sqs, statsd, timestream).
- `source` — ingress. Per-inlet changelogs:
[`kafka`](source/kafka/CHANGELOG.md), [`statemachine`](source/statemachine/CHANGELOG.md).

## Examples

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ stability label.
| `cmd/crucible` | Headless IR CLI: lint, render, diff, validate, eject. | v0.1.0 |
| `telemetry` | Vendor-neutral tracing/metrics seam, plus `slog`, `otel`, `datadog` adapters. | experimental |
| `sink` | Egress fan-out, fire-and-forget. 20+ destinations: SQL, Dynamo, S3, Kafka, NATS, Redis, StatsD, … | experimental |
| `source` | Ingress: consume streams and drive machines, ack on durable transition. Inlets: Kafka, JetStream, Redis, CloudEvents, CDC; opt-in retry/DLQ/idempotency/schema middleware. | experimental |
| `source` | Ingress: consume streams and drive machines, ack on durable transition. Inlets: Kafka, JetStream, Redis, CloudEvents, CDC; opt-in retry/DLQ/idempotency/schema middleware. | v1.0.0 (stable) |
| `durable` | Durable-execution runtime: record and replay to survive a crash. | experimental |
| `cluster` | Distribution runtime: remote actors, supervision, and live migration over a pluggable transport (in-memory transport tested; gRPC `transport` module behind the same interface). | experimental |
| `wasm` | Run state behaviors as WebAssembly: polyglot guards over a JSON ABI via wazero. | experimental |
Expand Down
10 changes: 8 additions & 2 deletions STABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ each module currently carries.

| Label | What it promises |
| --- | --- |
| **stable (v1.x)** | Frozen public contract. Breaking changes only in a new major version. `state` is here. |
| **stable (v1.x)** | Frozen public contract. Breaking changes only in a new major version. `state` and `source` are here. |
| **stable contract (pre-1.0)** | The module version is still `v0.x`, but a named part of its behavior is a committed contract that will not break silently. `state/expr` commits its expression *semantics* this way. |
| **released (v0.x)** | Released and usable, versioned independently, free to move at its own pace. Being pre-1.0, a minor release may still break. `gen` and `cmd/crucible` are here. |
| **advisory** | Ships inside a stable module but sits *outside* its frozen contract and may change in a minor release. The `state` subpackages (`analysis`, `evolution`, `conformance`, `verify`) are advisory. |
| **experimental** | Usable, tested, and benchmarked, but the API may change before it reaches v1. Pin a version and expect churn. The IO edges (`sink`, `source`) and host-side runtimes (`durable`, `cluster`, `transport`, `wasm`, `telemetry`) are experimental. |
| **experimental** | Usable, tested, and benchmarked, but the API may change before it reaches v1. Pin a version and expect churn. The IO edge (`sink`) and host-side runtimes (`durable`, `cluster`, `transport`, `wasm`, `telemetry`) are experimental. |
| **planned** | Not yet implemented. `broker` is planned. |

## Using pre-1.0 modules
Expand All @@ -30,3 +30,9 @@ Promoting an experimental module to v1 means committing to a frozen public contr
under the same terms as `state`. The graduation criteria and a per-module
compatibility matrix are tracked in
[issue #179](https://github.com/stablekernel/crucible/issues/179).

`source` graduated at v1.0.0 (with its Kafka inlet released as
`source/kafka` v1.0.0): the neutral surface is frozen, the delivery and drain
contracts are pinned by regression tests, and the frozen-contract terms are
recorded in [`source/CHANGELOG.md`](source/CHANGELOG.md) and
[`source/kafka/CHANGELOG.md`](source/kafka/CHANGELOG.md).
50 changes: 50 additions & 0 deletions source/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Changelog

All notable changes to `crucible/source` are documented here.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.0.0] — 2026-08-23

The first stable release. It freezes the neutral ingress surface — `Inlet`,
`Subscription`, `Message`/`Headers`/`Cursor`, `Handler`/`BatchHandler`/`Result`
with the `Ack`/`Nak`/`NakAfter`/`Term`/`Reject`/`Skip`/`InProgress`/`Manual`
vocabulary, `Middleware`/`Chain`, the `Hopper` engine, every `Option`, the
capability interfaces (`Seekable`, `ConsumerGroups`, `PartitionOrdered`,
`LagReporter`, `Transactional`, `Batched`, `Idempotent`), and the sentinel
errors — under the same frozen-contract terms as `state`: from 1.0.0 onward,
new capabilities arrive as additive packages, modules, and options rather than
breaking changes.

### Delivery contract (frozen)

- Delivery is at-least-once within a live subscription: a message is acked only
after its handler reports success, and `Nak` redelivers. Across process
restarts and rebalances, backends whose redelivery rides persisted offsets
(Kafka) resume from the last committed position, which a concurrently
committed higher offset can advance past a nacked record; those backends
redeliver exactly within a session and best-effort across restarts. Each
adapter documents its precise semantics (see `source/kafka/README.md`).
- `Subscription.Close` begins a graceful drain: after it returns, `Next`/
`NextBatch` never yield new messages — only already-buffered ones — and once
in-flight messages settle, `Next` reports `ErrDrained`.
- Duplicate `Settle` calls for one message are safe: they never corrupt
in-flight accounting or double-advance a backend position.

### Added (this cycle)

- `ErrDrained` drain semantics pinned by regression tests for both `Next` and
`NextBatch` (close-then-Next, redelivery storm with bounded goroutines,
drain-on-cancel with a mid-flight backend settle, duplicate-settle
idempotence).

### Changed

- `ActionNak` documentation now states per-backend redelivery semantics in one
voice with the adapters: JetStream naks natively (with optional delay);
Kafka pauses and re-seeks the partition so the record is fetched again
in-session, honoring `Result.Requeue` as a pause delay.

[Unreleased]: https://github.com/stablekernel/crucible/tree/main/source
[1.0.0]: https://github.com/stablekernel/crucible/releases/tag/source%2Fv1.0.0
86 changes: 68 additions & 18 deletions source/kafka/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,83 @@ All notable changes to `crucible/source/kafka` are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this module adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [1.0.0] — 2026-08-23

The first stable release. It freezes the adapter surface — `Inlet` with `New`
and the `With*` options, the `Subscription` and its capability
implementations, the ack model below, and the sentinel errors — with the
neutral `source` seam carrying no franz-go types. The typed power seams
(`WithSASL`, `WithBalancer`, `WithClientOptions`, `WithClient`, `Inlet.As`,
`Message.As`) deliberately expose franz-go and may track franz-go releases;
everything else follows the frozen-contract terms of `source` v1.0.0.

### Added

- Kafka source ingress adapter: an `Inlet` (built with `New` and functional
options — `WithSeedBrokers`, `WithSASL`, `WithTLS`, `WithBalancer`,
`WithClientID`, `WithDLQTopic`, `WithMaxPollRecords`, `WithTransactional`,
`WithClientOptions`, `WithClient`) opening a `source.Subscription` over
franz-go. The consume loop polls with `PollRecords`, hands records to the
engine as a neutral `source.Message`, and settles per the handler `Result`:
Ack marks for commit (`AutoCommitMarks`, commit-after-process), Nak declines
to mark (best-effort delay via pause + re-seek + resume), Term produces to the
configured dead-letter topic then marks, InProgress and Manual are no-ops.
- Capability interfaces, discovered by type assertion with no vendor types in
the exported surface: `Seekable` (live `SetOffsets`, time seeks via
`ListOffsets`), `ConsumerGroups` (assign/revoke/lost hooks with
drain-and-commit on revoke), `PartitionOrdered`, `LagReporter`, and
`WithStartOffset`, `WithClientOptions`, `WithClient`) opening a
`source.Subscription` over franz-go. The consume loop polls with
`PollRecords`, hands records to the engine as a neutral `source.Message`,
and settles per the handler `Result` (ack model below).
- Capability interfaces, discovered by type assertion: `Seekable` (live
`SetOffsets`, time seeks via `ListOffsets`; partitions enumerated from
committed offsets with broker-metadata fallback, so seeking works on a cold
group), `ConsumerGroups` (assign/revoke/lost hooks with drain-and-commit on
revoke), `PartitionOrdered`, `LagReporter` (end-minus-committed across
committed partitions; `ErrNoCommittedOffsets` on a cold group), and
`Transactional` (Kafka EOS). `BlockRebalanceOnPoll` provides a safe
processing window. The underlying `*kgo.Client` and `*kgo.Record` are
reachable only through `As`.
- Exactly-once consume-process-produce. `WithTransactional(id)` builds a
`GroupTransactSession` with read-committed fetch isolation and no auto-commit,
and the subscription's `source.Transactional.Begin(ctx, m, fn)` runs `fn`
inside a producer transaction: records produced through the handed `source.Tx`
are flushed and `m`'s consumed offset is committed in one atomic unit, or the
transaction aborts (on a work error or a rebalance fence) and the input is
redelivered. Produced records are neutral `source.ProducedRecord` values; no
franz-go type crosses the seam.
`GroupTransactSession` with read-committed fetch isolation and no
auto-commit, and `Begin` runs fn inside a producer transaction: records
produced through the handed `source.Tx` are flushed and `m`'s consumed
offset is committed in one atomic unit, or the transaction aborts (on a
work error or a rebalance fence) and the input is redelivered. Poison flows
through Begin too (see Term below).
- `WithStartOffset(StartEarliest | StartLatest)`: typed cold-start policy for
a brand-new consumer group (default earliest, franz-go's default);
partitions with commits always resume from the commit.
- Sentinel errors `ErrNoCommittedOffsets` and `ErrTermInsideTransaction`.

### Ack model (frozen)

- `Ack` marks the record for commit (commit-after-process); marks commit on
graceful drain and on revoke.
- `Nak` (plain or `NakAfter(d)`) never marks: the partition is paused,
re-seeked to the record's offset, resumed — deterministic redelivery within
the live subscription, with `d` as the pause delay (best-effort;
head-of-line-blocks the partition; concurrent commits can pass the nacked
offset, so cross-restart redelivery is best-effort).
- `Term` produces to the dead-letter topic, then marks. On a transactional
subscription, direct `Term` reports `ErrTermInsideTransaction`; route
poison through `Begin` (produce the DLQ record via `Tx`, return success) so
the DLQ write and the offset commit are atomic.
- `InProgress`/`Manual` are no-ops. Duplicate settles are safe; duplicate
marks are broker-idempotent.

### Changed

- `Nak` redelivery: plain `Nak` now re-seeks immediately (previously it only
declined to mark, deferring redelivery to the next restart/rebalance). This
makes the core at-least-once claim true within a live subscription.
- `SeekToStart/SeekToEnd/SeekToTime` no longer silently no-op before the first
offset commit (metadata fallback); `Lag` errors with
`ErrNoCommittedOffsets` instead of returning 0 for a cold group.
- After `Subscription.Close`, `Next`/`NextBatch` never poll again: they yield
only already-buffered records, then block until in-flight settles land and
return `source.ErrDrained` (previously a closed subscription with in-flight
records could yield new records).
- Package documentation corrects the vendor-boundary claim: no franz-go type
crosses the neutral seam; typed option seams deliberately expose franz-go.

### Removed

- The integration test's `testcontainers-go`, `redpanda`, and `kadm`
requirements left this module's `go.mod`: the RedPanda end-to-end leg now
lives in the test-only `source/kafka/integration` module, so downstream
module graphs inherit only franz-go.

[Unreleased]: https://github.com/stablekernel/crucible/tree/main/source/kafka
[1.0.0]: https://github.com/stablekernel/crucible/releases/tag/source%2Fkafka%2Fv1.0.0
16 changes: 15 additions & 1 deletion source/kafka/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,21 @@ one is injected with `WithClient`, in which case it is the caller's.

## Stability

Experimental (pre-v1). The API may change until the suite locks v1.0.0.
Stable at v1.0.0: the adapter surface is frozen (see the
[CHANGELOG](CHANGELOG.md) for the frozen-contract terms). The typed power
seams (`WithSASL`, `WithBalancer`, `WithClientOptions`, `WithClient`) may
track franz-go releases.

## Setup notes

- The dead-letter topic must exist before the first `Term` settles: the
adapter's DLQ producer does not enable topic auto-creation. Create it up
front (as the integration suite does with `CreateTopics`).
- Managed Kafka (e.g. MSK): authenticate by constructing franz-go
`sasl.Mechanism` values (`pkg/sasl/plain`, `.../scram`, `.../oauthbearer`,
or the AWS MSK IAM token provider) and passing them via `WithSASL` with
`WithTLS`; this means importing franz-go directly for the mechanism
constructors, which is expected.

## License

Expand Down
4 changes: 3 additions & 1 deletion source/kafka/kafka.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,9 @@
//
// # Stability
//
// Experimental (pre-v1); the API may change until the suite locks v1.0.0.
// Stable at v1.0.0: the adapter surface is frozen (see CHANGELOG.md for the
// frozen-contract terms). The typed power seams ([WithSASL], [WithBalancer],
// [WithClientOptions], [WithClient]) may track franz-go releases.
package kafka

import (
Expand Down