Event Processing Lab

A hands-on learning project comparing Kafka, Esper, Apache Flink, and Flink CEP for event processing.

Overview

The Event Processing Lab lets integration architects gain practical experience with different event processing technologies through one common use case: the order lifecycle (OrderCreated → PaymentReceived / PaymentFailed → ShipmentStarted / OrderCancelled). By implementing the same scenarios across several technologies, it enables direct comparison of scalability, operational complexity, state handling, and rule complexity.

The architecture, goals, decisions, quality scenarios, risks, and the Comparison Matrix live in the arc42 documentation under src/docs/ — see Architecture documentation. This README covers only how to set up, build, and run the lab.

Prerequisites

• Java 21 or later
• Gradle 8.5 or later (or use a locally installed Gradle; the wrapper jar is not committed)
• Apache Kafka (a local broker via the provided Docker Compose file is easiest)
• A container engine (Docker or Podman) if you want to build the docs with ./build.sh

Setup

1. Start Kafka

docker-compose -f docker-compose.kafka.yml up -d

Kafka defaults to localhost:9092; override via gradle.properties (kafka.bootstrap.servers).

2. Build

gradle build            # build and test all modules
gradle :event-model:build

Running the components

Each processing technology is a runnable module:

gradle :event-producer:run     # produce order-lifecycle events
gradle :kafka-consumer:run     # plain Kafka consumer baseline
gradle :esper-cep:run          # Esper CEP engine
gradle :flink-basic:run        # Flink DataStream processing
gradle :flink-cep:run          # Flink CEP pattern detection

Running a use case to evaluate the Comparison Matrix

The Comparison Matrix rates each technology along the comparison dimensions. Its values are a reviewable assessment — reproduce and evaluate them by running the shared use case against each processor:

1. Start Kafka (see Setup) so the order-events topic is available.
2. Start one processor to evaluate, e.g. the Esper CEP engine:

   gradle :esper-cep:run

3. Emit an order-lifecycle scenario from a second terminal:

   gradle :event-producer:run

   The producer publishes the order lifecycle events (OrderCreated, PaymentReceived/PaymentFailed, ShipmentStarted, OrderCancelled) that drive the runtime scenarios in Chapter 6 — Runtime View.
4. Observe the processor output (console/logs): which patterns it detects, how it keeps state, and the latency/throughput it reports. This is the evidence for the comparison dimensions.
5. Repeat for each processor (kafka-consumer, esper-cep, flink-basic, flink-cep) and compare the observations against the Comparison Matrix and the measurable quality scenarios in Chapter 10 — Quality Requirements.

Metrics are currently read from logs. An automated dashboard that surfaces the comparison metrics is on the roadmap (Phase 5), and unit/behaviour tests for the implementations are planned in Phase 4.

Project structure

event-processing-lab/
├── AGENTS.md                      # Thin agent contract routing to the toolkit
├── README.md                      # This file
├── build.sh                       # docs-toolbox task runner (validate/generate/build)
├── build.gradle / settings.gradle # Gradle multi-module build
├── metamodel/                     # Vendored artifact/relation schemas (contracts)
├── templates/                     # Vendored ADR/quality-scenario/risk templates
├── scripts/                       # Vendored validator + agent-adapter generators
├── adapters/                      # Generated agent adapters (codex, vibe, copilot, cursor)
├── features/                      # Gherkin behaviour specs (living documentation)
├── .github/workflows/             # CI (validate + tests) and GitHub Pages publish
├── src/docs/                      # arc42 architecture documentation (see below)
├── event-model/                   # Shared event model and serialization
├── event-producer/                # Kafka producer of order-lifecycle events
├── kafka-consumer/                # Plain Kafka consumer baseline
├── esper-cep/                     # Esper CEP engine
├── flink-basic/                   # Flink DataStream processing
└── flink-cep/                     # Flink CEP pattern detection

Each module holds its Java sources under src/main/java/com/example/<module>/. Derived documentation output under src/docs/**/generated/ and build/ is not committed; regenerate it with ./build.sh.

Technologies

Technology	Purpose	Module
Kafka	Event backbone & transport	event-producer, kafka-consumer
Esper	Lightweight CEP engine	esper-cep
Apache Flink	Distributed stream processing	flink-basic, flink-cep
Flink CEP	Pattern detection library	flink-cep

Architecture documentation

The complete architecture documentation follows the arc42 template and the docs-as-code-toolkit/architecture-knowledge-toolkit conventions:

• arc42 Architecture Documentation — assembled entry point
• Solution Strategy & Comparison Matrix
• Runtime View — the runtime scenarios
• Quality Requirements — quality goals and 15 measurable scenarios
• Architecture Decisions — ADRs
• Risks and Technical Debt
• Vision & Mission, Roadmap, Q&A

Validating and building the documentation

The docs are validated, generated, and rendered through the pinned docs-toolbox container image via ./build.sh:

./build.sh validate    # validate artifact metadata and relations
./build.sh generate    # regenerate derived fragments and indexes
./build.sh build       # render build/architecture/index.html

Set DOCS_TOOLBOX_LOCAL=1 to run against the host toolchain instead. CI (.github/workflows/ci.yml) validates every pull request and builds the Java modules; on main, the documentation is published to GitHub Pages (.github/workflows/pages.yml).

Related resources

• Apache Kafka Documentation
• Esper Documentation
• Apache Flink Documentation
• Flink CEP Documentation
• arc42 Architecture Template
• docs-as-code-toolkit / architecture-knowledge-toolkit