Durga

BPMN → Kafka code generation and process monitoring. Two tools:

• Scaffolder — reads a BPMN model and generates Kafka-oriented worker, gateway, orchestration, and topic setup skeletons.
• Monitoring app — a Quarkus SPA that tracks all running Durga processes via Kafka Streams, showing state, latency, stuck instances, and BPMN diagrams with live overlays.

Code-generation targets

The scaffolder generates the same process for either of two runtime targets, selected with --target (default java):

• Java (--target java) — a Quarkus/SmallRye Reactive Messaging project. Each task, gateway, and orchestrator becomes a CDI bean wired to Kafka channels; the shared plugin runtime is durga-runtime.
• Rust (--target rust) — a Cargo crate where each task and gateway becomes a binary under src/bin/, sharing a runtime in src/lib.rs and depending on the durga-rust plugin crate.

Both targets emit the same ProcessEvent wire format and the same topic layout, so the Java monitor observes a Rust-generated process unchanged and a mixed Java/Rust fleet is fully interoperable.

System manual | BPMN coverage matrix | Beta support boundary | Maturity plan | Release checklist | Operations hardening | Deployment guide | Plugin architecture | Testcontainers setup

Quick start

mvn -q clean package
java -jar durga-tools/target/durga-tools-0.1.0-beta.1.jar path/to/process.bpmn

Local Kafka setup

cd setup && docker compose up

Kafka UI will log transient connection errors until the broker is ready — this is normal. The broker listens on localhost:9094.

Running tests

# Unit tests and generated-project checks (no Docker):
mvn test -Dtest='!*IntegrationTest'

# Monitoring UI tests and build:
cd monitoring-ui && npm ci && npm test && npm run build

# Integration tests (require Docker):
mvn test -Dtest='*IntegrationTest'

# If Docker detection fails, use the Linux container fallback:
./setup/run-integration-tests.sh

See Testcontainers setup for details.

BPMN scaffolding

Output lands in generated/ by default:

• Java sources in generated/src/main/java/<package>/ (default package org.example.generated; override with --package), plus runtime contract classes under generated/src/main/java/org/gautelis/durga/
• topics.sh and summary.json
• task-payloads.json with sample input payloads
• pom.xml and README.md for the generated project
• runtime contracts for DataHandle, handle-aware plugin execution, and Vannak-compatible DataIndividualMetadataEvent records on vannak-metadata-events-{processId}
• Helper scripts: demo-scenario.sh, send-task-input.sh, complete-task.sh, fail-task.sh, escalate-task.sh, complete-call-activity.sh, send-message-event.sh, send-signal-event.sh, watch-process-events.sh, watch-task-output.sh

Flags:

• --dry-run — print summary.json, topics.sh, connect configs, and application YAML without writing files
• --out <dir> — custom output directory
• --event-topic <topic> — override the canonical lifecycle event topic (default: process-events-{processId}). Each pipeline gets an isolated topic by default; use this flag to share a topic across pipelines or use a custom name.
• --transactions — generate transactional workers using Kafka producer/consumer APIs
• --target java|rust — code-generation target (default java; see Code-generation targets)
• --validation — generate the process in validation mode: a complete shadow of an existing process that reads the same live input but writes to validation topics instead of the real ones (see Validation mode)

The generator skips existing files in src/main/java/, merges new channels into application.yml, and evaluates gateway conditions from BPMN conditionExpression at runtime.

BPMN sample catalog

All sample models live under durga-tools/src/test/resources/bpmn/. Run any with:

mvn -q clean package
java -jar durga-tools/target/durga-tools-0.1.0-beta.1.jar durga-tools/src/test/resources/bpmn/<model>.bpmn

Model	Feature
invoice_receipt.bpmn	Baseline process (start, service, review, approve/reject, notify)
order_fulfillment.bpmn	Legacy reference model
invoice_receipt_reminder.bpmn	Intermediate timer catch
invoice_message_exchange.bpmn	Message throw/catch
invoice_signal_exchange.bpmn	Signal throw/catch
invoice_review_deadline.bpmn	Interrupting timer boundary
invoice_review_reminder_non_interrupting.bpmn	Non-interrupting timer boundary
invoice_processing_error.bpmn	Interrupting error boundary
invoice_review_escalation.bpmn	Interrupting escalation boundary
invoice_call_activity.bpmn	Call activity request/reply
invoice_review_subprocess.bpmn	Embedded subprocess
invoice_nested_subprocess.bpmn	Nested subprocesses
invoice_subprocess_deadline.bpmn	Interrupting timer boundary on subprocess
invoice_subprocess_reminder_non_interrupting.bpmn	Non-interrupting timer boundary on subprocess
invoice_subprocess_error.bpmn	Interrupting error boundary on subprocess
invoice_event_subprocess_message.bpmn	Non-interrupting message-start event subprocess
invoice_event_subprocess_interrupting_message.bpmn	Interrupting message-start event subprocess
invoice_event_subprocess_timer.bpmn	Timer-start event subprocess
invoice_event_subprocess_error.bpmn	Error-start event subprocess
data_pipeline_demo.bpmn	Plugin-annotated pipeline (json-transform, field-filter, kv-enricher)
order_events_pipeline.bpmn	8-plugin order pipeline with XOR gateway; use --connect for source/sink
log_processing_pipeline.bpmn	Regex, template, flatten, validate, mask; use --connect
custom_activity_demo.bpmn	Custom activity with contract interface + delegating worker

Monitoring

A Kafka Streams topology consumes per-process lifecycle events and materializes:

• latest state per instance into process-state
• counts by state into process-state-counts
• active-instance index into process-active-state
• activity latency summaries into process-latency
• per-activity throughput (items processed) into process-activity-throughput
• coarse lifecycle trend buckets into process-trends
• BPMN model cache into process-models

Processes self-register by publishing their BPMN 2.0 XML to the process-models Kafka topic on startup. The monitor discovers processes from this registry — no pre-configured process ID list needed.

The dashboard overlays each task in the Process Diagram with its aggregate statistics — items processed and latency (avg/p95) — and layers any active alarm on top (driving the colour), rather than showing a single execution's state. The lifecycle events that feed latency (ACTIVITY_ENTERED/ACTIVITY_COMPLETED pairs) are emitted by the generated workers. Validation events (process-events-<processId>-validation) are excluded from the state and fault/alarm topologies so a running shadow never disturbs production monitoring.

Validation mode

Validation mode is a step on the path to a follow-up release. You regenerate the process from a modified BPMN model (a candidate — new plugin versions, changed routing, a different topology) with --validation, and run it as a complete shadow of the process that is currently in production (or of a prior implementation). The shadow exercises the candidate against real traffic and documents what it would have produced, so a supervisor can compare the candidate's behaviour against the running process before promoting it.

--validation is a mode, not an add-on: it produces one process, wired differently, rather than extra workers beside a functional process. A validation build:

• reads the same live input topics as the production process, but through a distinct consumer group ({processId}-validation), so it runs alongside the real process without stealing its messages or advancing its offsets (start at latest by default, or set DURGA_VALIDATION_OFFSET_RESET for a bounded recent replay);
• writes nothing to the real process topics. Every task output that would normally land on {processId}_{taskId}_output is diverted to a per-task validation topic {processId}_{taskId}_output-validation, and lifecycle events go to the per-process process-events-{processId}-validation topic instead of the live process-events-{processId};
• suppresses substantial side effects. Tasks that would mutate the outside world (e.g. writing to object storage) do not do so in validation mode — the plugin runs with a validation PluginExecutionContext and only records the alternative response it would have produced. Nothing outside the validation topics is modified.

Because each task reads the live production input for its position and re-emits its result, the comparison is done per task: a change in an early task never contaminates the comparison of later ones. The monitor pairs each task's validation ACTIVITY_COMPLETED (from process-events-<processId>-validation) against the live production ACTIVITY_COMPLETED for the same input and classifies each comparison as EQUAL, DIFF, PRIOR_MISSING, or CANDIDATE_ERROR using a normalized JSON diff with configurable ignore-paths (durga.validation.ignore.paths). Results are exposed via:

• GET /api/validation/summary?processId=<id> — per-task outcome counts
• GET /api/validation/results?processId=&taskId=&status= — individual comparisons
• GET /api/validation/instances/{instanceId} — comparisons for one instance
• a Validation Report panel in the dashboard (per-task summary + per-instance input/prior/candidate diff), and durga_validation_* Prometheus metrics

The validation wiring is applied for both the Java and Rust targets, and both targets chain tasks through identical topics (see Code-generation targets), so a candidate can be validated on either runtime and compared against a production process on either runtime — e.g. a Rust shadow validating a running Java process, task by task. The scripts/run-e2e-validation.sh helper starts a validation shadow alongside a running run-e2e-test.sh feed.

Full-stack dev demo (one command)

./setup/dev-up.sh

Starts everything — Kafka in Docker, the monitoring backend, auto-registers all BPMN models from durga-tools/src/test/resources/bpmn/, starts continuous feeds for invoice_receipt and order_fulfillment, and serves the Svelte SPA. Open http://localhost:8081. Press Ctrl+C to stop all services.

Multiple processes:

FEED_PIDS="invoice_receipt,order_fulfillment" ./setup/dev-up.sh

Fast restart (skip builds):

SKIP_BUILD=true ./setup/dev-up.sh

Configurable environment variables:

Variable	Default	Purpose
FEED_PIDS	invoice_receipt,order_fulfillment	Comma-separated process IDs to feed
BOOTSTRAP	localhost:9094	Kafka bootstrap servers
PORT	8081	Backend HTTP port (API + SPA)
START_KAFKA	true	Auto-start Kafka via Docker Compose
SKIP_BUILD	false	Skip Maven + npm build
BPMN_DIR	durga-tools/src/test/resources/bpmn	{pid}.bpmn directory for diagram fallback
FEED_INTERVAL	1000	Milliseconds between feed lifecycle completions

Manual setup

# Terminal 1 — Kafka
cd setup && docker compose up -d

# Terminal 2 — Build
cd monitoring-ui && npm install && npm run build && cd ..
mvn -q package -DskipTests
JAR="$(find durga-monitor/target -maxdepth 1 -name 'durga-monitor-*-runner.jar' -print -quit)"
DEMO_JAR="$(find durga-demo/target -maxdepth 1 -name 'durga-demo-*.jar' ! -name 'original-*' -print -quit)"

# Terminal 3 — Monitoring backend
java -Dquarkus.http.port=8081 -Ddurga.streams.state.dir=/tmp/kafka-streams-state \
  -jar "${JAR}" \
  localhost:9094 durga-monitor

# Terminal 4 — Register BPMN models
for f in durga-tools/src/test/resources/bpmn/*.bpmn; do
  java -cp "${DEMO_JAR}" \
    org.gautelis.durga.demo.BpmnModelPublisher \
    localhost:9094 "$(basename "$f" .bpmn)" "$f"
done

# Terminal 5 — Feed
java -cp "${DEMO_JAR}" \
  org.gautelis.durga.demo.ContinuousFeedPublisher \
  localhost:9094 invoice_receipt 1000

Open http://localhost:8081 for the dashboard. The SPA displays an overview of all registered processes with KPIs. Click any process row to drill down into latency, stuck instances, trends, instance detail, and the BPMN diagram.

HTTP API endpoints

Health is also available at the conventional root path:

• GET /health — Kafka Streams state (RUNNING, REBALANCING, etc.)

The API namespace also exposes:

• GET /api/health — Kafka Streams state (RUNNING, REBALANCING, etc.)
• GET /api/processes/list — all known process IDs (from models + counts)
• GET /api/instances/{processInstanceId} — latest state view for one instance
• GET /api/processes/{processId}/counts — state counts per process
• GET /api/processes/{processId}/latency — per-activity latency summaries
• GET /api/processes/{processId}/throughput — per-activity items-processed counts
• GET /api/processes/{processId}/trends — lifecycle trend buckets
• GET /api/counts — counts across all monitored processes
• GET /api/stuck?processId=<id>&olderThanSeconds=60 — stuck-instance detection
• GET /api/diagram?processId=<id> — BPMN 2.0 XML from the process-models cache
• GET /api/metrics — Micrometer metrics in Prometheus text format

Set DURGA_MONITORING_API_KEY or -Ddurga.monitoring.api.key=<key> to require Authorization: Bearer <key> on monitoring JSON endpoints. /api/metrics remains unauthenticated for Prometheus-style scrapes.

CLI client

JAR="$(find durga-monitor/target -maxdepth 1 -name 'durga-monitor-*-runner.jar' -print -quit)"

java -cp "${JAR}" \
  org.gautelis.durga.monitoring.ProcessMonitoringClient \
  http://localhost:8081 health

java -cp "${JAR}" \
  org.gautelis.durga.monitoring.ProcessMonitoringClient \
  http://localhost:8081 counts invoice_receipt

java -cp "${JAR}" \
  org.gautelis.durga.monitoring.ProcessMonitoringClient \
  http://localhost:8081 latency invoice_receipt

java -cp "${JAR}" \
  org.gautelis.durga.monitoring.ProcessMonitoringClient \
  http://localhost:8081 stuck invoice_receipt 60

java -cp "${JAR}" \
  org.gautelis.durga.monitoring.ProcessMonitoringClient \
  http://localhost:8081 instance <processInstanceId>

For an authenticated monitoring API, set DURGA_MONITORING_API_KEY in the client environment before running the commands.

Demo scenarios (without a generated process)

java -cp durga-demo/target/durga-demo-0.1.0-beta.1.jar \
  org.gautelis.durga.demo.ProcessEventScenarioRunner \
  localhost:9094 happy invoice_receipt register_invoice,review_invoice,notify_requester

# Also available: stuck, failed

Docker demo

docker compose -f setup/docker-compose.demo.yml up --build

Starts Kafka, the monitoring backend, and a continuous feed publisher. Open http://localhost:8081 for the dashboard (API + SPA), http://localhost:8080 for Kafka UI.

FEED_PROCESS_ID=order_fulfillment FEED_INTERVAL_MS=2000 \
  docker compose -f setup/docker-compose.demo.yml up --build

Generated project probes

Every scaffolded project includes producer and observer helpers:

./generated/demo-scenario.sh happy
./generated/send-task-input.sh register_invoice
./generated/complete-task.sh approve_invoice <instance-id>
./generated/fail-task.sh register_invoice <instance-id>
./generated/escalate-task.sh review_invoice <instance-id>
./generated/complete-call-activity.sh validate_invoice_process <instance-id>
./generated/send-message-event.sh invoice_review_response_message <instance-id>
./generated/send-signal-event.sh invoice_review_signal_signal <instance-id>
./generated/watch-process-events.sh
./generated/watch-task-output.sh register_invoice

Embedded subprocesses generate scope entry/completion services. Event subprocesses with message or signal starts generate start/completion services driven by external Kafka topics. Interrupting starts emit cancellation for the enclosing scope; non-interrupting starts branch alongside the parent flow. Timer, error, and escalation event subprocess starts are supported within embedded subprocesses.

User interfaces

Processes (or data pipelines) are managed using any BPMN modeler, such as Camunda Modeler.

The monitoring tool also uses the BPMN model as a backdrop to presenting statistics.