External bus providers and database data sources, via resident adapters by mmalkhatib · Pull Request #299 · simplify9/Bitween-api

mmalkhatib · 2026-09-10T13:20:04Z

Brings external systems into the pipeline as data sources: brokers Bitween does not own, and databases it talks to over a pooled connection. Both run as resident adapters — long-lived processes that dial out to the host over gRPC — behind one runtime interface, so the pipeline cannot tell which kind of adapter it is invoking.

43 non-merge commits, releases/r10.0 merged in three times along the way. 472 unit, 392 integration and 312 client tests pass.

External bus providers

A BusGateway can now consume from a customer's own broker rather than Bitween's.

RabbitMQ and SQS adapters (SW.Bitween.Adapters.Bus.*), published packages, not in-process.
Node placement — a broker connection is exclusive, so exactly one node holds it, through a lease with a database-issued fencing term. A node whose term is no longer current stops immediately rather than carrying on consuming.
Inbound deduplication, enforced rather than assumed, with a per-source window.
Memory and CPU ceilings per adapter, on SimplyWorks.Serverless 8.1.16+.
A configuration and health UI: what it is connected to, what it last said, and why it stopped.

Database data sources

Oracle and PostgreSQL adapters over a shared core (SW.Bitween.Adapters.Db.Core) — the command surface, paging, the statement allow-list and the polling receiver are written once; each engine contributes its connection string, catalogs and capability list.
Resident so the pool outlives the message: a connect, a TLS handshake and an authentication round trip are paid once rather than per Xchange.
Query, execute, call, batch — statements, stored procedures, functions and views.
A polling receiver in Kafka Connect's vocabulary (bulk, incrementing, timestamp, timestamp+incrementing) plus marker, a processed-flag column, which is what enterprise integration tables actually use. The cursor is host-held, so a restart does not replay; it advances per row, only after Bitween has durably accepted that row.
Describe and Discover: what the engine supports, and what is actually in the database.

Statements are records, not a settings box

SQL lives on the data source as DataSourceStatement, and a subscription names it. This is the security boundary: adapter property values have {{partner.X}} substituted into them before the adapter sees them, so SQL in a subscription's properties would be steerable by ordinary partner data. It also gives each statement its own permission (data-source-statements.*, separate from data-sources.edit — writing a query and rotating a password are different jobs), its own audit trail, and a usage count that makes dead SQL a countable fact.

Statements are validated when saved — the engine prepares them, parsed and planned, never run. A wrong placeholder prefix is named rather than left as "syntax error at position 76": :acid on PostgreSQL is answered with write @acid. TestConnection runs the same check across every configured statement and reports all the failures, not the first.

Where the receive settings live

Split on one principle: the SQL and the shape of its rows belong to the statement; the reading policy belongs to the subscription doing the reading. The polling SQL, cursor column and key column are the statement's — a statement reading where id > @cursor order by id has id as its cursor whoever polls it. Mode, batch size and the mark-processed statement are the subscription's. The columns are editable from the subscription that polls with them, so the whole receiver fits on one screen while there is still one place the value lives.

UI

Data source forms build themselves from the adapter. [AdapterSettings] / [AdapterSetting] are read out of the published package with MetadataLoadContext — read, never loaded, nothing in the adapter runs. Add a setting to an adapter and it appears, in the adapter's own order, with no front-end change.
A schema browser — grouped by schema, searched by name against the database rather than the page in hand, columns on demand. "Use in a statement" drafts SQL from a table, view, procedure, function or sequence.
Data sources moved to Configuration, first in the group.
Test result first on the page; a failed connection reports what the adapter said, stage by stage.

Fixes worth calling out

A shared receive cursor. Host state is keyed by (adapter, instance, name) and the instance is the connection, so a fixed cursor name meant two subscriptions polling one data source split the rows between them — no error, no warning. Scoped per subscription; the old unscoped value is inherited once, by whichever subscription asks first.
The adapter pool was keyed by adapter id, so the first renter's credentials reached later ones. Now keyed by id plus a hash of the ordered startup values.
Passwords hashed with SHA256 and compared in constant time.
Four write endpoints injected RequestContext and never checked a permission.
Scriban 7.4.0, clearing four advisories against the template engine.
A database adapter is a receiver and a handler, not only a data source — it was configurable but not choosable.
The test suite was deleting the dev environment's adapters: it shared the default bucket and teardown deletes it.

Migrations

Eight, per provider: external bus data sources, inbound deduplication, cluster leases, adapter memory limits, adapter CPU limits, database data sources, data source statements, receive columns on statements.

Notes for review

Bitween:BusProvidersEnabled gates every resident data source provider, brokers and databases alike. It is on in Development now; it was set out of band before, so a dev run came up with every data source untestable. The name predates database sources — operator-facing messages say what it gates rather than repeating it.
Oracle's integration tests pull a 4.8 GB image. Whether CI runs them per PR or nightly is still an open decision.
MySQL and SQL Server adapters are not built. Each is a driver, a connection string, a catalog query and a capability list over the existing core.
Push ingress (LISTEN/NOTIFY, Oracle CQN, Service Broker) is out of scope; ChangeNotification is declared false everywhere.

🤖 Generated with Claude Code

A BusGateway can now be fed by an external broker instead of the internal bus, through a resident serverless adapter. Nothing existing changes. BusGateway.DataSourceId is nullable and null still means the internal bus, so every gateway already in a database behaves exactly as before. The ExternalBusDataSources migration is additive: three columns and one table, no data migration. Shape external broker -> resident adapter (owns the connection) -> BusProviderEventSink (resolves the gateway, persists) -> XchangeService.SubmitFilterXchange -> filter, mapper, handler, auto-retry — unchanged -> ack returns -> adapter acknowledges its broker Past the sink, ingress from a broker and ingress from the API are the same thing; none of the pipeline is reimplemented. The adapter does not acknowledge its broker until Bitween has persisted, so a Bitween outage stops draining the customer's queue rather than losing their messages. Domain DataSource holds how to reach a system — endpoint, credentials, health — while the gateway keeps what the message MEANS. One data source serves many gateways, as one connection serves many queues. DataSourceKind declares Relational/Document/ObjectStore alongside Broker because the adapter contract is identical for them; only push-vs-poll differs. Runtime BusProviderSupervisor reconciles running adapters against active data sources and writes heartbeat health back to the row, so a broker that has gone away is visible without tailing logs. Off by default via BitweenOptions .BusProvidersEnabled — a broker connection is exclusive and node placement is not implemented yet, so every instance would otherwise fight for it. DataSource.OwnedByNode exists for that election to write into. Providers Adapters.Bus.RabbitMq someone else's RabbitMQ. autoAck:false, ack only after Bitween persists, nack-with-requeue on rejection. DeclareMode defaults to `assert` because silently creating queues on a customer's broker is not our call. Dedupe key is the broker message id, not the delivery tag — tags are per channel and reset on reconnect. Publish included, so egress works on external gateways even though the internal one lacks it. Adapters.Bus.Sqs SQS is polled and has no ack, only delete. Same contract: receive, persist, ONLY THEN DeleteMessage. A rejection resets visibility to 0 so it retries in seconds. TestConnection warns when the visibility timeout is short enough to redeliver mid-persist. This is also the transport for Amazon Selling Partner API notifications, which SP-API delivers by publishing to an SQS queue you own. UnwrapSellingPartnerNotification strips the envelope and promotes notificationType and metadata to headers, so a Bitween document schema need not carry Amazon's wrapper. SP-API request/response calls are ordinary HTTPS and belong in a mapper — this covers the push half only. Both live under the Adapters/Bus Providers solution folder and target net8.0 deliberately: adapters are separate processes and must stay runnable on hosts that predate r10's move to net10.0. Temporary SimplyWorks.Serverless and .Sdk package references are swapped for ProjectReferences into ../SW-Serverless while simplify9/SW-Serverless#108 and #109 await approval. Restore them once merged and published. This also pulled SW.Serverless.Sdk and .Contract into the solution file. Docs: docs/external-bus-providers.md, including what is deliberately not done — placement, the DataSource CRUD API and UI, secret protection at rest, and broker-backed integration tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…redentials from the container Extends BitweenFixture with a SECOND RabbitMQ standing in for a customer's own broker — reusing the internal one would let a test pass while the message actually travelled Bitween's own bus, which is the confusion this feature exists to avoid — plus ElasticMQ for the SQS API without LocalStack's weight. Both bus adapters are published into the local cloud store with protocol-2 metadata, so the tests exercise install-from-storage rather than a local path. ExternalBusGatewayTests covers ingress becoming an Xchange, endpoint-to-gateway resolution, egress, staged test-connection, heartbeat health reaching the health view, and the regression that matters most: a gateway with no DataSourceId is still an internal-bus gateway. Fixed while running them: the fixture hardcoded guest/guest, but RabbitMqBuilder generates random credentials, so every external-broker test failed with ACCESS_REFUSED on PLAIN. Read from the container's own connection string now. STATUS: 4 of 8 passing. The transport works end to end — the adapter installs, attaches, connects, declares topology, consumes and heartbeats, which is what TestConnection and the health test prove. What does not yet work is downstream of that: no Xchange is created. Under investigation; the feature is not proven. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Four test bugs, none of them in the feature. The infolink cache is a SINGLETON holding a ten-minute snapshot, so a Document created by a test is invisible to XchangeService: SubmitFilterXchange resolved it to null, built an Xchange from that, saved successfully, and returned. The sink therefore acked and nothing was ever queryable by DocumentId — which presented as "acked but no Xchange" and read like an ack-ordering bug. GatewayRoutingTests already documents this trap; the helper now revokes the cache after creating a Document, as it does. Each_gateway_receives_only_its_own_endpoint listed one queue on the data source and published to the other. The adapter consumes what the DATA SOURCE lists, so nothing was consuming the queue under test. Bitween_can_publish_out_to_an_external_broker measured depth on a queue the adapter was draining, so the message was consumed as fast as it was published and depth read 0 whether the publish worked or not. It now publishes to a queue the adapter does not consume. A_rejected_message_stays_on_the_broker_for_redelivery is replaced rather than fixed, because its premise was wrong. Bitween persists first and validates afterwards, so malformed content becomes an Xchange carrying a bad result — a pipeline outcome, not an ingest failure. No payload will make the sink reject. Split into the two contracts that do exist, both deterministic and neither needing a broker: The_sink_rejects_an_event_it_cannot_attribute_to_a_data_source The_sink_accepts_and_discards_an_event_no_gateway_claims The second matters as much as the first: an unclaimed endpoint must NOT be rejected, or it requeues for ever and the customer's queue never drains. That the adapter then nacks and the broker redelivers is already proven against a real broker in SW-Serverless (A_rejected_message_is_nacked_back_and_redelivered). What belongs here is Bitween's half of that contract. The temporary Diagnostic_ test that ruled out the gateway lookup is removed. The feature itself needed no changes: ingress from an external broker becomes an Xchange, endpoint-to-gateway resolution works, egress works, test-connection and heartbeat health work, and a gateway with no DataSourceId is still an internal-bus gateway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… lifecycles Five tests that run a message all the way to an XchangeResult, through subscriptions that actually execute serverless adapters rather than stopping at the Xchange row. An_external_broker_message_runs_a_subscription_and_produces_a_result resident adapter -> sink -> Xchange -> filter -> route -> subscription -> classic serverless MAPPER and HANDLER -> result. Asserts the output size equals the mapper's payload, which pins that the mapper ran rather than the original message passing through untouched. The_internal_bus_produces_the_same_outcome_as_an_external_broker The design claims broker ingress and internal ingress converge at SubmitFilterXchange and share everything after it. Same payload, same subscription shape, one fed by a customer's RabbitMQ and one fed directly — compared on OutputHash. This was a load-bearing assumption; it now has a test. A_filter_on_a_route_selects_which_subscription_runs Two routes, two filters, one message: exactly one subscription runs. A_resident_adapter_and_classic_adapters_serve_one_message_together Both lifecycles in one journey. The resident adapter has owned the broker connection since before the message existed; the mapper and handler are spawned per invocation and die with the scope. Asserts the resident is still Ready with RestartCount 0 afterwards, having outlived them. A_failing_handler_is_recorded_as_a_result_not_as_an_ingest_failure A handler error is Success=false with the message text AND the queue still drains, because the message was persisted and acked long before the handler ran. The failure belongs to the retry policy, not to the broker. Three wrong assumptions found while writing these, all in the tests: Every Subscription constructor sets Inactive = true, so a freshly created subscription is off and the filter never matches it. All five failed on this. The configurable sample adapter's failure switch is SimulateError with ErrorMessage, not the ThrowException I invented. A handler's return becomes the RESPONSE; only a mapper produces OUTPUT. With no mapper configured there was no output at all, so the original assertion could never hold. Fixing it made the test stronger — it now exercises both stages and pins the mapper's exact output. 14/14 with ExternalBusGatewayTests, sharing one fixture and database. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…upe check Nine tests against a real SQS API (ElasticMQ), covering what is SQS-specific rather than restating RabbitMQ's shape. SQS has no ack, only DELETE: a received message is invisible for the visibility timeout and reappears if it is not deleted. So the tests are about deletion and visibility. An Xchange existing while the queue is empty is the proof that deletion happened AFTER persistence. Queue depth counts ApproximateNumberOfMessagesNotVisible as well, or an in-flight message reads as delivered and the assertion means nothing. SP-API unwrapping is asserted on InputSize against the exact byte count of the compact payload, plus a guard that the envelope is more than twice that size — so a pass-through cannot satisfy it by accident. The substring checks this replaced would have accepted a partially unwrapped body. Plus test-connection with its visibility-timeout warning, discovery, egress to a queue the adapter is not polling, and heartbeat health. Fixture: ElasticMQ builds QueueUrl from its in-container port, so the URL it returns is unreachable from the test host. Only the path is trustworthy; the authority is rebuilt from the mapped endpoint. FINDING — deduplication is not enforced. Every adapter picks its key deliberately (broker message id, SP-API notification id, content hash) and BusProviderEventSink carries it onto the Xchange as a reference. Nothing checks it, so a redelivered message produces a second Xchange. At-least-once is not optional here — it is what persist-then-acknowledge buys, and duplicates are therefore normal rather than exceptional. The key is carried, which is the precondition; the enforcement is absent. Documented in docs/external-bus-providers.md and now the top item on the outstanding list. The test that covers it asserts only what is true and names the gap. An earlier draft asserted nothing meaningful and would have read as dedupe coverage while proving none. 23/23 with ExternalBusGatewayTests and PipelineEndToEndTests on one fixture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

At-least-once delivery is what persist-then-acknowledge buys — a crash between committing the Xchange and acknowledging the broker redelivers by design — so duplicates are normal and something has to recognise them. Every adapter already chose a key deliberately; nothing checked it, and a redelivered message produced a second Xchange. InboundMessage records the key, namespaced by DataSourceId. THE KEY IS THE PRIMARY KEY: a duplicate is detected by the insert FAILING, never by a lookup succeeding. Check-then-act races, so two concurrent deliveries of one key would both miss and both persist — which is the same defect CodeRabbit found in the test sink on the other repository. It commits with the Xchange in ONE transaction, by adding the row to the same DbContext SubmitFilterXchange writes through. Splitting them gives two failure modes and the worse one is silent: a dedupe row committing while the Xchange fails suppresses that message for ever. A duplicate is ACCEPTED, never rejected — rejecting would nack and redeliver a message that is by definition already handled. An event with no key is never deduplicated, because collapsing unidentified messages would lose data. DataSource.DeduplicationWindowDays (30, zero disables) with a nightly batched prune job. Forgetting too early is the dangerous direction, and the right window is a property of the customer's broker, so it lives on the data source. Unique-violation detection covers all three providers: PostgreSQL 23505, MySQL 1062, SQL Server 2601/2627. FOUND WHILE DOING THIS — SW.Bitween.PgSql.BitweenDbContext does not call base.OnModelCreating; it redeclares the model. Anything configured only in SW.Bitween.Api's context is INERT on the primary provider. DataSource reached the model anyway, by convention, through the BusGateway.DataSource navigation — so it worked while its intended configuration (unique index on Name, explicit lengths) was silently never applied. InboundMessage has no such navigation and did not exist at all, which is what the failing tests were reporting. Declared in the PgSql context now, with a comment recording the trap. Re-applying DataSource's intended configuration is on the outstanding list. 9 dedupe tests, including eight concurrent deliveries of one key to prove the constraint rather than a lucky lookup, and that a failed ingest does NOT remember its key. 32/32 across all four external-bus suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A broker connection is exclusive: two nodes consuming one queue is duplicate processing, which is the failure this design exists to prevent. Ownership is now granted per data source, so BusProvidersEnabled is safe on every node. THE LOCK is a RabbitMQ exclusive queue per data source. An exclusive queue belongs to one connection, so declaring it succeeds for exactly one node and the broker releases it the moment that connection dies — liveness with no lease renewal to get wrong and no clock to trust. THE FENCE is a monotonic term in cluster_lease, because the lock alone is not enough. A node paused by a long GC or a healing partition can have its queue reclaimed while it still believes it owns the resource, and RabbitMQ has no counter that reveals it. Acquiring bumps the term; a stale holder fails validation and stops. Per data source rather than globally, so load spreads without a scheduler and one node leaving does not move everything at once. ILeaderElection exists so the mechanism can be REPLACED when the internal bus is no longer RabbitMQ, not so two implementations live side by side. Three details that are load-bearing: AutomaticRecoveryEnabled is OFF on the election connection. A recovered connection silently re-declares the exclusive queue, so a node that lost ownership during an outage takes it back without bumping the term — two owners, neither aware. Releasing DELETES the queue. Found by the tests: an exclusive queue belongs to the CONNECTION, not the channel, so closing the channel released nothing. Four of six tests passed against that — mutual exclusion and crash failover both worked — while GRACEFUL handover never completed, which is exactly what a rolling restart or deactivating a data source does. The two handover tests were burning their full 15s retry windows; the suite went from 30s to 759ms. Losing a lease stops the adapter WITHOUT draining, because another node may already be consuming and finishing in-flight work would handle messages twice. A dedicated connection, separate from SW.Bus's: sharing it would tie every lease to the fate of ordinary message traffic, so one blip drops every broker connection the node owns at once. 6 election tests, including six nodes racing for one resource and a superseded holder failing validation. 38/38 across all five external-bus suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Replaces the nine local ProjectReferences into ../SW-Serverless now that #108, #109 and #111 are merged and published. 8.1.13 could not be used: SimplyWorks.Serverless.Sdk declared a dependency on SimplyWorks.Serverless.Contract 8.1.13, which was never published because the pack-push step enumerates projects explicitly and the new package had not been added to it. 8.1.14 publishes all three, so restore resolves. Also removes SW.Serverless.Sdk and .Contract from SW.Bitween.sln — they were pulled in as cross-repo entries when the local references were added, and have no place here now. Verified against the packages rather than assumed: clean restore with no NU warnings, clean build, and 38/38 integration tests — which exercise the resident runtime out of the packed assemblies, spawning adapter processes, attaching over the Unix socket, electing leaders and deduplicating. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…it properly The reconcile loop had no tests at all, despite a fixture comment claiming otherwise. Adding them, and pushing on the adapters under conditions that break a broker client rather than conditions that demonstrate one, turned up five real defects. 217 unit + 236 integration tests pass. ROUTING. BusProviderEventSink ordered candidate gateways by Endpoint descending to make an exact match beat the catch-all. PostgreSQL sorts NULLS FIRST on a descending order, so the catch-all won every time: a data source with both a catch-all and a specific gateway filed every message against the wrong Document, running the wrong subscriptions, with nothing in the audit trail saying so. Now ordered by the match itself, which means the same thing on all three providers. CONCURRENCY. RabbitBusHandler acked and nacked one shared IModel from concurrent thread-pool handlers, which RabbitMQ.Client does not support; only the publish channel was guarded, and even there CreateBasicProperties sat outside the lock. Both channels now have a gate, and shutdown goes through them too. IDENTITY. With no MessageId the adapter hashed the body for a dedupe key, so two legitimately identical messages — reorder the same SKU, the same daily totals — collapsed into one and the second was dropped for the whole deduplication window, thirty days by default. Unkeyed deliveries are handled once per delivery, with a warning, which trades a possible reprocess for never losing a real message. PROVIDERS. MySQL and SQL Server had no migrations for data_source, inbound_message or cluster_lease, so the feature only ever worked on PostgreSQL — MigrationDriftTests was already red on this branch, and its own remarks say the symptom is a pod crash-looping at startup. Generated both. CLUSTERING. ClusterLease was declared only in the PgSql context, so even with those migrations MySQL and SQL Server would have had no table for leader election, and two nodes could consume one queue silently. Moved to the base context and regenerated. Tests (21 new) - BusProviderSupervisorTests: start, endpoint derivation from gateways, a second gateway, no restart when nothing changed (the process id is the witness — a restart every thirty seconds would tear down every consumer twice a minute), restart when configuration changes, deactivate, lease release, placement, fencing, handover, one bad adapter not stopping the pass, health write-back, shutdown handover, inactive gateways. - RabbitBusAdapterTests: identical bodies, identity from the message id, integrity under concurrent delivery, concurrent publishes. - ExternalBusGatewayTests: exact match beats catch-all, catch-all still catches, and a data source feeding a gateway cannot be deleted. Five were mutation-checked: removing the fingerprint guard, the lease revalidation, the start-failure isolation and the inactive-gateway filter, and restoring the content-hash key, each make the intended test fail. Also: BusProviderSupervisor.ReconcileAsync is public — "reconcile now" is a real operation, the same shape as receivenow, and a loop whose only entry point is a thirty-second timer cannot be tested. BusGateway.EndpointProperties is passed through by the supervisor but read by no adapter; documented as inert rather than left to be discovered, and it no longer produces "Endpoint::key" for the catch-all. Corrected a stale comment in Startup.cs claiming placement across nodes was unimplemented. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The external bus feature had no API and no UI: it was reachable only by writing rows into the database, which is how every one of its tests set itself up. This adds both, in SW.Bitween.Web/ClientApp — the UI that ships inside the API — and the multi-integration tests the single-queue suite never covered. 217 unit + 258 integration + 90 ClientApp tests pass. API - /datasources: search, get, create, update, delete, plus test and telemetry. - Secrets are masked on the way out and merged back on save, so editing the prefetch cannot overwrite the password with a row of dots, and a data source copied out of a response cannot authenticate with the sentinel. A property NAMED like a credential is masked whether or not anyone declared it — the one nobody ticks is the one that ends up in a JSON response. - The list carries no connection settings at all. Even masked, sending every credential to render a table row is exposure with no purpose. - test runs the real adapter against the stored settings and reports stage by stage, but starts it with Consume=false so the customer's queues are inspected rather than drained. Without it the only way to find a wrong password is to save, wait out a reconcile, and read the health column. - BusGateway gained DataSourceId and Endpoint. An external gateway with no endpoint is refused — it would sit connected and never receive anything — and so is a second gateway on an endpoint another already reads, which would make every message a coin toss between them. Observability - /datasources/{id}/telemetry reads the heartbeat rather than the row, which only carries what the last reconcile wrote back. The panel keeps the two sources apart on purpose: adapter-reported figures (per-queue backlog, received/acked/nacked/failed) stop arriving the moment the adapter wedges, while host-observed ones (pid, memory, CPU, threads, uptime, restarts) need no cooperation and still answer. That is what separates "the broker is quiet" from "the adapter is stuck". - It is scoped to the node that answers and says so, because a broker connection is exclusive and no other node can see its figures. UI - A Data sources area: list with connection health and which node holds each, a create page, and a detail page with settings, the live panel and Test. - On the bus gateway, a Source control — internal bus or a broker, with the endpoint and that connection's health. A dialog rather than a canvas node: the canvas draws what happens to a message per route, and where messages come from belongs to the gateway. - Fixed two bugs the new column exposed: "Type not on bus" was shown for external gateways in both the list and the toolbar. It is meaningless there — an external gateway never touches Bitween's own bus — and it was hiding the gateway's real problem, which was that it had no routes. Adapters - Both gained Consume=false: connect and declare, but do not consume. This is what makes a connection test safe to run against a live queue. Tests - DataSourceApiTests (14): secrets never returned in clear, an undeclared credential masked anyway, a masked secret surviving an unrelated edit, a sentinel with nothing behind it dropped, and the gateway guards. - SharedBrokerTests (8): one Acme broker, five queues, three of them Bitween's. One connection and one process serve all three gateways; thirty interleaved messages each land on their own information type; a queue nobody pointed Bitween at stays untouched; a fourth integration joins without disturbing the others; deactivating one gateway stops only its queue and leaves its messages on the broker rather than acking them away; the same message id on two queues is two messages; two data sources on one broker are owned separately; and each queue runs its own subscription and records its own result. - The dedupe-scoping claim is mutation-verified: dropping the endpoint from the key makes that test fail. Also corrects CLAUDE.md, which documented the standalone Bitween-UI repo as the dashboard and never mentioned ClientApp at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…less 8.1.16 An adapter is a separate process holding a customer's broker connection, and Bitween constrained it in no way at all: one runaway payload was bounded only by the host, taking every other integration on the node with it. The runtime had supported memory ceilings all along and nothing passed them; CPU had no ceiling to pass. Both are now configurable per data source, from the UI. 217 unit + 265 integration + 90 ClientApp tests pass. Packages - SimplyWorks.Serverless[.Sdk] 8.1.14 -> 8.1.16 across all nine references, after checking the published assembly actually carries ResourceLimits, CpuPercentLimit, UpdateLimitsAsync, RestartAsync and CommandDetails rather than assuming the version number implied them. Ceilings - DataSource gains SoftMemoryLimitMb, HardMemoryLimitMb, CpuPercentLimit and CpuLimitSamples, migrated on all three providers. - The supervisor passes them into the AdapterSpec AND folds them into the fingerprint. Without the second half a raised limit would save cleanly, change nothing, and the adapter would keep running under the old one — with nothing anywhere to say it had not taken. - Guards that refuse a combination which can never fire: a soft memory ceiling above the hard one (the runtime fails the allocation first, so the graceful recycle never happens), and a CPU ceiling above 100% (the figure is a share of the whole node, so nothing can exceed it). Enforced in the handler, not only in the FluentValidation validator, because the validator runs in the HTTP pipeline and nothing else does. UI - Memory ceilings and a CPU ceiling on the data source page, each saying what crossing it actually does — drain versus kill — since that is the difference between messages going back to the broker and being lost. - The CPU copy carries the warning that the figure is a share of the WHOLE node rather than of one core: one core pegged on a sixteen-core node reads about 6%, so "50%" would allow eight cores. That misreading is not hypothetical — it broke the first version of the serverless test. The "for scale" hint says outright that it uses the browser's core count, not the node's, rather than implying knowledge of the server. Also adds /datasources/{id}/inspect, relaying Discover and GetStats to the adapter actually serving traffic — not a throwaway instance, the way the connection test does, because these are questions about the live connection. An allow-list rather than a passthrough: the adapter also exposes Publish, which writes to the customer's broker, and this endpoint is guarded by View. Tests - Changing either ceiling restarts the adapter (mutation-verified: removing the ceilings from the fingerprint makes it fail), both refusals, and the not-running answers for inspect. - Fixed a leak in the test helper that was the real cause of seven unrelated failures: it created a RabbitMqLeaderElection per test and never disposed it, so the exclusive queue that IS the lock stayed held and later reconciles found every data source owned by a node that no longer existed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…face A resident adapter could be picked from the handler, mapper, receiver or validator dropdown and could not work. The catalog lists by key prefix and never read the metadata, so residency made no difference to whether it was offered; the invoke path did not read it either and sent every packaged adapter down the classic route, where a resident one waits out the command timeout for an answer that never comes. The UI swallows the failed properties fetch, so what an operator saw was a selectable adapter with nothing to configure — not a broken one. 224 unit + 274 integration tests pass. One interface, three runtimes - IAdapterRuntime, asked in order, first to claim an adapter runs it: native (in-process), resident (rented from the pool), classic (spawned, and the catch-all, so it is asked last). IAdapterInvoker picks between them. - XchangeService, ReceivingJob and RetryAlertService now say WHAT they want run — id, role, method — with no branching on adapter kind anywhere in the pipeline. The native-versus-serverless if blocks are gone. - The role travels with the call because the id does not carry it: a mapper and a handler are both invoked as Handle, and only the role tells the native runtime which registration to resolve. Losing it would silently run the wrong adapter. - Sessions, not just calls. A receiver is Initialize, ListFiles, a GetFile and DeleteFile per item, then Finalize, and all of them have to reach ONE instance — renting per call would scatter that across pooled processes and Initialize would run somewhere the listing never sees. Catalog - SearchVersioned lists by the Kind stamped at publish time, falling back to the infolink6.<kind>s. prefix. The fallback stays because dropping it would empty the dropdown on any deployment that has not republished since the stamp existed. An adapter that declares its kind now appears whatever it is called, so reclassifying no longer means renaming — and a rename is not free, because every subscription stores the id. Tests - AdapterInvokerTests (7): routing, ordering, role and property passthrough, session release, and a session staying open until the caller closes it. Hand written fakes, no database or containers — 28ms, which is the seam this refactor was for. - ResidentPipelineAdapterTests (5): a real resident adapter answering the handler contract, the SAME instance serving three messages (a classic one is a new process per call and would answer 1 every time), a session pinned to one instance, and classic and native adapters still going through the same door. - ResidentAsPipelineAdapterTests (4): the dropdown behaviour, and both classification paths. - Mutation-verified: stopping the resident runtime claiming fails exactly the three resident tests; ignoring the stamped Kind fails exactly the metadata one. Known cost: listing by Kind reads metadata for anything the prefix did not already match. It is cached, but on a store with hundreds of adapters this is more work than the single prefixed LIST it replaces, and a registry table would be the better answer at that size. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

r10.0 replaced the per-entity trail tables with a single audit_entry table (AddAuditTrail + DropLegacyTrails); this branch added the external bus provider stack (data sources, cluster leases, inbound messages, resident adapter runtime). The two sets of migrations interleave by timestamp, so each provider gets an empty MergeExternalBusWithAuditTrail migration whose only job is to carry a snapshot that describes both halves. Verified: a fresh database and a database upgraded from r10 state produce identical schemas (366 columns each) on all three providers; MigrationDriftTests pass; 224 unit and 281 integration tests pass.

…ernal-bus-providers

… bus adapters Bumps the ten references from 8.1.16 and stamps [AdapterKind("bus")] on both bus handlers. The attribute did not exist in 8.1.16 — it ships with the installer work merged as simplify9/SW-Serverless#117. The installer now reads it at publish time and writes Kind and Lifecycle onto the package, which is the only way a host can find a provider it did not publish itself: Bitween's own are found by their "bitween." id prefix, and a prefix cannot possibly match a third party's adapter. Renaming an adapter to carry its role in the id is not an option either, since hosts store the id against every configuration that uses it. Bitween's two adapters carry the attribute so they are discoverable both ways and the prefix can eventually retire. Verified with the installer's own AdapterDescriber over both published packages: Lifecycle: resident, Kind: 'bus'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A failing data source reported "Adapter stream closed." — true, useless, and the same sentence for a wrong password, a wrong port and an unreachable host. The adapter's own exception went to stderr and nowhere else, and the row's LastException stayed null, because that field only ever held what a LIVE adapter reported about itself. AdapterFailureReader undoes the SDK's wire encoding, drops the capture timestamp and the stack frames, and keeps the exception's first line plus the innermost "--->". That last part is what makes it a diagnosis rather than a shrug: "None of the specified endpoints were reachable" is equally true of a wrong host, a wrong port or a firewall, while "Cannot determine the frame size" says TLS was spoken at a plaintext port and nothing else does. The adapter's reason leads and Bitween's own observation goes in brackets, so the useful half survives truncation in a narrow panel. Reading stderr at the moment of failure gets nothing: Bitween notices when the gRPC stream ends, while the reason is still being pumped on another thread. So SettledOutputAsync waits — bounded, ~2s — for the process to exit and the last error line to land. An adapter that is still alive is not waited on at all. The supervisor falls back to the same reader, and a start that throws outright is now recorded on the row instead of only reaching the node's log. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The front end carried its own copy of someone else's contract: which fields a RabbitMQ connection starts with, what DeclareMode accepts, which properties are credentials. A hand-kept copy drifts, and it did — the form advertised a "Tls" setting the adapter never read, so an operator could set it, see nothing wrong, and still be authenticating in the clear. QueueType, meanwhile, was a free-text box for a value the broker only accepts three of. An adapter now marks its options class [AdapterSettings(Kind, Label)] and each property [AdapterSetting(Hint/Default/AllowedValues/Secret/Required/Hidden)]. The contract is shared *source* rather than a package because adapters target net8.0 while the host is net10.0, and Bitween matches the attributes by name. DataSourceProviderCatalog reads them out of the published package with MetadataLoadContext — READ, never loaded, so nothing in a third party's adapter runs merely because Bitween described it — and GET /datasources/Providers serves the result. Add a setting to an adapter and it appears in the UI with no front-end change; delete the attributes and the form empties, which is the property the tests assert. Two things fall out of the same descriptor. Kind decides where a data source can be used: a bus gateway reads from a queue, so only a Broker can back one, enforced in Resources/BusGateways/{Create,Update} — the coming database adapters will be data sources too and must not be offerable there. And the detail page no longer wipes a half-typed setting on every poll: it reseeds the draft from a fingerprint of the editable fields rather than from object identity, which changed on every refetch whether or not anything had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

7.0.6 carried two high and two moderate severity advisories (GHSA-24c8-4792-22hx, GHSA-7jvp-hj45-2f2m, GHSA-6q7j-xr26-3h2c, GHSA-q6rr-fm2g-g5x8). Scriban is not a peripheral dependency here — it evaluates user-authored mapping templates against message payloads, so it is the component most directly exposed to input. The upgrade annotated TryGetValue's nullability, which left SmartArray's override no longer matching (CS8765); the override now declares the same nullability. 90 mapper and Scriban tests pass on the new version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The obsolete Rfc2898DeriveBytes constructor (SYSLIB0060) silently defaults to SHA1, so that is what every password was derived with. Replacing the call means naming an algorithm, and since nothing is deployed yet there is no reason to name the weak one: new passwords are PBKDF2-HMAC-SHA256, 32 bytes, 210,000 iterations, written as $SWHASH$V2$. Roughly 34ms per hash, which is the point of the cost. Two things found while in there, both worse than the deprecation that led here: - Verify compared byte by byte and returned at the first difference, which leaks through timing how much of a hash a guess got right. Now CryptographicOperations.FixedTimeEquals. - A malformed stored hash threw out of the sign-in endpoint. One corrupt row should not stop everyone else signing in, so it now fails to verify. V1 is still verified, so the seeded administrator and any account already in a developer's database keeps working. Nine tests cover that, including the literal seeded hash — this is a change where nothing fails to compile, nothing throws, and everyone silently stops being able to sign in. AESCryptoService is only de-deprecated, not re-secured: deriving key and IV from one PBKDF2 call is byte-for-byte what two successive GetBytes calls produced. Its salt is a hardcoded constant whose bytes spell "Ivan Medvedev" from an old MSDN sample, and its CBC is unauthenticated. Both are now documented where they are, because changing either is a ciphertext migration rather than an edit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The class redeclared a string Id it already had from GlobalAdapterValuesSetCreate (CS0108). Two properties of the same name on one object is not a style problem: a write through a base-typed reference lands in one slot while a serializer reads the other, and which one wins is not obvious from the call site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

HasError(line) and StripTimestamp(line) become IsError and WithoutTimestamp in an extension(string line) block, so the call sites ask something of the line rather than passing it to a helper. Only net10.0 projects can do this. The bus adapters and SampleResidentHandler target net8.0 so published adapters run on the older runtime, and SW.Bitween.Adapters.Shared is linked into them as source, so it is held to the same limit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Rider kept offering "Convert into primary constructor" because nothing recorded that we want it. The repo's .editorconfig held three lines of YAML indentation and no C# section at all, so every C# preference was whatever each IDE defaults to — which is how a suggestion nobody disagreed with stayed unapplied for months. The .editorconfig now states the choices, and IDE0290 is applied across the solution with `dotnet format style`: 166 constructors whose entire body assigned parameters to fields, gone. Net -1,100 lines. Null guards are preserved where they existed (_cache = cache ?? throw ...), so this changes shape, not behaviour. Two things the tool did that needed cleaning up after: - It put every parameter list on one line, up to 232 characters. Headers over 110 are wrapped, and a long base list goes on its own line. - Wrapping DuplicateDocumentFoundException split a string literal across two lines. Caught by the build; repaired. 242 unit and 292 integration tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`private readonly ILogger<RabbitBusHandler> _logger = logger;` is what dotnet format leaves behind: the parameter is already in scope for the whole class body, so the field adds an underscore and nothing else. 305 of them, gone. Kept where the field earns its place — `_options = options.Value` unwraps IOptions, and `_cache = cache ?? throw new ArgumentNullException(...)` is a guard, not a rename. 11 are left alone. In those files the bare parameter name is also declared locally (a `var host =`, or a nested type's own constructor parameter), so rewriting `_host` to `host` would bind to the wrong thing. They are listed by the script that did this rather than guessed at. The blank lines the removed field blocks left behind are tidied in the same pass. 242 unit and 292 integration tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The Sdk had no <Nullable> setting, so 23 warnings were the compiler pointing out that `string?` means nothing in a file where nullable is off. Turning it on replaced those with real findings — mostly in the three JSON converters, which parse configuration a user typed: - All three declared WriteJson/ReadJson non-nullable while JsonConverter<T> declares them nullable, then dereferenced the value they had promised was never null. - A Matcher whose "value", "pattern" or "path" was missing deserialized into a matcher with a null field. That is not a parse error today — it becomes a NullReferenceException later, during retry evaluation, naming nothing. It now fails at deserialization, naming the field and the matcher type. - OneOf/NotOneOf filtered nulls with Where, which the compiler cannot follow; OfType says the same thing in a way it can. - IsNullOrMissing(JToken) tested `token is null` in a body whose signature said that could not happen. NativeAdapters already had nullable enabled and 18 unattended warnings, all of them a misconfigured adapter turning into a NullReferenceException from inside the HTTP or S3 stack rather than a message naming the empty setting. A blank LoginUrl, ContentType, Verb, Url, ClientId or ClientSecret now says so. A login or OAuth2 endpoint that answers with something other than the expected JSON says that, instead of dereferencing null. The S3 receiver's clients, which are null until Initialize() runs, go through one accessor that says so once rather than a bare `!` at each use. Four Sdk properties are now `string?` because they always could be null: a test result's Error, a connection's LastError, a receive attempt's ErrorMessage. 205 CS8618 warnings remain, all of them serializer-populated DTO properties in the public model. Annotating those is a decision about the published contract — whether each field is optional — not a mechanical fix, so they are left visible rather than silenced with a blanket `= null!`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The last 205 CS8618 warnings, decided per property against how each field is actually populated rather than silenced in bulk: - Optional in the contract -> `Type?`, with a comment saying when it is null. Several were already documented as nullable in prose while the type said otherwise — AuditEntryModel.UserDisplayName's own summary says "or null for a change made with no signed-in user", SettingRow.Value says "Always null for secrets", BusGatewayRow.DataSourceName says "Null for an internal gateway". - Always populated -> `= null!`. No runtime change: it compiles to the null that is already assigned, and marks the field as one the server fills in. - Collections -> `= []`, except where null is load-bearing. `required` was the obvious choice for mandatory request fields and is the wrong one: it stops a test constructing an invalid request, which is exactly what a validation test does. Those are `= null!` with a comment instead. Two collections were reverted after `= []` broke a test. SubscriptionConfiguration .Schedules and DocumentCreate.PromotedProperties distinguish null ("the caller said nothing") from empty ("clear them") — the applier's own comment says so, and SetSchedules throws on a Receiving subscription with an empty schedule list. Both are nullable, with the distinction written down. Every other collection was checked for the same hazard; MergeWithOriginal coalesces null to empty, so the adapter property collections are safe. The Sdk now builds with zero warnings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… permission SubscriptionCategories Create, Update and Delete, and Mappers Preview, each took a RequestContext into their constructor and never called EnsurePermission. A viewer could rename the categories every subscription is filed under, delete them, and run the mapper preview — which loads every GlobalAdapterValuesSet in the deployment, where shared configuration lives. Injecting the thing that checks permissions and then not asking it reads exactly like a handler that does check. The compiler could not see it either: an assigned-but-unread private field is not a warning. It only surfaced because these became primary constructor parameters, where an unused one is CS9113. The permissions match the neighbours rather than being invented: this resource's own Search already requires Subscriptions.View, and SaveMapper — the other half of the mapping editor — requires Subscriptions.Edit. Four tests cover it, and they were checked by deleting the guards again: exactly the two whose guard was removed fail. Also removes five dependencies nothing reads, now that CS9113 names them — including AdapterSecretProperties on SaveAlertOverride, whose secret restoration goes through the static AdapterSecretProperties.Merge and never touched the injected instance. CS9113 is now zero. 242 unit and 296 integration tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A relational data source Bitween can read from, write to and look inside, served by a resident adapter process holding a real ADO.NET connection pool. Host plumbing: - DataSource.Placement (Auto/Exclusive/PerNode). A broker queue is consumed by one node; a connection pool must exist on every node that runs work, so reusing the exclusive lease would leave every other replica unable to run the Xchanges that need it. - Subscription.DataSourceId, one per subscription. A bound subscription invokes the data source's running instance rather than renting a pooled one, so the warm pool is the point of contact, and its own adapter properties travel per call. - adapter_state table behind IAdapterStateStore, where a polling receiver's cursor lives. An adapter cannot hold its own progress: it is restarted, and the next instance may be on another node. DataSourceStatement, an entity rather than a JSON field. Statements must live on the connection — a subscription's adapter property values have partner values templated into them, so SQL there is an injection surface — but a field on the data source meant writing a query needed the right that changes credentials. Separating it also buys namespacing, an owner, an audit trail and a usage count, so dead SQL is finally a provable fact. Adapters: SW.Bitween.Adapters.Db.Core carries the command surface, statement allow-list, paging, and the polling receiver; Oracle and PostgreSQL add a driver, a connection string, a catalog query and a capability list. The Oracle adapter is proven against a real Oracle 23 container — REF CURSOR, the data dictionary, and a cursor that survives the process restarting. Migrations for PgSql, MsSql and MySql. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three conflicts, none of them from the database work — they are where the two branches independently changed the same code. XchangeService: this branch had converted it to a primary constructor and routed adapters through IAdapterInvoker; r10 added the mapper-context path for NativeMapper and kept the field-based constructor. Kept both — r10's semantics, this branch's constructor — which meant taking their side of each hunk and reconciling the field names, plus taking NativeAdapterDiscoveryService as a constructor parameter since their code needs it. Startup and BitweenFixture: a union. This branch's three adapter runtimes in registration order (which IS the routing order) plus r10's MappingContextFactory. Verified by r10's own tests: 419 unit tests pass, including the 168 NativeMapper ones that would fail first if the resolution had changed their behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

PostgreSQL, proven against a real PostgreSQL: the same thirteen behaviours Oracle's suite pins, so the shared core is shown to behave identically and the engine differences show up where they should — in the capability list, and in a set-returning function being read with Query where Oracle needs a REF CURSOR through Call. Its own container rather than the fixture's application database, so a migration change cannot break these tests and a bad statement here cannot touch application data. UI: statements are managed on the data source page, for a Relational source only, because a statement is reached only through the connection it runs against — the same reasoning that puts data sources under bus gateways in the nav. Deliberately no new route and no nav entry. The panel answers to data-source-statements.*, not data-sources.edit. That is the point of the entity: writing a query must not need the right that rotates the credentials. Usage is fetched only when a row is expanded, because the answer costs a scan of the connection's subscriptions and a list of forty statements would run forty of them on first paint. Routing was the risk, since handler shape decides the URL. Each of the five is now matched to a proven pair in this codebase AND exercised over real HTTP against a real database: create, list filtered by data source, usage, update, delete, plus the case-insensitive duplicate guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Driven against a live stack: a PostgreSQL warehouse schema (customers, orders, order_lines, a polling outbox, a set-returning function, a procedure, a sequence), the API serving the built UI, and the adapter published to cloud storage so the provider catalogue finds it. The connection test had silently stopped covering statements. It builds its throwaway instance's startup values from DataSource.Properties, and when statements moved out of a property and into rows nothing there composed them — so the prepare-every-statement check quietly verified nothing and still reported success. That check is most of what the button is for. Now composed in Test as well as in the supervisor, and proven by a statement naming a column that does not exist: the test fails with 42703 and names it. Two buttons on the statements panel both read "Add statement" — the header toggle and the form's submit. Ambiguous to a person and indistinguishable to anything automating it; the form now reads Create statement / Save changes. The connection panel explained fencing tokens and exclusive ownership directly under "Held by: every node", which is the opposite of what a relational source does. Both that and the test panel's "the queues its gateways read are inspected, not drained" now say what is true for a pool. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The subscription half of the database work. A Connection picker and a Statement dropdown appear on the Source, Transformation and Delivery stages, but only when that slot's adapter is a relational provider — for anything else there is nothing to choose. The connection is the subscription's, so several stages pointed at one database share one warm pool; the statement and operation are the slot's, so a handler and a receiver can run different statements down the same connection. The dropdown is fed by that connection's statements, and there is no box to type SQL into: adapter property values have partner values templated into them before the adapter sees them, so SQL here would be steerable by ordinary partner data. A statement named here that the connection no longer defines is called out on the spot, because otherwise it surfaces as a failed message somewhere nobody is watching. Two things had to be fixed before any of it could work, and both were the same mistake in different places: describing a RESIDENT adapter by spawning it down the classic stdio path. A resident one dials out, so the probe never answers. - AdapterRequirements failed as "Received null data", which meant a subscription naming a resident adapter could not be saved at all. - The two secret-masking paths failed CLOSED and masked every property, so a subscription's chosen statement came back as "__private__" — invisible to the screen and unmatchable by its own dropdown. Nothing there is a secret: a resident adapter's credentials live on its data source. Both now go through one ResidentAdapters.IsResidentAsync, which reads the published metadata the same way the runtime picks a path. Driven end to end against a real warehouse database: bind, choose, save, reload, retire the statement and watch the warning appear. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The seeded demo data cannot be saved. Every subscription is a Receiving type with no receiver and no schedule, which the update validator refuses — so opening one in the UI and pressing Save fails, and nothing in the seed exercises a database data source at all. That is misleading enough to cost an afternoon, so this writes down what a working set looks like. tools/dev-warehouse.sql builds the shape an ERP integration actually meets: customers, orders, order lines, an outbox to drain, a set-returning function, a stored procedure and a sequence, with sixty orders and thirty unsent notifications. tools/dev-database.md is the rest of it, including the two things that cost the most time. Publish the adapter through ICloudFilesService rather than by copying a zip and a sidecar into the local store: the object's metadata is what AdapterInstaller reads, and a hand-written one reads back inconsistently — the adapter worked, then failed later as "missing 'EntryAssembly'" when a cache expired, which points at nothing useful. And an hourly schedule needs a minute offset, because hours: 1 is rejected as "Invalid hourly schedule". Verified by draining for real: thirty outbox rows became thirty exchanges over two polls, every row marked processed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

BitweenFixture called AddLocalTestsCloudFiles() with no bucket override, so it shared the default bucket with anything else using the local store on the machine — including a developer's running Bitween — and its teardown calls Cleanup(), which deletes the bucket outright. So running the integration suite silently unpublished the dev environment's adapters. The next thing anyone did there failed as "metadata at 'adapters/...' is missing 'EntryAssembly'", which points at the adapter, not at the test run that removed it. It reads as intermittent, because it depends on whether the metadata cache is still warm. The fixture now has its own bucket. Also, a statement holding a bare procedure name — which is what Operation=call takes, and on Oracle the only form that works — was failing TestConnection. The prepare step treats every statement as SQL text, so `release_order` is a syntax error every time and the check failed on a statement that is correct. It is now recognised and reported as "procedure name — existence not checked", which says what was and was not verified rather than quietly passing it. Both found by pressing Test on a real data source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

It sat beneath Bus gateways with a comment explaining why: a data source was only ever reached through one, so "where do these messages come from?" was a gateway's question. That stopped being true when a data source could be a database — a subscription runs statements against one with no gateway involved anywhere. What it describes is a connection to something outside Bitween, which is configuration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…re looked for The test's answer was the fourth box down, so pressing Test meant scrolling past the connection, the live panel and the statement list to find out what happened. It now sits directly under the button that produces it. The add-statement form was below the list, which is fine at zero statements and useless at twenty — the thing someone opened the panel to fill in was off the bottom of the screen. It comes first now, padded to the panel's gutter, which a direct child of Panel does not get on its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…source The DB adapters declared [AdapterKind("datasource")] alone. That kind is what makes one configurable as a connection; the receiver and handler pickers ask the catalog for their own kinds and got nothing back, so the adapter was configurable but not choosable — absent from the Scheduled jobs receiver list, and rendering as an empty select on a subscription whose receiver_id was in fact set. The adapter really is all three: one resident instance polls a table on a schedule and runs a statement on delivery. AdapterKindAttribute is AllowMultiple for exactly this. Also stop a blank select standing in for a configured adapter. Whatever is set is now offered as an option marked "Not in catalog" when the catalog does not list it — otherwise an unpublished adapter makes a wired-up subscription read as unconfigured, and the only way to save the page is to pick something else, silently replacing a working adapter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The form rendered Object.keys(properties), and those come back from the server as a dictionary whose order is whatever the database and the serializer between them produced — so the page asked for a password before the username it belongs to, even though the adapter declares UserName first. Render in the provider descriptor's order instead, which is the adapter's source order: connection, then credentials, then tuning. Properties added by hand are not declared anywhere, so they keep their own order and follow the declared set rather than being interleaved. With no catalog yet the order is left alone, so the fields do not jump when it arrives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two problems, one message. The switch is named BusProvidersEnabled because it predates database sources, and the test told an operator "External bus providers are turned off" when what they were testing was a PostgreSQL connection. Renaming the key would break every deployment that already sets it, so the message now says what the switch gates — resident data source providers — and names the key as the thing to turn on rather than as a description of what they configured. The second problem is that development had to set it out of band. Nothing in any appsettings file turned it on, so a dev run started without the environment variable came up with every data source untestable, failing in a way that reads as a fault in the data source rather than in how the app was launched. It is on in Development now, and still opt-in everywhere else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two things wrong with the panel on a relational source. It was headed "What is on the broker" and closed by saying nothing is consumed, acknowledged or published — broker vocabulary, on a page whose subject is a PostgreSQL connection. Both now follow the source's kind, the way the placement and test copy on this page already do. And it printed the whole answer. A database's catalog is every table, view and routine the role can see — 1,812 lines on the dev warehouse — so the panel buried the settings form under a page of JSON with no way back but the scrollbar. The block is capped and scrolls in place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…d at Discover already answered "what is in there" — with 1,812 lines of JSON on the first real database we pointed it at. Nobody finds a column that way, and the raw panel is now gone for relational sources because two ways to ask the same question, one of them unusable, is one too many. The browser groups by schema, filters by object type, and searches by name. The search goes to the database as nameLike rather than filtering the page in hand: filtering here would quietly search a 200-row window and report "nothing found" about a table that is there. Columns and routine parameters arrive when a row is opened, one object at a time, because the adapter refuses to make them the default and is right to. Which object types appear is the engine's answer, not a list this file keeps — Oracle has packages and PostgreSQL has materialised views, and hardcoding either offers it to the other. Two things it needed that did not exist: - Inspect invoked the command with no argument, so Discover's filters could not be reached at all. DataSourceInspectRequest.Arguments now carries them. It cannot widen what the endpoint does: the allow-list still holds only Describe, Discover and GetStats, and every one of those reads. - "Use in a statement" drafts SQL into the statements panel. That is the plan's statement picker arriving early, and it is what makes this a browser rather than a catalog viewer — a select naming the real columns, a call binding only the parameters the caller supplies, and a limit on anything that returns rows. Also makes Inspect's "not running here" kind-aware. For a broker that is the normal answer on every node but one; for a pooled database connection, held by every node, it means the adapter did not start — opposite reactions, so they no longer share a sentence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Host state is keyed by (adapter, instance, name), and for a data source the instance is the CONNECTION — one resident process serving every subscription pointed at it. The receiver's cursor name was the constant "receive.cursor", so it was one cursor for the whole connection: whichever subscription polled first advanced it, and the rows it took were invisible to the other. Half the rows each, no error, no warning. What hid it is the limitation this is a prerequisite for fixing. One receive statement per data source made the collision look like configuration rather than a bug, so moving those settings to the subscription without this would have turned an ergonomic limit into silent data loss. The name now carries the subscription, which the host stamps on every invocation as __subscriptionId__ — not stripped, unlike __dataSourceId__, because this one is for the adapter to read. The unscoped name is inherited ONCE, by whichever subscription asks first, and a marker records who took it. A plain fallback would have been worse than no migration: every subscription added later would have started life at wherever the original receiver had got to, skipping everything before that. The SDK's state API has no delete, hence a marker rather than removing the old key. The three receiving tests were already order-dependent for this reason and only passed on the order xUnit happened to pick. Each now polls as its own subscription, which is what makes them independent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…iption Six settings sat on the data source, which meant one connection could only ever feed one receiver — and the SQL among them was the last SQL still living in a settings box, with no permission of its own, no audit trail and no usage count. A connection is shared by everything pointed at it, so nothing that varies per reader can live there. What is left divides on one principle: the SQL and the shape of its rows belong to the STATEMENT; the reading policy belongs to the SUBSCRIPTION doing the reading. the polling SQL statement, named by the subscription CursorColumn statement — it describes what the query returns KeyColumn statement — same MarkProcessed subscription, naming a statement ReceiveMode subscription, defaulting to the data source ReceiveBatchSize subscription, defaulting to the data source A named statement supersedes the data source's own receive settings entirely, SQL and row shape together: taking SQL from one place and the cursor column from another is how the two come to disagree. The statement is a NAME and never text. A per-invocation property has {{partner.X}} substituted into it before the adapter sees it, so SQL there would be steerable by ordinary partner data — the same argument that put statements on the data source in the first place. An unknown name is refused, listing what is configured. Three things this needed: - The composed statement set grew a second shape. A statement nothing polls still composes to exactly the string it always did, so upgrading the host ahead of the adapters changes no value they already receive and moves no supervisor fingerprint; only a polled statement takes the object form. - The plan is read from the INVOCATION alone, with Options as an explicit fallback — not through ValueOf, which falls back to startup values. For ReceiveStatement the two mean different things: an invocation's is a name, the data source's legacy setting of that name is raw SQL. Reading through a fallback took the second and looked it up as the first. - The usage reader counted only Statement, so a statement a receiver polled every minute reported as unused — which is precisely the licence to delete it that the count exists to withhold. It now counts all three keys and says which job each does. The old settings still work when set, and are hidden from the form rather than removed, so a receiver configured before the split keeps running untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…at is not there Two reports, both about a field that exists and cannot be found. The cursor and key columns sat behind a text link reading "A receiver polls with this statement" — prose, not a control. It failed exactly as a disclosure does: the subscription bound to the statement warns "set its key column on the connection", and whoever followed that here found a sentence where the field should be. They are always shown now, under a caption saying who they are for. Two optional boxes is the cheaper mistake. The receive mode dropdown offered "The connection's default" whether or not the connection defined one. Taking it on a connection with no default produced a receiver that failed at poll time with "no ReceiveMode" — a fault reported nowhere near the screen that caused it. The option now names the default when there is one and says there is none when there is not, and a subscription left with neither is warned before it is saved. Also warns when a cursor column is set but the SQL never mentions the cursor, which polls from the beginning every time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@key

…lls it The key and cursor columns belong to the statement, and the SQL is why: a statement reading `where id > @cursor order by id` has `id` as its cursor whoever polls it, and a subscription nominating anything else would have the adapter save a value of one type and bind it into a comparison against another. The key column is welded the same way, to the mark-processed statement that binds it as @key — put it on the subscription and a three-way agreement is split across two screens with two of the three on one side. But storing them there was made to mean travelling there. The previous version only warned — "set its key column on the connection" — which sent whoever was configuring a receiver to a second screen to finish a job they had started on the first. The fields are on the receiver now, and the write goes to the statement. So this is the one control on that panel that does not edit the subscription. It saves immediately, to a record other subscriptions share, and both facts are on screen: the caption says who else it affects, and the action is a button rather than a blur, so it is never something that happened while you were looking elsewhere. Warnings read the SAVED statement rather than the typed value, so they do not disappear the moment someone starts typing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@acid

A statement could hold anything until someone pressed Test connection — or, if nobody did, until it failed as a message days later, pointing at a typo made by someone else. A typo belongs to whoever typed it, and the only moment that is true is while they are still looking at the form. The adapter grows ValidateStatement { sql } -> { ok, error, note }: the engine PREPARES it, parsed and planned, never run and never stored. Create and Update call it before writing. Update only when the SQL actually changed, so a rename is not refused because of a table someone dropped last week. Best-effort by construction. The adapter has to be running on this node to answer; a source that is stopped or still starting cannot be asked, and the save then proceeds unchecked. Blocking someone from saving a fix because the connection they are fixing it for is down would be exactly backwards. Two answers are not plain pass/fail: - A bare PROCEDURE NAME passes with a note saying its existence was not checked. Preparing it as text is a syntax error every time, so refusing it would block a statement that is perfectly correct. - The WRONG PLACEHOLDER PREFIX is named rather than left to a column number. ":acid on PostgreSQL" is what a statement copied from an Oracle data source looks like, and the driver calls it "syntax error at position 76" — true, and no help at all. The message now says: write @acid rather than :acid. TestConnection keeps checking every statement, and now reports ALL the failures instead of stopping at the first. Fixing one only to be told about the next, one test at a time, hides that the second is usually the first mistake repeated. The result panel says "1 of 4 statements", tints the failing rows, and keeps the driver's own line breaks — PostgreSQL puts the character offset on its own line, and collapsed into a paragraph it read as part of the sentence before it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

r10 refactored the adapter resources underneath this branch: AdapterStartupValues now owns the native-versus-published fork that six call sites used to write out, AdapterListing owns the listing, ServerlessAdapterDescriber caches and throttles the child processes, and Catalog is a new endpoint that describes a whole kind at once. Ten files conflicted, and two of this branch's fixes had to move rather than merge. The resident-adapter guard moved INTO AdapterStartupValues.Describe. It was three copies, one per caller that had independently got this wrong; r10 gave those callers one implementation to share, so the guard belongs in it. That matters more after the merge than before: Catalog describes every adapter of a kind on each load, so without it a single catalogue request would try to spawn every resident adapter over stdio and wait for each to time out. The Kind-declared discovery moved into AdapterListing, which r10 wrote using only the infolink6.<kind>s. convention. Left as it was, an adapter found by the Kind stamped on it at publish time — every third party's, and this branch's database adapters, which declare receiver and handler — would have vanished from the pickers again. Both SearchVersioned and the new Catalog go through AdapterListing, so this now serves both where it used to serve one. XchangeService merged clean and did not compile: r10's new retry-chain code refers to _dbContext while this side has the primary constructor's dbContext, and the result was a file half in each convention. 472 unit, 390 integration and 312 client tests pass on the result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The failure was raised to the panel header, four rows above the form and — on a connection with a long list of statements — off screen entirely, so a save that was refused looked like a save that did nothing. It sits under the fields now, where the eye already is when the button is pressed, and is no longer also sent to the header: one message in two places reads as two problems. The header keeps the DELETE failure, which has no form to sit under. Success said nothing at all. Creating a statement closes the form, which is its own answer, but saving an edit leaves every field exactly as it was — which is precisely why it needs saying. Both places that save a statement now flash a "Saved" in place of the button for two seconds, which is what the mapping editor already does; there is no toast in this codebase and this is not the change to introduce one in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three commits: the newest attempt of each retry chain, a dashboard panel for chains that are stuck, and a filter for live attempts. No conflicts — they touch the exchange and dashboard screens, which this branch does not. Two files they share with this branch merged on their own and were checked rather than assumed: the subscription studio's model keeps dataSourceId in the source stage's fields, and SubscriptionPage still mounts the data source binding. 472 unit, 392 integration and 312 client tests pass on the result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

coderabbitai · 2026-09-10T13:20:14Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: simplify9/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 55850baa-bc91-4188-8832-456cfb6f0f94

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gitguardian · 2026-09-10T13:20:38Z

⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
37030477	Triggered	Generic Password	`3ca63d1`	SW.Bitween.IntegrationTests/Tests/DataSourceApiTests.cs	View secret
3998894	Triggered	Generic Password	`c42bafb`	SW.Bitween.UnitTests/SecurePasswordHasherTests.cs	View secret
37030478	Triggered	Username Password	`3ca63d1`	SW.Bitween.IntegrationTests/Tests/DataSourceApiTests.cs	View secret
37147666	Triggered	Generic Password	`f12b63c`	SW.Bitween.IntegrationTests/Fixtures/OracleFixture.cs	View secret
37116004	Triggered	Generic Password	`bf7a62b`	SW.Bitween.UnitTests/NativeMapper/DocumentMapperTests.cs	View secret
37147666	Triggered	Generic Password	`e26445b`	SW.Bitween.IntegrationTests/Fixtures/PostgreSqlDbFixture.cs	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

r10 moved by one commit: the merge of PR #299, which is this branch's own earlier work. So the merge changes no file — `git diff --cached` is empty and the tree is identical to the branch head. What it records is ancestry, so the next pull request carries only the three commits since: the MySQL and SQL Server adapters, the per-engine guide, and outbound delivery to a customer's broker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mmalkhatib and others added 30 commits September 6, 2026 04:13

Merge remote-tracking branch 'origin/releases/r10.0' into feature/ext…

7da2da1

…ernal-bus-providers

mmalkhatib and others added 18 commits September 9, 2026 15:29

AhmadRAbuhussein approved these changes Sep 10, 2026

View reviewed changes

AhmadRAbuhussein merged commit d053088 into releases/r10.0 Sep 10, 2026
5 checks passed

mmalkhatib mentioned this pull request Sep 11, 2026

MySQL and SQL Server adapters, outbound delivery to external brokers, and a per-engine guide #300

Merged

Navigation Menu

Uh oh!

External bus providers and database data sources, via resident adapters - #299

External bus providers and database data sources, via resident adapters#299
AhmadRAbuhussein merged 48 commits into
releases/r10.0from
feature/external-bus-providers

mmalkhatib commented Sep 10, 2026

Uh oh!

coderabbitai Bot commented Sep 10, 2026

Uh oh!

gitguardian Bot commented Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mmalkhatib commented Sep 10, 2026