Skip to content

MySQL and SQL Server adapters, outbound delivery to external brokers, and a per-engine guide - #300

Merged
samerzughul merged 4 commits into
releases/r10.0from
feature/external-bus-providers
Sep 11, 2026
Merged

MySQL and SQL Server adapters, outbound delivery to external brokers, and a per-engine guide#300
samerzughul merged 4 commits into
releases/r10.0from
feature/external-bus-providers

Conversation

@mmalkhatib

Copy link
Copy Markdown
Contributor

Follows #299. Three things: the two remaining database engines, outbound delivery to a customer's broker, and a guide for a developer meeting one of these databases for the first time.

Along the way it fixes four bugs that were already in releases/r10.0, two of them silent.


MySQL and SQL Server adapters

Each is what the plan predicted — a driver, a connection string, a catalog query and a capability list over the existing core — and the core grew two hooks to fit them.

MySQL (serves MariaDB). A database is the schema, so discovery with no schema means the one this connection opened rather than every database on a shared server. No sequences, no MERGE, no RETURNING, no array type; a procedure returns rows by SELECTing, which is the capability PostgreSQL declares false.

SQL Server (serves Azure SQL). The default schema belongs to the login, not the connection, so a named schema is applied per connection through a new OnConnectionOpenedAsync hook — and not fatally, because impersonating a schema's owner is a privilege many service accounts lack. MERGE, OUTPUT and snapshot isolation are all real here.

Outbound: a delivery publishes to the customer's broker

Publish had existed in both bus adapters since they did, and nothing could reach it — the pipeline calls Handle on a handler and neither class had one. An integration could drain a customer's queue and had no way to answer on it. Both adapters now implement IInfolinkHandler and declare [AdapterKind("handler")].

Adding Handle was the smaller half. A broker data source is exclusive — one node holds the connection so the queue is drained once — while a delivery runs on whichever node picked the message up. Publishing would have failed on every node but the owner.

Exclusivity is about consuming, not connecting. When the owned instance is not on this node, the runtime opens a send-only connection instead: the data source's own settings, Consume=false, no Endpoints, its own pool key. It sends and never subscribes, so nothing is processed twice. Same Consume=false the connection test already uses, for the same reason.

Integration-tested on both brokers, including a test that publishes from a host holding no instance for the data source and asserts ownership was not taken on the way past.

Four bugs this found in r10

  • Oracle was never validating anything. ODP.NET's Prepare() is a client-side no-op — Oracle compiles at execute time — so the shared statement check passed everything, including select 1 from no_such_table. Its connection test's "prepares every statement" was vacuous. Now uses DBMS_SQL.PARSE, with the PL/SQL frames stripped so the answer is the one ORA- line about the operator's SQL.
  • Subscriptions/Get stopped projecting DataSourceId — dropped when an earlier r10 merge took their version of the file, which predates the field. Nothing failed: the API answered null while the row held an id, so the UI read every bound subscription as unbound and saving one from that screen wrote the null back. A test now asserts the read.
  • WriteBackHealthAsync keyed a dictionary by InstanceKey, unique for an exclusive instance and not for a pooled one. Publish-only instances made that routine, and a duplicate key took the whole reconcile pass down.
  • PostgreSQL claimed MERGE on every version. It arrived in 15; 13 and 14 are still widely run. Capabilities can now be narrowed once the server version is known — unit-tested rather than needing a container per release.

Plus: a never-ANALYZEd PostgreSQL table reported 0 rows (reltuples is -1 there, clamped to zero), so the browser said "no rows" about a table that might hold millions. It says unknown now.

UI

  • Brokers appear in the delivery picker with Publish to / Exchange / Routing key, a warning when the endpoint is empty, and a note explaining the send-only connection. Never offered in a receiver — ingress comes through a bus gateway.
  • The connection is subscription-wide, so choosing a broker where another stage needs a database now says so, instead of saving something that fails on its first message.
  • The schema browser generated SQL that did not parse on half the engines — hardcoded @name and limit 100. Oracle binds :name and takes fetch first; SQL Server puts top before the columns. Each adapter reports its own dialect now. It also drafted call proc(@a) for a procedure, which made a statement whose procedure name was that entire string.
  • A statement saved while its adapter was not running was never checked, silently. Both forms now say "Saved, but not checked."
  • The SQL hint no longer names two engines out of four — it reports the prefix the adapter declares.

Docs

  • New: docs/database-adapters-per-engine.md — for a developer who knows SQL but not this database. One section per engine answering the same five questions: what you must set, how to write a statement, what the types arrive as, how to call a procedure, and what the errors mean. Plus receive-mode selection and a minimal grant per engine.
  • external-bus-providers.md gains an Egress section: the properties, why it needed more than a method, and the one thing it cannot do.
  • database-adapters-api.md documents both new engines and both bespoke statement checks.
  • Per-engine dev warehouses (dev-warehouse-{mysql,sqlserver,oracle}.sql), same schema and rows as the PostgreSQL one so the differences you meet are the real ones.

Verified

475 unit · 431 integration (53 new) · 313 client.

End to end on a live two-node stack: ownership split correct (brokers on one node, relational pools on every node), no duplication with both live, failover with the fencing term incrementing (RabbitMQ 29→30, SQS 1→2), and messages flowing after takeover. All four databases polled through the real pipeline with cursors scoped per subscription.

Notes for review

  • A subscription has one dataSourceId, applied to every slot — so one subscription cannot read from a database and publish to a broker. Chain two with ResponseSubscriptionId. The UI now warns rather than letting it save.
  • The SQL Server test image is pinned to 2022; Testcontainers defaults to 2019, which has no arm64 build and exits instantly — surfacing as "container is not running", nothing about architecture.
  • MySQL's fixture needs --log-bin-trust-function-creators=1: creating a function requires SUPER while binary logging is on.
  • Oracle's image is 4.8 GB. Whether CI runs those per PR or nightly is still open.

🤖 Generated with Claude Code

mmalkhatib and others added 4 commits September 11, 2026 08:40
…really checked

Each is what the plan said it would be — a driver, a connection string, a
catalog query and a capability list over the existing core — and the core grew
two hooks to fit them.

MySQL, which serves MariaDB. A database IS the schema, so discovery with no
schema means the one this connection opened rather than every database on a
shared server. No sequences, no MERGE, no RETURNING, no array type; a procedure
returns rows by SELECTing, which is the capability PostgreSQL declares false.

SQL Server, which serves Azure SQL. The default schema belongs to the LOGIN
rather than the connection, so a data source that names one applies it per
connection — through the new OnConnectionOpenedAsync hook, and not fatally,
because impersonating a schema's owner is a privilege many service accounts do
not have. MERGE, OUTPUT and snapshot isolation are all real here.

The second hook is the one that mattered. The shared statement check prepares
the SQL, and two engines cannot be asked that way:

- SqlCommand.Prepare refuses unless every parameter has an explicit type, which
  a caller checking somebody else's SQL does not know. It asks
  sp_describe_undeclared_parameters instead.

- ODP.NET's Prepare is a client-side NO-OP — Oracle compiles a statement when it
  is executed, not when it is prepared. So Oracle has been reporting every
  statement as valid without looking at one, including a select from a table
  that does not exist, and its connection test's "prepares every statement" was
  vacuous. It uses DBMS_SQL.PARSE now, which compiles and resolves names while
  running nothing, and the PL/SQL frames that raises are stripped so the answer
  is the one ORA- line about the operator's SQL.

Two more things the engines made visible:

- The schema browser wrote @name and `limit 100` whatever the connection was.
  That is right for two engines and wrong for the other two — Oracle binds :name
  and takes `fetch first`, SQL Server puts `top` before the column list. The
  adapter reports its own prefix and limit style now, so a draft cannot use a
  syntax the connection will refuse. It also drafted "call proc(@A)" for a
  procedure, which made a statement whose procedure NAME was that whole string.

- A statement saved while its adapter was not running was never checked, and the
  save said nothing — indistinguishable from one that had been verified. Create
  and Update now answer with whether the database actually looked, and both
  forms say "Saved, but not checked" when it did not.

472 unit, 431 integration (35 new, across both engines) and 313 client tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…base

The settings tables said what each field is. They did not say which four you
have to fill in, which defaults to leave alone, how the engine writes a
placeholder or a row limit, what its types arrive as, whether a procedure can
return rows, or what its error messages mean. That is the whole of what someone
needs on the day they point Bitween at a database they have not met.

One section per engine, each answering the same five questions, plus the parts
that are the same everywhere, how to choose a receive mode, and a minimal grant
per engine. The reference links to it and it links back.

Writing it turned up two things that were not true:

- PostgreSQL claimed MERGE whatever the server was. It arrived in 15, and 13 and
  14 are both still widely run — so a capability list that said yes was a
  statement somebody writes and cannot parse. Capabilities can now be narrowed
  once the version is known, which is the only point at which the difference can
  be told, and the rule is unit-tested rather than needing a container per
  release.

- A PostgreSQL table that has never been ANALYZEd reports reltuples as -1, and
  that was clamped to 0 — so the browser said "no rows" about a table that might
  hold millions. Null means unknown, and null is what it says now.

And one thing the doc got wrong before the code was checked: Oracle reports
every NUMBER as decimal, including NUMBER(10,0). The hint is coarse on purpose.

475 unit, 431 integration, 313 client tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Publish had existed in both bus adapters since they did, and nothing could reach
it: Bitween's pipeline calls Handle on a handler, and neither class had one. So
an integration could drain a customer's queue and had no way to answer on it.
Both adapters now implement IInfolinkHandler and declare [AdapterKind("handler")],
so a subscription's delivery stage can target a queue.

Adding Handle was the smaller half. A broker data source is EXCLUSIVE — one node
holds the connection so the queue is drained once — while a delivery runs on
whichever node picked the message up. Publishing would therefore have failed on
every node but the owner, which is to say almost always.

Exclusivity is about consuming, not about connecting. When the owned instance is
not on this node, the runtime opens a send-only connection instead: built from
the data source's own settings, Consume=false, no Endpoints, its own pool key so
it can never be confused with the consuming one. It sends and never subscribes,
so nothing is processed twice. That is the same Consume=false the connection test
already uses, for the same reason.

The rented path also had to carry the call's properties. Startup values are the
connection's; where to publish is the subscription's, and passing only the spec
sent the message to an endpoint the connection had never heard of.

Two bugs this turned up:

- WriteBackHealthAsync keyed a dictionary by InstanceKey, which is unique for an
  exclusive instance and not for a POOLED one. Publish-only instances made that
  routine, and a duplicate key took the whole reconcile pass down. It now looks
  only at instances that ARE a data source.

- Subscriptions/Get stopped projecting DataSourceId — dropped when the r10 merge
  took their version of the file, which predates the field. Nothing failed: the
  API answered null while the row held an id, so the UI read every bound
  subscription as unbound and saving one from that screen wrote the null back.
  A test now asserts the read, because a projection is only as good as the thing
  that notices a field missing from it.

UI: a broker is offered in a delivery (never in a receiver — ingress comes
through a bus gateway), with Publish to / Exchange / Routing key, and a warning
when the endpoint is empty. The connection is subscription-wide, so choosing a
broker where another stage needs a database now says so rather than saving
something that fails on its first message. The SQL hint no longer names two
engines out of four — it reports the prefix the adapter itself declares.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
r10 moved by one commit: the merge of PR #299, which is this branch's own earlier
work. So the merge changes no file — `git diff --cached` is empty and the tree is
identical to the branch head. What it records is ancestry, so the next pull
request carries only the three commits since: the MySQL and SQL Server adapters,
the per-engine guide, and outbound delivery to a customer's broker.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Sep 11, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37147666 Triggered Generic Password 2e3d40b SW.Bitween.IntegrationTests/Fixtures/MySqlDbFixture.cs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository: simplify9/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0dbfffc6-1e42-49ae-8bc4-a07d0dfbc1b1


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@samerzughul
samerzughul merged commit 342c216 into releases/r10.0 Sep 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants