ovsdb-server: non-db-change-aware clients not disconnected after schema conversion when schema version is unchanged (clustered databases)

[halfcrazy]

Summary

When a clustered (raft) database is converted via the convert RPC and the new schema has the same version string as the current one (e.g., converting to identical schema content, or a content-only change without a version bump), ovsdb-server silently destroys all monitors of clients that have not issued set_db_change_aware, without disconnecting them and without sending any notification. The TCP connection stays up, echo requests keep being answered, and the client never receives monitor updates again. There is no way for such a client to detect this state.

This contradicts the documented behavior in ovsdb-server.7.rst (section 4.1.19, Schema Conversion):

If the conversion is successful, the server notifies clients that use the set_db_change_aware RPC introduced in Open vSwitch 2.9 and cancels their outstanding transactions and monitors. The server disconnects other clients, enabling them to notice the change when they reconnect.

Environment

• Multi-node raft cluster (3 nodes), reproduced on recent OVS; code inspection confirms the issue exists on current master (1d4157643, post-4.0.0) and, by inspection, on all versions since clustered databases were introduced.
• Client: libovsdb (github.com/ovn-kubernetes/libovsdb), connected to the cluster leader, using monitor_cond/update3.

Steps to Reproduce

1. Set up a 3-node clustered database (e.g., OVN_Northbound).
2. Connect a client and create a monitor on the database; verify updates are received.
3. From another connection, issue convert with a schema whose content is identical to the current one (or, more generally, any new schema that keeps the same version string):

   ovsdb-client convert tcp:<leader>:6641 /path/to/same-schema.ovsschema
   

4. Make further changes to monitored tables (e.g., via ovs-vsctl/northd or another client).

Expected: per the documentation, the non-change-aware client is disconnected and re-monitors upon reconnect.

Actual: no FIN/RST is sent to the client; echo continues to work; the client's monitors have been destroyed server-side and it receives no further updates and no monitor_canceled notification. Packet capture shows the connection remains healthy while monitor updates stop permanently.

Root Cause Analysis

1. The convert RPC handler does not require the new schema to differ from the current one; it only checks that the schema name matches (ovsdb_trigger_try, ovsdb/trigger.c). A same-content conversion succeeds.
2. The schema change is then unconditionally appended to the raft log: ovsdb_txn_propose_schema_change → ovsdb_storage_write_schema_change → raft_command_execute (ovsdb/storage.c, ovsdb_storage_write_schema_change). Every identical convert produces a new raft entry.
3. When the entry commits, every server applies it via read_db → parse_txn → update_schema (ovsdb/ovsdb-server.c). The reconnect of non-aware clients is gated on the schema version string:

   if (!db->schema || strcmp(schema->version, db->schema->version)) {
       ovsdb_jsonrpc_server_reconnect(config->jsonrpc, false, ...);
   }

   With an unchanged version, no session is disconnected.
4. However, the database replacement is not gated on the version: update_schema proceeds to ovsdb_replace() → ovsdb_monitor_prereplace_db() (ovsdb/monitor.c), which destroys every monitor on the database.
5. The monitor_canceled notification inside ovsdb_jsonrpc_monitor_destroy() is only sent to sessions with db_change_aware set (ovsdb/jsonrpc-server.c). Non-aware sessions get nothing.

Additionally, while investigating we noticed that in multi-node clusters the explicit reconnect_all on conversion commit in the main loop:

if (ovsdb_trigger_run(db->db, time_msec())) {
    /* The message below is currently the only reason to disconnect
     * all clients. */
    ovsdb_jsonrpc_server_reconnect(jsonrpc, false,
        xasprintf("committed %s database schema conversion", ...));
}

(ovsdb/ovsdb-server.c) appears to never fire for clustered databases: ovsdb_trigger_try() only returns true on the synchronous-completion path (standalone/single-node storage); for a multi-node raft proposal the trigger completes via the asynchronous "committing" state, which returns false, and converted_db has already been stolen by read_db (ovsdb_trigger_find_and_steal_converted_db) before ovsdb_trigger_run() executes. So for clustered databases, the version-string comparison in update_schema() is effectively the only mechanism that disconnects non-aware clients on conversion — and it fails when the version is unchanged.

By contrast, the notification path for db-change-aware clients is not gated on the version check, so it fires for every conversion, version change or not. update_schema() unconditionally calls ovsdb_replace() (ovsdb/ovsdb.c), whose first step cancels all monitors on the database:

void
ovsdb_replace(struct ovsdb *dst, struct ovsdb *src)
{
    /* Cancel monitors. */
    ovsdb_monitor_prereplace_db(dst);
    ...
}

ovsdb_monitor_prereplace_db() (ovsdb/monitor.c) destroys every front-end monitor with notify_cancellation = true:

LIST_FOR_EACH_SAFE (m, list_node, &db->monitors) {
    struct jsonrpc_monitor_node *jm;
    LIST_FOR_EACH_SAFE (jm, node, &m->jsonrpc_monitors) {
        ovsdb_jsonrpc_monitor_destroy(jm->jsonrpc_monitor, true);
    }
}

and ovsdb_jsonrpc_monitor_destroy() (ovsdb/jsonrpc-server.c) sends the notification to exactly those sessions that opted into change awareness:

void
ovsdb_jsonrpc_monitor_destroy(struct ovsdb_jsonrpc_monitor *m,
                              bool notify_cancellation)
{
    if (notify_cancellation) {
        struct ovsdb_jsonrpc_session *s = m->session;
        if (jsonrpc_session_is_connected(s->js) && s->db_change_aware) {
            struct jsonrpc_msg *notify = jsonrpc_create_notify(
                "monitor_canceled",
                json_array_create_1(json_clone(m->monitor_id)));
            ovsdb_jsonrpc_session_send(s, notify);
        }
    }
    ...
}

Net effect: on every conversion of a clustered database — with or without a version change — the conversion path always reaches ovsdb_jsonrpc_monitor_destroy(..., /* notify_cancellation = */ true), so a connected db-change-aware client receives one monitor_canceled for each of its monitors on the converted database, with no condition beyond change awareness itself (per ovsdb-server.7.rst section 4.1.7). The client can then recover by re-issuing its monitor requests or reconnecting — this is what the C client library (lib/ovsdb-cs.c, ovsdb_cs_handle_monitor_canceled()) does. A non-aware client, on the other hand, receives neither monitor_canceled (gated on s->db_change_aware above) nor a disconnect (gated on the version comparison in update_schema()), and is left with silently destroyed monitors. The two recovery mechanisms that the documented contract relies on — notification for aware clients, disconnection for the rest — are therefore not symmetric in this case.

Impact

Any long-lived OVSDB client that does not implement set_db_change_aware (the RPC only exists since OVS 2.9, and several client libraries do not implement it) can silently lose its monitors when a database is converted without a version bump — a realistic scenario during tooling-driven schema management or repeated convert invocations with a fixed schema file. Client-side inactivity probes do not help because the connection remains fully functional; only monitor updates stop. The stale cache then silently diverges from the database.

Suggested Fixes

The recovery mechanisms should be symmetric per the documented contract: just as the monitor_canceled notification for aware clients is unconditional (not gated on the version check), the disconnect for non-aware clients should be unconditional too. Any of the following restores that symmetry:

1. Disconnect non-change-aware sessions whenever a conversion replaces the database and destroys monitors — i.e., in update_schema(), fire the reconnect based on the fact that ovsdb_replace() is happening, not on the version string; or
2. Base the update_schema() check on an actual schema/content change instead of the version string, so same-version conversions still trigger the reconnect; or
3. Reject convert requests whose schema is identical to the current one (and document that conversions must bump the version), making the unreachable state explicit.

Note that simply sending monitor_canceled to non-aware clients is not a viable substitute: a client that never issued set_db_change_aware typically does not implement handling for this notification (that is the purpose of the opt-in), so it would ignore the notification and remain stuck. The disconnect path is the only recovery mechanism available to such clients.

Related Work

Client-side mitigation in libovsdb: ovn-kubernetes/libovsdb#456 implements set_db_change_aware + monitor_canceled handling, which protects libovsdb-based clients (the PR description and discussion there include the packet capture showing the healthy-but-silent connection). The server-side behavior for non-aware clients still deviates from the documentation, so a server-side fix is needed as well.