Skip to content

Cannot run against cluster-mode Redis: plain redis::Client does not follow MOVED #5

Description

@fcostaoliveira

The tool cannot connect to a cluster-mode-enabled Redis. It opens a plain redis::Client, which does not follow MOVED redirections, so every command against a slot owned by another node fails.

Verified against a local 3-node cluster (--cluster-replicas 0, cluster_state:ok):

$ resque-bench --host 127.0.0.1 --port 7001 --db 0 --workers 4 --jobs 500 --num-queues 8 \
    --queue '{jobs}default'

  redis://127.0.0.1:7001/0   jobs=500  queues=8 queues ({jobs}default_0…{jobs}default_7)

Error: Pipeline failures: [(Index 0, error: Moved: 9631 127.0.0.1:7002), ...]

redis = { features = ["tokio-comp", "tokio-rustls-comp", "tls-rustls-webpki-roots"] } — no cluster / cluster-async feature, and redis::Client::open is used throughout.

Hash tags alone are not enough

The run above already hash-tags every queue into one slot (9631). It still fails: the client simply has no way to learn that slot 9631 lives on port 7002. Cluster support needs both a cluster-aware connection and same-slot keys — the tags are necessary but not sufficient.

Without tags there is a second, independent failure: a multi-key BRPOP spanning slots is rejected outright.

$ redis-cli -c -p 7001 BRPOP queue:default_0 queue:default_1 queue:default_2 0.1
CROSSSLOT Keys in request don't hash to the same slot

That is correct server behaviour, but it means the default queue naming (queue:default_0…7, which spans 8 slots) can never work on a cluster regardless of the client.

Note on --db

Before v0.1.2 this failed even earlier, because the default URL carries /13 and cluster mode has only db 0:

Error: Redis server refused to switch database - Server(ResponseError): SELECT is not allowed in cluster mode

Fixed by the --db change, but worth recording: on a cluster the tool must resolve to db 0. A future cluster mode should probably reject a non-zero --db with a clear message rather than let the server produce that one.

Why this matters

Blocking-command benchmarks currently cannot cover clustered topologies at all — which is the topology where the interesting behaviour lives, because that is where multi-key BRPOP stops being a single-node operation. Real Sidekiq passes every configured queue to one BRPOP, and kombu expands each queue over 4 priority steps; neither hash-tags, so both are unrunnable on a cluster today. (That limitation is upstream-real too — it is why Sidekiq Pro fell back to polled LMOVE, redis/redis#1785.)

Suggested shape

  1. Add the cluster-async feature to the redis dependency and use redis::cluster_async::ClusterConnection when talking to a cluster.
  2. Detect rather than require a flag where possible — INFO cluster reports cluster_enabled:1, so the tool can pick the right client automatically and keep the CLI unchanged. An explicit --cluster override is still worth having for the ambiguous cases.
  3. Make the queue naming hash-tag aware in cluster mode, so the derived keys land in one slot without the caller having to know to pass --queue '{tag}name'.
  4. Fail fast with a clear message if a cluster endpoint is given a non-zero --db, or a queue set that would span slots.

Happy to send a PR if the shape above looks right — point (2) in particular is a design call I would not want to guess at.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions