Add RabbitMQ operations guide - #2532
Conversation
📝 WalkthroughAdds a RabbitMQ operations guide covering routine checks, quorum queue leader distribution, missing queue and stream replicas, and recovery considerations. Links the upgrade guidance to the new stream replica repair documentation. WalkthroughAdds a RabbitMQ operations guide to the documentation index. The guide covers routine checks, quorum queue leader balancing, missing queue and stream replicas, and recovery constraints. The upgrade guide now references this documentation for stream replica errors and repair procedures. ChangesRabbitMQ operations guidance
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: 🟡 Moderate · up to The guide currently includes incorrect queue-repair commands and replica checks that could cause operators to target the wrong queues or vhost, or perform unnecessary repairs. It is not merge-ready until these instructions are corrected; the maintenance-window wording should also be fixed. 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f530b752-d53b-4044-a375-0deb68155e90
📒 Files selected for processing (3)
doc/source/operations/index.rstdoc/source/operations/rabbitmq.rstdoc/source/operations/upgrading-openstack.rst
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: stackhpc/check
- GitHub Check: Ansible 2.20 lint with Python 3.12
- GitHub Check: Ansible 2.19 lint with Python 3.12
🔇 Additional comments (2)
doc/source/operations/index.rst (1)
23-23: LGTM!doc/source/operations/upgrading-openstack.rst (1)
49-51: LGTM!
| Quorum queues should have a member on every RabbitMQ node. To find quorum | ||
| queues with fewer members than the cluster has nodes, run the following on a | ||
| controller. Replace ``3`` with the number of RabbitMQ nodes in the cluster: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| sudo docker exec rabbitmq rabbitmqctl list_queues name type members --formatter json | \ | ||
| jq 'map(select(.type == "quorum") | select((.members // []) | length < 3))' | ||
|
|
||
| The output lists the affected queues and their current members. For example, | ||
| a queue with only ``rabbit@ctrl0`` as a member is missing replicas on the | ||
| other nodes. | ||
|
|
||
| An upstream Kolla-Ansible change is intended to add missing quorum queue | ||
| members automatically: `review 973110 | ||
| <https://review.opendev.org/c/openstack/kolla-ansible/+/973110>`__. | ||
| Until that change is available in the deployed Kolla-Ansible version, add the |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Use configured replica targets instead of the cluster-node count.
Both sections treat a replica missing from any RabbitMQ node as a fault. RabbitMQ supports explicit quorum-queue group sizes and stream replica sets that can be smaller than the cluster. (rabbitmq.com)
doc/source/operations/rabbitmq.rst#L60-L76: compare quorum members with the configured group size and qualify the CMR behaviour.doc/source/operations/rabbitmq.rst#L97-L106: repair only the intended stream replicas and avoid unconditional all-node growth.
📍 Affects 1 file
doc/source/operations/rabbitmq.rst#L60-L76(this comment)doc/source/operations/rabbitmq.rst#L97-L106
| sudo docker exec rabbitmq rabbitmqctl list_queues name type members --formatter json | \ | ||
| jq 'map(select(.type == "quorum") | select((.members // []) | length < 3))' |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
set -eu
docker exec rabbitmq rabbitmq-queues help grow
docker exec rabbitmq rabbitmq-queues help add_memberRepository: stackhpc/stackhpc-kayobe-config
Length of output: 213
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- relevant documentation ---'
sed -n '55,105p' doc/source/operations/rabbitmq.rst
printf '%s\n' '--- repository references ---'
rg -n --hidden -S 'rabbitmq-queues|list_queues|add_member|grow' . -g '!*.git'Repository: stackhpc/stackhpc-kayobe-config
Length of output: 11066
🌐 Web query:
RabbitMQ 3.13 rabbitmq-queues man page grow add_member vhost-pattern
💡 Result:
In RabbitMQ 3.13, the rabbitmq-queues command-line tool provides specific commands for managing quorum queue replicas. The grow and add_member commands serve different purposes in managing these replicas [1][2]. grow Command The grow command adds a new replica on a specified node for a set of quorum queues [1][2]. Syntax: rabbitmq-queues grow node selector [--vhost-pattern pattern] [--queue-pattern pattern] [--errors-only] Parameters: node: The target node (e.g., "rabbit@newhost") where the new replica should be added [1][2]. selector: Determines which queues to grow. Supported values include "all" (selects all quorum queues) and "even" (selects quorum queues with an even number of replicas) [2][3]. --vhost-pattern pattern: Optional. Filters the queues by virtual host using a pattern [1][2]. --queue-pattern pattern: Optional. Filters the queues by name using a pattern [1][2]. Example: rabbitmq-queues grow "rabbit@newhost" "all" --vhost-pattern "a-vhost" --queue-pattern ".*" [1][2] add_member Command The add_member command is used to add a quorum queue replica on a specific node for a single, designated queue [1][2]. Syntax: rabbitmq-queues add_member --vhost virtual-host queue node Parameters: --vhost virtual-host: Specifies the virtual host of the queue [1][2]. queue: The name of the quorum queue [1][2]. node: The target node where the replica should be added [1][2]. Example: rabbitmq-queues add_member --vhost "a-vhost" "a-queue" "rabbit@new-node" [1][2]
Citations:
- 1: https://www.rabbitmq.com/docs/3.13/man/rabbitmq-queues.8
- 2: https://github.com/rabbitmq/rabbitmq-website/blob/main/versioned_docs/version-3.13/man/rabbitmq-queues.8.md
- 3: https://manpages.opensuse.org/Tumbleweed/rabbitmq-server/rabbitmq-queues.8.en.html
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- RabbitMQ 3.13 command documentation ---'
curl -fsSL https://raw.githubusercontent.com/rabbitmq/rabbitmq-website/main/versioned_docs/version-3.13/man/rabbitmq-queues.8.md |
sed -n '/## grow/,/## /p;/## add_member/,/## /p' | head -220
printf '%s\n' '--- surrounding repository context ---'
sed -n '1,115p' doc/source/operations/rabbitmq.rstRepository: stackhpc/stackhpc-kayobe-config
Length of output: 4521
🌐 Web query:
site:rabbitmq.com/docs/3.13/man/rabbitmq-queues.8 "add_member" "-p" "--vhost"
💡 Result:
The command add_member is a subcommand of the rabbitmq-queues tool, used to add a quorum queue member (replica) to a specific node in a RabbitMQ cluster [1]. The command usage is: rabbitmq-queues add_member --vhost [virtual-host] [queue] [node] The --vhost flag is used to specify the virtual host (vhost) where the target queue resides [1]. The -p flag is not a standard option for this specific subcommand as documented; the vhost is explicitly defined using the --vhost flag followed by the vhost name [1]. If a command is being used as add_member -p, it is possible that -p is being used as a shorthand for --vhost in some configurations or older versions, but the authoritative documentation uses --vhost [1].
Citations:
🏁 Script executed:
#!/bin/sh
set -eu
url='https://raw.githubusercontent.com/rabbitmq/rabbitmq-website/main/versioned_docs/version-3.13/man/rabbitmq-queues.8.md'
curl -fsSL "$url" |
rg -n -C 8 'add_member|grow|vhost-pattern|queue-pattern|Usage|usage'Repository: stackhpc/stackhpc-kayobe-config
Length of output: 5065
🌐 Web query:
RabbitMQ 3.13 rabbitmq-queues add_member --vhost source usage
💡 Result:
In RabbitMQ 3.13, the rabbitmq-queues add_member command is used to add a replica (member) of a quorum queue to a specific node [1][2]. The basic syntax for the command is: rabbitmq-queues add_member --vhost virtual-host queue-name node Arguments: --vhost virtual-host: Specifies the virtual host (vhost) where the target queue resides [1][3]. queue-name: The name of the quorum queue to which you are adding a member [1][2]. node: The name of the RabbitMQ node (e.g., rabbit@node-name) where the new replica should be hosted [1][2]. Example usage: rabbitmq-queues add_member --vhost "my-vhost" "my-queue" "rabbit@new-node" Alternative syntax: Some documentation also references the use of the -p flag as a shorthand for the vhost parameter [4][5]: rabbitmq-queues add_member -p virtual-host queue-name node Important Considerations: 1. Operational Requirements: You must have a quorum of existing replicas available in the cluster to successfully add a new member, as membership changes are treated as queue state changes [4][6]. 2. Purpose: This command is used to manually manage quorum queue replicas. When a new node is added to a cluster, it does not automatically host quorum queue replicas; they must be explicitly added by the operator using commands like add_member or grow [4][7].
Citations:
- 1: https://www.rabbitmq.com/docs/3.13/man/rabbitmq-queues.8
- 2: https://github.com/rabbitmq/rabbitmq-website/blob/main/versioned_docs/version-3.13/man/rabbitmq-queues.8.md
- 3: https://www.rabbitmq.com/docs/man/rabbitmq-queues.8
- 4: https://www.rabbitmq.com/docs/3.13/quorum-queues
- 5: https://github.com/rabbitmq/rabbitmq-website/blob/main/versioned_docs/version-3.13/quorum-queues/index.md
- 6: https://www.rabbitmq.com/docs/4.2/quorum-queues
- 7: https://www.rabbitmq.com/docs/quorum-queues
Use valid per-queue commands and preserve the vhost scope.
rabbitmq-queues grow accepts a node and a selector, not a queue name. Use add_member --vhost <vhost> <queue-name> rabbit@ctrl0 for one queue. Add -p <vhost> to list_queues and --vhost-pattern <vhost> to the bulk grow command so both operations target the same vhost.
| * Rebalancing changes queue leadership and can create a short period of extra | ||
| RabbitMQ activity. Perform it during an appropriate maintenance window when | ||
| the deployment is busy. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Run leader rebalancing during a quiet maintenance window.
The phrase when the deployment is busy conflicts with the preceding warning about extra RabbitMQ activity. Replace it with when the deployment is quiet or during a low-activity maintenance window.
Proposed wording change
- the deployment is busy.
+ the deployment is quiet.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| * Rebalancing changes queue leadership and can create a short period of extra | |
| RabbitMQ activity. Perform it during an appropriate maintenance window when | |
| the deployment is busy. | |
| * Rebalancing changes queue leadership and can create a short period of extra | |
| RabbitMQ activity. Perform it during an appropriate maintenance window when | |
| the deployment is quiet. |
1 participant
No description provided.