Skip to content

Instantly share code, notes, and snippets.

@MarcialRosales

MarcialRosales/README.md

Last active January 5, 2024 06:37
Show Gist options
  • Save MarcialRosales/8cae6ef203b120ff400c4232ab8980e7 to your computer and use it in GitHub Desktop.
Save MarcialRosales/8cae6ef203b120ff400c4232ab8980e7 to your computer and use it in GitHub Desktop.
Download ZIP
Managing policies in RabbitMQ for PCF
Raw
README.md

Managing policies in RabbitMQ for PCF

This is a quick guide on how to manage RabbitMQ policies in RabbitMQ for PCF. We can make it extensible for any RabbitMQ, not necessarily for PCF, but the example and sample scripts provided in this guide are meant for PCF.

Prefer Policies over x-args

It is highly recommended that we do not pass any arguments (a.k.a x-args) when we declare AMQP queues or exchanges. Although the AMQP protocol allows us, it will not allow us to change those arguments afterwards. Say we declare a queue with a TTL of 60sec. Later on we decide that 60sec was too tight and we want it to be 90sec. Our application will fail to declare the queue with the new setting. There several ways to remedy this situation or on the other hand avoid it entirely.

To remedy the situation we would have to:

  • delete the entire if we dont care about losing the messages and let the application declare it again with the new settings
  • use a naming convention with a version in it (e.g. my-queue-v1) so that when we need to change the queue settings we ramp up the version number. Additionally, we would have to move the messages from the previous version to the new version

As we can see, a big hassle. We can entirely avoid that and use policies instead. We can change the policy definition at any time and RabbitMQ makes sure to set up the corresponding resources to match the policy.

Setting up Policies in Pre-Provision RabbitMQ

In Pre-Provision RabbitMQ, the operator can set up a default policy which will be applied to both, queues and exchanges. It is not possible to use pattern matching with this default policy. Policies will be applied to all queues and exchanges. This may be sufficient for some simple architectures where all queues and exchanges are configured the same way. But as the application architecture evolves clearly it is not enough and we need to set more policies.

The next section is about setting up policies in On-Demand RabbitMQ but it is equally applicable to Pre-Provision too.

Setting up Policies in On-Demand RabbitMQ

In On-Demand RabbitMQ, the operator cannot set a default policy either globally for all service plans neither for a concrete service plan. Instead, it is up to the service instance owner (i.e. the CF space developer who created the service instance) to fully manage the RabbitMQ cluster and this includes defining policies.

The way to set up policies in RabbitMQ is via the Manamangent API (or via the UI if we want to do via a browser). But in order to access the Management API we need a usename and password, in addition to the actual URL.

A space developer has two ways to get the credentials of a RabbitMQ user (at least via the cf-cli)

  • One is via service binding that we invoke it via cf bind-service command or automatically when we push the application if we declared the services block in the application's manifest file
  • Another is via service key that we create via cf create-service-key command

There is a third way however it is not via cf-cli directly but directly via the RabbitMQ management UI.

Both ways creates a RabbitMQ user with the following user tags: management and policymaker. The latter is the key user tag that permits a developer to manage (i.e. create & delete) its policies.

Lets go step by step to manage a set of policies.

1. Create a service instance

First, the space developer creates a service instance named rmq:

> cf create-service p.rabbitmq single-node rmq

2. Gather the RabbitMQ Management API credentials

The space developer, via one of the two mechanisms explained above, gets the RabbitMQ Management API credentials (http_api_uri, vhost, password and username).

Let's say we pushed an application called demo-app and bind it to the rmq service instance:

> mkdir demo-app
> cd demo-app
> touch Staticfile
> touch index.html
> echo "<html><body><h1>Hello world!</h1></body></html>" > index.html
> cd .. 
> cf push demo-app -p demo-app
> cf bind-service demo-app rmq

We can get the credentials for the user assigned to the application via:

> cf env demo-app

It prints out something like this:

...

System-Provided:
VCAP_SERVICES: {
 "p.rabbitmq": [
  {
    "credentials": {
      ...
      "http_api_uri":     "https://a9b37966-d67a-448f-b93d-744e3dc95890:hWD38Vvc9wdQAXG8Lh_AAjRI@rmq-a1d3c836-c670-4d22-9191-3e497b68a885.sys.philippinepink.cf-app.com/api/",  
      "password": "xxxxxx",
      "username": "a9b37966-d67a-448f-b93d-xxxxx",
      "vhost": "a1d3c836-c670-4d22-9191-3e497b68a885"
      ...
    }
  }
}
...

3. Declare RabbitMQ Policy via the Management API

We have, at least, 2 choices to declare policies via the management api:

  • One option is to use curl or other http client tool to send HTTP requests directly to the http_api_uri we gathered from the application or service-key. This is the option we implemented below
  • A second option is to use rabbitmqadmin that we can easily download from almost the same http_api_uri. We need to remove the api/ and replace it with cli/rabbitmqadmin. This tool though requires Python installed.

We would proceed as follows. We have created 2 scripts, set-policy and list-policies which rely on cf, curl and jq commands. You must have these commands installed.

set-policy script accepts as the first parameter the application's name from where to obtain the RabbitMq credentials. In our example, it would be demo-app. The second parameter is the name of the policy. The third parameter is the policy pattern and the last parameter is the definition. We can use this script to create a new policy and/or modify it.

TL;DR: The script assumes the application is bound to a single RabbitMQ Service instance.

Set up a HA policy

All queues whose name start with ha. will be mirrored with 1 replica and with automatic synchronization.

./set-policy demo-app ha "^ha\." '{"ha-mode":"exactly", "ha-params":2, "ha-sync-mode":"automatic"}'

Running the command ./list-policies demo-app we can see the policy we just created:

[
  {
    "vhost": "a1d3c836-c670-4d22-9191-3e497b68a885",
    "name": "test",
    "pattern": "^ha\\.",
    "apply-to": "all",
    "definition": {
      "ha-mode": "exactly",
      "ha-params": 2,
      "ha-sync-mode": "automatic"
    },
    "priority": 0
  }
]

POLICY PRIORITY: The set-policy script did not give us the chance to set up policy's priority. If we do not have overlapping policies then we do not need to explicitly set up the priority. RabbitMQ will assign priority 0 as we can see in the outcome of list-policies command. However, if two or more policies match the same queue and/or exchange, we have overlapping policies and RabbitMQ uses their priorities to determine which one to apply.

Set up a HA plus DLX policy

All queues named as ha-dlx.* will be mirrored with 1 replica, with automatic synchronization and with a dead-letter exchange. Notice that if we have to combine multiple features like in this case, ha and dlx, we have to create a policy that combines both.

./set-policy demo-app ha-dlx "^ha-dlx\." '{"ha-mode":"exactly", "ha-params":2, "ha-sync-mode":"automatic",  "dead-letter-exchange": "ha-dlx" }'

Set up a policy for temporal queues

All queues named as temp.* will have a maximum length and message ttl. When the queue is full, the broker rejects further publish attempts. If the publisher uses Publisher confirmation, it will get a nack.

./set-policy demo-app ha "^temp\." '{"max-length": 2, "overflow": "reject-publish", "message-ttl": 60000}'

Set up a policy for a concrete queue

A queue named event-log which must be configured as lazy queue.

./set-policy demo-app ha "^event-log$" '{"queue-mode":"lazy"}'

Setting policies from within the application

We started this guide with a recommendation whereby we should not programmatically declare the queues and exchanges with arguments (x-args) but instead via policies. Effectively, we moved configuration from the application code to somewhere else, outside of the application.

As we already know policies allows us to configure multiple queues/exchanges all at once, with a single definition, provided those resources stick to certain naming convention, a.k.a. policy pattern. It could be very daunting having to change the name of our queues and exchanges to follow certain pattern so that we can leverage the convenience offered by the policy pattern.

If you want to keep the queue/exchange names as they are you can still create a policy per resource whose policy pattern matches exactly the name of the resource as we did earlier.

And now comes the question whether we should programmatically setup the policy from within the application or else from outside as we just did it earlier.

Set up policies via application code

First, our application would have to read the http credentials from VCAP_SERVICES. Then, for each queue/exchange that we need to configure we use a http client library to send a POST request to set up the corresponding policy. We can do it before or after we declare the resource.

There is a Java Binding for the RabbitMQ management api that will make our job far easier compared to using a raw http client library.

Set up policies externally and declaratively

There could be two approaches:

  • One aproach is that each application defines its own policies
  • A second approach is where we define policies globally. In other words, we define a set of policies that follows certain naming convention and applications must adhere to such naming convention. For instance, queues named as ha.* will be configured with mirrored queues.

If we go with the first option, each application would declare its policies as configuration in one or many json files and the pipeline that deploys the application is responsible for declaring the policies in RabbitMQ.

Enable RabbitMQ features via policies

Here are all the features we can enable via Policies in RabbitMQ. The sample command to enable each feature uses the standard cURL command rather than the set-policy script we used in the previous sections.

  • Make queue highly available

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/ha -p
'{ "pattern":"^ha\.", "definition":{"ha-mode":"exactly", "ha-params":2, "ha-sync-mode":"automatic"}}'

  • Define queue lazy

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/event-log -p
'{ "pattern":"^event-log$", "definition":{"queue-mode":"lazy"}}'

    or to revert it back to a normal queue

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/event-log -p
'{ "pattern":"^event-log$", "definition":{"queue-mode":"default"}}'

  • Limit queue to a maximum length and reject further publish

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/temp -p
'{ "pattern":"^temp\.", "definition":{"max-length": 2, "overflow": "reject-publish"}}'

  • Define a queue with a dead-letter-exchange

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/ha-dlx -p
'{ "pattern":"^ha\.dlx\.", "definition":{ "dead-letter-exchange": "ha-dlx" }}'

    For brevity we have removed the ha definition. ha-dlx was meant to be a policy which combined both features, ha and dlx.

  • Define message TTL for queues

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/temp -p
'{ "pattern":"^temp\.", "definition":{"message-ttl"": 60000 }}'

  • Delete queue after N seconds

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/temp -p
'{ "pattern":"^temp\.", "definition":{"expires"": 1800000 }}'

  • Define an alternative route when an exchange cannot find a route/binding for a message

    curl -u <username>:<password> -X PUT <http_api_uri>/policies/<vhost>/critical -p
'{ "pattern":"\.critical\.", "definition":{"alternate-exchange": "ae" }}'
Raw
list-policies
#!/usr/bin/env bash
set -e
APP_NAME=${1:?First parameter must be the application from where to obtain the credentials}
CURL_FLAGS=${2:-"--insecure"}
which jq &> /dev/null || ( echo "Install jq"; exit 1)
APP_GUID="$(cf app $APP_NAME --guid)"
rmq_si=$(cf curl /v2/apps/$APP_GUID/env | jq '.system_env_json.VCAP_SERVICES | .["p.rabbitmq"][0]')
instance_name=$(echo $rmq_si | jq -r .name)
http_api_uri=$(echo $rmq_si | jq -r .credentials.http_api_uri)
username=$(echo $rmq_si | jq -r .credentials.username)
password=$(echo $rmq_si | jq -r .credentials.password)
vhost=$(echo $rmq_si | jq -r .credentials.vhost)
curl -s $CURL_FLAGS \
-u $username:$password \
${http_api_uri}policies/$vhost | jq .
Raw
set-policy
#!/usr/bin/env bash
set -e
APP_NAME=${1:?First parameter must be the application from where to obtain the credentials}
POLICY_NAME=${2:?Second parameter must be the policy name}
POLICY_PATTERN=${3:?Third parameter must be the policy pattern}
POLICY_DEFINITION=${4:?Forth parameter must be the policy definition}
CURL_FLAGS=${5:-"--insecure"}
which jq &> /dev/null || ( echo "Install jq"; exit 1)
APP_GUID="$(cf app $APP_NAME --guid)"
rmq_si=$(cf curl /v2/apps/$APP_GUID/env | jq '.system_env_json.VCAP_SERVICES | .["p.rabbitmq"][0]')
instance_name=$(echo $rmq_si | jq -r .name)
http_api_uri=$(echo $rmq_si | jq -r .credentials.http_api_uri)
username=$(echo $rmq_si | jq -r .credentials.username)
password=$(echo $rmq_si | jq -r .credentials.password)
vhost=$(echo $rmq_si | jq -r .credentials.vhost)
DEFINITION=$(cat <<EOF
{
"pattern": "$POLICY_PATTERN",
"definition": $POLICY_DEFINITION
}
EOF
)
curl $CURL_FLAGS \
-u $username:$password \
-H "Content-Type:application/json" \
-X PUT --data "$DEFINITION" \
${http_api_uri}policies/$vhost/$POLICY_NAME
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment