Skip to content

Instantly share code, notes, and snippets.

@ruhnet

ruhnet/KazooCentos7DigitalOcean.md

Last active August 23, 2024 01:04
Show Gist options
  • Save ruhnet/d1b1a5b02498ec36504f8fcead81b50e to your computer and use it in GitHub Desktop.
Save ruhnet/d1b1a5b02498ec36504f8fcead81b50e to your computer and use it in GitHub Desktop.
Download ZIP
Kazoo Centos 7 Cluster on Digital Ocean
Raw
KazooCentos7DigitalOcean.md

Kazoo Cluster on CentOS 7 ON DIGITALOCEAN

This file provided by "fink" in #2600hz on Freenode

Prerequisites

Set Hostname

Ensure the hostname is set appropriately. Modify as required and reboot, if needed.

Environment Variables

Set the following environment variables in a Bash shell:

PUBLIC_IP=$(dig @ns1.google.com TXT o-o.myaddr.l.google.com +short -4 2> /dev/null | sed s/\"//g)
PRIVATE_IP=$(ip addr show eth0 | grep -m 1 'inet ' | awk '{print $2}' | cut -f1 -d'/')
HOSTNAME=$(hostname)
RELEASE_BASE=https://packages.2600hz.com/centos/7/stable/2600hz-release
RELEASE_VER=4.3
META_PKG=2600hz-release-${RELEASE_VER}-0.el7.centos.noarch.rpm
LATEST_RELEASE=${RELEASE_BASE}/${RELEASE_VER}/${META_PKG}

Note: Replace eth0 with the appropriate interface name.

Upgrade / Install Packages

Install prerequisites:

yum -y upgrade
yum -y install yum-utils psmisc nano epel-release ${LATEST_RELEASE}

Update packages:

yum -y update
yum clean all

Configure DNS

Ensure DNS for the nodes is configured appropriately. Check that DNS records work before proceeding!

Configure Firewall

Self-explanatory.

Generate or Set Erlang Cookie

The following will generate an Erlang cookie. Note: The cookie must be the same across all nodes! Do not generate a new cookie for each node!

COOKIE=`date +%s | sha256sum | base64 | head -c 32`
echo COOKIE=$COOKIE

If you're adding a node to an existing cluster, obtain the existing cookie from a node using the following:

echo COOKIE=`cat /opt/kazoo/.erlang.cookie`

Set the existing cookie on the node that you'll be installing software on.

Verify Hostname / IP / Cookie

Verify that the hostname and IP are correct for the installation.

echo "Kazoo $RELEASE_VER / $HOSTNAME ($PRIVATE_IP / $PUBLIC_IP) (Cookie: $COOKIE)"

Hosts File

echo "${PRIVATE_IP} ${HOSTNAME} `hostname -s`" >> /etc/hosts
echo "127.0.0.1 ${HOSTNAME} `hostname -s`" >> /etc/hosts

Timezone

Set system timezone to UTC.

ln -fs /usr/share/zoneinfo/UTC /etc/localtime

NTP

Install NTP

yum -y install ntp
systemctl enable ntpd
systemctl restart ntpd

CouchDB

Background

CouchDB must be installed as a cluster before installing Kazoo.

Repo File

Add a repo file for CouchDB.

cat > /etc/yum.repos.d/bintray-apache-couchdb-rpm.repo << EOF
[bintray--apache-couchdb-rpm]
name=bintray--apache-couchdb-rpm
baseurl=http://apache.bintray.com/couchdb-rpm/el7/\$basearch/
gpgcheck=0
repo_gpgcheck=0
enabled=1
EOF

Install CouchDB

yum -y install couchdb

Configure CouchDB

Clustering

Modify CouchDB to listen on all interfaces (0.0.0.0).

sed -i "s/127\.0\.0\.1/0\.0\.0\.0/g" /opt/couchdb/etc/default.ini

Erlang VM

sed -i "s/127\.0\.0\.1/$HOSTNAME/g" /opt/couchdb/etc/vm.args
sed -i "s/monster/$COOKIE/g" /opt/couchdb/etc/vm.args

Start CouchDB

systemctl enable couchdb
systemctl start couchdb

Check CouchDB Status

curl -sX GET http://localhost:5984

The following should be returned:

{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"8c293bccf2265f7248fa1ed120326842","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

Create User Database

curl -X PUT http://localhost:5984/_users

The following should be returned:

{"ok":true}

Fauxton

Create Admin Users
echo "Create Fauxton Admin: http://$PUBLIC_IP:5984/_utils/#createAdmin"

Follow the above link to create a Fauxton (CouchDB's web-based administration tool) admin user. Create an admin user for yourself and one for Kazoo with a username of "kazoo". Generate a password for the "kazoo" user:

echo `date +%s | sha256sum | base64 | head -c 32`
Verify Admin User

Verify the kazoo admin user created above works.

curl -sX GET http://localhost:5984 --user kazoo

The following should be returned:

Enter host password for user 'kazoo': {"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"8c293bccf2265f7248fa1ed120326842","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
Configure Clustering

Open the following link in a browser:

http://host.name:5984/_utils/#/setup

Enter the information required as appropriate and click on "Add Node" then "Configure Cluster".

Note: Clustering only needs to be configured on one node. Ensure you click "Add Node" when configuring clustering, otherwise clustering will be enabled without any nodes.

Verify Clustering
curl -X GET "http://localhost:5984/_membership" --user kazoo

The following should be returned:

Enter host password for user 'kazoo': 
{"all_nodes":["[email protected]","[email protected]"],"cluster_nodes":["[email protected]","[email protected]"]}

HAProxy

Install Kazoo-wrapped HAProxy

yum -y install kazoo-haproxy

Configure HAProxy

Import Configuration

cat > /etc/kazoo/haproxy/haproxy.cfg << EOF
global
    log /dev/log local0 info
    maxconn 4096
    user haproxy
    group daemon
    stats socket /var/run/haproxy/haproxy.sock mode 777

defaults
    log global
    mode http
    option httplog
    option dontlognull
    option log-health-checks
    option redispatch
    option httpchk GET /
    option allbackups
    option http-server-close
    maxconn 2000
    retries 3
    timeout connect 6000ms
    timeout client 12000ms
    timeout server 12000ms

listen couchdb-data 0.0.0.0:15984
    balance roundrobin
        server <hostname1> <ip address1>:5984 check
        server <hostname2> <ip address2>:5984 check
        ...etc

listen haproxy-stats 0.0.0.0:22002
    mode http
    stats uri /
EOF

Note: Change the order of the roundrobin servers on the different nodes. The local node should always be listed first.

Remove Old Configuration File

rm -f /etc/haproxy/haproxy.cfg
ln -s /etc/kazoo/haproxy/haproxy.cfg /etc/haproxy/

Start HAProxy

systemctl enable kazoo-haproxy
systemctl start kazoo-haproxy

Ensure HAProxy Started

kazoo-haproxy status

The following should be returned:

|Host                      |Backend         |Status |Active |Rate   |1xx    |2xx    |3xx    |4xx    |5xx    |Ping   |
|sbc01.us-east-1.kazoo.tel |couchdb-data   |UP     |0      |0      |0      |0      |0      |0      |0      |138ms  |
|sbc02.us-west-1.kazoo.tel |couchdb-data   |UP     |0      |0      |0      |0      |0      |0      |0      |1ms    |

Ensure HAProxy Works

curl localhost:15984

The following should be returned:

{"couchdb":"Welcome","version":"2.3.0","git_sha":"07ea0c7","uuid":"8c293bccf2265f7248fa1ed120326842","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

curl localhost:15986

The following should be returned:

{"couchdb":"Welcome","uuid":"8c293bccf2265f7248fa1ed120326842","version":"2.3.0","vendor":{"name":"The Apache Software Foundation"}}

RabbitMQ

Install Kazoo-wrapped RabbitMQ

yum install -y kazoo-rabbitmq

Start RabbitMQ

systemctl enable kazoo-rabbitmq
systemctl start kazoo-rabbitmq

Ensure RabbitMQ Started

ss -lpn | grep 5672

The following should be returned:

tcp    LISTEN     0      128       *:5672                  *:*                   users:(("beam.smp",pid=1534,fd=16))
tcp    LISTEN     0      128       *:25672                 *:*                   users:(("beam.smp",pid=1534,fd=8))
tcp    LISTEN     0      128       *:15672                 *:*                   users:(("beam.smp",pid=1534,fd=17))

Ensure RabbitMQ Works

curl -i -u guest:guest http://localhost:15672/api/aliveness-test/%2F

The following should be returned:

HTTP/1.1 200 OK
cache-control: no-cache
content-length: 15
content-security-policy: default-src 'self'
content-type: application/json
date: Tue, 03 Dec 2019 01:17:39 GMT
server: Cowboy
vary: accept, accept-encoding, origin

{"status":"ok"}

Check RabbitMQ Broker Status

kazoo-rabbitmq status

The following should be returned:

Status of node 'kazoo-rabbitmq@sbc01' ...
[{pid,1691},
 {running_applications,
     [{rabbitmq_management,"RabbitMQ Management Console","3.7.19"},
      {rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.7.19"},
      {rabbitmq_consistent_hash_exchange,"Consistent Hash Exchange Type",
          "3.7.19"},
      {rabbitmq_management_agent,"RabbitMQ Management Agent","3.7.19"},
...

Kamailio

Install Kazoo-wrapped Kamailio

yum install -y kazoo-kamailio

Configure Kamailio

Configure Syslog

cat > /etc/rsyslog.d/11-kamailio.conf << EOF
:syslogtag, isequal, "[KAMAILIO]" /var/log/kamailio/kamailio.log
& stop
EOF
systemctl restart rsyslog

Update Hostname

sed -i "s/# # #!substdef \"!MY_HOSTNAME/#!substdef \"!MY_HOSTNAME/g" /etc/kazoo/kamailio/local.cfg
sed -i "s/kamailio\.2600hz\.com/${HOSTNAME}/g" /etc/kazoo/kamailio/local.cfg

Set / Advertise IP Addresses

sed -i "s/# # #!substdef \"!MY_IP_ADDRESS!127.0.0.1!g\"/#!substdef \"!MY_IP_ADDRESS!${PUBLIC_IP}!g\"/" /etc/kazoo/kamailio/local.cfg
sed -i "s/listen=UDP_SIP/listen=udp:${PRIVATE_IP}:5060 advertise ${PUBLIC_IP}:5060/g" /etc/kazoo/kamailio/local.cfg
sed -i "s/listen=TCP_SIP/listen=tcp:${PRIVATE_IP}:5060 advertise ${PUBLIC_IP}:5060/g" /etc/kazoo/kamailio/local.cfg
sed -i "s/listen=UDP_ALG_SIP/listen=udp:${PRIVATE_IP}:7000 advertise ${PUBLIC_IP}:5060/g" /etc/kazoo/kamailio/local.cfg
sed -i "s/listen=TCP_ALG_SIP/listen=tcp:${PRIVATE_IP}:7000 advertise ${PUBLIC_IP}:5060/g" /etc/kazoo/kamailio/local.cfg

Set Websocket Hostname

sed -i "s/##   #!substdef \"!MY_WEBSOCKET_DOMAIN!2600hz\.com!g\"/#!substdef \"!MY_WEBSOCKET_DOMAIN!kazoo\.tel!g\"/" /etc/kazoo/kamailio/local.cfg

Add Primary RabbitMQ Server

nano /etc/kazoo/kamailio/local.cfg

Uncomment out the following:

#!substdef "!MY_AMQP_URL!amqp://guest:[email protected]:5672!g"

Note: Replace 127.0.0.1 with the appropriate RabbitMQ IP address.

Add the following:

#!substdef "!MY_AMQP_ZONE!zone-name-here!g"

(https://forums.2600hz.com/forums/topic/9889-blf-across-zones)

Add Secondary RabbitMQ Server

nano /etc/kazoo/kamailio/local.cfg

Add the following information for each zone with a RabbitMQ server (note the addition of the "zone=" part in the middle):

#!substdef "!MY_AMQP_SECONDARY_URL!zone=us-west-1;kazoo://guest:[email protected]:5672!g"

Note: This must be configured for each zone! MY_AMQP_URL should always reflect the local RabbitMQ server and MY_AMQP_SECONDARY_URL should always reflect the remote RabbitMQ server.

Note: Replace 127.0.0.1 with the appropriate RabbitMQ IP address.

Start Kamailio

systemctl enable kazoo-kamailio
systemctl start kazoo-kamailio

Ensure Kamailio Started

ss -ln | egrep "5060|7000"

The following should be returned:

udp    UNCONN     0      0         *:5060                  *:*                  
udp    UNCONN     0      0         *:7000                  *:*                  
tcp    LISTEN     0      128       *:5060                  *:*                  
tcp    LISTEN     0      128       *:7000                  *:*

Check Kamailio Status

kazoo-kamailio status

The following should be returned:

error: 500 - No Destination Sets

Note: This error is expected as FreeSWITCH servers haven't been configured.

FreeSWITCH

Install Kazoo-wrapped FreeSWITCH

yum install -y kazoo-freeswitch

Configure FreeSWITCH

Set Cookie

sed -i "s/change_me/$COOKIE/g" /etc/kazoo/freeswitch/autoload_configs/kazoo.conf.xml

Change Local Network ACL

sed -i "s/localnet.auto/whatever/g" /etc/kazoo/freeswitch/sip_profiles/sipinterface_1.xml

Configure IPs

For DigitalOcean, FreeSWITCH must advertise the PUBLIC_UP for ext-sip-ip and ext-rtp-ip.

sed -i "s/<param name=\"ext-sip-ip\" value=\"auto\"\/>/<param name=\"ext-sip-ip\" value=\"${PUBLIC_IP}\"\/>/g" /etc/kazoo/freeswitch/sip_profiles/sipinterface_1.xml
sed -i "s/<param name=\"ext-rtp-ip\" value=\"auto\"\/>/<param name=\"ext-rtp-ip\" value=\"${PUBLIC_IP}\"\/>/g" /etc/kazoo/freeswitch/sip_profiles/sipinterface_1.xml

Start FreeSWITCH

systemctl enable kazoo-freeswitch
systemctl start kazoo-freeswitch

Check FreeSWITCH status

kazoo-freeswitch status

The following should be returned:

UP 0 years, 0 days, 0 hours, 0 minutes, 0 seconds, 243 milliseconds, 276 microseconds
FreeSWITCH (Version 1.6.20  64bit) is ready
0 session(s) since startup
0 session(s) - peak 0, last 5min 0 
0 session(s) per Sec out of max 200, peak 0, last 5min 0 
5000 session(s) max
min idle cpu 0.00/99.93
Current Stack Size/Max 244K/244K

Running mod_kazoo v1.4.0-1
Listening for new Erlang connections on 0.0.0.0:8031 with cookie XXXXXXXXXX
Registered as Erlang node [email protected], visible as freeswitch
No erlang nodes connected

Note: You will not see any connected Erlang modules.

Install Sipify Script

curl -o /usr/bin/sipify.sh https://raw.githubusercontent.com/2600hz/community-scripts/master/FreeSWITCH/sipify.sh
chmod 755 /usr/bin/sipify.sh

Note: This script is used for FreeSWITCH log parsing and is optional.

Kazoo Applications

Install Kazoo Applications

Note: This should be done on a dedicated application server or on an SBC. This should not be installed on a server running FreeSWITCH.

yum install -y kazoo-applications

Allow Kazoo Sudo Access

usermod -aG wheel kazoo

Configure Kazoo Applications

Core Configuration

Note: The Kazoo core configuration must be the same on all nodes.

cat > /etc/kazoo/core/config.ini << EOF 
[zone]
name = "us-east-1"
amqp_uri = "amqp://guest:guest@<ip address>"

[zone]
name = "us-west-1"
amqp_uri = "amqp://guest:guest@<ip address>"

[bigcouch]
compact_automatically = true
cookie = <cookie>
ip = "X.X.X.X"
port = 15984
username = "kazoo"
password = "<password>"
admin_port = 15986

[kazoo_apps]
cookie = <cookie>
zone = "us-east-1"
host = "sbc01.us-east-3.kazoo.tel"

[kazoo_apps]
cookie = <cookie>
zone = "us-west-1"
host = "sbc02.us-west-2.kazoo.tel"

[ecallmgr]
cookie = <cookie>
zone = "us-east-1"
host = "sbc01.us-east-3.kazoo.tel"

[ecallmgr]
cookie = <cookie>
zone = "us-west-1"
host = "sbc02.us-west-2.kazoo.tel"

[log]
syslog = info
console = notice
file = error
EOF

Configure Cookie

echo $COOKIE > /opt/kazoo/.erlang.cookie
chown kazoo:daemon /opt/kazoo/.erlang.cookie
chmod 400 /opt/kazoo/.erlang.cookie

Note: The Kazoo Erlang cookie must be the same on all nodes.

Start Kazoo Applications

systemctl enable kazoo-applications
systemctl start kazoo-applications

Check Databases

curl -s localhost:5984/_all_dbs

The following should be returned:

["_global_changes","_replicator","_users","accounts","acdc","alerts","anonymous_cdrs","dedicated_ips","faxes","global_provisioner","numbers/inum","numbers/managed","oauth","offnet","pending_notifications","port_requests","ratedeck","services","sip_auth","system_auth","system_config","system_data","system_media","system_schemas","tasks","token_auth","webhooks"]

Note: You should have at least 26 databases. You can check the number of databases returned with the following command:

curl -s localhost:15984/_all_dbs | python -mjson.tool | wc -l

Import System Media Prompts

sup kazoo_media_maintenance import_prompts /opt/kazoo/sounds/en/us/

Note: Importing system media prompts only needs to be performed in one region.

Refresh Databases

sup kapps_maintenance refresh

Note: The database refresh only needs to be performed in one region.

Check the Kazoo Cluster

kazoo-applications status

The following should be returned:

Node          : [email protected]
md5           : ufHyVeljJWt9aLE3-pNgaQ
Version       : 4.3.22 - 19
Memory Usage  : 84.41MB
Processes     : 1965
Ports         : 21
Zone          : us-east (local)
Broker        : amqp://1.2.3.4
Globals       : local (4)
Node Info     : kz_amqp_pool: 150/0/0 (ready)
WhApps        : blackhole(13m2s)         callflow(12m26s)         cdr(12m25s)              conference(12m24s)       
                crossbar(12m23s)         fax(12m15s)              hangups(11m45s)          media_mgr(11m45s)        
                milliwatt(11m45s)        omnipresence(11m45s)     pivot(11m45s)            registrar(11m44s)        
                reorder(11m44s)          stepswitch(11m42s)       sysconf(13m6s)           tasks(11m40s)            
                teletype(11m35s)         trunkstore(9m47s)        webhooks(9m46s)          

Node          : [email protected]
Version       : 5.1.5
Memory Usage  : 15.31MB
Zone          : us-east (local)
Broker        : amqp://1.2.3.4
WhApps        : kamailio(16m32s)         
Roles         : Dispatcher Presence Proxy Registrar 
Subscribers   : 
Subscriptions : 
Presentities  : presence (0)  dialog (0)  message-summary (0)  
Listening on  : 1.2.3.4 tcp (5060 7000) udp (5060 7000) 
Registrations : 0

...

ok

Kazoo ecallmgr

Start ecallmgr

systemctl enable kazoo-ecallmgr
systemctl start kazoo-ecallmgr

Check ecallmgr Status

kazoo-ecallmgr status

The following should be returned:

...
Node          : [email protected]
md5           : 1Crrwj3S0XDtlFSy2m29hA
Version       : 4.3.22 - 19
Memory Usage  : 56.69MB
Processes     : 1319
Ports         : 32
Zone          : us-east (local)
Broker        : amqp://1.2.3.4
Globals       : total (0)
Node Info     : kz_amqp_pool: 150/0/0 (ready)
WhApps        : ecallmgr(20s)            
Channels      : 0
Conferences   : 0
Registrations : 0
Media Servers : [email protected] (11s)
...

If you don't see the "Media Servers" line, you can explicitly add FreeSWITCH servers to ecallmgr using the following command:

sup -n ecallmgr ecallmgr_maintenance add_fs_node freeswitch@hostname

Note: Ensure public and/or private DNS is configured correctly for the hostname.

Check ACLs

sup -n ecallmgr ecallmgr_maintenance sbc_acls

The following should be returned:

+--------------------------------+--------------------+---------------+-------+------------------+----------------------------------+
| Name                           | CIDR               | List          | Type  | Authorizing Type | ID                               |
+================================+====================+===============+=======+==================+==================================+
| [email protected] | 4.3.2.1/32 | authoritative | allow | system_config    |                                  |
| [email protected] | 1.2.3.4/32    | authoritative | allow | system_config    |                                  |
+--------------------------------+--------------------+---------------+-------+------------------+----------------------------------+

Configure ACLs

If you don't see your kamailio servers listed:

sup -n ecallmgr ecallmgr_maintenance allow_sbc <hostname> <ip address>

The following should be returned:

updating authoritative ACLs sbc01.us-east-1.kazoo.tel(1.2.3.4/32) to allow traffic
issued reload ACLs to [email protected]

Note: Remember to add the hostname and IP for the local and remote nodes!

Check FreeSWITCH / ecallmgr Connection

kazoo-freeswitch status

The following should be returned:

...
Connected to:
  [email protected] (127.0.0.1:8031) up 0 years, 0 days, 0 hours, 6 minutes, 36 seconds
...

Check Kamailio / FreeSWITCH Visibility

kazoo-kamailio status

The following should be returned:

{
	NRSETS: 2
	RECORDS: {
		SET: {
			ID: 1
			TARGETS: {
				DEST: {
					URI: sip:1.2.3.4:11000
					FLAGS: AP
					PRIORITY: 0
				}
			}
		}
		SET: {
			ID: 2
			TARGETS: {
				DEST: {
					URI: sip:4.3.2.1:11000
					FLAGS: AP
					PRIORITY: 0
				}
			}
		}
	}
}

Master Account

Create Admin Account

sup crossbar_maintenance create_account \
    {Company Name} \
    {sip.company.com} \
    {username} \
    {password}

The following should be returned:

created new account 'f3a17c0c7b58f4fc0ffac54635624aa0' in db 'account%2Ff3%2Fa1%2F7c0ca5624aa07b58f4fc0c5463ff'
created new account admin user '8b2e1b5f39595eee8fec37342a26afed'

MonsterUI

Install MonsterUI, Apps, and Apache

yum -y install monster-ui* httpd

Initialize Monster Apps

sup crossbar_maintenance init_apps \
    /var/www/html/monster-ui/apps \
    http://api.kazoo.tel:8000/v2

Add Apache Virtual Host

echo "<VirtualHost *:80>
    DocumentRoot \"/var/www/html/monster-ui\"
    ServerName api.kazoo.tel
</VirtualHost>
" > /etc/httpd/conf.d/api.kazoo.tel.conf

Start Apache

systemctl enable httpd
systemctl start httpd

Check Crossbar

curl -s http://localhost:8000/v2

The following should be returned:

{"data":{"message":"invalid credentials"},"error":"401","message":"invalid_credentials","status":"error","timestamp":"2019-03-12T00:16:29Z","version":"4.3.22","node":"ufHyVeljJWt9aLE3-pNgaQ","request_id":"fefaa85fbf27728bcb182c69101e5ab3","auth_token":"undefined"}

Chek WebHooks

curl -s http://localhost:8000/v2/webhooks

The following should be returned:

{"page_size":7,"data":[{"name":"Object","description":"Receive notifications when objects (like JSON document objects) in Kazoo are changed"...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment