dsundarraj · March 15, 2016 18:39
diff --git a/lbaas.txt b/lbaas.txt
 Check for package installation:

 root@oblocknode04:/etc/contrail# dpkg -l | grep haproxy
 ii  haproxy                              1.5.4-1ppa1~precise                   amd64        fast and reliable load balancing reverse proxy
 root@oblocknode04:/etc/contrail# dpkg -l | grep iproute
 ii  iproute                              1:3.12.0-2                            all          transitional dummy package for iproute2
 ii  iproute2                             3.12.0-2                              amd64        networking and traffic control tools
 root@oblocknode04:/etc/contrail# dpkg -l | grep netns
 ii  python-opencontrail-vrouter-netns    2.21.2-28                             amd64        OpenContrail vrouter network namespace package

 In testbed.py specify
 env.enable_lbaas = True

 Workflow:

 - Create a VIP network

    - neutron net-create vipnet
    - neutron subnet-create —name vipsubnet vipnet 2.2.2.0/24
 - Create a pool network

    - neutron net-create poolnet
    - neutron subnet-create —name poolsubnet polite 3.3.3.0/24
 - Create a pool for HTTP

    - neutron lb-pool-create —lb-method ROUND_ROBIN —name test pool —protocol HTTP —subnet-id poolsubnet
 - Add members to the pool

    - neutron lb-member-create —address 3.3.3.2 —protocol-port 80 testpool
    - neutron lb-member-create —address 3.3.3.3 —protocol-port 80 testpool
 - Create a VIP for HTTP and associate it to the testpool

    - neutron lb-vip-create —name testvip —protocol-port 80 —protocol HTTP —subnet-id vipsubnet testpool
 - Associating a FIP to a VIP 😃


    - Similar to any other floatingip-associate

        - neutron floatingip-associate <vip-uuiid> <floatingip-uuid>
 - Create healthmonitor

    - neutron lb-healthmonitor-create —delay 20 —timeout 10 —max-retries 3 —type HTTP
 - Associate healthmonitor

    - neutron lb-healthmonitor-associate <healthmonitor-uuid> testpool
 - For SSL

    - Manually copy ssl-cert.pem to compute (or all computes). We only support single cert today.
    - Update /etc/contrail/contrail-vrouter-agent.conf

            - haproxy_ssl_cert_path=<path to ssl-cert.pem copied in step above>
    - Restart vrouter-agent
    - neutron lb-vip-create —name testvip —protocol-port 443 —protocol HTTP —subnet-id vipsubnet testpool

 Scheduling options

 - Round Robin: go in turns according to set weight.
 - Least connection: pool members with least connections are preferred.
 - Source ip: source ip hash based scheme

 Pool members liveness check

 - TCP: LB makes a tcp connection
 - HTTP: LB makes HTTP request on the established TCP connection
 - PING: LB pings pool members

 Neutron lbaas commands: https://wiki.openstack.org/wiki/Neutron/LBaaS/CLI

  lb-healthmonitor-create        Create a health monitor
  lb-healthmonitor-delete        Delete a given health monitor
  lb-healthmonitor-list          List health monitors that belong to a given tenant
  lb-healthmonitor-show          Show information of a given health monitor
  lb-healthmonitor-update        Update health monitor's information

  lb-healthmonitor-associate     Associate a health monitor with a pool
  lb-healthmonitor-disassociate  Disassociate a health monitor with a pool

  lb-member-create               Create a member
  lb-member-delete               Delete a given member
  lb-member-list                 List members that belong to a given tenant
  lb-member-show                 Show information of a given member
  lb-member-update               Update member's information

  lb-pool-create                 Create a pool
  lb-pool-delete                 Delete a given pool
  lb-pool-list                   List pools that belong to a given tenant
  lb-pool-show                   Show information of a given pool
  lb-pool-update                 Update pool's information

  lb-pool-stats                  Retrieve traffic statistics for pool

  lb-vip-create                  Create a vip
  lb-vip-delete                  Delete a given vip
  lb-vip-list                    List vips that belong to a given tenant
  lb-vip-show                    Show information of a given vip
  lb-vip-update                  Update vip's information

 Config files:

 Set log_level to SYS_DEBUG in contrail-svc-monitor.conf, contrail-api.conf, contrail-schema.conf and contrail-svc-monitor.conf to see transactional records for the LB operations above.

 Multiple availability zones can be specified here as availability_zone

 contrail-svc-monitor.conf
 [DEFAULTS]
 ifmap_server_ip=172.16.80.4
 ifmap_server_port=8443
 ifmap_username=svc-monitor
 ifmap_password=svc-monitor
 api_server_ip=172.16.80.4
 api_server_port=8082
 api_server_use_ssl=False
 zk_server_ip=172.16.80.4:2181
 log_file=/var/log/contrail/contrail-svc-monitor.log
 cassandra_server_list=172.16.80.4:9160
 disc_server_ip=172.16.80.4
 disc_server_port=5998
 region_name=RegionOne
 log_local=1
 log_level=SYS_NOTICE
 rabbit_server=172.16.80.4
 rabbit_port=5672

 [SECURITY]
 use_certs=False
 keyfile=/etc/contrail/ssl/private_keys/svc_monitor_key.pem
 certfile=/etc/contrail/ssl/certs/svc_monitor.pem
 ca_certs=/etc/contrail/ssl/certs/ca.pem

 [SCHEDULER]
 analytics_server_ip=172.16.80.4
 analytics_server_port=8081

 On the vrouter: look at /etc/contrail/contrail-vrouter-agent.conf

 [SERVICE-INSTANCE]
 # Path to the script which handles the netns commands
 netns_command=/usr/bin/opencontrail-vrouter-netns

 # Number of workers that will be used to start netns commands
 #netns_workers=1

 # Timeout for each netns command, when the timeout is reached, the netns
 # command is killed.
 #netns_timeout=30

 Logs:
 /var/log/contrail/contrail-api.log
 /var/log/contrail/contrail-api-0-stdout.log
 /var/log/contrail/api-0-zk.log

 /var/log/contrail/schema-zk.log
 /var/log/contrail/contrail-schema.log
 /var/log/contrail/contrail-schema-stdout.log

 /var/log/contrail/contrail-svc-monitor.log
 /var/log/contrail/contrail-svc-monitor-stdout.log
 /var/log/contrail/svc-monitor.err
 /var/log/contrail/svc-monitor-zk.log

 LB Configs:

 - ip netns list
 - Check /var/lib/contrail/loadbalancer/<uuid>/etc/haproxy/haproxy.cfg

 Introspect:
 http://10.87.25.250:8088/Snh_ServiceInstanceList?si_name=
 http://10.87.25.250:8082/loadbalancer-pools
 http://10.87.25.250:8081/analytics/uves/service-instances

 Issues:

 Issue 1: A vip does not launch any haproxy instance.

    - Look at contrail-api log and schema log first
    - Is the svc_monitor process stuck or hung, look for its logs
    - Restart contrail_svc_monitor, see if haproxy instance got created correctly.

 -----------------------------

 Issue 2: Killing haproxy on hypervisor and see it it spawns a new haproxy somewhere else, but I don't see that happening.

 - Check haproxy process on vrouter "ps -ef | grep haproxy"
 - Check contrail config for VM back-refs pointing to the old haproxy instance.
 - The database still points to the old active/standby instances which does not seem to be correct.

 -----------------------------

 Issue 3: Multiple haproxy processes getting spawned for a single LBaaS.

 Has been observed when LB config is updated. Old haproxy handles old sessions and new one is spawned for new sessions and new configs.

 -----------------------------

 Issue 4: SG applied for VIP must apply for the SI VM as well. Check that.

 Other tools:

 - Run contrail-svc-monitor manually

    - ps aux | grep contrail-svc-monitor
 contrail  2320  0.0  0.0 311980 60364 ?        Sl   00:59   0:36 /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
    - root     13181  0.0  0.0  10472  2196 pts/25   S+   12:17   0:00 grep --color=auto contrail-svc-monitor
    - See where it crashes

        - /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
 - tcpdump on the corresponding veth interfaces for data packet flow
 - echo "show stat" | nc -U 1d663edb-cab2-45bd-a041-4eb019695ba1.haproxy.cfg.sock

 _________________________

 Details of Implementation:

 You can read more about how it exactly works at  /usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py

 -----------------------------
	Check for package installation:

	root@oblocknode04:/etc/contrail# dpkg -l \| grep haproxy
	ii haproxy 1.5.4-1ppa1~precise amd64 fast and reliable load balancing reverse proxy
	root@oblocknode04:/etc/contrail# dpkg -l \| grep iproute
	ii iproute 1:3.12.0-2 all transitional dummy package for iproute2
	ii iproute2 3.12.0-2 amd64 networking and traffic control tools
	root@oblocknode04:/etc/contrail# dpkg -l \| grep netns
	ii python-opencontrail-vrouter-netns 2.21.2-28 amd64 OpenContrail vrouter network namespace package

	In testbed.py specify
	env.enable_lbaas = True

	Workflow:

	- Create a VIP network

	- neutron net-create vipnet
	- neutron subnet-create —name vipsubnet vipnet 2.2.2.0/24
	- Create a pool network

	- neutron net-create poolnet
	- neutron subnet-create —name poolsubnet polite 3.3.3.0/24
	- Create a pool for HTTP

	- neutron lb-pool-create —lb-method ROUND_ROBIN —name test pool —protocol HTTP —subnet-id poolsubnet
	- Add members to the pool

	- neutron lb-member-create —address 3.3.3.2 —protocol-port 80 testpool
	- neutron lb-member-create —address 3.3.3.3 —protocol-port 80 testpool
	- Create a VIP for HTTP and associate it to the testpool

	- neutron lb-vip-create —name testvip —protocol-port 80 —protocol HTTP —subnet-id vipsubnet testpool
	- Associating a FIP to a VIP 😃


	- Similar to any other floatingip-associate

	- neutron floatingip-associate <vip-uuiid> <floatingip-uuid>
	- Create healthmonitor

	- neutron lb-healthmonitor-create —delay 20 —timeout 10 —max-retries 3 —type HTTP
	- Associate healthmonitor

	- neutron lb-healthmonitor-associate <healthmonitor-uuid> testpool
	- For SSL

	- Manually copy ssl-cert.pem to compute (or all computes). We only support single cert today.
	- Update /etc/contrail/contrail-vrouter-agent.conf

	- haproxy_ssl_cert_path=<path to ssl-cert.pem copied in step above>
	- Restart vrouter-agent
	- neutron lb-vip-create —name testvip —protocol-port 443 —protocol HTTP —subnet-id vipsubnet testpool

	Scheduling options

	- Round Robin: go in turns according to set weight.
	- Least connection: pool members with least connections are preferred.
	- Source ip: source ip hash based scheme

	Pool members liveness check

	- TCP: LB makes a tcp connection
	- HTTP: LB makes HTTP request on the established TCP connection
	- PING: LB pings pool members

	Neutron lbaas commands: https://wiki.openstack.org/wiki/Neutron/LBaaS/CLI

	lb-healthmonitor-create Create a health monitor
	lb-healthmonitor-delete Delete a given health monitor
	lb-healthmonitor-list List health monitors that belong to a given tenant
	lb-healthmonitor-show Show information of a given health monitor
	lb-healthmonitor-update Update health monitor's information

	lb-healthmonitor-associate Associate a health monitor with a pool
	lb-healthmonitor-disassociate Disassociate a health monitor with a pool

	lb-member-create Create a member
	lb-member-delete Delete a given member
	lb-member-list List members that belong to a given tenant
	lb-member-show Show information of a given member
	lb-member-update Update member's information

	lb-pool-create Create a pool
	lb-pool-delete Delete a given pool
	lb-pool-list List pools that belong to a given tenant
	lb-pool-show Show information of a given pool
	lb-pool-update Update pool's information

	lb-pool-stats Retrieve traffic statistics for pool

	lb-vip-create Create a vip
	lb-vip-delete Delete a given vip
	lb-vip-list List vips that belong to a given tenant
	lb-vip-show Show information of a given vip
	lb-vip-update Update vip's information

	Config files:

	Set log_level to SYS_DEBUG in contrail-svc-monitor.conf, contrail-api.conf, contrail-schema.conf and contrail-svc-monitor.conf to see transactional records for the LB operations above.

	Multiple availability zones can be specified here as availability_zone

	contrail-svc-monitor.conf
	[DEFAULTS]
	ifmap_server_ip=172.16.80.4
	ifmap_server_port=8443
	ifmap_username=svc-monitor
	ifmap_password=svc-monitor
	api_server_ip=172.16.80.4
	api_server_port=8082
	api_server_use_ssl=False
	zk_server_ip=172.16.80.4:2181
	log_file=/var/log/contrail/contrail-svc-monitor.log
	cassandra_server_list=172.16.80.4:9160
	disc_server_ip=172.16.80.4
	disc_server_port=5998
	region_name=RegionOne
	log_local=1
	log_level=SYS_NOTICE
	rabbit_server=172.16.80.4
	rabbit_port=5672

	[SECURITY]
	use_certs=False
	keyfile=/etc/contrail/ssl/private_keys/svc_monitor_key.pem
	certfile=/etc/contrail/ssl/certs/svc_monitor.pem
	ca_certs=/etc/contrail/ssl/certs/ca.pem

	[SCHEDULER]
	analytics_server_ip=172.16.80.4
	analytics_server_port=8081

	On the vrouter: look at /etc/contrail/contrail-vrouter-agent.conf

	[SERVICE-INSTANCE]
	# Path to the script which handles the netns commands
	netns_command=/usr/bin/opencontrail-vrouter-netns

	# Number of workers that will be used to start netns commands
	#netns_workers=1

	# Timeout for each netns command, when the timeout is reached, the netns
	# command is killed.
	#netns_timeout=30

	Logs:
	/var/log/contrail/contrail-api.log
	/var/log/contrail/contrail-api-0-stdout.log
	/var/log/contrail/api-0-zk.log

	/var/log/contrail/schema-zk.log
	/var/log/contrail/contrail-schema.log
	/var/log/contrail/contrail-schema-stdout.log

	/var/log/contrail/contrail-svc-monitor.log
	/var/log/contrail/contrail-svc-monitor-stdout.log
	/var/log/contrail/svc-monitor.err
	/var/log/contrail/svc-monitor-zk.log

	LB Configs:

	- ip netns list
	- Check /var/lib/contrail/loadbalancer/<uuid>/etc/haproxy/haproxy.cfg

	Introspect:
	http://10.87.25.250:8088/Snh_ServiceInstanceList?si_name=
	http://10.87.25.250:8082/loadbalancer-pools
	http://10.87.25.250:8081/analytics/uves/service-instances

	Issues:

	Issue 1: A vip does not launch any haproxy instance.

	- Look at contrail-api log and schema log first
	- Is the svc_monitor process stuck or hung, look for its logs
	- Restart contrail_svc_monitor, see if haproxy instance got created correctly.

	-----------------------------

	Issue 2: Killing haproxy on hypervisor and see it it spawns a new haproxy somewhere else, but I don't see that happening.

	- Check haproxy process on vrouter "ps -ef \| grep haproxy"
	- Check contrail config for VM back-refs pointing to the old haproxy instance.
	- The database still points to the old active/standby instances which does not seem to be correct.

	-----------------------------

	Issue 3: Multiple haproxy processes getting spawned for a single LBaaS.

	Has been observed when LB config is updated. Old haproxy handles old sessions and new one is spawned for new sessions and new configs.

	-----------------------------

	Issue 4: SG applied for VIP must apply for the SI VM as well. Check that.

	Other tools:

	- Run contrail-svc-monitor manually

	- ps aux \| grep contrail-svc-monitor
	contrail 2320 0.0 0.0 311980 60364 ? Sl 00:59 0:36 /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
	- root 13181 0.0 0.0 10472 2196 pts/25 S+ 12:17 0:00 grep --color=auto contrail-svc-monitor
	- See where it crashes

	- /usr/bin/python /usr/bin/contrail-svc-monitor --conf_file /etc/contrail/contrail-svc-monitor.conf --conf_file /etc/contrail/contrail-keystone-auth.conf
	- tcpdump on the corresponding veth interfaces for data packet flow
	- echo "show stat" \| nc -U 1d663edb-cab2-45bd-a041-4eb019695ba1.haproxy.cfg.sock

	_________________________

	Details of Implementation:

	You can read more about how it exactly works at /usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py

	-----------------------------