Error with STOMP client and port 61613 for candlepin

candlepin_events_breakfix.yml

Requirements: 

* Satellite 6.10 ( latest minor release ) installed on RHEL 7
* Organization RedHat and Location GSS

Breakfix Playbook:

~~~
---
- name: Creating breakfix for candlepin_events failure in Satellite 6.10
  hosts: repro_satellite ## mention the IP of the satellite here
  gather_facts: true
  tasks:

    - name: Check the presence of ssl-build directory.
      stat:
        path: "/root/ssl-build"
      register: ssl_build

    - name: Create the ssl-build directory is it was not present.
      file:
        path: /root/ssl-build
        state: directory
        mode: '0755' 
      when: ssl_build is defined and not ssl_build.stat.exists

    - name: Run installer to generate certs into ssl-build directory.
      command: bash -c "satellite-installer -S satellite"
      when: ssl_build is defined and not ssl_build.stat.exists
    
    - name: Read the /etc/tomcat/cert-users.properties file
      command: bash -c "cat /etc/tomcat/cert-users.properties"
      register: tomcat_properties
    
    - name: Decide if the file content requires modification or not.
      set_fact:
        need_update_answers: >-
          {%- if tomcat_properties.stdout == 'katelloUser=CN=localhost, OU=, O=candlepin, ST=, C=IN' -%}false
          {%- else -%}true
          {%- endif -%}

    - name: Modify the satellite-answers file.
      lineinfile:
        path: /etc/foreman-installer/scenarios.d/satellite-answers.yaml
        regexp: "{{ item.source }}"
        line: "{{ item.replace }}"
      loop:
        - { source: "^  country: US", replace: "  country: IN" }
        - { source: "^  org: Katello", replace: "  org: RedHat" }
        - { source: "^  org_unit: SomeOrgUnit", replace: "  org_unit: ''" }
        - { source: "^  state: North Carolina", replace: "  state: ''" }
      register: answers_updated
      when: need_update_answers is defined and need_update_answers is true

    - name: Run installer to break the system.
      command: >
        satellite-installer -S satellite
        --certs-regenerate true --certs-update-all
        --certs-update-server --certs-update-server-ca
      register: installer_run
      when: need_update_answers is defined and need_update_answers is true
      
    - name: Stop tomcat and foreman services
      systemd:
        name: "{{ item }}"
        state: stopped
      loop:
        - tomcat
        - foreman.service
      register: stop_services
      when: need_update_answers is defined or installer_run is defined

    - name: Break port settings for candlepin inside selinux
      seport:
        ports: 61613
        proto: tcp
        setype: "{{ item.type }}"
        state: "{{ item.state }}"
      loop:
        - { type: "candlepin_activemq_port_t", state: "absent" }
        - { type: "foreman_proxy_port_t", state: "present" }
      register: break_candlepin_port
      when: stop_services is defined and stop_services is success and need_update_answers is defined

    - name: Start back the services.
      systemd:
        name: "{{ item }}"
        state: started
      loop:
        - tomcat
        - foreman.service
      register: start_services
      when: break_candlepin_port is defined

    - name: Confirm that the candlepin component is broken
      uri:
        url: "https://{{ ansible_fqdn }}/katello/api/ping"
        return_content: yes
        timeout: 360
        validate_certs: false
      register: ping_status
      when: start_services is defined and start_services is success
      failed_when: '"FAIL" not in ping_status.content'
~~~


Issue: "hammer ping" shows candlepin_auth in failed state

~~~
candlepin_events: 
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 1ms
~~~

Task: 

* Ensure that the "hammer ping" output shows everything is OK
* Executing satellite-installer command should not revert back the fixes applied i.e. satellite should remain healthy. 

Error message 1:

2022-12-29 18:29:59,913 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue 
jobs in AddressSettings
2022-12-29 18:30:01,503 [thread=localhost-startStop-1] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224104: Error starting the Acceptor stomp {need
ClientAuth=true, trustStorePassword=TSztuMXXr9TbFMUGb8uNGt2yfSzfiuWw, scheme=tcp, port=61613, keyStorePassword=sfZoA7ZFHZnZZ9qaL2KhL6qXLaiMBRLE, sslEnabled=true, host=loc
alhost, trustStorePath=/etc/candlepin/certs/truststore, protocols=STOMP, useEpoll=false, keyStorePath=/etc/candlepin/certs/keystore}
2022-12-29 18:30:01,531 [thread=localhost-startStop-1] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224000: Failure in initialisation
java.lang.IllegalStateException: AMQ229230: Failed to bind acceptor stomp to localhost:61613
        at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.startServerChannels(NettyAcceptor.java:741)
        at org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.start(NettyAcceptor.java:522)
..
..
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.SocketException: Permission denied

Solution 1:

# systemctl stop tomcat
# semanage port -d 61613 -p tcp
# yum reinstall candlepin-selinux  --disableplugin=foreman-protector
# semanage port -l | grep 61613
# systemctl start tomcat

Error Message 2:

2022-12-29 20:48:00,862 [thread=Thread-5 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from /127.0.0.1:50108. Username: null; SSL certificate subject DN: CN=localhost, O=candlepin, C=IN
2022-12-29 20:48:00,863 [thread=Thread-5 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.protocol.stomp - AMQ332069: Sent ERROR frame to STOMP client /127.0.0.1:50108: Security Error occurred: User name [null] or password is invalid

Solution 2: ( improvement of this KB

##  Ensure that following file has the expected content of these parameters:

# grep country -A4 /etc/foreman-installer/scenarios.d/satellite-answers.yaml
  country: US
  state: North Carolina
  city: Raleigh
  org: Katello
  org_unit: SomeOrgUnit

## Run installer

# satellite-installer -S satellite --certs-regenerate true --certs-update-all --certs-update-server --certs-update-server-ca