mattkasun/mq_troubleshooting.md

Last active April 17, 2023 10:03

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/mattkasun/face2a7c1f32031a2126ff7243caad12.js"></script>
Save mattkasun/face2a7c1f32031a2126ff7243caad12 to your computer and use it in GitHub Desktop.

Download ZIP

Netmaker v0.13.0 MQ Troubleshooting

Raw

mq_troubleshooting.md

not applicable post v0.16.1

Basic Troubleshooting/Cert Regeneration

Check docker-compose.yml

if MQ is proxied

- image: eclipse-mosquitto:2.0.14-opensssl
- expose
  - 8883"

if MQ is not proxied

- image: eclipse-mosquitto:2.0.14-openssl
- ports:
  - 127.0.0.1:1883:1883
  - 8883:8883

volumes

MQ

  - /root/mosquitto.conf:/mosquitto/config/mosquitto.conf
  - mosquitto_data:/mosquitto/data
  - mosquitto_logs:/mosquitto/log
  - shared_certs:/mosquitto/certs

Netmaker

  - dnsconfig:/root/config/dnsconfig
  - sqldata:/root/data
  - shared_certs:/etc/netmaker

mosquitto.conf

per_listener_settings true

listener 8883
allow_anonymous false
require_certificate true
use_identity_as_username true
cafile /mosquitto/certs/root.pem
certfile /mosquitto/certs/server.pem
keyfile /mosquitto/certs/server.key

listener 1883 
allow_anonymous true

make sure the broker is reachable:

nslookup broker.NETMAKER_BASE_DOMAIN must resolve to the netmaker server

if mq is not proxied

port 8883 must be open on the container

make sure certs are generated properly.

run "docker logs mq"  and check for following startup messages

1651234045: mosquitto version 2.0.14 starting
1651234045: Config loaded from /mosquitto/config/mosquitto.conf.
1651234045: Opening ipv4 listen socket on port 8883.
1651234045: Opening ipv6 listen socket on port 8883.
1651234045: Opening ipv4 listen socket on port 1883.
1651234045: Opening ipv6 listen socket on port 1883.
1651234045: mosquitto version 2.0.14 running

If there is a certificate issue, eg.

1651234143: mosquitto version 2.0.14 starting
1651234143: Config loaded from /mosquitto/config/mosquitto.conf.
1651234143: Opening ipv4 listen socket on port 8883.
1651234143: Opening ipv6 listen socket on port 8883.
1651234143: Error: Unable to load server certificate "/mosquitto/certs/server.pem". Check certfile.


OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

Delete Certs

(may have to install sqlite3 on server)

sqlite3 /var/lib/docker/volumes/root_sqldata/_data/netmaker.db 'delete from certs;'

Restart netmaker:

docker restart netmaker

Restart mq:

docker restart mq

check mq logs again, make sure it has started appropriately

run

netclient pull -n <network>

on all clients

Detailed troubleshooting for valid certs

Using Openssl

on server

openssl verify -verbose -CAfile /root/certs/root.pem /root/certs/server.pem

on client

openssl verify -verbose -CAfile /etc/netclient/<broker.domain>/root.pem /etc/netclient/<broker.domain>/client.pem

Using mosquitto_pub

on server -

mosquitto_pub -d -t test -m "hello world" -h broker.<domainname> -p 8883 --cafile /root/certs/root.pem --cert /root/certs/server.pem --key /root/certs/server.key

on client

mosquitto_pub -d -t test -m "hello world" -h broker.domainname -p 8883 --cafile /etc/netclient/broker.<domainname>/root.pem --cert /etc/netclient/broker.<domainname>/client.pem --key /etc/netclient/client.key

Good Result

Client mosq-KOrMCTPqn1rejoTFSl sending CONNECT
Client mosq-KOrMCTPqn1rejoTFSl received CONNACK (0)
Client mosq-KOrMCTPqn1rejoTFSl sending PUBLISH (d0, q0, r0, m1, 'test', ... (11 bytes))
Client mosq-KOrMCTPqn1rejoTFSl sending DISCONNECT

afeiszli commented May 6, 2022

One key part that people seem to be missing is 8883 reachability. 8883 must be reachable publicly on the container and the host. If broker.domain:8883 is not reachable from the clients, setup will fail.

woshizilong commented Jun 2, 2022

Every time I join a new node, I need to regenerate the certificate according to the above steps, and each node has to pull the certificate again.
Feels like a bug with netmaker 0.14.1

zhangnew commented Jul 26, 2022

Just open 8883 instead of use Caddy/Nginx is ok? I don't find a way to proxy the 8883 with Caddy

Author

mattkasun commented Aug 27, 2022

Just open 8883 instead of use Caddy/Nginx is ok? I don't find a way to proxy the 8883 with Caddy

yes, caddy does not support proxing anything but http(s)

jackxuhass commented Aug 30, 2022

docker-compose run it！
why？
：[netmaker] 2022-08-30 14:40:09 error determining containerization: open /proc/1/sched: no such file or directory

atlasloewenherz commented Oct 26, 2022

Hi @mattkasun,

thanks you for sharing your notes here!

i have the following setup:






  ┌──────────────────────────────────────────────────────────┐
  │                                                          │
  │                FIREWALL Open Ports: 80,443,8883          │
  │                                                          │
  └───────────┬───────────────────┬──────────────────┬───────┘
              │                   │                  │
              │                   │                  │
 (1)  Port 80 │    (2)   Port: 443│  (3) Port: 8883  │
              │                   │                  │
              │                   │                  │
              │                   │                  │
   ┌──────────▼───────────────────▼──────────────────▼───────┐
   │                                                         │
   │                                                         │
   │                                                         │
   │                                                         │
   │          NATIVE TRAEFIK BOX                             │           1) certificates management via traefik certresolvers
   │                                                         │
   │                                                         │
   │                                                         │
   │                                                         │           2) https to all backends includings API
   └───────────┬──────────────────┬─────────────────┬────────┘
               │                  │                 │
               │                  │                 │                    3) this port is mapped to 8883 in the target container ( on a different machine) Mosquitto
               │                  │                 │
         https │            8883  │           https │                       traefik configuration: see next chapter
      to api   │         to mq    │        to ui    │
               │                  │                 │
               │                  │                 │
               │                  │                 │
    ┌──────────▼──────────────────▼─────────────────▼──────────┐         4) api.nm.domain.tld:443
    │                                                          │
    │                                                          │         5) mq.nm.domain.tld:8883
    │                                                          │
    │                                                          │         6) ui.nm.domain.tld:443
    │  ┌───────────────┐ ┌───────────────┐ ┌───────────────┐   │
    │  │  (4)          │ │  (5)          │ │  (6)          │   │
    │  │               │ │               │ │               │   │
    │  │  NETMAKER (API) │  MOSQUITTO    │ │ NETMAKER-UI   │   │
    │  │               │ │               │ │               │   │
    │  ├───────────────┤ ├───────────────┤ ├───────────────┤   │
    │ ┌┴───────────────┴─┴───────────────┴─┴───────────────┴─┐ │
    │ │                                                      │ │
    │ │         DOCKER RUNTIME ENVIRONMENT                   │ │
    │ │                                                      │ │
    │ ├──────────────────────────────────────────────────────┤ │
    └─┴──────────────────────────────────────────────────────┴─┘

following are traefik file based configs as mentioned in the diagramm:

entryPoints:
  web:
    address: ":80"
    proxyProtocol:
      trustedIPs:
        - "10.210.1.0/16"
    forwardedHeaders:
      trustedIPs:
        - "10.210.1.0/16"
  websecure:
    address: ":443"
    proxyProtocol:
      trustedIPs:
        - "10.210.1.0/16"
    forwardedHeaders:
      trustedIPs:
        - "10.210.1.0/16"
  emq:
    address: ":8883/tcp"


tcp:
  routers:
    emq_router:
      rule: "HostSNI(`*`)"
      entrypoints:
        - emq
      service: nm_mq
  services:
    nm_mq:
      loadbalancer:
        servers:
          - address: "10.210.1.21:8883"
          - address: "10.210.1.22:8883"

in the environment variables of netmaker/api container i have the following for :

CORS_ALLOWED_ORIGIN: "{{  NM_CORS_ALLOWED_ORIGIN }}"

i tried all the combinations i could think of for

NM_CORS_ALLOWED_ORIGIN: *.nm.domain.tld 👎

NM_CORS_ALLOWED_ORIGIN: ui.nm.domain.tld 👎

NM_CORS_ALLOWED_ORIGIN: *.domain.tld 👎

NM_CORS_ALLOWED_ORIGIN: * 👎

all the variation ends with the problem that ui.nm.domain.tld is resticted accessing api.nm.domain.tld so i end up with :

Access to XMLHttpRequest at 'api.nm.domain.tld:443/api/users/adm/hasadmin' from origin 'https://ui.nm.nm.domain.tld' has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, isolated-app, chrome-extension, chrome, https, chrome-untrusted.

do you or anyone else from the community happen to habe an idea what i could be missing here?

Thanks
Yassine

atlasloewenherz commented Oct 26, 2022

i have placed my request as an issue for netmaker since i think this is something everyone could benefit from ill update the outcome here