How to edit mosquitto systemd service restart interval to fix dns resolution error on startup

EDIT_MOSQUITTO_SYSTEMD_SERVICE.md

Option 1: Override mosquitto systemd service

This solution relies on using systemd override functionality require allows additional settings to be applied to existing systemd configuration files without having to redefined the entire systemd service definition.

Editing the mosquitto systemd definition

1. Edit the mosquitto service definition

   sudo systemctl edit mosquitto

   systemctl will open up the default text editor

2. Add the following contents to the prompted section (e.g. read the instructions presented in the text edit)

   [Unit]
   After=nss-lookup.target
   
   [Service]
   RestartSec=5

3. Save the file

   This will add the following file: /etc/systemd/system/mosquitto.service.d/override.conf

   [Service]
   RestartSec=5

4. Restart the device to verify that mosquitto connect normally after a reboot

Verify the custom mosquitto service definition

You can verify if the systemd "drop-in" is in place by using the following systemctl:

systemctl status mosquitto

● mosquitto.service - Mosquitto MQTT Broker
     Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mosquitto.service.d
             └─override.conf
     Active: active (running) since Mon 2023-07-17 13:33:07 CEST; 31min ago
       Docs: man:mosquitto.conf(5)
             man:mosquitto(8)
   Main PID: 843 (mosquitto)
      Tasks: 1 (limit: 8755)
        CPU: 1.577s
     CGroup: /system.slice/mosquitto.service
             └─843 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Jul 17 13:33:07 raspberrypi systemd[1]: Starting Mosquitto MQTT Broker...
Jul 17 13:33:07 raspberrypi mosquitto[843]: 1689593587: Loading config file /etc/tedge/mosquitto-conf/c8y-bridge.conf
Jul 17 13:33:07 raspberrypi mosquitto[843]: 1689593587: Loading config file /etc/tedge/mosquitto-conf/tedge-mosquitto.conf
Jul 17 13:33:07 raspberrypi mosquitto[843]: 1689593587: Note: It is recommended to replace `message_size_limit` with `max_packet_size`.
Jul 17 13:33:07 raspberrypi systemd[1]: Started Mosquitto MQTT Broker.

The output should include references to the override.conf file.

Drop-In: /etc/systemd/system/mosquitto.service.d
         └─override.conf

Reverting

The mosquitto customization can be reverted using the following command.

systemctl revert mosquitto

Option 2: Using new service which waits for network to be online before allows mosquitto to start

This solution relies on creating a new service definition which adds a dependency to mosquitto force it to start after the new service is ready.

The dummy service uses the After=network-online.target setting, to prevent mosquitto from starting too early.

The advantage with this method is that the service can be added and removed without having to edit anything to do with mosquitto.

1. Create a new systemd service file

   file: /lib/systemd/system/tedge-wait.service

   [Unit]
   Description=thin-edge.io online waiter. Prevent mosquitto from starting before the network has been established.
   After=network-online.target
   Before=mosquitto.service
   
   [Service]
   Type=oneshot
   ExecStart=/bin/sleep 0.5
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target mosquitto.service
   

2. Reload systemd

   sudo systemctl daemon-reload

3. Enable the service on startup

   sudo systemctl enable tedge-wait

4. Restart the device

5. Verify that mosquitto started without any problems

   journalctl -fu mosquitto --boot