I recently had this annoying issue with Mosquitto (a MQTT broker) that just wouldn't start.
The issue was simple - since version 2.x, Mosquitto will try to write a PID file which by default lives in
/var/run/mosquitto/mosquitto.pid (in previous versions it doesn't crash if it can't create this file). This is a
tmpfs drive so it resets with boot, and the Mosquitto user doesn't own the parent folder, so it can't create a subfolder in which it can store it's pid file.
The default systemd service definition for Mosquitto in Debian doesn't seem to create this folder before starting the service. The fix is simple, just place this file in
systemctl daemon-reload and restart the service with
systemctl restart mosquitto.service.
[Unit] Description=Mosquitto MQTT message broker Documentation=https://mosquitto.org/man/mosquitto-8.html After=network-online.target Wants=network-online.target systemd-networkd-wait-online.service [Service] Restart=on-failure RestartSec=2 User=mosquitto Group=mosquitto # Here's the important change ExecStartPre=+/bin/mkdir -m 740 -p /var/log/mosquitto ExecStartPre=+/bin/chown mosquitto:mosquitto /var/log/mosquitto ExecStartPre=+/bin/mkdir -m 740 -p /var/run/mosquitto ExecStartPre=+/bin/chown mosquitto:mosquitto /var/run/mosquitto ExecStart=/usr/sbin/mosquitto --config-file /etc/mosquitto/mosquitto.conf ExecReload=/bin/kill -HUP $MAINPID PrivateTmp=true PrivateDevices=true ProtectHome=true ProtectSystem=full [Install] WantedBy=multi-user.target
…and Bob's your uncle.
Update: I stole the unit file from this issue on GitHub.