systemd Service Active but Not Working? A Reliable Log Watchdog Fix

systemd-service-active-but-not-working.md
title systemd Service Active but Not Working? A Reliable Log Watchdog Fix
date
author VahaC
read 9 min read
category Self-Hosting
tags #automation #Homelab #Linux #lxc #Onedrive #Proxmox #Self-hosted #Telegram
systemd service active but not working

A systemd service active but not working is the worst kind of failure: systemctl status shows a green active (running), nothing restarts, nothing alerts, and the service quietly does nothing for hours. In my homelab a one-minute DNS outage left the OneDrive client in a Proxmox LXC exactly like that for ten hours. 😶

This guide uses that incident to show how to spot a systemd service active but not working and fix it with a log-based watchdog, a systemd timer and a Telegram alert. The example is the client from my OneDrive sync on a ZFS mirror guide, but the pattern works for any service that gets stuck after a network blip.

Why Restart=on-failure Misses a Stuck Service 🧠

systemd restarts a unit only when its process exits. A systemd service active but not working never exits; it waits for something that will not come.

Here is how that looks with the abraunegg OneDrive client (v2.5.11) in --monitor mode. It hears about remote changes through a Microsoft Graph WebSocket and polls only every monitor_interval seconds, which my OneDrive disk IO fix raised to cut disk writes.

If the network is down at startup, the client retries and recovers. But when the network dropped while it was running, it logged a few errors and went silent:

[2026-Oct-13 10:33:13] Cannot connect to the Microsoft OneDrive Service - Network Connection Issue: Could not resolve hostname
[2026-Oct-13 10:33:13]   Error Message:       Could not resolve hostname
[2026-Oct-13 10:33:58] Cannot connect to the Microsoft OneDrive Service - Network Connection Issue: Timeout was reached
[2026-Oct-13 10:33:58]   Error Message:       Timeout was reached

No new WebSocket, no retry, no exit: a textbook systemd service active but not working. Changes stopped until the next poll, up to a day away, and Immich, which reads photos from the synced folder, saw nothing new.

⚠️ Do not trust the green status. For a systemd service active but not working, systemctl status looks perfect. Only the log tells the truth.

How to Diagnose a systemd Service Active but Not Working 🔍

Look at the last lines of the service log, not at the unit state. For the OneDrive client:

tail -n 20 /var/log/onedrive/root.onedrive.log

The log file is named after the user that runs the client, so it is root.onedrive.log when the service runs as root. A healthy client logs WebSocket signals as files change. Network errors followed by silence are the signature of a systemd service active but not working.

The manual fix is a restart:

systemctl restart onedrive-personal.service

After it you should see Enabled WebSocket support and Sync with Microsoft OneDrive is complete in the log.

⚠️ Do not fix this by editing the client config. As explained in my disk IO post, any config change trips the client’s --resync safety check and the service will not start until you resync. A plain restart needs no resync.

Prerequisites ✅

  • A service that writes a timestamped log. Here: the OneDrive client as a systemd unit in --monitor mode with enable_logging = "true", in a Debian 13 LXC on Proxmox.
  • Root access to the machine or container. On Proxmox use pct enter <CTID>.
  • curl installed.
  • A Telegram bot token and your chat ID. I explain how to create a bot in my qBittorrent Telegram notifications guide.

Step 1: Store the Telegram Credentials 🔐

Keep the bot token out of the script, in a file that only root can read:

cat > /etc/service-watchdog.env <<'EOF'
TG_BOT_TOKEN=123456789:replace-with-your-bot-token
TG_CHAT_ID=123456789
EOF
chmod 600 /etc/service-watchdog.env

⚠️ Do not skip chmod 600. Anyone who can read the token can send messages as your bot.

Step 2: Create the Log-Based Watchdog Script 🛠️

This script is the actual fix for a systemd service active but not working. Save it as /usr/local/bin/service-watchdog.sh and set SERVICE and LOG to your unit name and log path.

#!/bin/bash
# service-watchdog.sh: restart a service that is "active" but stuck
# after a network error, then report the result to Telegram.

# ---- What to watch (this example: the OneDrive client) ----
SERVICE="${SERVICE:-onedrive-personal.service}"
LOG="${LOG:-/var/log/onedrive/root.onedrive.log}"
ERROR_REGEX='Cannot connect to the Microsoft OneDrive Service|Error Message:'
RECOVERED_REGEX='Enabled WebSocket support'
PROBE_URL='https://graph.microsoft.com/v1.0/'
# ------------------------------------------------------------

WD_LOG="/var/log/service-watchdog.log"
GRACE=300     # give the service 5 minutes to recover on its own
SETTLE=120    # wait this long after a restart before checking
DRY_RUN="${DRY_RUN:-0}"

# TG_BOT_TOKEN and TG_CHAT_ID
[ -r /etc/service-watchdog.env ] && . /etc/service-watchdog.env

say() { echo "[$(date '+%F %T')] $*" >> "$WD_LOG"; }

notify() {
    [ -z "$TG_BOT_TOKEN" ] && return
    curl -s -m 20 -o /dev/null \
        --data-urlencode "chat_id=$TG_CHAT_ID" \
        --data-urlencode "text=$1" \
        "https://api.telegram.org/bot$TG_BOT_TOKEN/sendMessage" \
        || say "Telegram delivery failed"
}

# Prints "<unix time of the last log line> <the line itself>"
last_line() {
    local line ts
    line=$(tail -n 200 "$LOG" | grep -v '^\[[^]]*\] *$' | tail -n 1)
    # "[2026-Oct-13 10:33:58.48]" -> "13 Oct 2026 10:33:58" (UTC)
    ts=$(echo "$line" | sed -E 's/^\[([0-9]{4})-([A-Za-z]{3})-([0-9]{2}) ([0-9:]{8}).*/\3 \2 \1 \4/')
    echo "$(date -u -d "$ts" +%s 2>/dev/null || echo 0) $line"
}

if [ "$1" = "--test-notify" ]; then
    notify "🟦 Watchdog for $SERVICE on $(hostname): test message"
    exit 0
fi

# Not running at all? That is systemd's job (Restart=on-failure)
systemctl is-active --quiet "$SERVICE" || exit 0

read -r err_epoch last <<< "$(last_line)"

# Only act if the last line is an error...
echo "$last" | grep -qE "$ERROR_REGEX" || exit 0
[ "$err_epoch" -gt 0 ] || exit 0

# ...that is older than GRACE seconds
age=$(( $(date -u +%s) - err_epoch ))
[ "$age" -lt "$GRACE" ] && exit 0

# Network still down? A restart would not help, try again next run
curl -s -m 15 -o /dev/null "$PROBE_URL" || exit 0

if [ "$DRY_RUN" = 1 ]; then
    echo "DRY_RUN: would restart $SERVICE (error ${age}s old)"
    exit 0
fi

say "Silent for ${age}s after: $last. Restarting $SERVICE"
before=$(wc -l < "$LOG")
systemctl restart "$SERVICE"
sleep "$SETTLE"

if tail -n +"$((before + 1))" "$LOG" | grep -qE "$RECOVERED_REGEX"; then
    say "Recovered"
    notify "🟧 $SERVICE on $(hostname) was stuck for $((age / 60)) min after a network error. Restarted, it is back."
else
    say "Did NOT recover"
    notify "🟥 $SERVICE on $(hostname) was restarted after a network error but did not recover. Check $LOG"
fi
chmod +x /usr/local/bin/service-watchdog.sh

It restarts a systemd service active but not working only when all of these are true:

  1. The service is active. A crash is systemd’s job.
  2. The last meaningful log line matches ERROR_REGEX.
  3. That error is older than 5 minutes.
  4. PROBE_URL answers again. While the network is down, a restart would not help.

Two minutes after the restart it looks for RECOVERED_REGEX in the new log lines and sends 🟧 if the service came back, or 🟥 if it did not. A healthy log never ends with an error, so the watchdog leaves it alone and cannot restart in a loop.

Run a self-hosted notification gateway? Point the curl call in notify() at it instead.

Step 3: Run the Watchdog Every 5 Minutes ⏱️

Create /etc/systemd/system/service-watchdog.service:

[Unit]
Description=Restart a stuck service after a network error

[Service]
Type=oneshot
ExecStart=/usr/local/bin/service-watchdog.sh

And /etc/systemd/system/service-watchdog.timer:

[Unit]
Description=Run the service watchdog every 5 minutes

[Timer]
OnBootSec=10min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

Enable the timer:

systemctl daemon-reload
systemctl enable --now service-watchdog.timer

A systemd timer beats cron here: output lands in the journal and a run still in its two-minute check is never started twice. A systemd service active but not working is now caught within about ten minutes.

Step 4: Test the Watchdog Before You Need It 🧪

First, check that Telegram delivery works:

/usr/local/bin/service-watchdog.sh --test-notify

Next, simulate a systemd service active but not working with a fake log and DRY_RUN=1, so nothing is restarted:

cat > /tmp/fake.log <<'EOF'
[2026-Oct-13 07:31:23.4149240] Sync with Microsoft OneDrive is complete
[2026-Oct-13 10:33:58.4808634] Cannot connect to the Microsoft OneDrive Service - Network Connection Issue: Timeout was reached
[2026-Oct-13 10:33:58.4809209]   Error Message:       Timeout was reached
EOF
LOG=/tmp/fake.log DRY_RUN=1 /usr/local/bin/service-watchdog.sh

Expected output (your number of seconds will differ):

DRY_RUN: would restart onedrive-personal.service (error 37795s old)

Against the real, healthy log it must print nothing. Only a systemd service active but not working triggers a restart:

DRY_RUN=1 /usr/local/bin/service-watchdog.sh
systemctl list-timers service-watchdog.timer

Use the Same Watchdog for Any systemd Service Active but Not Working ♻️

Nothing in the logic is OneDrive-specific. A systemd service active but not working is common in sync clients, MQTT bridges, VPN tunnels and API pollers: anything that holds a long-lived connection and does not rebuild it after a blip. To adapt the watchdog, answer three questions:

  • What does it log when it gets stuck? Put it into ERROR_REGEX, copied from a real incident.
  • What does it log when it is healthy again? Put that into RECOVERED_REGEX.
  • Which endpoint does it depend on? Put its URL into PROBE_URL.

If your service logs to journald, replace last_line() with this version; short-unix output already starts with a Unix timestamp:

last_line() {
    journalctl -u "$SERVICE" -n 1 -o short-unix --no-pager -q \
        | sed -E 's/^([0-9]+)\.[0-9]+ /\1 /'
}

For the recovery check, save started=$(date +%s) before the restart and replace the tail -n +... line with journalctl -u "$SERVICE" --since "@$started" -q | grep -qE "$RECOVERED_REGEX". For several services, copy the script and both units once per service. Even with central Proxmox LXC monitoring, an up/down check cannot see a systemd service active but not working; this watchdog can.

Troubleshooting 🧯

If the watchdog misses a systemd service active but not working, check these first:

  • No Telegram message: press Start in your bot once and check the token and chat ID. Errors go to /var/log/service-watchdog.log.
  • The dry run prints nothing on the fake log: your timestamps differ from [2026-Oct-13 10:33:58]; adjust the sed line in last_line().
  • You get 🟥: a restart did not help, so this is no longer a plain network blip. Read the log.
  • See what the timer did: run journalctl -u service-watchdog.service -n 20.

Final Thoughts

A systemd service active but not working slips past every check that only asks whether the process is alive. Watching the log for silence after an error closes that gap with about 80 lines of bash. For my OneDrive client, a network blip now costs about ten minutes and one Telegram message instead of a day of silent drift. 🏠

Change five variables, and every systemd service active but not working in your homelab gets the same safety net.

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.