Alarms and remote monitoring
From a Modbus or MQTT value to an SMS alarm, end to end
Updated
An alarm is more than if value > limit: send_sms(). That line sends a text every poll while the value sits above the limit, sends dozens more when the value hovers at the limit, stays silent when the sensor dies, and leaves no record of who knew what and when. This tutorial builds a small service that fixes all four: a setpoint with deadband and delay, latching until someone acknowledges, repeat texts until they do, a communications-fault alarm for every point, and a JSON event log. It reads values from MQTT or polls Modbus TCP directly, and sends SMS through Twilio’s HTTP API.
Assumed here: Raspberry Pi OS Bookworm or Trixie (Python 3.11 or later, for tomllib), paho-mqtt 2.2.0, pymodbus 3.16.2, requests 2.34.2, and a Twilio account with an SMS-capable number. Any Linux machine with systemd works the same way. The code was run on 2026-11-19 against a local MQTT broker and a simulated Modbus TCP device with SMS in dry-run mode; the live Twilio call was not exercised.
The alarm states
Each alarm moves between four states, a simplified form of the alarm state model in ANSI/ISA-18.2, which also defines shelved, suppressed and out-of-service states:
| State | Meaning | Leaves when |
|---|---|---|
normal |
Condition false, nothing to acknowledge | Condition holds for delay seconds: active_unack |
active_unack |
Condition true, nobody has acknowledged | Acknowledged: active_ack. Condition clears: cleared_unack (latched) or normal |
active_ack |
Condition true, someone owns it | Condition clears: normal |
cleared_unack |
Condition came and went before anyone acknowledged | Acknowledged: normal |
Latching matters most at night. A pump that tripped at 03:10 and recovered at 03:12 still needs a person to look at why, and a latched alarm stays visible until someone says they have.
Three settings decide when the condition is true:
- Setpoint. A
highalarm trips at or above it, alowalarm at or below it. - Deadband. The value must move this far back past the setpoint before the alarm clears. A 28.0 degC alarm with 0.5 deadband clears below 27.5, so a reading wobbling between 27.9 and 28.1 makes one alarm, not twenty.
- Delay. The condition must hold this many seconds before the alarm is raised. It filters spikes and the seconds a level sensor sloshes after a pump starts. Size it from the process: a delay longer than the time to damage defeats the alarm.
Comms faults are alarms too
A monitoring system that stops receiving data looks exactly like a system where everything is fine. The service therefore creates one extra alarm per point, <point>_comms, which trips when no good reading has arrived for stale_after seconds. While a point has no good data, its threshold alarms hold their current state instead of clearing on missing data. With the Modbus gateway, a failed RTU read is published with "quality": "bad", which this service treats as no reading.
Install
sudo apt install -y python3-venv
sudo mkdir -p /opt/alarm-service /etc/alarm-service
sudo python3 -m venv /opt/alarm-service/venv
sudo /opt/alarm-service/venv/bin/pip install paho-mqtt==2.2.0 pymodbus==3.16.2 requests==2.34.2
Credentials from the environment
The service reads Twilio credentials from environment variables, never from the config file, so the config can go into version control and the secrets cannot. Twilio’s Messages API accepts an API key and secret as HTTP Basic auth, which you can revoke without changing the account’s main token. Create a key in the Twilio console, then:
sudo install -m 600 /dev/null /etc/alarm-service/env
sudo tee /etc/alarm-service/env > /dev/null <<'EOF'
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_KEY=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_API_SECRET=your-api-key-secret
TWILIO_FROM=+15555550199
EOF
systemd reads this root-only file before it starts the service as an unprivileged user. If any of the four variables is missing, the service logs each text as notify_dry_run instead of sending it, which is how to test thresholds without paying for messages.
For US recipients from an ordinary 10-digit number, carriers require A2P 10DLC registration; unregistered traffic is charged extra carrier fees and filtered more heavily. Register before you depend on it, and test delivery to every on-call phone.
The service
Save as /opt/alarm-service/alarms.py:
"""Turn Modbus or MQTT readings into latched, acknowledged SMS alarms with an event log."""
import json
import logging
import os
import queue
import sys
import threading
import time
import tomllib
from dataclasses import dataclass, field
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse
import paho.mqtt.client as mqtt
import requests
from pymodbus import ModbusException
from pymodbus.client import ModbusTcpClient
log = logging.getLogger("alarms")
NORMAL, ACTIVE_UNACK, ACTIVE_ACK, CLEARED_UNACK = "normal", "active_unack", "active_ack", "cleared_unack"
@dataclass
class Alarm:
name: str
point: str
message: str
condition: str = "high" # "high", "low" or "comms" (no good reading for stale_after seconds)
setpoint: float = 0.0
deadband: float = 0.0
delay: float = 0.0 # seconds the condition must hold before the alarm is raised
latch: bool = True # stay in alarm after the value recovers, until someone acknowledges
state: str = NORMAL
condition_since: float | None = None
last_notified: float = 0.0
repeats_sent: int = 0
def in_alarm_condition(self, value: float | None, stale: bool, active: bool) -> bool | None:
"""True or False for the condition, or None when there is no good value to judge."""
if self.condition == "comms":
return stale
if value is None:
return None
# The deadband applies only on the way back, so a value hovering at the setpoint cannot chatter.
if self.condition == "high":
return value > self.setpoint - self.deadband if active else value >= self.setpoint
return value < self.setpoint + self.deadband if active else value <= self.setpoint
@dataclass
class PointValue:
value: float | None = None
received: float = 0.0
@dataclass
class Engine:
alarms: dict[str, Alarm]
stale_after: float
repeat_every: float
max_repeats: int
notify: "Notifier"
events: "EventLog"
points: dict[str, PointValue] = field(default_factory=dict)
def on_value(self, point: str, value: float | None, now: float) -> None:
# A bad reading is not stored: the last good value ages out after stale_after seconds instead.
if value is not None:
self.points[point] = PointValue(value, now)
def is_stale(self, point: str, now: float) -> bool:
pv = self.points.get(point)
return pv is None or now - pv.received > self.stale_after
def current_value(self, point: str, now: float) -> float | None:
return None if self.is_stale(point, now) else self.points[point].value
def evaluate(self, now: float) -> None:
for alarm in self.alarms.values():
active = alarm.state in (ACTIVE_UNACK, ACTIVE_ACK)
value, stale = self.current_value(alarm.point, now), self.is_stale(alarm.point, now)
condition = alarm.in_alarm_condition(value, stale, active)
if condition is None:
continue # bad or stale data: hold the current state; the comms alarm reports the outage
if condition and not active:
alarm.condition_since = alarm.condition_since or now
if now - alarm.condition_since >= alarm.delay:
self.raise_alarm(alarm, now)
elif not condition:
alarm.condition_since = None
if active:
self.clear_alarm(alarm, now)
if alarm.state == ACTIVE_UNACK and alarm.repeats_sent < self.max_repeats:
if now - alarm.last_notified >= self.repeat_every:
alarm.repeats_sent += 1
self.send(alarm, f"REPEAT {alarm.repeats_sent}: {alarm.message}", now)
def raise_alarm(self, alarm: Alarm, now: float) -> None:
alarm.state, alarm.repeats_sent = ACTIVE_UNACK, 0
self.events.write("raised", alarm, value=self.current_value(alarm.point, now))
self.send(alarm, f"ALARM: {alarm.message}", now)
def clear_alarm(self, alarm: Alarm, now: float) -> None:
unacknowledged = alarm.state == ACTIVE_UNACK
alarm.state = CLEARED_UNACK if (alarm.latch and unacknowledged) else NORMAL
self.events.write("cleared", alarm, value=self.current_value(alarm.point, now))
self.send(alarm, f"CLEARED: {alarm.message}", now)
def acknowledge(self, name: str, by: str) -> None:
alarm = self.alarms[name]
if alarm.state == ACTIVE_UNACK:
alarm.state = ACTIVE_ACK
elif alarm.state == CLEARED_UNACK:
alarm.state = NORMAL
else:
return
self.events.write("acknowledged", alarm, by=by)
def send(self, alarm: Alarm, text: str, now: float) -> None:
alarm.last_notified = now
stamp = datetime.fromtimestamp(now, timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
self.notify.outbox.put((alarm.name, f"{text} [{alarm.name}, {stamp}]"))
def snapshot(self) -> dict:
return {a.name: a.state for a in self.alarms.values()}
class EventLog:
def __init__(self, path: str):
self.path = path
self.lock = threading.Lock()
def write(self, event: str, alarm: Alarm | None = None, **details) -> None:
record = {"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"), "event": event}
if alarm:
record |= {"alarm": alarm.name, "state": alarm.state}
record |= details
line = json.dumps(record)
log.info(line)
with self.lock, open(self.path, "a") as f:
f.write(line + "\n")
class Notifier(threading.Thread):
"""Sends queued texts through the Twilio Messages API, retrying so the main loop never waits on the network."""
RETRY_DELAYS = (5, 30, 120)
def __init__(self, recipients: list[str], events: EventLog):
super().__init__(daemon=True)
self.outbox: queue.Queue = queue.Queue()
self.recipients = recipients
self.events = events
self.account_sid = os.environ.get("TWILIO_ACCOUNT_SID")
self.auth = (os.environ.get("TWILIO_API_KEY"), os.environ.get("TWILIO_API_SECRET"))
self.sender = os.environ.get("TWILIO_FROM")
self.dry_run = not all((self.account_sid, *self.auth, self.sender))
if self.dry_run:
log.warning("Twilio variables not set: messages are logged, not sent")
def run(self) -> None:
while True:
alarm_name, text = self.outbox.get()
for to in self.recipients:
self.deliver(alarm_name, to, text)
def deliver(self, alarm_name: str, to: str, text: str) -> None:
if self.dry_run:
self.events.write("notify_dry_run", alarm=None, alarm_name=alarm_name, to=to, text=text)
return
url = f"https://api.twilio.com/2010-04-01/Accounts/{self.account_sid}/Messages.json"
for attempt, delay in enumerate((0, *self.RETRY_DELAYS)):
time.sleep(delay)
try:
response = requests.post(url, data={"To": to, "From": self.sender, "Body": text}, auth=self.auth, timeout=15)
response.raise_for_status()
self.events.write("notified", alarm_name=alarm_name, to=to, sid=response.json().get("sid"))
return
except requests.RequestException as exc:
log.warning("SMS to %s failed (attempt %d): %s", to, attempt + 1, exc)
self.events.write("notify_failed", alarm_name=alarm_name, to=to, text=text)
def start_mqtt_source(cfg: dict, inbox: queue.Queue) -> None:
prefix = cfg["topic_prefix"].rstrip("/")
def on_connect(client, userdata, flags, reason_code, properties):
client.subscribe(f"{prefix}/+", qos=1)
def on_message(client, userdata, message):
point = message.topic.rsplit("/", 1)[-1]
if point == "status":
return
try:
payload = json.loads(message.payload)
value = payload["value"] if payload.get("quality", "good") == "good" else None
except (ValueError, KeyError, TypeError):
log.warning("unreadable payload on %s", message.topic)
value = None
inbox.put(("value", point, None if value is None else float(value)))
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=cfg.get("client_id", "alarm-service"))
if os.environ.get("MQTT_USERNAME"):
client.username_pw_set(os.environ["MQTT_USERNAME"], os.environ.get("MQTT_PASSWORD"))
if cfg.get("tls", False):
client.tls_set()
client.on_connect = on_connect
client.on_message = on_message
client.connect_async(cfg["host"], cfg.get("port", 1883), keepalive=30)
client.loop_start()
def start_modbus_source(cfg: dict, inbox: queue.Queue) -> None:
client = ModbusTcpClient(cfg["host"], port=cfg.get("port", 502), timeout=2, retries=1)
def poll() -> None:
while True:
for p in cfg["points"]:
inbox.put(("value", p["name"], read_modbus_point(client, p)))
time.sleep(cfg.get("poll_interval", 5.0))
threading.Thread(target=poll, daemon=True).start()
def read_modbus_point(client: ModbusTcpClient, p: dict) -> float | None:
if not client.connected and not client.connect():
return None
read = client.read_holding_registers if p.get("table", "holding") == "holding" else client.read_input_registers
datatype = client.DATATYPE[p.get("type", "int16").upper()]
try:
response = read(p["address"], count=datatype.value[1], device_id=p.get("device_id", 1))
except ModbusException as exc:
log.warning("%s: %s", p["name"], exc)
return None
if response.isError():
log.warning("%s: Modbus exception code %s", p["name"], response.exception_code)
return None
raw = client.convert_from_registers(response.registers, datatype, word_order=p.get("word_order", "big"))
return raw * p.get("scale", 1.0)
def start_ack_server(cfg: dict, inbox: queue.Queue, alarm_names: set[str], status: dict) -> None:
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if urlparse(self.path).path != "/alarms":
return self.reply(404, {"error": "not found"})
self.reply(200, status["alarms"])
def do_POST(self):
url = urlparse(self.path)
name = url.path.removeprefix("/ack/")
if not url.path.startswith("/ack/") or name not in alarm_names:
return self.reply(404, {"error": f"unknown alarm {name!r}"})
by = parse_qs(url.query).get("by", ["unknown"])[0]
inbox.put(("ack", name, by))
self.reply(202, {"acknowledging": name, "by": by})
def reply(self, code: int, body: dict) -> None:
data = json.dumps(body).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, *args):
pass
server = ThreadingHTTPServer((cfg.get("bind", "127.0.0.1"), cfg.get("port", 8081)), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
def main(config_path: str) -> None:
with open(config_path, "rb") as f:
cfg = tomllib.load(f)
logging.basicConfig(level=cfg.get("log_level", "INFO"), format="%(levelname)s %(message)s")
events = EventLog(cfg["event_log"])
notifier = Notifier(cfg["notify"]["to"], events)
alarms = {a["name"]: Alarm(**a) for a in cfg["alarms"]}
for point in {a.point for a in alarms.values()}:
alarms[f"{point}_comms"] = Alarm(name=f"{point}_comms", point=point, condition="comms",
delay=0, message=f"No good reading from {point}")
engine = Engine(alarms, cfg.get("stale_after", 60), cfg["notify"].get("repeat_minutes", 15) * 60,
cfg["notify"].get("max_repeats", 3), notifier, events)
inbox: queue.Queue = queue.Queue()
status = {"alarms": engine.snapshot()}
source = cfg["source"]
if source["kind"] == "mqtt":
start_mqtt_source(source, inbox)
elif source["kind"] == "modbus_tcp":
start_modbus_source(source, inbox)
else:
sys.exit(f"unknown source kind {source['kind']!r}")
start_ack_server(cfg.get("ack", {}), inbox, set(alarms), status)
notifier.start()
events.write("started", alarms=sorted(alarms))
# Each point gets stale_after seconds from startup to deliver a first reading before its comms alarm fires.
for point in {a.point for a in alarms.values()}:
engine.points[point] = PointValue(None, time.time())
while True:
try:
kind, name, payload = inbox.get(timeout=1.0)
if kind == "value":
engine.on_value(name, payload, time.time())
elif kind == "ack":
engine.acknowledge(name, payload)
except queue.Empty:
pass
engine.evaluate(time.time())
status["alarms"] = engine.snapshot()
if __name__ == "__main__":
main(sys.argv[1] if len(sys.argv) > 1 else "/etc/alarm-service/alarms.toml")
How the parts fit:
- The main loop owns all alarm state. MQTT messages, Modbus polls and acknowledgements arrive on one queue; nothing else changes an alarm, so there are no locks around the state machine.
- The loop evaluates every alarm at least once a second, so delays and stale timers expire even when no new reading arrives.
- Texts go to a separate thread with retries after 5, 30 and 120 s. A slow or failed Twilio call never delays evaluation, and a message that fails every attempt is logged as
notify_failed. - Every transition and every send is one JSON line in the event log, also echoed to the journal.
Configure alarms
Save as /etc/alarm-service/alarms.toml. The point names match the topics published by the gateway tutorial:
log_level = "INFO"
event_log = "/var/lib/alarm-service/events.jsonl"
stale_after = 60 # seconds without a good reading before a point's comms alarm fires
[source]
kind = "mqtt" # reads the topics published by the Modbus gateway
host = "localhost"
port = 1883
topic_prefix = "site1/modbus"
client_id = "alarm-service-site1"
# To poll a Modbus TCP device directly instead, replace [source] with:
# [source]
# kind = "modbus_tcp"
# host = "192.168.10.20"
# port = 502
# poll_interval = 5.0
# [[source.points]]
# name = "tank1_temperature"
# device_id = 1
# table = "input"
# address = 0
# type = "int16"
# scale = 0.1
[notify]
to = ["+15555550100", "+15555550101"]
repeat_minutes = 15 # resend while nobody has acknowledged
max_repeats = 3
[ack]
bind = "127.0.0.1"
port = 8081
[[alarms]]
name = "tank1_temp_high"
point = "tank1_temperature"
condition = "high"
setpoint = 28.0
deadband = 0.5 # clears below 27.5
delay = 60 # must stay at or above 28.0 for 60 s
latch = true
message = "Tank 1 temperature high (28.0 degC)"
[[alarms]]
name = "tank1_level_low"
point = "tank1_level"
condition = "low"
setpoint = 0.40
deadband = 0.05 # clears above 0.45
delay = 30
latch = true
message = "Tank 1 level low (0.40 m)"
With kind = "mqtt", the service subscribes to <topic_prefix>/+ and reads the value and quality fields of each JSON message. With kind = "modbus_tcp", it polls the listed registers itself, read-only.
Run it under systemd
Save as /etc/systemd/system/alarm-service.service:
[Unit]
Description=Threshold alarms with SMS notification
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/opt/alarm-service/venv/bin/python /opt/alarm-service/alarms.py /etc/alarm-service/alarms.toml
EnvironmentFile=/etc/alarm-service/env
Restart=always
RestartSec=5
DynamicUser=yes
StateDirectory=alarm-service
NoNewPrivileges=yes
ProtectHome=yes
[Install]
WantedBy=multi-user.target
StateDirectory gives the service a writable /var/lib/alarm-service for the event log while DynamicUser keeps the rest of the system read-only to it (systemd.exec).
sudo systemctl daemon-reload
sudo systemctl enable --now alarm-service
journalctl -u alarm-service -f
Acknowledge an alarm
The service listens on 127.0.0.1:8081, so only someone logged in to the Pi (or through an SSH tunnel) can acknowledge:
curl -s http://127.0.0.1:8081/alarms
curl -s -X POST 'http://127.0.0.1:8081/ack/tank1_temp_high?by=dana'
Acknowledging stops the repeat texts and records who did it. It does not fix anything: the alarm stays active_ack until the value is back inside the deadband. Acknowledging by replying to the SMS needs a public webhook that Twilio can call, and on-call rotation needs more than a static list; both are covered in acknowledgement and on-call rotations.
Test it before you trust it
- Leave the Twilio variables unset and start the service. Publish a value above the setpoint:
mosquitto_pub -t site1/modbus/tank1_temperature -m '{"value": 29.0, "quality": "good"}', and repeat it for longer thandelay. - Check
/var/lib/alarm-service/events.jsonlforraisedandnotify_dry_run, then aREPEAT 1text afterrepeat_minutes. - Publish 27.8 (inside the deadband: the alarm stays) and then 27.0 (clears: the state becomes
cleared_unack). - Acknowledge and confirm the state returns to
normaland the log records your name. - Stop publishing for
stale_afterseconds and confirm the_commsalarm fires. - Set the Twilio variables, restart, and repeat step 1 with a real phone. Then pull the Pi’s network cable during an alarm to see what your notification path does without internet.
A sample of the log from step 2 onward:
{"ts": "2026-11-19T14:55:42+00:00", "event": "raised", "alarm": "tank1_temp_high", "state": "active_unack", "value": 29.0}
{"ts": "2026-11-19T14:55:48+00:00", "event": "cleared", "alarm": "tank1_temp_high", "state": "cleared_unack", "value": 27.0}
{"ts": "2026-11-19T14:55:49+00:00", "event": "acknowledged", "alarm": "tank1_temp_high", "state": "normal", "by": "dana"}
{"ts": "2026-11-19T14:55:54+00:00", "event": "raised", "alarm": "tank1_temperature_comms", "state": "active_unack", "value": null}
What this does not cover
- The Pi itself failing. If the Pi loses power, nothing here sends a text. A watchdog outside the site, such as a cloud service that alarms when heartbeats stop, is the only fix; see power-failure alarms.
- Internet outage. SMS through an HTTP API needs internet. A cellular modem or a second path is in cellular backup for alarms.
- Escalation. Repeats go to the same list. Calling a second person when the first does not acknowledge is in alarm escalation by SMS and voice call.
- Restart state. Alarm states live in memory. After a restart, a condition that is still true is raised again, which errs on the side of telling someone.
Sources
- ISA-18 series of standards (ANSI/ISA-18.2, management of alarm systems), accessed 2026-11-19
- Twilio Messaging API, Message resource (create a message), accessed 2026-11-19
- Twilio, A2P 10DLC registration, accessed 2026-11-19
- paho-mqtt on PyPI (version 2.2.0), accessed 2026-11-19
- pymodbus on PyPI (version 3.16.2), accessed 2026-11-19
- requests documentation, accessed 2026-11-19
- Python tomllib documentation, accessed 2026-11-19
- systemd.exec manual page (EnvironmentFile, DynamicUser, StateDirectory), accessed 2026-11-19