Self-hosted uptime monitoring bot (HTTP, TCP, Ping) integrated with Delta Chat. Features per-chat isolation, smart failure retry alerts, multi-transport SMTP failover, and responsive status pages.
  • Python 99.2%
  • HTML 0.6%
  • Shell 0.2%
Find a file
2026-11-20 21:51:52 +00:00
.github/workflows v2.25.0: deploy and publish releases by tag (#3) 2026-10-24 16:28:17 +01:00
tests Wait for readers still opening the old database file when it is replaced 2026-11-20 21:51:52 +00:00
tools/loadbench Pool read connections per thread and cut the incident and status query cost 2026-11-20 19:38:47 +00:00
webxdc Recover live status connections after bot restarts 2026-11-18 14:07:21 +00:00
webxdc_peer Add peer realtime channel health checks 2026-11-19 00:02:05 +00:00
.dockerignore Initial commit: Delta Chat Uptime Bot 2026-06-29 18:51:28 +01:00
.gitignore Initial commit: Delta Chat Uptime Bot 2026-06-29 18:51:28 +01:00
blockcheck.py v2.39.0: Freeze? hint for bodies that stop after a few KB, diagnosis inside the overall deadline 2026-11-17 15:51:16 +00:00
bot.py Release 3.0.1: cut the database and CPU cost of periodic work and fix the issues found on the way 2026-11-20 21:33:31 +00:00
CHANGELOG.md Wait for readers still opening the old database file when it is replaced 2026-11-20 21:51:52 +00:00
database.py Wait for readers still opening the old database file when it is replaced 2026-11-20 21:51:52 +00:00
dc_fingerprint.py v2.28.0: core 2.66.0, own address from the self contact 2026-10-28 20:02:36 +00:00
docker-compose.yml v2.15.0: scale to 1000+ monitors (hung targets, batched cross-checks, quiet logs) 2026-10-17 17:44:42 +01:00
Dockerfile v2.24.0: degraded path detection for ping monitors, automatic account creation 2026-10-24 15:56:19 +01:00
icon.jpg Add compressed 512x512 JPG icon 2026-06-29 19:06:01 +01:00
icon.png Apply custom transparent icon.png for web and favicon, keeping icon.jpg for Delta Chat avatar 2026-06-29 21:15:26 +01:00
LICENSE v2.12.1: harden peer federation, DNS rebinding protection, MIT license 2026-10-16 13:26:06 +01:00
monitor_names.py Pool read connections per thread and cut the incident and status query cost 2026-11-20 19:38:47 +00:00
outbox.py Report sent mail kinds and slow idle peer telemetry 2026-11-18 21:49:45 +00:00
peer_rt.py Remove per-row journal writes, batch the retention cleanup, stop pruning local measurements and fix realtime ACK arming 2026-11-20 17:47:50 +00:00
README.md Release 3.0.1: cut the database and CPU cost of periodic work and fix the issues found on the way 2026-11-20 21:33:31 +00:00
requirements.txt v2.28.0: core 2.66.0, own address from the self contact 2026-10-28 20:02:36 +00:00
set_admin.py Initial commit: Delta Chat Uptime Bot 2026-06-29 18:51:28 +01:00
transports_view.py Show the relay actually used for sending in /transports 2026-10-14 14:12:20 +01:00
update.sh v2.25.0: deploy and publish releases by tag (#3) 2026-10-24 16:28:17 +01:00

Delta Chat Uptime Bot

Delta Chat Uptime Bot is a self-hosted uptime monitoring bot (similar to Uptime Kuma) integrated directly with Delta Chat. It monitors resources (websites, APIs, mail servers, TCP ports, or ping targets) and alerts you inside Delta Chat if they go offline.

Additionally, it automatically generates a secure, beautiful web status dashboard for each chat.

Features

  • 🛡️ Secure Administration: Claim ownership with /initadmin via private chat when no owner is configured in the database or environment. Once an admin fingerprint is configured, commands require that exact key; an unavailable sender fingerprint is refused and logged at most once per sender per hour. Email-only matching applies while no fingerprint is configured.
  • 💬 Per-Chat Isolation: Each chat (private or group) maintains its own separate list of monitored resources.
  • 🚨 Incident-Based Alerting & In-Place Dynamic Updates:
    • Instead of flooding the chat with dozens of separate DOWN/UP messages, outages trigger a unified Incident per chat.
    • As multiple monitors fail or recover, the bot edits the same incident message in-place with real-time status and duration metrics.
    • Tiered Rate-Limiting: Status transitions edit the incident message immediately; the live duration is refreshed every 5 minutes during the first day and hourly after that (every edit is a mail to the whole chat).
    • When all services recover, the incident message is updated to Resolved with total downtime duration. A definite core rejection of an initial incident or partial-outage notice allows up to three send attempts, at least five minutes apart; uncertain sends keep their at-most-once claims.
  • 🔍 Content & Keyword Assertion (Zero-Config + Custom Keywords):
    • Zero-Config Error Detection: In the background, automatically scans HTTP responses for silent failure signatures wrapped in 200 OK responses (e.g. database connection errors, 502/503 wrapped in HTML, Cloudflare error screens).
    • Custom Keyword Matching: Assert that response body contains specific text (e.g. /add https://api.site.com Health "status:ok" or /keyword 1 "Welcome"). Chats monitoring the same URL keep their own keyword requirements for local checks and peer votes. When the keyword is missing, the error shows a short response excerpt with URL credentials and secret query values redacted, or a byte count for binary data.
  • ⏸️ Smart Maintenance Windows & Alert Snoozing:
    • Pause monitoring and mute outage alerts during planned maintenance without skewing 30-day uptime metrics: /pause <id|url> [duration] (e.g. /pause 1 30m, /pause https://example.com 2h).
    • Supports replying /pause [duration] directly to incident alerts.
    • Automatically resumes normal monitoring when the maintenance window expires, or resume early with /resume.
  • ⚡ Universal Latency Measurement & Response Time Tracking:
    • Measures probe latency in milliseconds for HTTP/HTTPS, SMTP, TCP sockets, and ICMP Ping.
    • Real-time latency badges displayed on the Web Status Dashboard (⚡ 124ms) and in /list.
  • 📜 Detailed Outage & Incident History:
    • /events (or /incidents) — View the chat's historical incident log, active outages, and total downtime durations.
    • /history [id] — Inspect recent downtime events for a specific monitor with failure reasons, error codes, and recovery timestamps.
  • 🌐 Host Outage Protection & Circuit Breaker:
    • Automatically verifies host internet connectivity via high-speed canary checks (1.1.1.1, 8.8.8.8, 9.9.9.9, 1.0.0.1) before declaring any resource DOWN.
    • If the bot host itself loses internet access, false mass-downtime alerts and false downtime logs are suppressed, keeping 30-day uptime metrics accurate.
  • 🌍 IPv6: IPv6 addresses work as targets (2a00::1, [2a00::1]:443, https://[2a00::1]/). A node without IPv6 (no route, or Docker without enable_ipv6) does not report IPv6 targets as DOWN: its checks are skipped (⚪ in /ping, no alert, status unchanged) and other regions answer instead. Names with both A and AAAA records are checked over IPv4 there, so an unreachable IPv6 address no longer hides the real error.
  • 🚦 Rate-limited targets (429): a target that answers 429 Too Many Requests is not counted as down: that node abstains for the check (no vote, no alert, status unchanged), since a site limiting request rates refuses some nodes and answers others. 5xx errors, timeouts and refused connections still count.
  • 📮 SMTP / STARTTLS Checks (smtp://host[:port]): Checks a mail server the way a sending server does: it reads the 220 greeting, sends EHLO, requires STARTTLS in the reply, switches to TLS and verifies the certificate against trusted CAs and the target host name (what MTA-STS senders in mode: enforce require), then says QUIT. Port 25 is the default (smtp://chatmail.uk), smtp://host:587 works the same way, and port 465 uses TLS from the first byte. A plain host:25 TCP check only shows that the port is open; this one fails with the reason when the server stops offering STARTTLS (SMTP: STARTTLS not offered), its certificate expired or does not match the name (SSL: Hostname mismatch, ...), or it greets with an error. The session ends with QUIT, so the server logs an ordinary connection (ehlo=1 starttls=1 quit=1) that fail2ban does not count. The name sent in EHLO is the host's FQDN (or localhost); set SMTP_EHLO_NAME to change it.
    • Outbound port 25 must be open on the node. Many hosting and home providers block it; a node there sees every SMTP target as down. Use /probeignore *:25 on that node (it covers smtp:// targets on port 25 too) so it abstains instead of voting DOWN.
    • Peers running older versions do not get smtp:// targets for cross-checks (they announce no smtp_check capability) and do not vote on them.
  • 🔒 SSL Certificate Expiration Monitoring: Automatically tracks SSL/TLS certificate expiration for HTTPS and SMTP (smtp://) targets:
    • Periodic checks cached to run at most once per hour.
    • Staged proactive alerts sent to chat at 7 days, 3 days, and 24 hours (1 day) before expiration, as well as upon expiration.
    • Automatic alert state reset when certificate is renewed.
    • Real-time expiration countdown displayed in /list and on the Web Status Dashboard.
  • 🧹 Stale Resource Notices & 30-Day Auto-Cleanup:
    • A resource counts as unreachable for N days when its uptime over the last N days is under 1%. A short recovery (a flap) does not restart the count.
    • 7-Day Notice: Sends a notice after 7 days unreachable, suggesting removal if decommissioned.
    • 14-Day Warning: Sends a warning after 14 days, advising that monitors unreachable for 30 days are automatically removed.
    • 30-Day Auto-Cleanup: Deletes resources unreachable for 30 days and notifies the chat. Removal happens at least 3 days after the 14-day warning, so the warning always comes first.
    • Notices start over only after a resource has been up for a full day.
  • ⏸️ Chats the Bot Left Are Paused: Every hour (and at start) the bot checks that it can still write to each chat with monitors. If it was removed from the group, the chat was deleted or the 1:1 contact blocked it, that chat's monitors are paused: no checks, no alerts, not shared with peers. If the bot is added back they resume with all settings; after 30 days they are deleted with their history. The admin is notified each time, and /peers shows the number of paused chats.
  • 🐢 Slower Checks for Long Outages: A resource that stays down is checked less often: every 5 minutes after 1 day, every 15 minutes after 7 days, every hour after 14 days (never more often than its own interval). Recovery is still noticed within that time. /list and incident messages show it (🐢 checked every 15 min). Targets mirrored from peers slow down the same way and are checked once a day after 30 days, so a dead domain on one node does not keep the whole mesh busy.
  • 🛰️ Distributed Multi-Node Peering & Cross-Probe Verification:
    • Link multiple bot instances across different regions/servers as remote probes via private 1:1 Delta Chat DMs. Run /invitepeer on the other bot and send the /addpeer <link> token=<token> [node_name] command it prints to this bot; both bots then exchange telemetry.

    • Closed Peering: A bot only becomes a peer if the admin added it with /addpeer, or if it presents a single-use /invitepeer token (valid 24h). Telemetry and cross-check messages are only accepted from configured peers, in a 1:1 chat, end-to-end encrypted.

    • Peers Identified by Key: A peer is bound to its OpenPGP key fingerprint (taken from the SecureJoin invite or the first encrypted message), not to its email address. A bot with several relays may send from any of them and stays the same peer; a message from the peer's address signed by another key is rejected.

    • Controlled Disclosure (/peershare): By default peers only see monitors from chats the administrator is a member of (admin); all shares every chat, none shares no local monitors. Monitor names are never sent, and targets a peer asked this node to probe are only reported back to that peer, never relayed to others.

    • Zero Group Spam: All protocol handshakes, background telemetry, and instant cross-checks happen in private DMs between bots.

    • Cross-Probe Verification: Outages are verified across remote probes in real-time before alerting, distinguishing global downtime from regional/routing reachability issues.

    • Quorum Alerting (/alertmode quorum, default): When a target fails, this node and its live peers vote. If most nodes cannot reach it, it is 🔴 DOWN (incident). If only some cannot (e.g. 1–2 of 4, or only this node), it is 🟠 DEGRADED: one grouped notice in the chat, no incident and no downtime. Alerts show which nodes can and cannot reach the target and why they failed (🔴 DE, RU (Timeout) · 🟢 IT (196ms); with different reasons per node: 🔴 DE (Timeout), RU (SSL: self-signed certificate)). Nodes that are offline, cannot check the target (IPv6) or ignore it (/probeignore) abstain; with fewer than two voters this node decides alone. DEGRADED is entered and left after two consecutive votes. /alertmode local keeps the previous behaviour (this node decides).

      Periodic votes for ongoing outages and peer-reported problems are grouped on shared time ticks. Revotes wait at least 2, then 5, then 10 minutes between rounds, resetting when the result changes. New failures and recovery confirmations still run promptly. After three consecutive vote timeouts, a peer abstains from routine votes for 5 minutes, with a single probe at the next eligible round. Further timeouts extend the pause to 10, 20, then at most 30 minutes, and any vote answer clears it. If every peer would be paused, the two whose answers are oldest continue to receive votes. See /stats and /peers for active pauses.

    • Degraded Paths (ICMP Ping): A ping check sends 5 echoes (about 3 s at most). If 2 or more are lost, or the median round trip is at least 3 times this node's usual one for that address and 100 ms above it, the target is reachable but its path is bad (e.g. a congested peering between this node's provider and the target's network): the node votes 🟠 degraded instead of up, once its path has been bad for 3 series in a row over at least 3 minutes (a short burst on the node's own uplink is not a bad path); until then the check stays UP and its details show the slow round trip. In quorum chats this makes a DEGRADED notice, never an incident or downtime: 🟠 HOME (561 ms, loss 80%, usual 60 ms; trace: +589 ms between hop 6 171.75.9.210 and hop 7 213.19.194.214) · 🟢 DE (44ms). The usual round trip is learned per address from healthy checks (degraded ones are ignored, so a long bad period does not become the new normal) and kept across restarts. The trace: hint comes from one traceroute of the bad path (cached for 15 min) and names the link where the round trip rises most. A node that runs only as a peer reports degraded paths in its telemetry, so a home node is enough to notice that only its own connection is affected.

    • Scales With the Mesh: Every peer costs mail, and the relay allows 60 a minute. A vote asks up to 7 peers: all of them in smaller meshes; beyond that the node whose report started the vote plus the most stable others (not unstable now, fewest votes in the last day where they alone failed the target, most recently heard from).

    • Unstable Nodes Do Not Cause Notices: A node that fails or degrades 5 or more targets within 5 minutes that at least two other nodes reach has a network problem of its own (a home connection, provider filtering). It is marked unstable: its DOWN and DEGRADED votes abstain (⚪ RU (node network unstable, not counted)) and partial outage notices it alone caused are dropped, until it has met that threshold no more for 10 minutes. The admin gets one notice when this starts and one when it ends. Outages most nodes see are still reported. While a node is unstable, its telemetry reports of a target that a vote already found it alone failing are ignored for 30 minutes (10 minutes when the node itself answers UP to that vote: a short burst, a stale result or a bug of an older version; doubled each time in a row up to 2 hours, reset when it confirms a report; kept across restarts), so a noisy node does not make the others vote (and mail every peer) every two minutes. A peer also abstains, instead of voting DOWN, on a target whose name resolves to a private address there (split-horizon DNS at home, a VPN). A peer whose report names 10 or more targets that are UP here (broken DNS or network, which it confirms when asked again) is ignored altogether for 30 minutes, so it cannot make the node vote for dozens of targets every few minutes.

    • Blocking Hints (SNI block?, HTTP Host block?, DNS spoofing?): When an HTTP(S) check fails, the node probes the same address once more with the real name and with a neutral one (no SNI / example.com). A handshake or request that is reset or hangs only with the real name is shown as SNI block? (https) or HTTP Host block? (http); a system resolver that answers other addresses than 1.1.1.1 / 8.8.8.8, where only the public answer works (for https: with a valid certificate), as DNS spoofing?. The hint appears in the node's vote line (🔴 RU (SNI block? Connection reset by peer) · 🟢 DE (44ms)) and changes nothing else. If both probes fail the address itself is unreachable, which may be a dead server or a block by IP: no hint, and the votes of other nodes tell which. These are heuristics, hence the question mark (a CDN answering by region can look like a DNS difference only while its address in this region is also dead). The diagnosis runs at most once per target in 5 minutes. BLOCKCHECK=0 disables it; BLOCKCHECK_RESOLVERS (default 1.1.1.1,8.8.8.8) sets the resolvers, and a network that intercepts plain DNS to them can hide a spoofing.

    • Freeze (Freeze?): Russian TSPU equipment can "freeze" connections to some hosting networks: the connection opens, a few KB arrive, then nothing, without a reset. A GET check that has a Content-Length and sees the body stop below 64 KB reports Freeze? Stalled after 10 KB of 2.0 MB (DOWN); a stop above 64 KB is a slow link and gets the text without the tag. HEAD checks and responses without a length (streams) cannot show it: a freeze is found only where a body is read. This is the same signature the freeze-check script measures with 256 KB downloads.

    • Redirects in /ping: When the check is redirected, the reply shows the hops after the response line (↪️ Redirect: 301 → https://example.com/). It is this node's own view; the lines of other regions stay as they were.

    • Errors per Address Family: When an HTTP check of a host with IPv4 and IPv6 addresses fails, aiohttp reports only its last attempt, often an IPv6 address, which hid what happened over IPv4. The check is then repeated over IPv4 only: if that works the target is up, otherwise both reasons are shown (IPv4: Timeout · IPv6: Connect call failed).

    • Failures Seen Only by Peers: If a peer reports a target DOWN in its telemetry while this node still reaches it, the target's next check asks all peers and votes too, so a target most nodes cannot reach becomes DOWN even when this node can (and DEGRADED if only some cannot).

    • Status Page & History: DEGRADED monitors are shown in orange with the last vote; partial outages are listed on the status page, in /events and in /history <id> (not counted as downtime).

    • Batched Cross-Checks: Monitors that fail within the same 5 seconds go to each peer in one message (up to 100 targets) and come back in one, instead of a message per target and peer. In a mass outage this keeps the bots far below the chatmail relays' send limit (60 messages a minute). Bots announce the feature (caps in telemetry and check messages); peers that have not announced it yet get the classic one-target requests, so nodes can be updated one at a time.

    • Inbound Probe Limits: Single and batch requests share 20 probe slots, with at most 10 outstanding probes and 120 requests per minute per peer. Duplicate request IDs are dropped for 10 minutes. Batch workers reserve their slots before scheduling; excess requests receive no reply.

    • Peer Telemetry: Over mail, peers get a snapshot every five minutes while monitor and probe statuses stay the same. When statuses change, the bot waits at least two minutes after the previous snapshot. Both schedules honor longer intervals based on peer count or a peer’s request. Snapshots carry up to 200 targets and larger sets rotate with changes first. In /stats, Peers shows idle, changed and realtime METRICS counts for the past 24 hours.

    • Peer measurement storage keeps at most 1,000 rows per peer identity for seven days from local receipt. Claimed check times are clamped to seven days ago through 60 seconds ahead; mirrored targets unseen for seven days are removed. Handshakes and renames cannot take this node's or another peer's name.

    • Aggregated Web Dashboard: Web status pages show latency and status badges for all active probe locations ([📍 Frankfurt-DE: 18ms] [🛰️ RU-Moscow: 45ms]).

  • 🤖 Identified User-Agent: Sends a custom User-Agent header (e.g. Delta Chat Uptime Monitor/2.12.1 (+https://git.gluek.info/gluek/deltachat_uptime)) during HTTP checks so server administrators can easily identify monitoring requests in server logs. The name deliberately avoids the word "bot", because some services (e.g. Hiddify subscription links) answer 400 Bad Request to bot User-Agents.
  • 🚀 High-Concurrency Scaling Architecture:
    • Two-Tier HTTP Probing (HEAD -> GET with Auto-Memorization): Zero body download (0 bytes) and 0 CPU decoding for standard HTTP checks via HEAD, with automatic 16 KB GET fallback on 405 or errors (not on a timeout or connection error, which a GET would only repeat), and 128 KB limit for keyword assertions. When a server responds with 405 Method Not Allowed to HEAD (e.g. GoToSocial, Mastodon, specialized APIs), the bot automatically memorizes GET for that monitor and database record so all subsequent checks use GET directly without generating repeated 405 log entries. The same happens when HEAD gets any other 4xx/5xx while the GET right after it succeeds (e.g. changedetection.io answers 500 to HEAD /login), so the server no longer logs a failing HEAD on every check; if both fail, the target is simply down and nothing is memorized.
    • Async DNS Resolution & 5m TTL Cache: aiodns support and a bounded system-resolver pool keep DNS off the event loop, with a 5-second overall lookup deadline. Cached answers follow DNS TTLs. Each probe has one 30-second overall deadline (three times its 10-second request timeout, capped at 45 seconds), including DNS, redirects, HEAD/GET fallback, body reading and cleanup. Automatic title fetching for /add runs in one of four background workers with a 5-second total deadline; timeout or full worker capacity adds the monitor without a fetched title.
    • Native Async ICMP Ping (aioping): Native in-process raw socket pinging eliminates subprocess creation overhead. Without raw socket permissions (hosted containers without CAP_NET_RAW) it uses an unprivileged ICMP datagram socket (allowed by net.ipv4.ping_group_range, the Docker default), then /bin/ping. A node with none of these skips ping checks (⚪ in /ping, abstains in votes) instead of reporting them as DOWN. So does a node in Docker Desktop (macOS / Windows, detected by its linuxkit kernel): its network proxy drops ICMP replies as soon as several echoes are in flight (68-89% lost in tests, 0% on a Linux host), which would be false DOWN and degraded votes. PING_ON_DOCKER_DESKTOP=1 checks them anyway; for path monitoring from a Mac run the bot natively instead.
    • Deterministic Time Slot Staggering: Uniform phase slotting ((r_id * 11) % interval) spreads checks across 5-second windows, preventing thundering-herd spikes on startup and interval boundaries.
    • Lock-Free SQLite WAL Concurrent Reads & Persistent Writer: Dedicated _write_lock protects write transactions with a persistent connection (_writer_conn) while enabling concurrent lock-free reads for web dashboards and chat status commands.
    • Zero-Churn Disk I/O (synchronous=NORMAL): Uses SQLite WAL mode with PRAGMA synchronous = NORMAL, eliminating fsync overhead during commits and reducing continuous disk writes by >98%.
    • Non-Blocking Semaphore: Concurrency semaphore slots are held exclusively for the milliseconds of network probes; retry backoffs (30s) and remote peer checks execute asynchronously without starving healthy checks.
    • 200 Checks at Once: Checks are async, so the limit is sockets, not CPU (about 0.15 ms of CPU per check, plus 1-3 ms for a TLS handshake: 1000 monitors take a few percent of one core). Up to 200 run at once (CHECK_CONCURRENCY, connection pool 250, nofile 8192 in docker-compose.yml): 300 targets that accept connections but never answer finish in about 20 s instead of 130 s, so a dead data center does not delay the other checks.
    • One Transaction per Check: The status of every monitor of a target and the local measurement for the peers are written in one SQLite transaction.
    • Paced Outbox (send limit): chatmail relays accept 60 messages a minute per sender (burst 10) and answer 450 Too much mail above that; the core's retries count too, so a mass outage could delay alerts by minutes. Every message, edit and reaction therefore goes through one queue (outbox.py): 50 a minute with bursts of 5, sent by priority - alerts (new / resolved incidents, status changes, SSL, stale, partial outage, admin) → command replies → duration refreshes → peer protocol → telemetry. A queued update of the same incident message (or reaction, or telemetry to the same peer) is merged into the waiting one, so it goes out once with the latest text; peer requests and answers that waited past the peer's timeout are dropped. The minute summary in the log shows a backlog (outbox waiting 12 (alert 2, peer 10)). Pending capacity is 2,000 messages: overload drops command replies first, then telemetry and duration refreshes, protecting alerts and peer protocol traffic while routine items remain; drops are counted and warned at most once a minute.
    • Quiet Logs: Only DOWN results, status changes and retries are logged at INFO; routine UP results are DEBUG, and one line a minute sums up all checks (Checks in the last 60s: 148 (146 up, 2 down, 0 skipped), avg latency 87 ms, slowest …). With Docker's 30 MB log rotation the history then covers days instead of hours.
    • Dedicated Thread Pools: Separate thread executors for database queries (uptime_db) and Delta Chat JSON-RPC / SMTP calls (uptime_rpc) prevent slow email delivery from stalling database operations.
    • Single-Query Batch Metrics & In-Memory TTL Cache: Single SQL batch queries and 60-second TTL caching eliminate N+1 queries across web status pages, /list, and /status.
    • Optimized SQLite Indexes: Comprehensive indexes on downtime intervals, incident states, and peer telemetry ensure sub-millisecond query execution even with tens of thousands of historical records.
  • 🔄 Failure Resiliency & Retry Logic:
    • Checks resources once a minute.
    • If a resource check fails, the bot does not alert immediately. It retries 2 more times at 30-second intervals.
    • Alerts are only triggered if all 3 checks fail, avoiding false positives.
    • Once a DOWN resource recovers, it is marked UP on the first successful check.
  • 📊 Uptime Dashboards: Generates a secure, 12-character unguessable base62 URL (e.g. https://up.example.com/k8D2x9mPqL1a) hosting a modern dark-themed web status dashboard with active status, latency metrics, SSL countdowns, and recent incident logs for each chat.
  • ✉️ Multiple Mail Relays: Supports multiple mail servers. Relay selection and failover are handled by the Delta Chat core (2.61+), which sends via the newest relay first and falls back to the next one if a relay is unreachable.

Health Endpoint

GET /health (and HEAD) answers {"status": "ok", "version": "X.Y.Z"} without touching the database or building a page, for monitors and other nodes. / is a static page cached in memory.

Commands

User Commands (Public per-chat)

These commands are available to any member of a chat. They support suffixes (e.g. /add@up, /status@uptime) to route commands correctly if multiple bots exist in the same chat.

A plain /help sent in a group chat is answered in a private 1:1 chat with the sender, so several bots don't flood the group with help texts. Use /help@uptime to show the help in the group itself.

  • /ping [-4|-6] <target> ["keyword"] (or /check, /test) — Perform an immediate on-demand reachability and diagnostics test without adding to monitoring (rate-limited to 1 request per 15s per chat for non-admins; supports HTTP/HTTPS, SMTP STARTTLS (/ping smtp://chatmail.uk), SSL certificate inspection, keyword assertions, TCP port, ICMP Ping, and multi-region cross-checks). -6 / -4 check a name only over IPv6 / IPv4 (also on the other regions), e.g. /ping -6 chatmail.uk.
  • /add <target> [name] ["keyword"] — Add a monitor. Target formats: • https://google.com Google (HTTP/HTTPS check) • https://api.site.com Health "status:ok" (HTTP with keyword assertion) • smtp://chatmail.uk Mail (SMTP: STARTTLS + certificate; smtp://host:465 for implicit TLS) • google.com:443 Google TCP (TCP port check) • google.com Google Ping (ICMP Ping check) • 2a00::1 Server / [2a00::1]:22 SSH (IPv6 ping / TCP port; with a port the address goes in brackets, as in URLs)
  • /remove <id|url> (or /delete, /rm, /del) — Stop monitoring a resource by ID or target URL. You can also reply /remove directly to any incident or outage alert message to remove the affected monitor without knowing its ID.
  • /pause <id|url> [dur] (or /mute, /maintenance) — Mute outage alerts during maintenance (e.g. /pause 1 30m, /pause https://example.com 2h, or reply /pause to an alert).
  • /resume <id|url> (or /unpause) — Resume active monitoring after maintenance.
  • /keyword <id|url> [keyword|none] — Configure or remove expected keyword assertion.
  • /list — List monitored resources, status, latency, and SSL expiration in this chat.
  • /status — View monthly uptime statistics (online / degraded / offline, alert mode) and get the link to the chat's secure Web Status Page.
  • /report [all|partial <N|off>|full <N>|app off|on|only] — Configure per-chat outage notices and the status app; see below.
  • /alertmode [quorum|local] — How this chat is alerted: quorum (default) — DOWN when most nodes cannot reach a target, DEGRADED notice when some cannot; local — this node's checks only.
  • /events — View recent incidents and active outages for this chat.
  • /history [id] — View downtime history for monitors.
  • /sync — Synchronize monitored resources with other bots in the same chat (rate-limited to 1/minute for non-admins).
  • /donate — Support bot development ❤️
  • /help — View available commands and system information.

Outage report delays (/report)

Any chat member can change when this chat receives outage notifications:

  • /report shows the current settings and usage.
  • /report partial <N|off> waits N minutes before announcing a partial outage (default: 10). off disables partial notices.
  • /report full <N> waits N minutes of continuous DOWN before announcing an incident (default: 2).
  • /report all sets both delays to 0 for immediate notifications.

N must be an integer from 0 to 1440. Short outages that recover before the delay stay silent, and recovery messages are sent only for outages already announced. Detection, incident history, partial outage history and uptime statistics remain immediate. Pending delays survive restarts; uncertain sends are dropped to avoid duplicates. The existing scheduler checks pending notices about every 30 seconds, and the outbox may delay delivery further.

Admin-Only Commands

These commands are only executable by the configured administrator.

  • /url — View current base external status URL.

  • /url <url> — Update the base external status URL (e.g., /url https://up.gluek.info) to generate correct status links.

  • /probeignore [url|host|pattern] — On this node, stop probing targets it cannot reach (e.g. blocked in its region or by its ISP) and abstain when peers vote on them. Accepts a target as shown in alerts (https://nine.testrun.org/, dnd.wb.ru:25), a host (nine.testrun.org: all its targets) or a pattern with * matched against the target, its host and host:port (*:25 — all SMTP port checks, *.testrun.org). Without arguments lists the entries; /probeunignore <entry> removes one.

  • /nodeignore [node_pattern] [target_pattern] — Centrally exclude targets from being checked on remote peer nodes matching node_pattern (wildcards * supported, e.g. /nodeignore RU* facebook.*, /nodeignore *home *:25, /nodeignore Node-1 https://site.org/). Cross-check requests for matching targets are not sent to those nodes (saving network messages), their telemetry reports for them are ignored in quorum, and any matching stale peer measurements are cleared. Without arguments lists all rules; with node_pattern lists matching rules.

  • /nodeunignore <node_pattern> [target_pattern] — Remove a node ignore rule (e.g. /nodeunignore RU* facebook.*), or remove all rules for a node pattern (e.g. /nodeunignore RU*).

  • /nodename [name] — View or set the local probe node identifier (e.g. /nodename Frankfurt-DE). Peers (2.17.2+) pick up a new name from the next telemetry message; a name already used by another node is refused. Without DISPLAY_NAME the bot's display name follows it: Uptime Bot (<node name>).

  • /invitepeer — Generate a SecureJoin E2E encrypted invite link (https://i.delta.chat/#...) with a single-use pairing token (valid 24h) for linking another bot. Requires an encrypted private 1:1 chat with the administrator.

  • /peers (or /probes) — List distributed monitoring peers, node names, key fingerprints, the relay address each one last used, their bot version (⚠️ when older than this node) and last seen activity.

  • /stats — Bot statistics and diagnostics: version and uptime, CPU of the bot and the Delta Chat core (last minutes and average), database sizes, monitors by type and state, checks per minute, mail received/sent in the last hour, outbox counters and queue, peers (online, unstable, which run an older version), telemetry interval, votes and held reports in the last hour, relays.

    The Storage section shows free disk space, database and blob sizes, and the largest files and directories under the data and application folders. Results are cached for five minutes. The bot warns the admin at most once per day when free space drops below 10% or 30 MB. Set STORAGE_WARN=0 to disable these alerts.

    On supported systems the bot disables core dumps for itself and its Delta Chat RPC server. At startup and every ten minutes, it checks the top level of the application, working, and data directories for existing dumps. New dumps are reported once to the admin and listed in /stats. Set CORE_DUMP_WARN=0 to disable core dump notices.

    The peer section shows incoming and outgoing mail counts by message kind over the last 24 hours, plus median and 95th-percentile round-trip times for peer pings, vote replies, and full vote rounds. Timeouts are counted separately. Peer metrics refresh once a minute.

  • /rmcore — Delete detected core dump files and report how much space was freed. Send in a private chat with the bot.

  • /addpeer <email|link> [token=<token>] [node_name] — Link another Delta Chat Uptime bot as a remote probe using the command printed by its /invitepeer (e.g. /addpeer https://i.delta.chat/#... token=... RU). Without a token (e.g. /addpeer ruptimebot@chat.gluek.info RU) the other bot's admin must add this bot too.

  • /rmpeer <email|key> — Remove a remote peer probe by its current address or key fingerprint (the first 8+ hex digits shown by /peers are enough). Also stops probing targets it mirrored to this node.

  • /peershare [admin|all|none] — Choose which local monitors are disclosed to peers (default admin: only chats the administrator is a member of).

  • /accounts — List active bot accounts.

  • /rmaccount <id> — Delete a bot account.

  • /transports — Show mail relays: relay used for sending, send queue, incoming status and storage.

  • /addtransport — Add backup mail relays (either chatmail URIs or address/password). Restricted to private 1:1 chat with the bot for security.

  • /rmtransport <addr> — Remove backup mail relay.

Relay selection and failover are handled by the Delta Chat core (2.61+): it keeps sending through the relay that last worked and moves to the next one only if it cannot connect, so the relay used for sending can differ from the account's main address. /transports shows which relay that is, the core's send queue, and per relay the incoming status, the last received message and mailbox storage. The former /setprimary and /resilient commands are deprecated and only reply with this explanation.


Peer channel (preview)

Privacy: the channel is off by default. When enabled for a pair of bots, they connect over iroh (directly or through relays), so each bot can learn the other's IP address. Votes and telemetry stay end-to-end authenticated, and mail remains the fallback. Enable it only for peers you trust with that information.

The iroh peer channel preview is off by default. Use /peerrt to see the current setting, /peerrt off|canary|on to change it, or /peerrt <peer name|email> on|off|auto to set a peer's preview mode. In canary mode, only peers explicitly set to on are eligible. With the global setting on, peers set to auto are eligible too.

Eligible peers that announce rt1 and have a bound key exchange a small Uptime peer link app in their encrypted 1:1 chat. Nodes exchange capabilities during pairing and can announce them even when they have no monitor results to share. A capability change goes out on the next telemetry tick (within five seconds on an idle node), with up to three change attempts per peer in any ten minutes. Startup announcements and requests for unknown capabilities keep the ten-minute cooldown. The last verified rt1 capability survives restarts. The peer with the lower fingerprint sends the app and replaces it every 14 days. Cleanup preserves the current app until its replacement is accepted. Deletion triggers a check on the next telemetry tick, and a periodic check runs once a minute. If the recipient loses the app, it sends a reset by mail at most once an hour. A reset for an accepted app allows the sender to offer a replacement on the next tick. Failed or deleted pending offers still have a ten-minute retry floor, and pending offers still in the chat can be accepted for up to 14 days before a new offer is due. Re-enabling the preview can also replace a previously accepted app immediately. All offers count toward a limit of three per node per hour.

The bots join through Delta Chat Core without opening the app. Each side mails one hidden address advertisement after accepting the app and on restart, with up to 60 seconds of jitter and at least 60 seconds between attempts per carrier. An advertisement triggers an authenticated hello on the channel. Each side sends its own ping every 30 seconds, even while answering the other side. Both frames receive a pong that echoes the sender's epoch and sequence number. /peers shows the carrier age while waiting to join, UP with this bot's own measured RTT when available, or DOWN after 90 seconds without a valid frame or two consecutive missed acknowledgments. /stats totals these channels. If the peer is still active by mail, the bot tries to rejoin after five minutes down, then after 15 minutes and hourly.

An UP channel carries cross-check requests and answers, including batched votes and /ping. A missing acknowledgment after three seconds, or a channel going DOWN, sends the same check by mail within the original wait. The first answer from each peer counts once. Peers without rt1, disabled channels and oversized checks use mail. /stats shows mail and realtime round-trip percentiles and the number of mail fallbacks.

An UP channel also carries full telemetry snapshots at the peer interval, two minutes by default. Monitor and probe status changes are grouped in a ten-second window and can send a partial snapshot containing the changed targets. Valid frames keep the peer online. The bot sends one hourly mail snapshot while UP, uses mail when a snapshot exceeds the channel's size limit, and resumes the regular mail schedule on the next telemetry tick after a channel goes DOWN.

Status app (/report app)

The optional webxdc app gives each chat a mobile-friendly status view inside Delta Chat, with light/dark themes, full monitor names and URLs, all node results, a live name/URL filter, incidents, partial outages and the last 20 journal events. The bot posts the app followed by a plain text status line and pins that line once it is delivered to the server and at least 60 seconds old. Pending pins retry every 30 seconds; the app receives its first state after pinning. The pinned line names failing monitors using short names: for example, 🔴 2 down: cm.dc09.xyz, mail.x · 🟠 1 partial: site · 🟢 80 up. It is edited in place only when its text changes. Paused monitors are excluded from failure counts; each category shows at most three names, with a count of any others, within 120 characters. The app bubble shows counts only. It is off by default.

  • /report app on posts the app and its pinned status line, then updates them alongside normal chat notices (with the configured report delays).
  • /report app only uses the app for outage reporting and suppresses partial/full outage chat notices. Short system info lines still announce outage starts and recoveries after the configured report delays, in both only and on modes. Up to three events are combined per update, with a count of any others; monitor edits do not produce info lines. Monitoring, incidents and uptime statistics continue. Pending notices from app-only time are dropped when chat notices resume, including still-open outages.
  • /report app off stops app updates and leaves the existing app message in place.
  • /report live on|off allows or disables optional realtime updates for this chat (default: off). Any chat member can change it. With live mode on, each device can choose Live updates inside the app; this shares its IP address with the bot and other live members. The choice is remembered per app instance on that device. The app reconnects after about three minutes without a reply, including after a bot restart.

While the current app instance is sending live heartbeats, non-urgent snapshot mails wait at least 30 minutes. After three minutes without live activity, mail pacing returns to the interval for the chat's size and pinned status line updates resume. Incident and monitor changes remain urgent. STATUS_REALTIME_CHATS (comma-separated chat IDs) overrides /report live off for testing.

/report, /status and /help show the setting or command. Updates carry complete snapshots, are sent when monitor/node status, monitor details, chat name or history changes, normally wait at least 60 seconds between sends, and share a cap of 30 app updates and status line edits per chat per rolling hour. Chats with at least 10 members and no active live session wait five minutes between non-urgent snapshot mails and share a lower cap of 12 app updates and status line edits per rolling hour, even with live mode enabled. Urgent incident and monitor changes bypass the longer interval and lower cap; the core interval and existing 30/h safety cap still apply. Latency, node error text and uptime changes refresh alongside other updates, or after 30 minutes if values changed. A fresh app’s first state bypasses the hourly cap and is still counted toward it. Capped updates and edits log one warning per chat per hour and appear in /stats. Monitor additions, removals, renames, target/type changes, pauses and maintenance changes bypass the 60-second debounce and are audited within 30 seconds; the core send interval and hourly cap still apply. Each update is mail to all chat members, and new devices replay the history. Large snapshots omit partial outages, older incidents and events before node details while retaining monitor status flags; the app shows when this happens.

Every ten minutes the bot locally deletes messages older than one day from non-peer chats, incoming and outgoing, while keeping the newest 20 messages, the current status app, its status line and messages for ongoing incidents. Each run examines at most 2,000 older messages. The core's 30-day device retention remains a safety net. Peer chats are trimmed separately; these deletions do not delete messages for other members.

The app is re-posted when it is older than seven days, has received more than 500 updates, or its local message is missing, at most once per hour. Rotation posts a new app followed by a new status line, pins the new line and unpins the old line and app. A layout version change also creates a fresh app. Upgrading to 2.34.0 gives enabled chats a new app and pinned status line once, unpinning the previously pinned app. Ordinary bot upgrades update the existing app's data; toggling /report app off and on resumes the same instance. Journal entries are retained for 30 days. Incidents cover the last 30 days (up to 20); partial outages show the last 10. The webxdc/ assets are bundled into status.xdc in the data directory on demand and are included in Docker builds.

Deployment

Prerequisites

  • Docker and Docker Compose installed.
  • Optional: a dedicated email address for the bot (e.g. uptimebot@yourdomain.com). Without one the bot creates its own chatmail account at first start.
  • A domain name pointing to your host for the status pages (e.g. up.gluek.info).

1. Build and Prepare

Clone the repository, enter the directory, and build the Docker container:

cd deltachat_uptime
docker compose build

2. Configure Email and Admin (Optional)

Skip this step and the bot creates its account by itself at first start: the Delta Chat core (2.62+) picks the fastest chatmail relay from its built-in list and later adds further relays in the background.

To choose the account yourself, initialize it with your email and password, or with DCACCOUNT:server.tld for an account on a specific chatmail relay:

docker compose run --rm uptime_bot python bot.py init uptimebot@yourdomain.com "your_email_password"
docker compose run --rm uptime_bot python bot.py init DCACCOUNT:server.tld

Configure your admin email address and your cryptographic fingerprint on the server (or do it later inside the bot chat with the /initadmin command):

docker compose run --rm uptime_bot python set_admin.py --email admin@yourdomain.com
docker compose run --rm uptime_bot python set_admin.py --fingerprint 1234ABCD1234ABCD1234ABCD1234ABCD1234ABCD

ADMIN_DC_EMAIL and ADMIN_DC_FINGERPRINT provide environment fallbacks. Either configured owner value prevents /initadmin from replacing ownership; if neither is set, only the first claim succeeds.

3. Run the Bot

Start the bot daemon in background:

docker compose up -d

4. Claim Ownership inside Delta Chat

  1. Scan the bot's secure join QR code printed in the logs (docker compose logs uptime_bot), open the root web dashboard (/) to scan the high-resolution vector QR code, or copy the invite link / email address into Delta Chat.
  2. Send /initadmin to the bot in a private message.
  3. The bot will automatically verify your identity and associate your cryptographic fingerprint. You are now the administrator!

5. Set up Base URL

Tell the bot your public status domain so that it generates correct dashboard links:

/url https://up.gluek.info

6. Updates

Run ./update.sh (e.g. from cron) to deploy the newest release: the highest vX.Y.Z tag, from GitHub or, when GitHub is unreachable, from the Forgejo mirror; -f rebuilds the container without changes. Commits on main that are not released yet are not deployed. In .env.local, UPDATE_REF=v2.24.0 stays on that release (to roll back; remove it to follow new releases again), UPDATE_REF=main follows the branch, and MONITOR_URL pings a healthcheck on start, success and failure.

Releasing: merge into main, then git tag -a vX.Y.Z -m "..." <commit> && git push origin vX.Y.Z. Pushing the tag publishes the GitHub release with that version's CHANGELOG.md section (bot.py must have that VERSION) and sends the ntfy notification and the Home Assistant add-on sync.

Configuration & Profile Customization

You can customize the bot's name, avatar, and status text by passing environment variables (e.g. in your .env file or docker-compose.yml):

  • DISPLAY_NAME — Customize the display name of the bot (default: Uptime Bot (<node name>), updated by /nodename while the bot runs).
  • STATUS_TEXT — Customize the status/about text of the bot.
  • AVATAR_PATH — Path to an image file (PNG/JPG) to use as the bot's profile avatar (default: falls back to icon.png or icon.jpg in the project root).
  • MAX_MONITORS_PER_CHAT — Maximum monitors when adding to a chat (default: 300).
  • MAX_MONITORS_TOTAL — Maximum monitors when adding across the bot (default: 3000). Existing monitors keep running if either limit is lowered below the current count; these settings never delete or pause monitors.
  • TRUSTED_PROXIES — Comma-separated proxy IPs/CIDRs allowed to supply X-Forwarded-For (default: 127.0.0.0/8,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16). The right-most untrusted address in that chain identifies the client. Headers from other socket peers are ignored; an empty setting disables forwarding-header trust.

New monitor inputs are limited to 2,048 characters for URLs, 100 for names and 200 for keywords, including sync and peer inputs. Automatically derived names are shortened to 100 characters. Incoming sync JSON is capped at 64 KiB; peer protocol messages at 8 MiB, enough for the largest 200-entry telemetry batch with maximum-length Unicode fields escaped by JSON. Non-admin senders can run 30 commands per minute per chat (sliding window); excess commands are silently dropped after one “Slow down” reply per minute per chat. Admins are exempt, and /ping checks its existing 15-second chat cooldown before validation or DNS.

Web client rate buckets expire after inactivity and retain at most 10,000 keys. Dashboard HTML is rendered in a background database worker, cached per token for 15 seconds (up to 1,000 entries), and limited to four concurrent renders; excess requests receive 503 with Retry-After: 15. Status responses retain their no-store and other security headers.


Reverse Proxy with Caddy

If you use Caddy on your host (like for your ntfy bot), you can expose the status pages by adding the following config to your /etc/caddy/Caddyfile:

up.gluek.info {
    reverse_proxy 127.0.0.1:8080
}

Reload Caddy to apply changes:

sudo systemctl reload caddy

Development

python3 -m unittest discover -s tests runs the test suite. tools/loadbench builds a synthetic database of production size (10,000 monitors, 300 chats, 20 peers, 90 days of history) and measures what the periodic and per-check work costs per simulated minute (CPU time, database connections, write transactions, rows read). Use it before and after a change that touches a timer path:

python -m tools.loadbench.gen_db --db /tmp/lb.db
python -m tools.loadbench.run --db /tmp/lb.db --functions --markdown /tmp/lb.md

License

MIT, see LICENSE.